I gave a talk at the Capital Factory Demo Day event to a group of investors and entrepreneurs last week. The event was recorded but the video is still being edited which I’ll post as an update later. and you’ll find the full video down below.

The key message from the talk was:

Lean Startup is a rigorous process for iterating from Plan A to a plan that works.

I started by outlining the key behaviors of Lean Startups but my main objective was to raise the bar on how Lean Startup is practiced today. Through my workshops, I have come across a number of well-intentioned startups that want to practice Lean. They understand that Lean Startups are NOT about being cheap but speed. They get metrics, MVP, and Continuous Deployment but struggle with one of the most fundamental concept that undermines their efforts – the ability to correctly define and run experiments.

I argue that without proper hypotheses formulation and measurement, all you’re left with are a bunch of tactical techniques that don’t deliver on learning and might actually do more harm than good. Some of this difficulty is understandable as Lean Startups attempt to apply the Scientific Method to a startup which isn’t an exact science. Others take the view that “Lean Startup” is simply a framework or collection of techniques that help guide their journey to product/market fit. I disagree. Even though startups are not an exact science, Lean Startup is a rigorous process or at a minimum a rigorous meta-process that requires the same level of objectivity you would demand from a scientific inquiry.

Here are some rules I presented on running experiments:

Build a Lean Canvas

Before doing anything else you need to capture your business model hypotheses. By capture, I mean write down (or type up). I covered my adaptation on Alex Osterwalder’s business model canvas last time which is my preferred method.

Here it is again:

The canvas is your startup’s roadmap which is systematically tested through tightly defined experiments. I really like the 1-page business model format because it not only forces you to be concise, but is a lot more portable and accessible than a business plan – which means it will be read by more people and probably updated more frequently.

Start with what matters

Where you start on the canvas is based on the stage of your startup. The very first thing that matters is making sure you have a problem worth solving which translates to working on the Customer Segment, Problem, and Solution boxes. So for instance, before achieving Problem/Solution fit, expending a lot of effort optimizing channels is a form of waste and not the best use of time.

But also tackle the riskiest parts early

That said, it’s just as critical to identify the riskiest parts of the business model and start tackling those early. One area, for instance, that does trip a lot of startups is the channel or path to customer. Too many startups simply use fillers like Adwords or SEO here only to find out later that these channels aren’t viable for their business. So while a lot of effort shouldn’t be spent optimizing channels, building a significant path to customers is something every startup should be testing from day 1. The good news is that if you follow a Customer Development process, you have to do this any way to find prospects. Test your channel assumptions at a smaller scale.

Formulate falsifiable hypotheses

The next step then is formulating a set of falsifiable hypotheses which is the area I see startups struggle with. Randy Komisar/John Mullins describe a great technique in their book: “Getting to Plan B” for going from what they describe as a “Leap of Faith” to a testable hypothesis. What most people write down as business model hypotheses are really leaps of faith and they miss the step of converting them into testable hypotheses. The goal here is clearly defining the conditions under which a hypothesis can be absolutely proved or disproved – QUICKLY. Speed is key. Otherwise, you simply accumulate just enough evidence to convince yourself that the hypothesis is correct.

Here is an example of what I mean:

The first statement here is an example of hypothesis that cannot be disproved because the expected outcome of driving early adopters is too vague. How many early adopters do I need to sign-up to prove this hypothesis – 10, 100, 1000? The second statement not only has a specific expected outcome but is also based on a repeatable action which makes it testable.

Build an accessible dashboard

The final step is then attaching a measurable metric to the hypothesis and tracking it on a weekly dashboard that looks something like this:

An important point here is to anchor the hypothesis around a key macro metric versus a vanity metric or if the key macro event is “too far in the future” (e.g. actual purchase) come up with an acceptable proxy for a macro event. Again, speed is key. So in this example, what’s being tested is that a blog post could drive enough interest in a new product that leads to customer interviews for the customer discovery process.

Make the results auditable

Testing hypotheses can be scary for founders especially when they are proven wrong. I’ve found the best way to keep the results objective and minimize on some of the drama is to make the results auditable by anyone on the team. I not only make the dashboards available but all the data behind it which includes the people (customers) behind the numbers. That way if someone has a question on a particular result, they can simply dig in deeper or even arrange to pick up the phone and call the customer directly.

Run board meetings in a lessons learned format

But the best way to measure progress (learning) in a Lean Startup is by frequently reporting on the results of experiments using a Lessons Learned format like this one.

The left hand side summarizes the key hypotheses and insights from the last iteration and highlights the next set of experiments that need to be run. Progress is tracked using the graph on the right which shows the key macro metric that drives this startup. In this case this happens to be revenue.

Lean Startup is a rigorous process for iterating from Plan A to a plan that works.

Communicating progress this way lets the startup stay grounded in learning while iterating towards a plan that works. A Lean Startup is fact-based not faith-based. But without this level of rigor and objectivity, there is a danger of creating a data-based pattern-seeking organization that might be just as bad or even worse at furthering “rationalized dogma”. What do you think?

Side-note: The screenshots in this post were all taken from a home-grown tool I use for tracking my own experiments which is now available here: http://www.leancanvas.com.