Customer Development Checklist for My Web Startup – Part 1

February 16, 2010

Last year, Steve Blank threw out a challenge to Lean Startup Circle members to update his customer development checklist (Appendix B in his book) for their specific business. His checklist is built for Enterprise Software which doesn’t readily translate to other types of startups including web startups (especially when you get to Customer Validation).

After the talk, David Binetti pulled together a great group of people from Lean Startup Circle to create a Customer Development Checklist for web startups. I was honored to be asked. However, after a few exchanges trying to nail down the type of web startup to tackle first, we uncovered a number of tactical differences that made me come to the realization that defining such a model in a group setting is too hard to do. We would either end up with a highly generalized model or no model. I, for one, am not comfortable extrapolating a model from third-party accounts of what might have worked for other companies (like 37signals), and especially not without first-hand experience building a similar type of startup.

So I decided to take a stab at defining a Customer Development Checklist for my web startup.

First Some Background

This model is based on my experiences building and running 2 products: BoxCloud and CloudFire. Both use a subscription pricing model. BoxCloud was built using a release-early release-often development model and was initially launched with a Freemium pricing model – later changed to a free-trial only model. CloudFire is being built using a lean-startup/customer development model and was launched with a free-trial only model.

For a SaaS product like mine, I strongly believe you need to
a) charge for your service, and
b) validate pricing sooner rather than later.

Free trials, Freemium, free introductory periods, etc. are all tactics to lower sign-up friction and should be applied (split-tested) judiciously on a case-by-case basis. However, my key takeaway is that even if you’re considering Freemium, you should validate the premium part of Freemium first before giving anything away.

I’ll cover Customer Discovery in Part 1 and Customer Validation in Part 2. Hopefully, I’ll get to write Parts 3 and 4 one day.

Customer Discovery: What should I build and for whom?

Here’s my Customer Discovery Flow (you’ll probably want to click to enlarge and skim it before reading on).

Click to Enlarge

The 3,000 Foot View
For a web startup, the purpose of Customer Discovery is to identify a problem worth solving, defining the “right” minimum viable product to build, and testing the business model using 3 separate Build/Measure/Learn loops. Most web startups rely on a product website for distribution and blogs, SEO, SEM, for initial customer acquisition channels- leaving price as the biggest unknown in the business model.

Sidebar: There is a somewhat loose definition of how the term MVP gets used. Many have used it (myself included) to refer to anything (a landing page, a problem presentation, screenshots, etc.) that allows you to learn about customers with the least effort. Here, I am using the stricter definition of MVP to mean the minimum set of features needed to learn from earlyvangelists. In other words, Release 1 of your Product.

It All Starts With Stating Your Assumptions
State your hypotheses

Before you can test what you think you know, you have to write it down. It’s normal to try and short this step but I found it to be a very worthwhile exercise. Apart from minor terminology changes, most of Steve’s questions in his hypothesis worksheets hold up even for a web startup. I am not going to reproduce them here but if there is interest, I’ll make my versions available as a separate download.

Test the Problem
Test Problem

This will look very similar to Steve Blank’s flow. That’s because when it comes to Customer Discovery I haven’t found a more effective way for maximizing validated learning than “Getting out of the Building”. This is coming from someone who used to prefer to stay in the building and talk to customers over email. Now, I have an 800 number tied to my mobile phone and schedule for face-time opportunities with customers.

With my last product, I used a teaser/landing page with some pre-launch buzz to collect email addresses and measure interest. While it was encouraging to have interested users, it told me nothing about what problem they had, who they were, what I should ship first, or what I should charge. What does 10 email addresses a day tell me? What does 20, 40, 100? Why did the other 70% abandon my landing page? Was it the product, was it the copy, graphics, something else. What?

Building a good landing page is hard. Unless you have exceptional customer insight (or are your own customer), iterating without talking to customers is slow and painful as you split-test one page against another with very little initial test traffic. Yes, it feels like more legwork to find people who will have a conversation with you but a 15 minute unscripted conversation has more validated learning pound for pound than all the data you can crunch from web analytics.

I have never tried using surveys with my landing page registrations because I hate filling them out myself. Plus, to make them easy to fill, you have to be very specific which assumes you know exactly what you want to know, which is hardly the case.

The beauty of Steve’s process, is that it tests the problem separately from your solution.
To paraphrase Dave McClure:

Customers care about their problems NOT your solution.

During the “Problem Presentation”, you state the top 3 problems, then shut-up and listen which is key to getting it to work. It works because you aren’t asking customers to validate or design a solution which addresses the “Customers don’t know what they want” argument. It works because it isn’t a pitch. Actually, I take that back. It is a pitch. But it’s the customer that’s pitching their problems to you. I know I’ve hit the right “problem nerve” based on how passionate a customer gets during an interview.

I like to structure my “Problem Presentation” like this:

1. State the top 3 problems
2. Ask customer to prioritize problems and identify any higher priority problems
3. Have customer describe how they solve the problem today
4. Very briefly describe how you might solve the problem
5. Ask Customer whether your approach would solve their problem
6. Would they use your solution if it were free?
7. Would they pay $X/yr?
8. Ask for referrals to other customers

Build Your MVP
Build MVP

Unlike Enterprise Software, which can be chock full of features, web startups need to focus on the smallest feature set needed to learn from earlyvangelists or the MVP. After the first reality check, you should end up with a prioritized top 3 problem list which drives the features for your MVP. I stress the importance of then building out the MVP to the point where it’s demo-able. It will be hard for customers to visualize your solution without one. Screenshots and mockups may be used as stand-ins only if a demo is absolutely out of the question.

Test Your MVP
Test MVP

With the MVP built out, you then go test it against the original set of interviewees plus some.
I like to structure my “Product Presentation” like this:

1. State the problem
2. Use the demo to tell a story of how your solution solves the problem
3. Test pricing again
4. Ask for referrals to other customers
5. End with a call to action: sign-up, or commitment to sign-up

Tip: I practice delivering the demo using screencasting software which not only lets me iterate till it’s short and crisp, but I also end up with a video I can use later on the product website.

Iterate or Exit
Verify

The last step in Customer Discovery is to summarize what was learned and make a decision to iterate or exit.

What’s Next?

Next time, I’ll cover my flow for Customer Validation which I promise will look very different (from Steve’s) for a web startup.

While I built this model specifically for my web startup, I feel the fundamentals outlined here apply to any type of web startup, with the exception of startups that *primarily* rely on high network effects, like Twitter and Facebook, where initial success metrics are user acquisition versus revenue based.

What do you think?

Related posts:

  1. Customer Development Checklist for My Web Startup – Part 2
  2. How I learnt to grok Customer Development

Tagged as: customer development, Customer Discovery

  • Awesome post! As well as part II! I come back and refer to it often and am finally getting around to creating a visual of my own flow inspired by yours.

    Question: If you're using screencasting software for your problem presentation, what would be the difference between this and using screenshots of key front-end designs in a problem solving story? There is a difference in the visual flow and clearly having an MVP with a screencast is better, but how much of a difference does this really make in the feedback you get?

    Cheers,
    Luke
  • Better late than never: This is what I currently use do document my business model hypotheses: http://www.ashmaurya.com/wp-content/uploads/2010/06/BMHypothesis.txt

    I've also been using Osterwalder's business model canvas to then capture the essence of the model on a single page. Rob Fitzpatrik has a great online adaptation here: http://thestartuptoolkit.com/

    I highly encourage dating your hypotheses/canvases so you can easily document learnings and pivots over time.
  • Ddsharma
    I am new to Customer Dev but pretty well seasoned in RAD, iterative methods, etc etc. Reading through the Epiphany book, I just could not get out of my head: so when do we touch the keyboard? when do put the chisel against the marble? The Customer Development checklist seemed like just as involved a process (though with a different focus) as traditional product development. And it kept on bugging me that how far and for how long potential customers give input without being able to visualize some draft version of the product (or service -- we may say that web-based startup is really a service startup and this realization could open up other perspectives). I was about to take the Agile model and work on synthesizing it with Customer development.

    Ash .. you saved me the effort. Thanks.

    A question for you and may be Steve: Would you see Steve Jobs using this approach? I mean, not him personally, but would you see an iPhone or a iPad innovation emerge out of this type of exercise?
  • Vince
    Hello
    Very interesting article.
    Would you able to post some sample top 3 problems? The issue we're having is we feel we are 'leading' the customer in the Test the Problem phase.

    For example, do you use something like:

    - do all members in your house find it difficult to find a fire extinguisher in your home?

    or

    - most people can't find a fire extinguisher in their home, do you?

    It would really help.

    Thanks
  • Vince -

    A good problem statement sets the context and resonates with the customer's worldview. That is not "leading". What you should say here is what you would put on a landing page. They key is to keep it brief (under a minute), and then let the customer talk - asking them to agree/disagree with your problem statement and following up with open ended questions to understand how they solve the problem today. Even if they initially fake agreeing with your problem statement out of politeness, the follow-up questions should help uncover that.

    The opening gambit is important and I've found Jerry Weissman's book: Presenting to Win: The Art of Telling Your Story, Updated and Expanded Edition pretty helpful for that.

    In your case, I like the second version better because it specifically connects with the interviewee, but without understanding your product, it's hard to judge. Are you building a service that helps people more easily find a fire extinguisher? What if they don't have one at all? Are you selling fire extinguishers?
  • Vince
    Ash -

    The product/service would be remote fire protection services (e.g. in-house surpression /remote web monitoring etc.).

    Your format of 'closed question/open question follow up" is the key here, the latter I think must be included after their agreement/disagreement with the problem statement - but before asking how they would solve it. Essentially this is the 'why' they think the way they do. Perhaps I was following your problem statement outline too rigidly.

    So a problem statement might be:
    - fire extinguishers are seldom maintained and hard to find in an emergency
    followed by the open 'why do you think that way'.

    I think including 'others' in problem statements i.e " most people can't find..." creates a social bias ( this is a concept in R. Caldini''s book Influence, Science and Practice" and might skew results) - but wondered how other's were doing it, especially up and running business owners such as yourself.

    Many thanks for your reply.
    Vince
  • Hi Ash, enjoying your blog very much. You mentioned you would put up a link to your worksheets in the comments but I don't see it. Would you mind please doing this? Thanks.
  • Hi Ash ... fantastic work on this and thoroughly enjoying putting your process into practice in the projects I'm working on ... having gone through one iteration of Customer Discovery with Involvd I thought you might like to see my thoughts on the problem presentation interview

    http://www.involvd.com/blog/2010/04/28/customer-discovery-interviews-in-the-lean-startup-development-process/

    Nick
  • Great job Ash!

    I am an inspiring french entrepreneur following Steve since a year at least and planning to launch a new web service in France. I was looking for a practical methodology to launch a web startup following Steve's customer development model but was not satisfied with the flow. I was trying to find the best compromise between Steve model, Guy Kawasaki, Eric Ries and Sean Ellis methodology. I think you gave me the right model. Thanks so much for your valuable posts that will certainly save me a lot of time and money.

    I really think this is the new way to start a web company without passing through the useless business plan and market researchs. Thanks again. I will update later on my own experience if you agree.

    Philippe
  • I'm glad it helped and yes please share your own experiences when you are ready.

    Cheers.
  • I'm reading Steve's book now and posted a review on this site today. I thought I'd leave the office and get some feedback on my idea. So far I've made three calls and got two prospective customers that really like the idea....and would be prepared to pay for it. The hardest part is getting the meetings.

    I'm sold on the method and now need to get UK entrepreneurs up the learning curve on the Four Steps and Lean Start-up method and process. What was really funny on both calls was that the customer wanted to know the big picture before they would answer the questions ...they both have problems our product will potentially solve.
  • Congratulations on getting out of the building and taking the first step towards problem/solution fit.
    Yes, coordinating meetings is really hard work but the payoff is well worth it.
  • Ash, great post! I've been following your blog for some time now and it's one of the most sincere and open blogs in this space. Keep up the good work!

    We are in the process of releasing our first iteration of MVP and getting into the what you refer to as "Test MVP" phase. Since our product has two distinct groups of customers, consumers and retailers, it becomes even more challenging to analyze minimum MVP, distribution, pricing, demand creation, market type and competitive landscape.

    You have to take both groups into account, understand all the underlying drivers and then come up with a product that works for both. The whole chicken and egg problem becomes a real challenge, but that's why we're here, right :)) Would definitely like to hear any of your insights into products that fit into this category.
  • Marko - Thanks for the kind words.

    Are both groups of users paying customers? Even so, I'd recommend creating a value stream map and start at the head. My guess is retailers but I might be making the wrong assumption.

    Even for CloudFire, we have 2 sets of users - sharers, and viewers. Even though sharers are the primary paying customers, over time viewers will possibly be more valuable. For one, there are more of them. They are the ones that will help spread the product and the could possibly become secondary paying customers (e.g. order prints). But since sharers have to logically come first, we had to start with that group first.
  • Only retailers are going to be paying, but the only reason they will pay is if there are consumers on the platform. We are initially going to start with a micro segment of retailers in order to minimize the amount of time and effort spent on bringing retailers on board and focus our attention to consumers and their experience. By building the consumer base we will make the platform appealing to a larger number of retailers. Once we expand beyond our retailer micro segment, we will expand our focus to both groups.

    Have you got any good resources around value stream maps that are specific to software products?
  • You just described your value stream. Your success metrics are a little different than a traditional SaaS since you initially value user engagement more than revenue. I would spend most of the effort during "Test MVP" on consumers first and yes keep working with a small segment of retailers to understand their problems. But getting consumers on board and even through Customer Validation is pre-requisite to building anything for retailers.
  • Enjoyed this read Ash. I also really agree with a sentiment you expressed in the comments above -- it isn't about empirical numbers, but continuing to ask and learn until you feel like you aren't learning anything new, then move on to testing something new.

    In my own situation, we're past the main wave of customer interviews to understand a problem and focus our efforts, and now about to get an alpha up and running -- I'm not even sure I would qualify it as MVP but we need to get beyond what people say and really find out what they will do. I'm optimistic that even an insanely minimal experience will shed meaningful light.
  • Ryder - You're right on the link. I'll be correcting that shortly.

    Thanks.
  • Great post. Thanks for sharing it! I found you from hackernews and I'll definitely be printing this flowchart out. I noticed one possible bug though. Is "1st Advisory Board" supposed to link to "Verify the Problem" ?

    Cheers,
    Ryder
  • This is a fantastic post, Ash. It is the clearest explanation for how Customer Discovery should be done, but that aside, you had me at the flowchart (I'm a sucker for those).

    One question: When formulating your customer hypotheses, how do you suggest narrowing down a target niche when the hypothesized problem could apply to a large segment of the consumer web market (think: anybody that purchases online)? I'm struggling with how to better define a customer niche and moreover, how to find those people. Any thoughts?
  • Thank you Brian.

    Without knowing the specifics of the problem you are solving, it is hard to make any specific recommendations, but I can share some general thoughts from my experience. First, I don't know if most people noticed this, but I switched the order of the first two hypotheses from Steve Blank's flow. Steve had you thinking about your product before the customer problem. I think it should be the other way around, otherwise you are a solution looking for a problem to solve.

    Once you have a clear hypothesis on the problem you are are solving, try to identify subsets of users/verticals and rate the level of pain. Using my first product BoxCloud as an example, which is a dead-simple file sharing product, I looked for people that struggled with sharing large files. Graphic designers, game developers, architects, lawyers, doctors, etc. etc. all came to mind. I ranked them based on which had the most pain and would be easiest to reach. Medical, for instance, is hard to crack because of HIPPA compliance. To me, graphic designers felt like the best place to start so I started there. The exit criteria of Customer Development is determining if you've found the right problem to solve for the right customer. If not, rinse and repeat with another customer group. With my second product CloudFire, I considered 2 customer groups - parents, and professional photographers. I chose the first because I had just become a parent, felt the pain first-hand, and could readily reach other parents like me.

    We all want to build mainstream products but you only get there by starting razor focussed. I would look for ways to sub-target your online purchasers based on perceived pain and your ability to identify and reach them. If you want to get into more specifics email me directly and we could chat offline.
  • Ash, a somewhat naive question - but how do you get customers to *speak* to you? I have a feeling that there would be too much friction in getting them to fix a time for a meeting. No? Also, you briefly mentioned in your comment that you get some of your customer leads through a landing page. Can you tell how do you follow up with them to fix a meeting? Perhaps an example?

    The key concern of mine is how to get people to talk for customer validation. A case study or an example on this would be fantastic.
  • Paras -

    All my customer interviews have been done either face-to-face or over the phone. The added legwork pays off after you've talked to them.

    I used a landing page in my last product to collect email addresses but I didn't use that list for CD but rather to announce the product once it was launched. I guess you could send out an email and see if people would be up for a 15 minute call. After all, they already expressed interest in the product by registering.

    For CloudFire, because I am targeting parents like myself, I first tapped into friends, then my kids' daycare, then birthday parties. Kids, their parents, and cameras are easy to spot.

    Getting customers to talk is slightly different. A lot of my early customers are local and people I know so I directly get in touch with them and meet over coffee or lunch. I also have an 800 number which is a great way to have ad-hoc conversations with customers. They usually call in with an email or question. After answering, I ask them a question or two.

    KISSmetrics has an interesting way of preparing their private beta customers for CD making them aware that surveys and short calls are to be expected in exchange for early access. You can't of course make people stick to this.

    In the end, I'd say it's key to build relationships with your early customers. Just having 5 or 10 really engaged customers can teach you a lot in your quest for product/market fit (maybe there is a blog post in here).
  • ammosov
    Ash: what if you are separated from your customers by thousands of miles (which is quite typical both in Web, social gaming and in mobile space) and will never see them or speak to them in course of normal and quite profitable business? My feel is that it is cheaper to trace their problems by actually launching what you believe is an MVP and then exchange messages with those that start using it. In terms of cost, it is far less costly. Am I wrong?
  • What you are describing is the "release early, release often" approach. While it can work (and has worked before), I feel the chances of building an MVP with the right problem/solution fit out the gate is lower than when you talk to a representative set of customers first. If you build the wrong MVP, you may never get the chance to "exchange messages" with users because they may never even use your product or respond to your inquiries.

    The "release early, release often" approach starts by building a guess. Why not validate it first before building? You don't have to talk to everyone, just a representative set of customers.
  • ammosov
    Ash: thank you. Who would then be customers if the site or app is free and relies on ads (in case of freemium, things get more complex)? Is it visitors/users or advertisers? Are Google, FB or Twitter users "customers" or part of product sold to corporate customers?
  • Yes, *desperately* looking forward to you expanding this comment into a blog post. I guess getting people to talk (and not just exchange messages over email) for customer validation is a topic which many people can benefit from.
  • Ash -

    Thank you for you answer. My comment is probably too "Enterprise" specific where developments are often longer and where the development of infrastructure and basic parts can start without even thinking about the added value features themselves.

    But, correct me if I am wrong, unless you are in a totally new market, you always have some guesses about the MVP looking around you for existing solutions.

    Look at your competitors, their messages, ... it is already a point to start your hypothesis and developments.

    Based on your experiences, are your conclusions different ? Is it just "educated" guesses ?
  • Yves –

    I think the answer depends on market type and yes there are cases where you might be able to start laying out a foundation for the product because it is independent of CD but I would still exercise some caution there.

    For example, even if think you are in an existing market building a knock-off product for half the price, without CD you wouldn’t know whether price alone is really enough to win over customers. Chances are the existing product has a boat-load of features. Do you need to implement all of them, or only 20% to start? Which 20%? This is not always obvious.

    Before I went through CD for CloudFire, I thought everyone would want prints and even started researching APIs to print providers, pricing, etc. After CD, I found very few people cared that much about prints and even if we had it, it would just be a feature on a checklist and be no different from our competitors. Being able to share all your photos from a folder in seconds would be different.

    The ultimate goal of customer development is coming up with a unique value proposition that matters. Until you know that, it’s hard to start building an MVP.
  • Dave Copps
    Nice Ash, Almost looks like a Rails dev methodology - build/test/validate, build/test/validate. Thanks for sharing.
  • Thanks for reading Dave.
  • Steve is talking about this right now in his class at UC Berkeley's Haas School of Business
  • Cool - I'll most likely be speaking at your class in April. Hope to meet you then.
  • Not sure about the Build "MVP" block.

    While CD is going on, the product development is still happening from day one.

    Unless you are alone, there should be guys developing the software even when you start stating your hypothesis in a more formal way.

    I would say, it is more like a milestone/synchronization between CD and project development teams that has to happen at that time, because, as you rightly say, it will be difficult in a web startup world to create any real interest without a real demo during the MVP presentation.
  • Yves -

    I disagree. Building a product based on guesses is potentially taking you down the path of building features no one wants which is a form of waste. I would hold off coding until the customer problem and feature prioritization is identified. Being a technologist myself, I know this is hard to follow as we tend to measure progress in terms of what we build. If you do have a product development team already staffed during CD, have them start laying out the infrastructure for measuring metrics and continuous deployment. Once the first MVP is built, you need to constantly be building/measuring/learning. Most people only focus on the first part - building.
  • Great post. One quick question.

    When you say state the top 3 problems and then listen, are you starting with your problem hypothesis and the 3 drivers you think are most important and then iterating? So for sharing photos (I believe this is the website you are discussing for this) would that be 1) Difficulty of upload 2) difficulty organizing 3) having to sign up to share?

    Thanks again for the great post.
  • Lucas -

    Yes that is correct. You want to pick the 1-3 most compelling problems you think customers have (ranked in priority) and have them re-rank them/change them. The point is to set a stage and get them talking.

    I detail the specific problems I started with and what I ended up with in my "How I built my Minimum Viable Product" post.
  • This stuff is great stuff. As an entrepreneur, I'm trying to embrace it... but still love to build first.

    I built PrintFriendly.com. I'm happy to find it on your page!

    Cheers
  • Dror/Craig - Let me get my worksheets into a downloadable format. I'll add a link to the bottom of the post when I'm ready.

    I used a landing page to gather emails for my last product. With the latest one, since it was targeted at new parents, I first hit friends and family, then my kids' daycare, then referrals. Any event with kids, like birthday parties, are prime for casual customer development.

    A landing page can't give you this level of micro-targeting.
  • Dror Engel
    Great work Ash!
    Please upload the hypothesis worksheets .... :)
    did you collect your soon-to-be customers only by emails via landing page?

    Thanks
    Dror
  • Ash, great post. We're going through a similar process for a new product right now and it's good to have some validation for our own processes. We'll have some lean posts coming up soon as well.
  • Vitaliy - We definitely need more practitioners sharing their stories. Looking forward to it.
  • Ash,

    Thanks for doing this. Unbelievably useful. Your slide is going to show up at my Haas class tonight.

    steve
  • Thanks Steve. This one was easy. Wait till you see the next slide on Customer Validation. I'm still trying to make sense of it...
  • Joshua Ho
    Thanks for sharing Ash. Really look forward to your posts on customer validation, customer creation and company building.

    One question. In his book, Steve recommends speaking to at least 20 customers. How many did you actually speak to, considering that yours is a web startup?
  • Good question. I was going to cover that point but I guess I forgot.

    I had my "problem presentation" pipeline set up with 50 people but after about 30 I felt I wasn't learning anything new so I stopped and waited to talk to the rest when I had the "product presentation" ready. As I'll cover in the next post, you never want to stop talking to new customers and I am still doing a more later-stage variation of "product presentation" even today.
  • Great post, I think a lot of us would appreciate also seeing your hypothesis questions if you care to share them. It's very helpful to see the process in action.

    Just a heads up - looks like a dangling sentence in Test the Problem: "This is coming from someone who would rather have"
  • Thank you Craig.
blog comments powered by Disqus

Previous post:

Next post: