Zombie Users in a Lean Startup

In the previous post we explained that only new users should be taken into account when validating your Lean Startup hypotheses, where by definition a new user has never been in contact with previous product versions.

OK, so we focus on new users and we are done, right? Not quite so, according to what we've learnt with browseye there are still some important caveats: In this post we'll talk about freshmen, veterans and zombies!

Gotcha: Any given new user goes through two distinct modes: freshman and veteran. Depending on the hypothesis you want to validate you'll want to work with new users in the first or the second mode. Mixing them can lead to fatal hypothesis validation errors.



Example: Browseye is a tool to filter web contents so that you can find very fast what you are looking for on a web page. Initially, we didn't distinguish between freshman and veteran users. This made our activation rate artificially (but happily) high. When we decided that we were not interested in freshmen (read below why) and measured the activation rate again we found that it was in fact much lower than we had thought. The news were not nice to hear, but at least we got an accurate and realistic view on what was going on with our users.

So, who are these freshmen and veterans?

The Freshman Mode

The freshman mode begins right after the user installs or uses the app for the first time and typically lasts 1 day. In this mode the user acts like a freshman in college. Everything is new, so she explores and tries many different things. If a user in freshman mode clicks on a button is not because she likes the functionality behind that button, but because she wants to explore and find out what it does.

Working with users in freshman mode may be useful for testing a hypothesis related to the user experience, like for instance:

Hypothesis: we believe we'll get more registrations if we place the register button here instead of there.

For testing a value or growth hypothesis it's more likely that you'll be interested in measuring user behavior in the veteran mode.

The Veteran Mode

The veteran mode begins right after the freshman mode ends. Users in veteran mode already have a reasonable idea about what the app can and can't do. This is very interesting because it means that you can measure which parts of the app bring them value and which don't.

If a user thinks the app doesn't bring her any value she'll either uninstall it or become a zombie user.

Zombies are those users who have your app installed but don't use it because they see no value in it. 

 If a user finds your app valuable she will become an active user who will access and use it regularly.

Active users are perfect for testing your value and growth hypotheses.

Gotcha: Be careful with zombie users. If you have a nonnegligible number of them they will bias your measurements and make you think that your active users are less active than they really are.

Example: At browseye, once it took us a week to deploy a new version. This made our new users become rather old and some of them become zombies. So, when we were measuring the activation rate (in our case how many users filtered pages relative to the number of pages they visited) we got unusually low numbers because of the bias introduced by our small zombie community. Our numbers smelled really bad, luckily we could identify where the smell was coming from...