A simple template for your startup growth experiments

As every experiment needs to follow a clear process, a successful growth experiment includes indispensable elements including: a well-defined objective, a hypothesis, and a structured experimental design. In this post, we will assess the significance of growth experiments and identify the deciding elements of a successful one. We will also offer you a growth experiment template we have used to work with our clients.

A growth experiment is a cheap and organic way to boost company’s numbers, compared to traditional initiatives that require significant planning and financial resources. Although many businesses have been successful with certain growth hacks, it is still important that you experiment with your hacks to actually evaluate whether a growth tactic works in favor of your company’s characteristics.

Growth experiments are very popular among startups not for arbitrary reasons. The most important rationale is that throughout this growth experimentation process, companies can have a comprehensive overview about their business and are thus able to set a more structured growth objective and plan a well-designed experiment to achieve business goal. In contrary to random trials that might be used by startups with limited resources, those growth experiments provide a systematic way of assessing which factor of the funnel that the startup should focus on.

Furthermore, growth experiments are usually quicker to implement than traditional growth methods. Traditional methods (such as advertising and PR) usually consume plenty of resources, take a long time to be implemented and to be assessed in terms of business impact. Instead, with growth experiments, startups’ marketing executives can decide which growth experiments to prioritize first based on the available resource constraints of the company. Those growth experiments can vary from one week to six months. During that period, companies can use their tracking metrics to analyze whether their goal has been reached and make corresponding adjustments to improve their current and future experiments.

Lastly, those experiments are typically low-risk. Similarly to how marketers can choose experiment intervals, they can also choose growth tactics depending on their available resources, for example, the number of full-time marketers, the technology and the financial resources the company is processing. Therefore, even though the experiments might not come out as expected, it will not be of a big loss to the startup itself. Additionally, those controlled experiments can help marketers assess whether the introduction of the tactic is the only factor leading to the observed results. For example, in using A/B test, controlled experiments isolate the impact of the product change while controlling for external factors such as the day of the week, or the season those experiments are conducted.

However, it is also important to emphasize that growth is not one trick that is implemented in a short interval of time and can result in hyper growth for every hack. Businesses might be affected by the myth that many startup success stories such as Airbnb’s Craigslist integration are based solely on one hack. Instead, growth is a constant process of experiment.

Before implementing any growth experiments, marketers need to ensure that the following elements are set out and agreed within the team: objectives, hypothesis and experimental design.

The objective of an experiment is what you are actually trying to figure out by conducting this experiment. Objectives should be tied to some of your business’s KPIs or derived from them. There is little point in conducting a growth experiment without having KPIs to track the experiment’s performance. Furthermore, objectives should also be succinctly defined so that marketers would not be distracted mid-way through the experiment, and the whole team can work towards a shared goal. One question you can ask to determine your growth experiment’s objective is “If you were given 40 days to improve one aspect of your startup, what would you focus on?”. Examples of clear objectives are:

In contrast, some pitfalls of objectives can be too vague (Increase signup conversion) or not tied to any KPIs (Improve retention).

Hypothesis

The second element is a hypothesis (what you think will happen by conducting this experiment). To effectively measure improvements, you need to come up with a concrete outcome-based prediction that you can easily test. It should ideally include the metric you want to move and your guess at how much you expect the metric to move by. This is where you are advised to also include any assumptions you make. It is because this will help your company understand why an experiment works, and whether it can be duplicated or scaled up in the future.

The template of how to set a hypothesis is: “If successful, I predict (the metric you are testing) will increase by (% or units of the metric you are testing) because (initial assumption). To have a benchmark on the % or units change of the metric you are testing, you can look at past experiments of other businesses, or your startup’s metrics. Some good examples of a clear hypothesis are:

In contrast, “Signup increase” and “Email deliverability increase” are too ill-defined.

Experiment Design

Once you have identified your growth experiment’s objective and hypothesis, you can move on to design your experiment. As mentioned above, objective and hypothesis should both be closely linked to KPIs, so the first step of this stage is to identify KPIs you would measure on. For example, it could be the subscription number or subscription rate. Next, you can decide what method you would use to conduct the experiment.

The most popular method is A/B testing, where you compare the result observed from the control group (without the change) and that of the treatment group (benefitting from the change). Another frequently-used method is to compare the performance pre and post-treatment; however, since many factors might have changed during this period, we can get biased results. The third step is to list out the resources required to conduct the experiment, from human resources, financial resources to required cooperation with other brands or influencers.

Before you start, you need to ensure that your conversion funnel is tracked from the beginning to end, to see where people are dropping out of your conversion funnel and why it happens like that. After you have identified above elements, you can run the experiment for the previously-set time frame. Once the experiment ends, you can use various tools, such as Google Analytics, Crazy Egg, Compete, Google Website Optimizer, Optimizely, KISSmetrics, ClickTale and Facebook Insights to assess whether your hypothesis is true.

If it is, you can consider doubling-down on the experiment, or see if it provides more insights to other experiments. Otherwise, you can explore why the experiment does not come out the way you expect it to be, and continue refining the experiment until you see some positive results. It is also recommended that you wait for the data to be statistically significant before rejecting the result completely.

However, the experiment might not end here. There are some other questions you might want to answer before moving on to a new experiment; for instance, the long-term effect of the change, or any mistakes you could have made. Types of confounding mistakes could be that another experiment is being tested with the same user set, a big event happens at that time, the sample size is too small, and too many variables between the control group and treatment groups are tested. If none of the issues are at place, we can be more confident in the validity of the experiment.

To illustrate above points, we will assess two successful cases of growth experiments

Airbnb — Referral program

In short, it is important that every startup should have a defined template before implementing any growth experiment. We have also attached a growth experiment template so you can construct experiment in a more structured way. The document is structured in a similar way as the outline above, and includes space where you can elaborate on the results and future learnings from the experiment. Before you conduct a growth experiment, it is recommended you fill in every part of the document and anticipate how the experiment will be carried out in real life.

About Fast Forward Growth Services

Related posts:

I stand here alone. “Stars” is published by Scents of Roses.

Making sense of client or user feedback can be a tricky task, and recently I learned how time consuming and painstaking it can really be. I’m a Product Designer at Ve Global, a marketing and…

Socrates was a Greek philosopher who lived in Athens in 4th century BC during the infancy of democracy as we know it today. He is best known for his critical thinking and the Socratic method, which…