Experimentation Workflow

Successful experimentation is a structured process that begins with defining objectives and concludes with analyzing results and making data-driven decisions. This section provides an overview of the experimentation workflow to guide you through each stage of the process.

1. Define Your Experiment Objectives

   Before you start any experiment, it's essential to have clear and specific objectives in mind. What are you trying to achieve with the experiment? Whether it's optimizing pricing, increasing user engagement, or improving a specific aspect of your mobile app, defining your objectives is the first step.

2. Create a Hypothesis

   Formulate a hypothesis based on your objectives. A hypothesis is a testable statement that predicts the outcome of your experiment. For example, "Changing the pricing model will lead to a higher conversion rate."

3. Design Your Experiment

   In this stage, you'll need to:

   1. Select Metrics: Choose the metrics you'll use to evaluate the success of your experiment. These metrics should directly align with your objectives.
   2. Create Variants: Develop the different versions of your mobile app that you want to test. Variants represent the changes or features you're experimenting with.
   3. Define the Control Group: Determine the segment of your users that will serve as the control group – the baseline against which you'll compare the performance of your experimental variants.
   4. Segment Your Audience: Consider using audience segmentation to target specific user groups with your experiment. This is particularly useful if you want to test changes on a specific subset of your user base.

4. Set Up Randomization

   To ensure unbiased results, it's crucial to employ randomization. The platform will automatically assign users to variants in a random and statistically sound manner, preventing any inherent biases in the assignment process.

5. Run the Experiment

   Initiate your experiment by starting it within the platform. Monitor its progress and data collection throughout the designated time frame.

6. Collect and Analyze Data

   Throughout the experiment, collect data on the defined metrics for both the control group and the experimental variants. The platform automatically analyses the data to assess the impact of the changes on your objectives. The platform also automatically evaluates the statistical significance of the experiment results. A high level of statistical significance indicates that the observed differences are unlikely to be due to chance.

7. Make Informed Decisions

   Based on the experiment results and statistical significance, make informed decisions about whether to implement the changes tested in your variants or stick with the control group. Your decisions should align with your initial objectives.

8. Implement Changes

   If the experiment shows that one or more variants significantly outperform the control group and align with your objectives, it's time to implement these changes in your mobile app.

9. Monitor Ongoing Impact

   Even after implementing changes, it's important to continuously monitor the performance of your app to ensure that the positive effects observed in the experiment are sustained in real-world conditions.

10. Iterate and Refine

    Experimentation is an iterative process. Based on the insights gained from one experiment, you can design and run subsequent experiments to further refine your mobile app and optimise its performance.

11. Document and Share

    Keep a record of your experiments, results, and decisions. Sharing these insights and best practices with your team can help improve collaboration and alignment on optimisation strategies.

By following this experimentation workflow, you can systematically optimise your mobile app's performance, whether it's revenue, user engagement, or other key objectives.