Key Concepts

Before you dive into using the Glassfy Experimentation Platform, it's important to grasp some key concepts that are central to experimentation and optimisation. These fundamental concepts will serve as the building blocks of your experimentation journey.

1. A/B Testing

   A/B Testing, also known as split testing, is the practice of comparing two or more variants of a webpage, mobile app, or other user experiences to determine which one performs better. In the context of the Experimentation Platform, A/B testing allows you to compare different versions of your offering to identify which variant drives higher revenue, engagement, or other desired outcomes.

2. Control Group

   A Control Group is a segment of your user base that is not exposed to any experimental changes. It serves as the baseline against which you compare the performance of your experimental variants. The control group helps you measure the impact of changes by providing a reference point for what would happen without the experiment.

3. Variants

   Variants are the different versions of your offering that you create to test specific changes. For example, in a pricing experiment, you might have variants with different pricing models or strategies - e.g. different price, different duration, different trial length, etc... Variants allow you to compare the performance of these different versions.

4. Metrics

   Metrics are the quantitative measurements used to assess the success of your experiments. They can include key performance indicators (KPIs) such as revenue or conversion rate. You define which metrics matter most for your experiments, and the platform helps you track and analyze them.

5. Statistical Significance

   Statistical Significance is a measure of the likelihood that the differences observed in your experiment are not due to random chance. It helps you determine whether the observed variations in metrics between variants are statistically meaningful. A high level of statistical significance is essential to make confident decisions based on experiment results.

6. Randomization

   Randomization is the process of randomly assigning users to different variants in your experiment. It ensures that the experiment groups are comparable and eliminates bias. The platform automatically handles randomization to maintain the integrity of your experiments.

7. Hypothesis

   A Hypothesis is a statement that outlines the expected impact of a change you want to test in your experiment. It provides a clear prediction of what you believe will happen if you implement the change. Experimentation allows you to test and validate hypotheses.

8. Confidence Level

   The Confidence Interval the threshold for the Statistical Significance for the test to reject the null hypothesis that the variant offering is the same as the control offering, and accept the alternative hypothesis that they are different. Specifically, in the Glassfy Experimentation Platform we use a kind of test that only checks if the variant has a more efficient metric with respect to the control (i.e. higher revenue, higher conversion rate, etc...)
   The Significance Level = 1-Confidence Level represents the probability of Type I error, or False Positive, i.e. the probability of rejecting the null hypothesis while it's true.

9. Power

   The Power of an hypothesis test is the probability that the test correctly rejects the null hypothesis when a specific alternative hypothesis is true. It represents the chances of a true positive detection conditional on the actual existence of an effect to detect. 1 - Power represents the probability of Type II error, or False Negative, i.e. the probability of failing to reject the null hypothesis while it is false. Increasing the Power of a test reduces the likelihood of this type of error.

10. Minimum Detectable Effect (MDE)

    The Minimum Detectable Effect (MDE) is the minimum percentage variation of the contro metric with respect to the control group that should make the test positive. Confidence Level, Power and MDE are necessary for the calculation of the Sample Size.

11. Audience Segmentation

    Audience Segmentation involves dividing your user base into specific groups based on criteria like store or (soon) market. Segmenting your audience allows you to target experiments to specific user groups.

12. Duration and Sample Size

    The Duration of an experiment is the period during which the experiment is conducted. The Sample Size is the number of users included in the experiment. The duration and sample size are important considerations in experiment design and can impact the reliability of results.