--- title: AB Test Utils keywords: fastai sidebar: home_sidebar summary: "AB Test Utils." description: "AB Test Utils." nb_path: "nbs/utils/ab_testing.ipynb" ---
{% raw %}
{% endraw %}

Outline for A/B Tests:

1. Set up the experiment.
2. Run the test and record the success rate for each group.
3. Plot the distribution of the difference between the two samples.
4. Calculate the statistical power.
5. Evaluate how sample size affects A/B tests.

{% raw %}
{% endraw %}

## Stats

{% raw %}

#### `pooled_prob`[source]

> `pooled_prob`(`N_A`, `N_B`, `X_A`, `X_B`)

Returns pooled probability for two samples

{% endraw %} {% raw %}

#### `pooled_SE`[source]

> `pooled_SE`(`N_A`, `N_B`, `X_A`, `X_B`)

Returns the pooled standard error for two samples

{% endraw %} {% raw %}

#### `confidence_interval`[source]

> `confidence_interval`(`sample_mean`=`0`, `sample_std`=`1`, `sample_size`=`1`, `sig_level`=`0.05`)

Returns the confidence interval as a tuple

{% endraw %} {% raw %}

#### `z_val`[source]

> `z_val`(`sig_level`=`0.05`, `two_tailed`=`True`)

Returns the z value for a given significance level

{% endraw %} {% raw %}

#### `ab_dist`[source]

> `ab_dist`(`stderr`, `d_hat`=`0`, `group_type`=`'control'`)

Returns a distribution object depending on group type Examples: Parameters: stderr (float): pooled standard error of two independent samples d_hat (float): the mean difference between two independent samples group_type (string): 'control' and 'test' are supported Returns: dist (scipy.stats distribution object)

{% endraw %} {% raw %}

#### `min_sample_size`[source]

> `min_sample_size`(`bcr`, `mde`, `power`=`0.8`, `sig_level`=`0.05`)

Returns the minimum sample size to set up a split test Arguments: bcr (float): probability of success for control, sometimes referred to as baseline conversion rate mde (float): minimum change in measurement between control group and test group if alternative hypothesis is true, sometimes referred to as minimum detectable effect power (float): probability of rejecting the null hypothesis when the null hypothesis is false, typically 0.8 sig_level (float): significance level often denoted as alpha, typically 0.05 Returns: min_N: minimum sample size (float) References: Stanford lecture on sample sizes http://statweb.stanford.edu/~susan/courses/s141/hopower.pdf

{% endraw %} {% raw %}

#### `p_val`[source]

> `p_val`(`N_A`, `N_B`, `p_A`, `p_B`)

Returns the p-value for an A/B test

{% endraw %} {% raw %}
{% endraw %}

## Plotting

{% raw %}

#### `plot_norm_dist`[source]

> `plot_norm_dist`(`ax`, `mu`, `std`, `with_CI`=`False`, `sig_level`=`0.05`, `label`=`None`)

Adds a normal distribution to the axes provided Example: plot_norm_dist(ax, 0, 1) # plots a standard normal distribution Parameters: ax (matplotlib axes) mu (float): mean of the normal distribution std (float): standard deviation of the normal distribution Returns: None: the function adds a plot to the axes object provided

{% endraw %} {% raw %}

#### `plot_binom_dist`[source]

> `plot_binom_dist`(`ax`, `n`, `p`, `label`=`None`)

Adds a binomial distribution to the axes provided Example: plot_binom_dist(ax, 0, 1) # plots a standard normal distribution Parameters: ax (matplotlib axes) mu (float): mean of the normal distribution sig (float): standard deviation of the normal distribution Returns: None: the function adds a plot to the axes object provided

{% endraw %} {% raw %}

#### `plot_CI`[source]

> `plot_CI`(`ax`, `mu`, `s`, `sig_level`=`0.05`, `color`=`'grey'`)

Calculates the two-tailed confidence interval and adds the plot to an axes object. Example: plot_CI(ax, mu=0, s=stderr, sig_level=0.05) Parameters: ax (matplotlib axes) mu (float): mean s (float): standard deviation Returns: None: the function adds a plot to the axes object provided

{% endraw %} {% raw %}

#### `plot_null`[source]

> `plot_null`(`ax`, `stderr`)

Plots the null hypothesis distribution where if there is no real change, the distribution of the differences between the test and the control groups will be normally distributed. The confidence band is also plotted. Example: plot_null(ax, stderr) Parameters: ax (matplotlib axes) stderr (float): the pooled standard error of the control and test group Returns: None: the function adds a plot to the axes object provided

{% endraw %} {% raw %}

#### `plot_alt`[source]

> `plot_alt`(`ax`, `stderr`, `d_hat`)

Plots the alternative hypothesis distribution where if there is a real change, the distribution of the differences between the test and the control groups will be normally distributed and centered around d_hat The confidence band is also plotted. Example: plot_alt(ax, stderr, d_hat=0.025) Parameters: ax (matplotlib axes) stderr (float): the pooled standard error of the control and test group Returns: None: the function adds a plot to the axes object provided

{% endraw %} {% raw %}

#### `abplot`[source]

> `abplot`(`N_A`, `N_B`, `bcr`, `d_hat`, `sig_level`=`0.05`, `show_power`=`False`, `show_alpha`=`False`, `show_beta`=`False`, `show_p_value`=`False`, `show_legend`=`True`)

Example plot of AB test Example: abplot(n=4000, bcr=0.11, d_hat=0.03) Parameters: n (int): total sample size for both control and test groups (N_A + N_B) bcr (float): base conversion rate; conversion rate of control d_hat: difference in conversion rate between the control and test groups, sometimes referred to as minimal detectable effect when calculating minimum sample size or lift when discussing positive improvement desired from launching a change. Returns: None: the function plots an AB test as two distributions for visualization purposes

{% endraw %} {% raw %}

#### `show_area`[source]

> `show_area`(`ax`, `d_hat`, `stderr`, `sig_level`, `area_type`=`'power'`)

Fill between upper significance boundary and distribution for alternative hypothesis

{% endraw %} {% raw %}

#### `zplot`[source]

> `zplot`(`area`=`0.95`, `two_tailed`=`True`, `align_right`=`False`)

Plots a z distribution with common annotations Example: zplot(area=0.95) zplot(area=0.80, two_tailed=False, align_right=True) Parameters: area (float): The area under the standard normal distribution curve. align (str): The area under the curve can be aligned to the center (default) or to the left. Returns: None: A plot of the normal distribution with annotations showing the area under the curve and the boundaries of the area.

{% endraw %} {% raw %}

#### `abplot_CI_bars`[source]

> `abplot_CI_bars`(`N`, `X`, `sig_level`=`0.05`, `dmin`=`None`)

Returns a confidence interval bar plot for multivariate tests Parameters: N (list or tuple): sample size for all groups X (list or tuple): number of conversions for each variant sig_level (float): significance level dmin (float): minimum desired lift; a red and green dashed lines are shown on the plot if dmin is provided. Returns: None: A plot of the confidence interval bars is returned inline.

{% endraw %} {% raw %}

#### `funnel_CI_plot`[source]

> `funnel_CI_plot`(`A`, `B`, `sig_level`=`0.05`)

Returns a confidence interval bar plot for multivariate tests Parameters: A (list of tuples): (sample size, conversions) for control group funnel B (list of tuples): (sample size, conversions) for test group funnel sig_level (float): significance level Returns: None: A plot of the confidence interval bars is returned inline.

{% endraw %} {% raw %}
{% endraw %}

## 1. Set Up The Experiment

We will run an A/B test for a hypothetical company that is trying to increase the amount of users that sign up for a premium account. The goal of running an A/B test is to evaluate if a change in a website will lead to improved performance in a specific metric. You may decide to test very simple alternatives such as changing the look of a single button on a webpage or testing different layouts and headlines. You could also run an A/B test on multi-step processes which may have many differences. Examples of this include the steps required in signing up a new user or processing the sale on an online marketplace. A/B testing is a huge subject and there are many techniques and rules on setting up an experiment.

### Baseline Conversion Rate and Lift

Before running the test, we will know the baseline conversion rate and the desired lift or increase in signups that we would like to test. The baseline conversion rate is the current rate at which we sign up new users under the existing design. For our example, we want to use our test to confirm that the changes we make to our signup process will result in at least a 2% increase in our sign up rate. We currently sign up 10 out of 100 users who are offered a premium account.

{% raw %}

    bcr = 0.10  # baseline conversion rate
    d_hat = 0.02  # difference between the groups

{% endraw %}

### Control Group (A) and Test Group (B)

Typically, the total number of users participating in the A/B test make up a small percentage of the total amount of users. Users are randomly selected and assigned to either a control group or a test group. The sample size that you decide on will determine how long you might have to wait until you have collected enough. For example, websites with large audiences may be able to collect enough data very quickly, while other websites may have to wait a number of weeks. There are some events that happen rarely even for high-traffic websites, so determining the necessary sample size will inform how soon you can assess your experiment and move on to improving other metrics.

Initially, we will collect 1000 users for each group and serve the current signup page to the control group and a new signup page to the test group.

{% raw %}

    N_A = 1000
    N_B = 1000

{% endraw %}

## 2. Run the Test

Because this is a hypothetical example, we will need “fake” data to work on. We will use a function that will generate data for our simulation.

{% raw %}

    def generate_data(N_A, N_B, p_A, p_B, days=None, control_label='A',
                      test_label='B'):
        """Returns a pandas dataframe with fake CTR data
        Example:
        Parameters:
            N_A (int): sample size for control group
            N_B (int): sample size for test group
                Note: final sample size may not match N_A provided because the
                group at each row is chosen at random (50/50).
            p_A (float): conversion rate; conversion rate of control group
            p_B (float): conversion rate; conversion rate of test group
            days (int): optional; if provided, a column for 'ts' will be included
                to divide the data in chunks of time
                Note: overflow data will be included in an extra day
            control_label (str)
            test_label (str)
        Returns:
            df (df)
        """

        # initiate empty container
        data = []

        # total amount of rows in the data
        N = N_A + N_B

        # distribute events based on proportion of group size
        group_bern = scs.bernoulli(N_A / (N_A + N_B))

        # initiate bernoulli distributions from which to randomly sample
        A_bern = scs.bernoulli(p_A)
        B_bern = scs.bernoulli(p_B)

        for idx in range(N):
            # initite empty row
            row = {}
            # for 'ts' column
            if days is not None:
                if type(days) == int:
                    row['ts'] = idx // (N // days)
                else:
                    raise ValueError("Provide an integer for the days parameter.")
            # assign group based on 50/50 probability
            row['group'] = group_bern.rvs()

            if row['group'] == 0:
                # assign conversion based on provided parameters
                row['converted'] = A_bern.rvs()
            else:
                row['converted'] = B_bern.rvs()
            # collect row into data container
            data.append(row)

        # convert data into pandas dataframe
        df = pd.DataFrame(data)

        # transform group labels of 0s and 1s to user-defined group labels
        df['group'] = df['group'].apply(
            lambda x: control_label if x == 0 else test_label)

        return df

{% endraw %} {% raw %}

    ab_data = generate_data(N_A, N_B, bcr, bcr+d_hat)
    ab_data.head(10)

| group | converted
--- | --- | ---
0 | A | 0
1 | A | 0
2 | B | 0
3 | B | 0
4 | A | 0
5 | B | 0
6 | A | 0
7 | A | 0
8 | A | 0
9 | A | 0

{% endraw %}

The converted column indicates whether a user signed up for the premium service or not with a 1 or 0, respectively. The A group will be used for our control group and the B group will be our test group.

Let’s look at a summary of the results using the pivot table function in Pandas.

{% raw %}

    ab_summary = ab_data.pivot_table(values='converted', index='group', aggfunc=np.sum)
    # add additional columns to the pivot table
    ab_summary['total'] = ab_data.pivot_table(values='converted', index='group', aggfunc=lambda x: len(x))
    ab_summary['rate'] = ab_data.pivot_table(values='converted', index='group')
    ab_summary

| converted | total | rate
--- | --- | --- | ---
group |  |  |
A | 90 | 961 | 0.093652
B | 106 | 1039 | 0.102021

{% endraw %}

It looks like the difference in conversion rates between the two groups is 0.024 which is more than the lift we initially wanted of 0.02. This is a good sign that we should adopt the new design, but this is not enough evidence for us to confidently accept the new design. At this point we have not measured how confident we are in this result. This can be mitigated by looking at the distributions of the two groups.

## 3. Compare the Two Groups

We can compare the two groups by plotting the distribution of the control group and calculating the probability of getting the result from our test group. We can assume that the distribution for our control group is binomial because the data is a series of Bernoulli trials, where each trial only has two possible outcomes (similar to a coin flip).

{% raw %}

    A_converted = ab_summary.loc['A','converted']
    A_total = ab_summary.loc['A','total']
    A_cr = ab_summary.loc['A','rate']

    B_converted = ab_summary.loc['B','converted']
    B_total = ab_summary.loc['B','total']
    B_cr = ab_summary.loc['B','rate']

{% endraw %} {% raw %}

    fig, ax = plt.subplots(figsize=(12,6))
    x = np.linspace(A_converted-49, A_converted+50, 100)
    y = scs.binom(A_total, A_cr).pmf(x)
    ax.bar(x, y, alpha=0.5)
    ax.axvline(x=B_cr * A_total, c='blue', alpha=0.75, linestyle='--')
    plt.xlabel('converted')
    plt.ylabel('probability')

    Text(0, 0.5, 'probability')

{% endraw %}

The distribution for the control group is shown in red and the result from the test group is indicated by the blue dashed line. We can see that the probability of getting the result from the test group is very high. However, the probability does not convey the confidence level of the results. It does not take the sample size of our test group into consideration. Intuitively, we would feel more confident in our results as our sample sizes grow larger. Let’s continue and plot the test group results as a binomial distribution and compare the distributions against each other.

### Binomial distribution

{% raw %}

    fig, ax = plt.subplots(figsize=(12,6))
    xA = np.linspace(A_converted-49, A_converted+50, 100)
    yA = scs.binom(A_total, A_cr).pmf(xA)
    ax.bar(xA, yA, alpha=0.5)
    xB = np.linspace(B_converted-49, B_converted+50, 100)
    yB = scs.binom(B_total, B_cr).pmf(xB)
    ax.bar(xB, yB, alpha=0.5)
    plt.xlabel('converted')
    plt.ylabel('probability')

    Text(0, 0.5, 'probability')

{% endraw %}

### Compare the Null Hypothesis vs. the Alternative Hypothesis

Let’s start off by defining the null hypothesis and the alternative hypothesis.

- The null hypothesis is the position that the change in the design made for the test group would result in no change in the conversion rate.
- The alternative hypothesis is the opposing position that the change in the design for the test group would result in an improvement (or reduction) in the conversion rate.

{% raw %}

    # use the actual values from the experiment for bcr and d_hat
    # p_A is the conversion rate of the control group
    # p_B is the conversion rate of the test group
    bcr = A_cr
    d_hat = B_cr - A_cr
    abplot(N_A, N_B, bcr, d_hat)

{% endraw %}

Fortunately, both curves are identical in shape, so we can just compare the distance between the means of the two distributions. We can see that the alternative hypothesis curve suggests that the test group has a higher conversion rate than the control group. This plot also can be used to directly determine the statistical power.

## 4. Statistical Power and Significance Level

I think it is easier to define statistical power and significance level by first showing how they are represented in the plot of the null and alternative hypothesis. We can return a visualization of the statistical power by adding the parameter show_power=True

{% raw %}

    abplot(N_A, N_B, bcr, d_hat, show_power=True)

{% endraw %} {% raw %}

    abplot(N_A, N_B, bcr, d_hat, show_beta=True)

{% endraw %}

The gray dashed line that divides the area under the alternative curve into two also directly segments the area associated with the significance level, often denoted with the greek letter alpha.

{% raw %}

    abplot(N_A, N_B, bcr, d_hat, show_alpha=True)

{% endraw %}

Experiments are typically set up for a minimum desired power of 80%. If our new design is truly better, we want our experiment to show that there is at least an 80% probability that this is the case. We know that if we increase the sample size for each group, we will decrease the pooled variance for our null and alternative hypothesis. This will make our distributions much narrower and may increase the statistical power. Let’s take a look at how sample size will directly affect our results.

## 5. Sample Size

If we run our test again with a sample size of 2000 instead of 1000 for each group, we get the following results.

{% raw %}

    abplot(2000, 2000, bcr, d_hat, show_power=True)

{% endraw %}

Many people calculate Z from tables such as those shown here and here. However, I am more of a visual learner and I like to refer to the plot of the Z-distribution from which the values are derived.

{% raw %}

    zplot()

{% endraw %} {% raw %}

    zplot(0.80,False,True)

{% endraw %} {% raw %}

    min_sample_size(bcr=0.10, mde=0.02)

    3842.026629963879

{% endraw %} {% raw %}

    abplot(3843, 3843, 0.10, 0.02, show_power=True)

{% endraw %}

The calculated power for this sample size was approximately 0.80. Therefore, if our design change had an improvement in conversion of about 2 percent, we would need at least 3843 samples in each group for a statistical power of at least 0.80.

That was a very long but basic walkthrough of A/B tests. Once you have developed an understanding and familiarity with the procedure, you will probably be able to run an experiment and go directly to the plots for the null and alternative hypothesis to determine if your results achieved enough power. By calculating the minimum sample size you need prior to the experiment, you can determine how long it will take to get the results back to your team for a final decision.