{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# default_exp utils.ab_testing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# AB Test Utils\n",
    "> AB Test Utils."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Outline for A/B Tests:\n",
    "1. Set up the experiment.\n",
    "2. Run the test and record the success rate for each group.\n",
    "3. Plot the distribution of the difference between the two samples.\n",
    "4. Calculate the statistical power.\n",
    "5. Evaluate how sample size affects A/B tests."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#hide\n",
    "from nbdev.showdoc import *"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#export\n",
    "import scipy.stats as scs\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Stats"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#export\n",
    "def pooled_prob(N_A, N_B, X_A, X_B):\n",
    "    \"\"\"Returns pooled probability for two samples\"\"\"\n",
    "    return (X_A + X_B) / (N_A + N_B)\n",
    "\n",
    "\n",
    "def pooled_SE(N_A, N_B, X_A, X_B):\n",
    "    \"\"\"Returns the pooled standard error for two samples\"\"\"\n",
    "    p_hat = pooled_prob(N_A, N_B, X_A, X_B)\n",
    "    SE = np.sqrt(p_hat * (1 - p_hat) * (1 / N_A + 1 / N_B))\n",
    "    return SE\n",
    "\n",
    "\n",
    "def confidence_interval(sample_mean=0, sample_std=1, sample_size=1,\n",
    "                        sig_level=0.05):\n",
    "    \"\"\"Returns the confidence interval as a tuple\"\"\"\n",
    "    z = z_val(sig_level)\n",
    "\n",
    "    left = sample_mean - z * sample_std / np.sqrt(sample_size)\n",
    "    right = sample_mean + z * sample_std / np.sqrt(sample_size)\n",
    "\n",
    "    return (left, right)\n",
    "\n",
    "\n",
    "def z_val(sig_level=0.05, two_tailed=True):\n",
    "    \"\"\"Returns the z value for a given significance level\"\"\"\n",
    "    z_dist = scs.norm()\n",
    "    if two_tailed:\n",
    "        sig_level = sig_level/2\n",
    "        area = 1 - sig_level\n",
    "    else:\n",
    "        area = 1 - sig_level\n",
    "\n",
    "    z = z_dist.ppf(area)\n",
    "\n",
    "    return z\n",
    "\n",
    "\n",
    "def ab_dist(stderr, d_hat=0, group_type='control'):\n",
    "    \"\"\"Returns a distribution object depending on group type\n",
    "    Examples:\n",
    "    Parameters:\n",
    "        stderr (float): pooled standard error of two independent samples\n",
    "        d_hat (float): the mean difference between two independent samples\n",
    "        group_type (string): 'control' and 'test' are supported\n",
    "    Returns:\n",
    "        dist (scipy.stats distribution object)\n",
    "    \"\"\"\n",
    "    if group_type == 'control':\n",
    "        sample_mean = 0\n",
    "\n",
    "    elif group_type == 'test':\n",
    "        sample_mean = d_hat\n",
    "\n",
    "    # create a normal distribution which is dependent on mean and std dev\n",
    "    dist = scs.norm(sample_mean, stderr)\n",
    "    return dist\n",
    "\n",
    "\n",
    "def min_sample_size(bcr, mde, power=0.8, sig_level=0.05):\n",
    "    \"\"\"Returns the minimum sample size to set up a split test\n",
    "    Arguments:\n",
    "        bcr (float): probability of success for control, sometimes\n",
    "        referred to as baseline conversion rate\n",
    "        mde (float): minimum change in measurement between control\n",
    "        group and test group if alternative hypothesis is true, sometimes\n",
    "        referred to as minimum detectable effect\n",
    "        power (float): probability of rejecting the null hypothesis when the\n",
    "        null hypothesis is false, typically 0.8\n",
    "        sig_level (float): significance level often denoted as alpha,\n",
    "        typically 0.05\n",
    "    Returns:\n",
    "        min_N: minimum sample size (float)\n",
    "    References:\n",
    "        Stanford lecture on sample sizes\n",
    "        http://statweb.stanford.edu/~susan/courses/s141/hopower.pdf\n",
    "    \"\"\"\n",
    "    # standard normal distribution to determine z-values\n",
    "    standard_norm = scs.norm(0, 1)\n",
    "\n",
    "    # find Z_beta from desired power\n",
    "    Z_beta = standard_norm.ppf(power)\n",
    "\n",
    "    # find Z_alpha\n",
    "    Z_alpha = standard_norm.ppf(1-sig_level/2)\n",
    "\n",
    "    # average of probabilities from both groups\n",
    "    pooled_prob = (bcr + bcr+mde) / 2\n",
    "\n",
    "    min_N = (2 * pooled_prob * (1 - pooled_prob) * (Z_beta + Z_alpha)**2\n",
    "             / mde**2)\n",
    "\n",
    "    return min_N\n",
    "\n",
    "\n",
    "def p_val(N_A, N_B, p_A, p_B):\n",
    "    \"\"\"Returns the p-value for an A/B test\"\"\"\n",
    "    return scs.binom(N_A, p_A).pmf(p_B * N_B)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Plotting"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#export\n",
    "def plot_norm_dist(ax, mu, std, with_CI=False, sig_level=0.05, label=None):\n",
    "    \"\"\"Adds a normal distribution to the axes provided\n",
    "    Example:\n",
    "        plot_norm_dist(ax, 0, 1)  # plots a standard normal distribution\n",
    "    Parameters:\n",
    "        ax (matplotlib axes)\n",
    "        mu (float): mean of the normal distribution\n",
    "        std (float): standard deviation of the normal distribution\n",
    "    Returns:\n",
    "        None: the function adds a plot to the axes object provided\n",
    "    \"\"\"\n",
    "    x = np.linspace(mu - 12 * std, mu + 12 * std, 1000)\n",
    "    y = scs.norm(mu, std).pdf(x)\n",
    "    ax.plot(x, y, label=label)\n",
    "\n",
    "    if with_CI:\n",
    "        plot_CI(ax, mu, std, sig_level=sig_level)\n",
    "\n",
    "\n",
    "def plot_binom_dist(ax, n, p, label=None):\n",
    "    \"\"\"Adds a binomial distribution to the axes provided\n",
    "    Example:\n",
    "        plot_binom_dist(ax, 0, 1)  # plots a standard normal distribution\n",
    "    Parameters:\n",
    "        ax (matplotlib axes)\n",
    "        mu (float): mean of the normal distribution\n",
    "        sig (float): standard deviation of the normal distribution\n",
    "    Returns:\n",
    "        None: the function adds a plot to the axes object provided\n",
    "    \"\"\"\n",
    "    x = np.linspace(0, n, n+1)\n",
    "    y = scs.binom(n, p).pmf(x)\n",
    "    ax.plot(x, y, label=label)\n",
    "\n",
    "\n",
    "def plot_CI(ax, mu, s, sig_level=0.05, color='grey'):\n",
    "    \"\"\"Calculates the two-tailed confidence interval and adds the plot to\n",
    "    an axes object.\n",
    "    Example:\n",
    "        plot_CI(ax, mu=0, s=stderr, sig_level=0.05)\n",
    "    Parameters:\n",
    "        ax (matplotlib axes)\n",
    "        mu (float): mean\n",
    "        s (float): standard deviation\n",
    "    Returns:\n",
    "        None: the function adds a plot to the axes object provided\n",
    "    \"\"\"\n",
    "    left, right = confidence_interval(sample_mean=mu, sample_std=s,\n",
    "                                      sig_level=sig_level)\n",
    "    ax.axvline(left, c=color, linestyle='--', alpha=0.5)\n",
    "    ax.axvline(right, c=color, linestyle='--', alpha=0.5)\n",
    "\n",
    "\n",
    "def plot_null(ax, stderr):\n",
    "    \"\"\"Plots the null hypothesis distribution where if there is no real change,\n",
    "    the distribution of the differences between the test and the control groups\n",
    "    will be normally distributed.\n",
    "    The confidence band is also plotted.\n",
    "    Example:\n",
    "        plot_null(ax, stderr)\n",
    "    Parameters:\n",
    "        ax (matplotlib axes)\n",
    "        stderr (float): the pooled standard error of the control and test group\n",
    "    Returns:\n",
    "        None: the function adds a plot to the axes object provided\n",
    "    \"\"\"\n",
    "    plot_norm_dist(ax, 0, stderr, label=\"Null\")\n",
    "    plot_CI(ax, mu=0, s=stderr, sig_level=0.05)\n",
    "\n",
    "\n",
    "def plot_alt(ax, stderr, d_hat):\n",
    "    \"\"\"Plots the alternative hypothesis distribution where if there is a real\n",
    "    change, the distribution of the differences between the test and the\n",
    "    control groups will be normally distributed and centered around d_hat\n",
    "    The confidence band is also plotted.\n",
    "    Example:\n",
    "        plot_alt(ax, stderr, d_hat=0.025)\n",
    "    Parameters:\n",
    "        ax (matplotlib axes)\n",
    "        stderr (float): the pooled standard error of the control and test group\n",
    "    Returns:\n",
    "        None: the function adds a plot to the axes object provided\n",
    "    \"\"\"\n",
    "    plot_norm_dist(ax, d_hat, stderr, label=\"Alternative\")\n",
    "\n",
    "\n",
    "def abplot(N_A, N_B, bcr, d_hat, sig_level=0.05, show_power=False,\n",
    "           show_alpha=False, show_beta=False, show_p_value=False,\n",
    "           show_legend=True):\n",
    "    \"\"\"Example plot of AB test\n",
    "    Example:\n",
    "        abplot(n=4000, bcr=0.11, d_hat=0.03)\n",
    "    Parameters:\n",
    "        n (int): total sample size for both control and test groups (N_A + N_B)\n",
    "        bcr (float): base conversion rate; conversion rate of control\n",
    "        d_hat: difference in conversion rate between the control and test\n",
    "            groups, sometimes referred to as **minimal detectable effect** when\n",
    "            calculating minimum sample size or **lift** when discussing\n",
    "            positive improvement desired from launching a change.\n",
    "    Returns:\n",
    "        None: the function plots an AB test as two distributions for\n",
    "        visualization purposes\n",
    "    \"\"\"\n",
    "    # create a plot object\n",
    "    fig, ax = plt.subplots(figsize=(12, 6))\n",
    "\n",
    "    # define parameters to find pooled standard error\n",
    "    X_A = bcr * N_A\n",
    "    X_B = (bcr + d_hat) * N_B\n",
    "    stderr = pooled_SE(N_A, N_B, X_A, X_B)\n",
    "\n",
    "    # plot the distribution of the null and alternative hypothesis\n",
    "    plot_null(ax, stderr)\n",
    "    plot_alt(ax, stderr, d_hat)\n",
    "\n",
    "    # set extent of plot area\n",
    "    ax.set_xlim(-8 * stderr, 8 * stderr)\n",
    "\n",
    "    # shade areas according to user input\n",
    "    if show_power:\n",
    "        show_area(ax, d_hat, stderr, sig_level, area_type='power')\n",
    "    if show_alpha:\n",
    "        show_area(ax, d_hat, stderr, sig_level, area_type='alpha')\n",
    "    if show_beta:\n",
    "        show_area(ax, d_hat, stderr, sig_level, area_type='beta')\n",
    "\n",
    "    # show p_value based on the binomial distributions for the two groups\n",
    "    if show_p_value:\n",
    "        null = ab_dist(stderr, 'control')\n",
    "        p_value = p_val(N_A, N_B, bcr, bcr+d_hat)\n",
    "        ax.text(3 * stderr, null.pdf(0),\n",
    "                'p-value = {0:.3f}'.format(p_value),\n",
    "                fontsize=12, ha='left')\n",
    "\n",
    "    # option to show legend\n",
    "    if show_legend:\n",
    "        plt.legend()\n",
    "\n",
    "    plt.xlabel('d')\n",
    "    plt.ylabel('PDF')\n",
    "    plt.show()\n",
    "\n",
    "\n",
    "def show_area(ax, d_hat, stderr, sig_level, area_type='power'):\n",
    "    \"\"\"Fill between upper significance boundary and distribution for\n",
    "    alternative hypothesis\n",
    "    \"\"\"\n",
    "    left, right = confidence_interval(sample_mean=0, sample_std=stderr,\n",
    "                                      sig_level=sig_level)\n",
    "    x = np.linspace(-12 * stderr, 12 * stderr, 1000)\n",
    "    null = ab_dist(stderr, 'control')\n",
    "    alternative = ab_dist(stderr, d_hat, 'test')\n",
    "\n",
    "    # if area_type is power\n",
    "    # Fill between upper significance boundary and distribution for alternative\n",
    "    # hypothesis\n",
    "    if area_type == 'power':\n",
    "        ax.fill_between(x, 0, alternative.pdf(x), color='green', alpha=0.25,\n",
    "                        where=(x > right))\n",
    "        ax.text(-3 * stderr, null.pdf(0),\n",
    "                'power = {0:.3f}'.format(1 - alternative.cdf(right)),\n",
    "                fontsize=12, ha='right', color='k')\n",
    "\n",
    "    # if area_type is alpha\n",
    "    # Fill between upper significance boundary and distribution for null\n",
    "    # hypothesis\n",
    "    if area_type == 'alpha':\n",
    "        ax.fill_between(x, 0, null.pdf(x), color='green', alpha=0.25,\n",
    "                        where=(x > right))\n",
    "        ax.text(-3 * stderr, null.pdf(0),\n",
    "                'alpha = {0:.3f}'.format(1 - null.cdf(right)),\n",
    "                fontsize=12, ha='right', color='k')\n",
    "\n",
    "    # if area_type is beta\n",
    "    # Fill between distribution for alternative hypothesis and upper\n",
    "    # significance boundary\n",
    "    if area_type == 'beta':\n",
    "        ax.fill_between(x, 0, alternative.pdf(x), color='green', alpha=0.25,\n",
    "                        where=(x < right))\n",
    "        ax.text(-3 * stderr, null.pdf(0),\n",
    "                'beta = {0:.3f}'.format(alternative.cdf(right)),\n",
    "                fontsize=12, ha='right', color='k')\n",
    "        \n",
    "\n",
    "def zplot(area=0.95, two_tailed=True, align_right=False):\n",
    "    \"\"\"Plots a z distribution with common annotations\n",
    "    Example:\n",
    "        zplot(area=0.95)\n",
    "        zplot(area=0.80, two_tailed=False, align_right=True)\n",
    "    Parameters:\n",
    "        area (float): The area under the standard normal distribution curve.\n",
    "        align (str): The area under the curve can be aligned to the center\n",
    "            (default) or to the left.\n",
    "    Returns:\n",
    "        None: A plot of the normal distribution with annotations showing the\n",
    "        area under the curve and the boundaries of the area.\n",
    "    \"\"\"\n",
    "    # create plot object\n",
    "    fig = plt.figure(figsize=(12, 6))\n",
    "    ax = fig.subplots()\n",
    "    # create normal distribution\n",
    "    norm = scs.norm()\n",
    "    # create data points to plot\n",
    "    x = np.linspace(-5, 5, 1000)\n",
    "    y = norm.pdf(x)\n",
    "\n",
    "    ax.plot(x, y)\n",
    "\n",
    "    # code to fill areas\n",
    "    # for two-tailed tests\n",
    "    if two_tailed:\n",
    "        left = norm.ppf(0.5 - area / 2)\n",
    "        right = norm.ppf(0.5 + area / 2)\n",
    "        ax.vlines(right, 0, norm.pdf(right), color='grey', linestyle='--')\n",
    "        ax.vlines(left, 0, norm.pdf(left), color='grey', linestyle='--')\n",
    "\n",
    "        ax.fill_between(x, 0, y, color='grey', alpha=0.25,\n",
    "                        where=(x > left) & (x < right))\n",
    "        plt.xlabel('z')\n",
    "        plt.ylabel('PDF')\n",
    "        plt.text(left, norm.pdf(left), \"z = {0:.3f}\".format(left), fontsize=12,\n",
    "                 rotation=90, va=\"bottom\", ha=\"right\")\n",
    "        plt.text(right, norm.pdf(right), \"z = {0:.3f}\".format(right),\n",
    "                 fontsize=12, rotation=90, va=\"bottom\", ha=\"left\")\n",
    "    # for one-tailed tests\n",
    "    else:\n",
    "        # align the area to the right\n",
    "        if align_right:\n",
    "            left = norm.ppf(1-area)\n",
    "            ax.vlines(left, 0, norm.pdf(left), color='grey', linestyle='--')\n",
    "            ax.fill_between(x, 0, y, color='grey', alpha=0.25,\n",
    "                            where=x > left)\n",
    "            plt.text(left, norm.pdf(left), \"z = {0:.3f}\".format(left),\n",
    "                     fontsize=12, rotation=90, va=\"bottom\", ha=\"right\")\n",
    "        # align the area to the left\n",
    "        else:\n",
    "            right = norm.ppf(area)\n",
    "            ax.vlines(right, 0, norm.pdf(right), color='grey', linestyle='--')\n",
    "            ax.fill_between(x, 0, y, color='grey', alpha=0.25,\n",
    "                            where=x < right)\n",
    "            plt.text(right, norm.pdf(right), \"z = {0:.3f}\".format(right),\n",
    "                     fontsize=12, rotation=90, va=\"bottom\", ha=\"left\")\n",
    "\n",
    "    # annotate the shaded area\n",
    "    plt.text(0, 0.1, \"shaded area = {0:.3f}\".format(area), fontsize=12,\n",
    "             ha='center')\n",
    "    # axis labels\n",
    "    plt.xlabel('z')\n",
    "    plt.ylabel('PDF')\n",
    "\n",
    "    plt.show()\n",
    "\n",
    "\n",
    "def abplot_CI_bars(N, X, sig_level=0.05, dmin=None):\n",
    "    \"\"\"Returns a confidence interval bar plot for multivariate tests\n",
    "    Parameters:\n",
    "        N (list or tuple): sample size for all groups\n",
    "        X (list or tuple): number of conversions for each variant\n",
    "        sig_level (float): significance level\n",
    "        dmin (float): minimum desired lift; a red and green dashed lines are\n",
    "            shown on the plot if dmin is provided.\n",
    "    Returns:\n",
    "        None: A plot of the confidence interval bars is returned inline.\n",
    "    \"\"\"\n",
    "\n",
    "    # initiate plot object\n",
    "    fig, ax = plt.subplots(figsize=(12, 3))\n",
    "\n",
    "    # get control group values\n",
    "    N_A = N[0]\n",
    "    X_A = X[0]\n",
    "\n",
    "    # initiate containers for standard error and differences\n",
    "    SE = []\n",
    "    d = []\n",
    "    # iterate through X and N and calculate d and SE\n",
    "    for idx in range(1, len(N)):\n",
    "        X_B = X[idx]\n",
    "        N_B = N[idx]\n",
    "        d.append(X_B / N_B - X_A / N_A)\n",
    "        SE.append(pooled_SE(N_A, N_B, X_A, X_B))\n",
    "\n",
    "    # convert to numpy arrays\n",
    "    SE = np.array(SE)\n",
    "    d = np.array(d)\n",
    "\n",
    "    y = np.arange(len(N)-1)\n",
    "\n",
    "    # get z value\n",
    "    z = z_val(sig_level)\n",
    "    # confidence interval values\n",
    "    ci = SE * z\n",
    "\n",
    "    # bar to represent the confidence interval\n",
    "    ax.hlines(y, d-ci, d+ci, color='blue', alpha=0.35, lw=10, zorder=1)\n",
    "    # marker for the mean\n",
    "    ax.scatter(d, y, s=300, marker='|', lw=10, color='magenta', zorder=2)\n",
    "\n",
    "    # vertical line to represent 0\n",
    "    ax.axvline(0, c='grey', linestyle='-')\n",
    "\n",
    "    # plot veritcal dashed lines if dmin is provided\n",
    "    if dmin is not None:\n",
    "        ax.axvline(-dmin, c='red', linestyle='--', alpha=0.75)\n",
    "        ax.axvline(dmin, c='green', linestyle='--', alpha=0.75)\n",
    "\n",
    "    # invert y axis to show variant 1 at the top\n",
    "    ax.invert_yaxis()\n",
    "    # label variants on y axis\n",
    "    labels = ['variant{}'.format(idx+1) for idx in range(len(N)-1)]\n",
    "    plt.yticks(np.arange(len(N)-1), labels)\n",
    "\n",
    "\n",
    "def funnel_CI_plot(A, B, sig_level=0.05):\n",
    "    \"\"\"Returns a confidence interval bar plot for multivariate tests\n",
    "    Parameters:\n",
    "        A (list of tuples): (sample size, conversions) for control group funnel\n",
    "        B (list of tuples): (sample size, conversions) for test group funnel\n",
    "        sig_level (float): significance level\n",
    "    Returns:\n",
    "        None: A plot of the confidence interval bars is returned inline.\n",
    "    \"\"\"\n",
    "\n",
    "    # initiate plot object\n",
    "    fig, ax = plt.subplots(figsize=(12, 3))\n",
    "\n",
    "    # initiate containers for standard error and differences\n",
    "    SE = []\n",
    "    d = []\n",
    "    # iterate through X and N and calculate d and SE\n",
    "    for idx in range(len(A)):\n",
    "        X_A = A[idx][1]\n",
    "        N_A = A[idx][0]\n",
    "        X_B = B[idx][1]\n",
    "        N_B = B[idx][0]\n",
    "        d.append(X_B / N_B - X_A / N_A)\n",
    "        SE.append(pooled_SE(N_A, N_B, X_A, X_B))\n",
    "\n",
    "    # convert to numpy arrays\n",
    "    SE = np.array(SE)\n",
    "    d = np.array(d)\n",
    "    print(d)\n",
    "\n",
    "    y = np.arange(len(A))\n",
    "\n",
    "    # get z value\n",
    "    z = z_val(sig_level)\n",
    "    # confidence interval values\n",
    "    ci = SE * z\n",
    "\n",
    "    # bar to represent the confidence interval\n",
    "    ax.hlines(y, d-ci, d+ci, color='blue', alpha=0.35, lw=10, zorder=1)\n",
    "    # marker for the mean\n",
    "    ax.scatter(d, y, s=300, marker='|', lw=10, color='magenta', zorder=2)\n",
    "\n",
    "    # vertical line to represent 0\n",
    "    ax.axvline(0, c='grey', linestyle='-')\n",
    "\n",
    "    # invert y axis to show variant 1 at the top\n",
    "    ax.invert_yaxis()\n",
    "    # label variants on y axis\n",
    "    labels = ['metric{}'.format(idx+1) for idx in range(len(A))]\n",
    "    plt.yticks(np.arange(len(A)), labels)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Set Up The Experiment\n",
    "We will run an A/B test for a hypothetical company that is trying to increase the amount of users that sign up for a premium account. The goal of running an A/B test is to evaluate if a change in a website will lead to improved performance in a specific metric. You may decide to test very simple alternatives such as changing the look of a single button on a webpage or testing different layouts and headlines. You could also run an A/B test on multi-step processes which may have many differences. Examples of this include the steps required in signing up a new user or processing the sale on an online marketplace. A/B testing is a huge subject and there are many techniques and rules on setting up an experiment."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Baseline Conversion Rate and Lift\n",
    "Before running the test, we will know the baseline conversion rate and the desired lift or increase in signups that we would like to test. The baseline conversion rate is the current rate at which we sign up new users under the existing design. For our example, we want to use our test to confirm that the changes we make to our signup process will result in at least a 2% increase in our sign up rate. We currently sign up 10 out of 100 users who are offered a premium account."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# code examples presented in Python\n",
    "bcr = 0.10  # baseline conversion rate\n",
    "d_hat = 0.02  # difference between the groups"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Control Group (A) and Test Group (B)\n",
    "Typically, the total number of users participating in the A/B test make up a small percentage of the total amount of users. Users are randomly selected and assigned to either a control group or a test group. The sample size that you decide on will determine how long you might have to wait until you have collected enough. For example, websites with large audiences may be able to collect enough data very quickly, while other websites may have to wait a number of weeks. There are some events that happen rarely even for high-traffic websites, so determining the necessary sample size will inform how soon you can assess your experiment and move on to improving other metrics.\n",
    "\n",
    "Initially, we will collect 1000 users for each group and serve the current signup page to the control group and a new signup page to the test group."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# A is control; B is test\n",
    "N_A = 1000\n",
    "N_B = 1000"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Run the Test\n",
    "Because this is a hypothetical example, we will need “fake” data to work on. We will use a function that will generate data for our simulation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def generate_data(N_A, N_B, p_A, p_B, days=None, control_label='A',\n",
    "                  test_label='B'):\n",
    "    \"\"\"Returns a pandas dataframe with fake CTR data\n",
    "    Example:\n",
    "    Parameters:\n",
    "        N_A (int): sample size for control group\n",
    "        N_B (int): sample size for test group\n",
    "            Note: final sample size may not match N_A provided because the\n",
    "            group at each row is chosen at random (50/50).\n",
    "        p_A (float): conversion rate; conversion rate of control group\n",
    "        p_B (float): conversion rate; conversion rate of test group\n",
    "        days (int): optional; if provided, a column for 'ts' will be included\n",
    "            to divide the data in chunks of time\n",
    "            Note: overflow data will be included in an extra day\n",
    "        control_label (str)\n",
    "        test_label (str)\n",
    "    Returns:\n",
    "        df (df)\n",
    "    \"\"\"\n",
    "\n",
    "    # initiate empty container\n",
    "    data = []\n",
    "\n",
    "    # total amount of rows in the data\n",
    "    N = N_A + N_B\n",
    "\n",
    "    # distribute events based on proportion of group size\n",
    "    group_bern = scs.bernoulli(N_A / (N_A + N_B))\n",
    "\n",
    "    # initiate bernoulli distributions from which to randomly sample\n",
    "    A_bern = scs.bernoulli(p_A)\n",
    "    B_bern = scs.bernoulli(p_B)\n",
    "\n",
    "    for idx in range(N):\n",
    "        # initite empty row\n",
    "        row = {}\n",
    "        # for 'ts' column\n",
    "        if days is not None:\n",
    "            if type(days) == int:\n",
    "                row['ts'] = idx // (N // days)\n",
    "            else:\n",
    "                raise ValueError(\"Provide an integer for the days parameter.\")\n",
    "        # assign group based on 50/50 probability\n",
    "        row['group'] = group_bern.rvs()\n",
    "\n",
    "        if row['group'] == 0:\n",
    "            # assign conversion based on provided parameters\n",
    "            row['converted'] = A_bern.rvs()\n",
    "        else:\n",
    "            row['converted'] = B_bern.rvs()\n",
    "        # collect row into data container\n",
    "        data.append(row)\n",
    "\n",
    "    # convert data into pandas dataframe\n",
    "    df = pd.DataFrame(data)\n",
    "\n",
    "    # transform group labels of 0s and 1s to user-defined group labels\n",
    "    df['group'] = df['group'].apply(\n",
    "        lambda x: control_label if x == 0 else test_label)\n",
    "\n",
    "    return df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "  <div id=\"df-68dbec69-4f7e-421d-a667-f0541ccbbcf8\">\n",
       "    <div class=\"colab-df-container\">\n",
       "      <div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>group</th>\n",
       "      <th>converted</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>A</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>A</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>B</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>B</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>A</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>B</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>A</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>A</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>A</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>A</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>\n",
       "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-68dbec69-4f7e-421d-a667-f0541ccbbcf8')\"\n",
       "              title=\"Convert this dataframe to an interactive table.\"\n",
       "              style=\"display:none;\">\n",
       "        \n",
       "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
       "       width=\"24px\">\n",
       "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
       "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
       "  </svg>\n",
       "      </button>\n",
       "      \n",
       "  <style>\n",
       "    .colab-df-container {\n",
       "      display:flex;\n",
       "      flex-wrap:wrap;\n",
       "      gap: 12px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert {\n",
       "      background-color: #E8F0FE;\n",
       "      border: none;\n",
       "      border-radius: 50%;\n",
       "      cursor: pointer;\n",
       "      display: none;\n",
       "      fill: #1967D2;\n",
       "      height: 32px;\n",
       "      padding: 0 0 0 0;\n",
       "      width: 32px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert:hover {\n",
       "      background-color: #E2EBFA;\n",
       "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
       "      fill: #174EA6;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert {\n",
       "      background-color: #3B4455;\n",
       "      fill: #D2E3FC;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert:hover {\n",
       "      background-color: #434B5C;\n",
       "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
       "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
       "      fill: #FFFFFF;\n",
       "    }\n",
       "  </style>\n",
       "\n",
       "      <script>\n",
       "        const buttonEl =\n",
       "          document.querySelector('#df-68dbec69-4f7e-421d-a667-f0541ccbbcf8 button.colab-df-convert');\n",
       "        buttonEl.style.display =\n",
       "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
       "\n",
       "        async function convertToInteractive(key) {\n",
       "          const element = document.querySelector('#df-68dbec69-4f7e-421d-a667-f0541ccbbcf8');\n",
       "          const dataTable =\n",
       "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
       "                                                     [key], {});\n",
       "          if (!dataTable) return;\n",
       "\n",
       "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
       "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
       "            + ' to learn more about interactive tables.';\n",
       "          element.innerHTML = '';\n",
       "          dataTable['output_type'] = 'display_data';\n",
       "          await google.colab.output.renderOutput(dataTable, element);\n",
       "          const docLink = document.createElement('div');\n",
       "          docLink.innerHTML = docLinkHtml;\n",
       "          element.appendChild(docLink);\n",
       "        }\n",
       "      </script>\n",
       "    </div>\n",
       "  </div>\n",
       "  "
      ],
      "text/plain": [
       "  group  converted\n",
       "0     A          0\n",
       "1     A          0\n",
       "2     B          0\n",
       "3     B          0\n",
       "4     A          0\n",
       "5     B          0\n",
       "6     A          0\n",
       "7     A          0\n",
       "8     A          0\n",
       "9     A          0"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ab_data = generate_data(N_A, N_B, bcr, bcr+d_hat)\n",
    "ab_data.head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The converted column indicates whether a user signed up for the premium service or not with a 1 or 0, respectively. The A group will be used for our control group and the B group will be our test group.\n",
    "\n",
    "Let’s look at a summary of the results using the pivot table function in Pandas."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "  <div id=\"df-6153f0c5-585c-4118-a700-18486c95f506\">\n",
       "    <div class=\"colab-df-container\">\n",
       "      <div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>converted</th>\n",
       "      <th>total</th>\n",
       "      <th>rate</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>group</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>A</th>\n",
       "      <td>90</td>\n",
       "      <td>961</td>\n",
       "      <td>0.093652</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>B</th>\n",
       "      <td>106</td>\n",
       "      <td>1039</td>\n",
       "      <td>0.102021</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>\n",
       "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-6153f0c5-585c-4118-a700-18486c95f506')\"\n",
       "              title=\"Convert this dataframe to an interactive table.\"\n",
       "              style=\"display:none;\">\n",
       "        \n",
       "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
       "       width=\"24px\">\n",
       "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
       "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
       "  </svg>\n",
       "      </button>\n",
       "      \n",
       "  <style>\n",
       "    .colab-df-container {\n",
       "      display:flex;\n",
       "      flex-wrap:wrap;\n",
       "      gap: 12px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert {\n",
       "      background-color: #E8F0FE;\n",
       "      border: none;\n",
       "      border-radius: 50%;\n",
       "      cursor: pointer;\n",
       "      display: none;\n",
       "      fill: #1967D2;\n",
       "      height: 32px;\n",
       "      padding: 0 0 0 0;\n",
       "      width: 32px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert:hover {\n",
       "      background-color: #E2EBFA;\n",
       "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
       "      fill: #174EA6;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert {\n",
       "      background-color: #3B4455;\n",
       "      fill: #D2E3FC;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert:hover {\n",
       "      background-color: #434B5C;\n",
       "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
       "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
       "      fill: #FFFFFF;\n",
       "    }\n",
       "  </style>\n",
       "\n",
       "      <script>\n",
       "        const buttonEl =\n",
       "          document.querySelector('#df-6153f0c5-585c-4118-a700-18486c95f506 button.colab-df-convert');\n",
       "        buttonEl.style.display =\n",
       "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
       "\n",
       "        async function convertToInteractive(key) {\n",
       "          const element = document.querySelector('#df-6153f0c5-585c-4118-a700-18486c95f506');\n",
       "          const dataTable =\n",
       "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
       "                                                     [key], {});\n",
       "          if (!dataTable) return;\n",
       "\n",
       "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
       "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
       "            + ' to learn more about interactive tables.';\n",
       "          element.innerHTML = '';\n",
       "          dataTable['output_type'] = 'display_data';\n",
       "          await google.colab.output.renderOutput(dataTable, element);\n",
       "          const docLink = document.createElement('div');\n",
       "          docLink.innerHTML = docLinkHtml;\n",
       "          element.appendChild(docLink);\n",
       "        }\n",
       "      </script>\n",
       "    </div>\n",
       "  </div>\n",
       "  "
      ],
      "text/plain": [
       "       converted  total      rate\n",
       "group                            \n",
       "A             90    961  0.093652\n",
       "B            106   1039  0.102021"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ab_summary = ab_data.pivot_table(values='converted', index='group', aggfunc=np.sum)\n",
    "# add additional columns to the pivot table\n",
    "ab_summary['total'] = ab_data.pivot_table(values='converted', index='group', aggfunc=lambda x: len(x))\n",
    "ab_summary['rate'] = ab_data.pivot_table(values='converted', index='group')\n",
    "ab_summary"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It looks like the difference in conversion rates between the two groups is 0.024 which is more than the lift we initially wanted of 0.02. This is a good sign that we should adopt the new design, but this is not enough evidence for us to confidently accept the new design. At this point we have not measured how confident we are in this result. This can be mitigated by looking at the distributions of the two groups."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Compare the Two Groups\n",
    "We can compare the two groups by plotting the distribution of the control group and calculating the probability of getting the result from our test group. We can assume that the distribution for our control group is binomial because the data is a series of Bernoulli trials, where each trial only has two possible outcomes (similar to a coin flip)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "A_converted = ab_summary.loc['A','converted']\n",
    "A_total = ab_summary.loc['A','total']\n",
    "A_cr = ab_summary.loc['A','rate']\n",
    "\n",
    "B_converted = ab_summary.loc['B','converted']\n",
    "B_total = ab_summary.loc['B','total']\n",
    "B_cr = ab_summary.loc['B','rate']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Text(0, 0.5, 'probability')"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "[encoded data, 11372 characters]\n",
      "text/plain": [
       "<Figure size 864x432 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig, ax = plt.subplots(figsize=(12,6))\n",
    "x = np.linspace(A_converted-49, A_converted+50, 100)\n",
    "y = scs.binom(A_total, A_cr).pmf(x)\n",
    "ax.bar(x, y, alpha=0.5)\n",
    "ax.axvline(x=B_cr * A_total, c='blue', alpha=0.75, linestyle='--')\n",
    "plt.xlabel('converted')\n",
    "plt.ylabel('probability')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The distribution for the control group is shown in red and the result from the test group is indicated by the blue dashed line. We can see that the probability of getting the result from the test group is very high. However, the probability does not convey the confidence level of the results. It does not take the sample size of our test group into consideration. Intuitively, we would feel more confident in our results as our sample sizes grow larger. Let’s continue and plot the test group results as a binomial distribution and compare the distributions against each other."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Binomial distribution"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Text(0, 0.5, 'probability')"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "[encoded data, 10420 characters]\n",
      "text/plain": [
       "<Figure size 864x432 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig, ax = plt.subplots(figsize=(12,6))\n",
    "xA = np.linspace(A_converted-49, A_converted+50, 100)\n",
    "yA = scs.binom(A_total, A_cr).pmf(xA)\n",
    "ax.bar(xA, yA, alpha=0.5)\n",
    "xB = np.linspace(B_converted-49, B_converted+50, 100)\n",
    "yB = scs.binom(B_total, B_cr).pmf(xB)\n",
    "ax.bar(xB, yB, alpha=0.5)\n",
    "plt.xlabel('converted')\n",
    "plt.ylabel('probability')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Compare the Null Hypothesis vs. the Alternative Hypothesis"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let’s start off by defining the null hypothesis and the alternative hypothesis.\n",
    "- The null hypothesis is the position that the change in the design made for the test group would result in no change in the conversion rate.\n",
    "- The alternative hypothesis is the opposing position that the change in the design for the test group would result in an improvement (or reduction) in the conversion rate."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 33708 characters]\n",
      "text/plain": [
       "<Figure size 864x432 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# define the parameters for abplot()\n",
    "# use the actual values from the experiment for bcr and d_hat\n",
    "# p_A is the conversion rate of the control group\n",
    "# p_B is the conversion rate of the test group\n",
    "bcr = A_cr\n",
    "d_hat = B_cr - A_cr\n",
    "abplot(N_A, N_B, bcr, d_hat)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Fortunately, both curves are identical in shape, so we can just compare the distance between the means of the two distributions. We can see that the alternative hypothesis curve suggests that the test group has a higher conversion rate than the control group. This plot also can be used to directly determine the statistical power."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Statistical Power and Significance Level\n",
    "I think it is easier to define statistical power and significance level by first showing how they are represented in the plot of the null and alternative hypothesis. We can return a visualization of the statistical power by adding the parameter show_power=True"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 36836 characters]\n",
      "text/plain": [
       "<Figure size 864x432 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "abplot(N_A, N_B, bcr, d_hat, show_power=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 37684 characters]\n",
      "text/plain": [
       "<Figure size 864x432 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "abplot(N_A, N_B, bcr, d_hat, show_beta=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The gray dashed line that divides the area under the alternative curve into two also directly segments the area associated with the significance level, often denoted with the greek letter alpha."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 35836 characters]\n",
      "text/plain": [
       "<Figure size 864x432 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "abplot(N_A, N_B, bcr, d_hat, show_alpha=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Experiments are typically set up for a minimum desired power of 80%. If our new design is truly better, we want our experiment to show that there is at least an 80% probability that this is the case. We know that if we increase the sample size for each group, we will decrease the pooled variance for our null and alternative hypothesis. This will make our distributions much narrower and may increase the statistical power. Let’s take a look at how sample size will directly affect our results."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. Sample Size\n",
    "If we run our test again with a sample size of 2000 instead of 1000 for each group, we get the following results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 36396 characters]\n",
      "text/plain": [
       "<Figure size 864x432 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "abplot(2000, 2000, bcr, d_hat, show_power=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Many people calculate Z from tables such as those shown here and here. However, I am more of a visual learner and I like to refer to the plot of the Z-distribution from which the values are derived."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 29756 characters]\n",
      "text/plain": [
       "<Figure size 864x432 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "zplot()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 29240 characters]\n",
      "text/plain": [
       "<Figure size 864x432 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "zplot(0.80,False,True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3842.026629963879"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "min_sample_size(bcr=0.10, mde=0.02)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 37336 characters]\n",
      "text/plain": [
       "<Figure size 864x432 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "abplot(3843, 3843, 0.10, 0.02, show_power=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The calculated power for this sample size was approximately 0.80. Therefore, if our design change had an improvement in conversion of about 2 percent, we would need at least 3843 samples in each group for a statistical power of at least 0.80."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That was a very long but basic walkthrough of A/B tests. Once you have developed an understanding and familiarity with the procedure, you will probably be able to run an experiment and go directly to the plots for the null and alternative hypothesis to determine if your results achieved enough power. By calculating the minimum sample size you need prior to the experiment, you can determine how long it will take to get the results back to your team for a final decision."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Author: Sparsh A.\n",
      "\n",
      "Last updated: 2021-12-27 03:57:43\n",
      "\n",
      "recohut: 0.0.7\n",
      "\n",
      "Compiler    : GCC 7.5.0\n",
      "OS          : Linux\n",
      "Release     : 5.4.144+\n",
      "Machine     : x86_64\n",
      "Processor   : x86_64\n",
      "CPU cores   : 2\n",
      "Architecture: 64bit\n",
      "\n",
      "matplotlib: 3.2.2\n",
      "IPython   : 5.5.0\n",
      "scipy     : 1.4.1\n",
      "numpy     : 1.19.5\n",
      "pandas    : 1.1.5\n",
      "\n"
     ]
    }
   ],
   "source": [
    "#hide\n",
    "%reload_ext watermark\n",
    "%watermark -a \"Sparsh A.\" -m -iv -u -t -d -p recohut"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "name": "python3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}
