{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_id": "00000-666a014d-7212-40f0-88ab-4adcf136da76",
    "deepnote_cell_type": "markdown",
    "output_cleared": false,
    "tags": []
   },
   "source": [
    "# Introduction \n",
    "\n",
    "Automatic differentiation (AD) is a set of techniques to numerically evaluate the derivative of a function specified by a computer program. In mathematics and computer fields, a large number of numerical methods require derivatives to be calculated(Margossian, 2019). For example, in machine learning, gradient plays an essential role. Neural networks are able to gradually increase accuracy with every training session through the process of gradient descent(Wang, 2019).  In physics, differentiation is needed to quantify the rate of change in the equations of motion. However, for many problems, calculating derivatives analytically is both time consuming and computationally impractical. Automatic differentiation can gives exact answers in constant time(Saroufim, 2019), which is a powerful tool to automate the calculation of derivatives, especially when differentiating complex algorithms and mathematical functions(Margossian, 2019).  \n",
    "\n",
    "There are three main methods of calculating derivatives: numerical differentiation, symbolic differentiation, and automatic differentiation. Specifically, symbolic differentiation involves computing exact expressions for the derivatives in terms of the function variables by representing the expression as a tree and manipulating it using the rules of differentiation. The derivatives can quickly become very complicated for complex functions and higher-order derivatives. Thus, for complicated functions, symbolic differentiation is not easily generalizable and becomes computationally inefficient due to redundant evaluations of different components of the function . Numerical differentiation aims to approximate derivatives through finite differentiating, but the solution accuracy is greatly affected by the truncation and round-off errors associated with finite difference formulas. Automatic differentiation is more computationally efficient and generalizable than symbolic differentiation and can find exact derivatives as opposed to numerical differentiation.\n",
    "\n",
    "In this library, we will describe the mathematical background and concepts in the Background section, explain the software organization, and discuss the implementation of the forward mode of automatic differentiation.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_id": "00001-1e174171-09b4-4c3a-8aab-fa8f970172e1",
    "deepnote_cell_type": "markdown",
    "output_cleared": false,
    "tags": []
   },
   "source": [
    "# Background\n",
    "\n",
    "Automatic differentiation assumes that we are working with a differentiable function composed of a finite number of elementary functions and operations with known symbolic derivatives. The table below shows some examples of elementary functions and their respective derivatives:\n",
    "\n",
    "| Elementary Function    | Derivative              |\n",
    "| :--------------------: | :----------------------:|\n",
    "| $x^3$                  | $3 x^{2}$               | \n",
    "| $e^x$                  | $e^x$                   |\n",
    "| $\\sin(x)$              | $\\cos(x)$               |\n",
    "| $\\ln(x)$               | $\\frac{1}{x}$           |\n",
    "\n",
    "Given a list of elementary functions and their corresponding derivatives, the automatic differentiation process involves the evaluations of the derivatives of complex compositions of these elementary functions through repeated applications of the chain rule:\n",
    "\n",
    "$ \\frac{\\partial}{\\partial x}\\left[f_1 \\circ (f_2 \\circ \\ldots (f_{n-1} \\circ f_n)) \\right] = \n",
    "\\frac{\\partial f_1}{\\partial f_2} \\frac{\\partial f_2}{\\partial f_3} \\ldots \\frac{\\partial f_{n-1}}{\\partial f_n}\\frac{\\partial f_n}{\\partial x}$\n",
    "\n",
    "This process can be applied to the evaluation of partial derivatives as well thus allowing for computing derivatives of multivariate and vector-valued functions.\n",
    "\n",
    "## Computational Graph\n",
    "\n",
    "The forward mode automatic differentiation process described above can be visualized in a computational graph, a directed graph with each node corresponding to the result of an elementary operation.\n",
    "\n",
    "For example, consider the simple problem of evaluating the following function and its derivative at $x=2$:\n",
    "$$\n",
    "f(x) = x^3 +2x^2\n",
    "$$\n",
    "The evaluation of this function can be represented by the evaluation trace and computational graph below where the numbers in orange are the function values and the numbers in blue are the derivatives evaluated after applying each elementary operation:\n",
    "\n",
    "<img src=\"https://raw.githubusercontent.com/anita76/playground/master/src/ex_comp_graph.png\" width=\"75%\" />\n",
    "\n",
    "| Trace       | Elementary Function     | Value   | Derivative                | Derivative Value      |\n",
    "| :---------: | :----------------------:| :------:| :------------------------:| :--------------------:|\n",
    "| $v_1$       | $x$                     | $2$     | $\\dot{x}$                 | $1$                   |\n",
    "| $v_2$       | $v_1^2$                 | $4$     | $2v_1 \\dot{v}_1$          | $4$                   |\n",
    "| $v_3$       | $v_1^3$                 | $8$     | $3v_1^2 \\dot{v}_1$        | $12$                  |\n",
    "| $v_4$       | $2v_2$                  | $8$     | $2 \\dot{v}_2$             | $8$                   |\n",
    "| $v_5$       | $v_3 + v_4$             | $16$    | $ \\dot{v}_3 + \\dot{v}_4$  | $20$                  |\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_id": "00002-de6540a6-d222-4d8b-b063-a392b8a9b7d4",
    "deepnote_cell_type": "markdown",
    "output_cleared": false,
    "tags": []
   },
   "source": [
    "# How to Use West Coast AD\n",
    "\n",
    "### Installation with GitHub\n",
    "1. Clone the package repository:\n",
    "\n",
    "    `git clone https://github.com/West-Coast-Differentiators/cs107-FinalProject.git`\n",
    "\n",
    "2. If you do not have Anaconda installed, follow the instructions on their [website](https://deepnote.com/project/07e48b43-3d96-4960-83bf-911527fb55ea?cellId=00002-de6540a6-d222-4d8b-b063-a392b8a9b7d4#/milestone2-2.ipynb) to install it.\n",
    "3. Create a new conda environment:\n",
    "\n",
    "    `conda create --name <env_name> python=3.8` \n",
    "\n",
    "4. Activate your virtual environment:\n",
    "\n",
    "    `conda activate <env_name>`\n",
    "\n",
    "5. Navigate to the base repository:\n",
    "\n",
    "    `cd cs107-FinalProject`\n",
    "\n",
    "6. Install the package and its requirements:\n",
    "\n",
    "    `pip install ./`\n",
    "\n",
    "7. You may check that the installation was successful by running the package tests:\n",
    "\n",
    "    `python -m unittest discover -s WestCoastAD/test -p '*_test.py'`\n",
    "\n",
    "### Using the Package\n",
    "\n",
    " Having installed the package, users can evaluate derivatives of functions written in terms of WestCoastAD variables. For instance, the derivatives of $f(x) = x^3+ 2x^2$ and $f(x) = sin(x) + cos(x)$ can be evaluated as follows:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "cell_id": "00005-bd0d4a8c-1297-4a71-9dd8-e4ffe13c11b4",
    "deepnote_cell_type": "code",
    "execution_millis": 237,
    "execution_start": 1603314710130,
    "output_cleared": true,
    "source_hash": null,
    "tags": []
   },
   "outputs": [],
   "source": [
    "# 1) import WestCoastAD\n",
    ">>> from WestCoastAD import Variable\n",
    "# 2) define the variables of your function as WestCoastAD variables \n",
    "#    with the value at which you want to evaluate the derivative and \n",
    "#    a derivative seed value\n",
    ">>> x = Variable(value=2, derivative_seed=1) \n",
    "# 3) define your function in terms of the WestCoastAD variable objects\n",
    ">>> f = x**3 + 2* x**2\n",
    "# 4) access the computed derivative and the value of the function \n",
    ">>> print(f.value)\n",
    "16\n",
    ">>> print(f.derivative)\n",
    "20\n",
    "# 5) For other mathematical operations that are not part of the standard python library,\n",
    "#    the function could be called with numpy or directly on the variable itself.\n",
    "#    For a full list of supported elementary functions, refer to the software implementation\n",
    "#    section of this document.\n",
    ">>> import numpy as np\n",
    ">>> x = Variable(value=np.pi/2, derivative_seed=1)\n",
    "# 6) As an example, below we directly call sin on the variable and use numpy to call cos\n",
    ">>> f = x.sin() + np.cos(x)\n",
    ">>> print(f.value)\n",
    "1.0\n",
    ">>> print(f.derivative)\n",
    "-0.9999999999"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_id": "00006-fae5795b-fb45-4755-9897-c40dd13122b7",
    "deepnote_cell_type": "markdown",
    "output_cleared": false,
    "tags": []
   },
   "source": [
    "# Software Organization\n",
    "## Directory Structure\n",
    "```bash\n",
    "cs107-FinalProject  \n",
    "             setup.py\n",
    "             .travis.yml \n",
    "             README.MD\n",
    "             docs/\n",
    "                             __init__.py\n",
    "                             milestone1.ipynb\n",
    "                             milestone2.ipynb   \n",
    "                             milestone2A.ipynb\n",
    "                             milestone2_progress.ipynb  \n",
    "                             ...\n",
    "             WestCoastAD/  \n",
    "                             __init__.py\n",
    "                             src/    \n",
    "                                     __init__.py\n",
    "                                     forward_mode_AD.py\n",
    "                                     gradient_descent_optimizer.py\n",
    "                                     ...\n",
    "\n",
    "                             tests/ \n",
    "                                     __init__.py\n",
    "                                     forward_mode_AD_test.py \n",
    "                                     gradient_descent_optimizer_test.py\n",
    "                                     ...\n",
    "                     ...\n",
    "```\n",
    "## Modules and Basic Functionality\n",
    "\n",
    "#### Docs Module\n",
    "* This module contains the documentation for the package. The documentations include the information about the package including the purpose, the organization of the software, how to use the package and how to install the package.\n",
    "\n",
    "#### Setup Module\n",
    "* This module contains configurations to allow users to install our package. Inside the setup.py file, we use the setuptools library for packaging the modules and installing the package dependencies. Our package requires that numpy be installed.\n",
    "\n",
    "#### Forward Mode Module\n",
    "* This module includes the implementation of the forward mode of automatic differentiation. The module defines a Variable class that will take the value to be evaluated and the derivative seed value. The derivative and value of functions defined in terms of the Variable class, can be accessed by viewing the derivative and value attributes of this class.\n",
    "\n",
    "#### Gradient Descent Optimization Module\n",
    "* This is the extension module that will be added in a future release to include our extension for the AD package. The future section of the documentation below, describes our AD extension.\n",
    "\n",
    "#### Test Modules\n",
    "* This module contains all the tests of our implementation including unit tests and acceptance tests. We used Python's unittest to define the tests and check our coverage with CodeCov. Having successfully installed the package following the instruction in the \"How to Use WestCoastAD\" section of this documentation, the tests can be run using the following command:\n",
    "`python -m unittest discover -s WestCoastAD/test -p '*_test.py'`. Alternatively, you can run individual test files using: `python WestCoastAD/test/<TEST_FILE_NAME>.py`\n",
    "\n",
    "## Packaging and Build Pipeline\n",
    "\n",
    "#### Packaging\n",
    "* We packaged our module with setuptools following standard python packaging protocols as described in the following tutorial: [https://python-packaging-tutorial.readthedocs.io/en/latest/setup_py.html](https://python-packaging-tutorial.readthedocs.io/en/latest/setup_py.html)\n",
    "\n",
    "#### Distribution\n",
    "* WestCoastAD is currently distributed via Github. In the \"How to Use WestCoast AD\" section of this document, there is information about where to get the package, how to install a conda environment and how to install the package itself.\n",
    "\n",
    "#### Travis CI\n",
    "* Travis CI allows for automatic building and testing of commits that are pushed to remote. It requires that all pull request be tested and if all the tests are passed, it will update the build automatically. Travis CI also contains a badge that will inform the user if the package is working or if it is down. The travis.yml file is located in our main directory.\n",
    "\n",
    "#### CodeCov\n",
    "* CodeCov provides details about our test coverage. This allows us to identify where tests need to be added and will help us discover possible bugs in the code. CodeCov also contains a badge that will inform the user of the percentage code coverage of our tests.\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_id": "00007-f3a9f884-c1c8-4c92-af2f-9bc61ee4ba42",
    "deepnote_cell_type": "markdown",
    "output_cleared": false,
    "tags": []
   },
   "source": [
    "# Software Implementation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_id": "00008-a0bd0996-c324-43bb-9640-c616961ae782",
    "deepnote_cell_type": "markdown",
    "output_cleared": false,
    "tags": []
   },
   "source": [
    "This section covers the class and function reference for `WestCoastAD`.\n",
    "\n",
    "`Variable` is the base class used to instantiate gradient objects and to perform derivatives.\n",
    "The instances of the `Variable` class should be used to express the scalar or vector function to be differentiated.\n",
    "\n",
    "| Class Attributes        | Usage        |\n",
    "| :----------------------- |:-------------|\n",
    "| `value`                 | Value of the variable (`int` or `float`). The setter and getter for value are defined using the  `property` decorator.|\n",
    "| `derivative`            | Derivative of the function (`int` or `float`). A `seed derivative` can be provided at the time of instantiation. This attribute is set and retrieved using a `property` decorator.|\n",
    "\n",
    "<br>\n",
    "\n",
    "**Class Methods**\n",
    "\n",
    "###### `__add__`\n",
    "Dunder method for overloading the \"+\" operator. This method computes the value and the derivative of the summation operation on `int`, `float` & `Variable` instances. This method returns a new `Variable` object.\n",
    "\n",
    "###### `__radd__`\n",
    "Same method as `__add__` with reversed operands.\n",
    "\n",
    "###### `__mul__`\n",
    "Dunder method for overloading the \"*\" operator. This method computes the value and the derivative following the [product rule](https://en.wikipedia.org/wiki/Product_rule) on `int`, `float` & `Variable` instances. It returns a new `Variable` object.\n",
    "\n",
    "###### `__rmul__`\n",
    "Same method as `__mul__` with reversed operands.\n",
    "\n",
    "###### `__sub__`\n",
    "Dunder method for overloading the \"-\" operator. This method computes the value and the derivative of the substraction operation on `int`, `float` & `Variable` instances. It returns a new `Variable` object.\n",
    "\n",
    "###### `__rsub__`\n",
    "Same method as `__sub__` with reversed operands.\n",
    "\n",
    "###### `__truediv__`\n",
    "Dunder method for overloading the \"/\" operator. This method computes the value and the derivative following the [quotient rule](https://en.wikipedia.org/wiki/Quotient_rule) on `int`, `float` & `Variable` instances. It returns a new `Variable` object. This method throws a `ZeroDivisionError` if the divisor is 0.\n",
    "\n",
    "###### `__rtruediv__`\n",
    "Same method as `__truediv__` with reversed operands.\n",
    "\n",
    "###### `__pow__`\n",
    "Dunder method for overloading the \"**\" operator. This method computes the value and the derivative following the [power rule](https://en.wikipedia.org/wiki/Power_rule) if exponent is `int` or `float`.\n",
    "It returns a new `Variable` object. This method throws a `ValueError` if:\n",
    "* a negative valued variable/function is raised to a non-integer power (e.g. $x^{1.2}, x=-2$)\n",
    "* a zero valued variable/function is raised to a non-positive power (e.g. $x^{-5}, x = 0$)\n",
    "* a non-positive valued variable/function is raised to the power of another variable/function (e.g. $x^{sin(x)}, x=-1$). This is because the generalized power rule of differentiation takes the logarithm of the base and the logarithm is not defined for non-positive values.\n",
    "\n",
    "###### `__rpow__`\n",
    "Dunder method which handles the differentiation of equations of the form $a^{f(x)}$. It returns a new `Variable` object. This method throws a `ValueError` if:\n",
    "* zero is raised to the power of a variable/function with a non-positive value (e.g. $0^{x}, x=0$). \n",
    "* a negative number is used as the base (e.g. $(-2)^x$)\n",
    "\n",
    "###### `__neg__`\n",
    "Dunder method for overloading the negation operation. This method computes the value and derivative of the negation operation and returns a new `Variable` object.\n",
    "\n",
    "###### `__abs__`\n",
    "Dunder method for overloading the abs function. This method computes the value and derivative of the absolute value of a variable and returns a `Variable` object. It will throw a `ValueError` if given a value of zero since the derivative of absolute value is not defined at zero.\n",
    "\n",
    "###### `__repr__`\n",
    "Dunder method for overloading the `repr` function. Returns the object representation of the class `Variable` in the form `Variable(value=value, derivative=derivative)`\n",
    "\n",
    "###### `log`\n",
    "This method computes the value and derivative of `log` function and returns a new `Variable` object. A `ValueError` is thrown for non-positive values.\n",
    "\n",
    "###### `exp`\n",
    "This method computes the value and derivative of `exp` function and returns a new `Variable` object\n",
    "\n",
    "###### `sqrt`\n",
    "Square root is a special case of pow and therefore calls `__pow__` with exponent `1\\2`.\n",
    "\n",
    "###### `sin`\n",
    "\n",
    "This method computes the value and the derivative of the sine function. It returns a new `Variable` object.\n",
    "\n",
    "###### `cos`\n",
    "\n",
    "This method computes the value and the derivative of the cosine function. It returns a new `Variable` object.\n",
    "\n",
    "###### `tan`\n",
    "\n",
    "This method computes the value and the derivative of the tangent function. It returns a new `Variable` object. This method will throw a `ValueError` if given a value that is an odd multiple of $\\frac{\\pi}{2}$\n",
    "\n",
    "###### `sinh`\n",
    "\n",
    "This method computes the value and the derivative of the hyperbolic sine function. It returns a new `Variable` object.\n",
    "\n",
    "###### `cosh`\n",
    "\n",
    "This method computes the value and the derivative of the hyperbolic cosine function.  It returns a new `Variable` object.\n",
    "\n",
    "###### `tanh`\n",
    "\n",
    "This method computes the value and the derivative of the hyperbolic tangent function.  It returns a new `Variable` object.\n",
    "\n",
    "###### `arcsin`\n",
    "\n",
    "This method computes the value and the derivative of the arcsin function.  It should be given a value in  (-1,1) for the derivative to be defined, otherwise a `ValueError` will be raised. It returns a new `Variable` object.\n",
    "\n",
    "###### `arccos`\n",
    "\n",
    "This method computes the value and the derivative of the arccos function. It should be given a value in  (-1,1) for the derivative to be defined, otherwise a `ValueError` will be raised. It returns a new `Variable` object.\n",
    "\n",
    "###### `arctan`\n",
    "\n",
    "This method computes the value and the derivative of the arctan function. It returns a new `Variable` object.\n",
    "\n",
    "\n",
    "#### External Dependencies\n",
    "We use `numpy` to evaluate elementary functions which were not available as part of the Python standard library (e.g. sin, cos, etc). We will also be accepting numpy as an input in later versions when we extend the implementation to also compute gradients and Jacobians of multivariable and vector-valued functions.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_id": "00009-3218d8d3-967c-4548-b593-dbf2a2b25e53",
    "deepnote_cell_type": "markdown",
    "tags": []
   },
   "source": [
    "# Future Features\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_id": "00008-10d31b5b-b5dd-4e38-9f40-2e7061e5ca28",
    "deepnote_cell_type": "markdown",
    "tags": []
   },
   "source": [
    "### Vector valued and multivariable functions \n",
    "In addition to using floats and ints as input, we will accept numpy arrays as input to the Variable class. This is to accommodate for the computation of gradients of multivariable functions as well as the Jacobians of vector-valued functions. This will require modifications to the methods defined in the Variable class as well as the implementation of new elementary operations such as matrix multiplication.\n",
    "### Extention class:  optimization with gradient descent built upon the AD core\n",
    "\n",
    "Gradient-based optimization is an essential method in the field of machine learning. It provides a simple and iterative algorithm for finding local minima of a real-valued function $F$ numerically. Given an objective function, gradient-based methods make use of the fact that the objective function decreases steepest if one goes in the direction of the negative gradient: $-\\nabla F(x)$. The iteration step of the algorithm is defined in terms of a step size parameter $\\eta$ as:\n",
    "$$\n",
    "x_{i+1} = x_i - \\eta\\nabla F(x_i) \n",
    "$$\n",
    "Automatic differentiation  provides an efficient way for gradient computation by creating computation graph. We will be implementing the algorithm for gradient descent in a new module under our WestCoastAD package and using our forward_mode_AD module to compute the derivative for each iteration. the `gradient_descent_optimizer.py` file, will implement a function called `optimize_gd` which will take as input the function to be optimized and a list of all the WestCoastAD variables used to define the function. The function will return the optimum values for the variables in the function after performing gradient descent. The solution will be of the same shape as the input variables.  Gradient descent parameters such as the number of iterations and the learning rate can also be optionally passed into the `optimize_gd` function if the user does not want to use the default values for these parameters.\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_id": "00010-4ce99ca0-99b0-4707-b4ab-aae77a1b3715",
    "deepnote_cell_type": "markdown",
    "output_cleared": false,
    "tags": []
   },
   "source": [
    "# References\n",
    "-  Margossian, C. C. (2019). A review of automatic differentiation and its efficient implementation. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 9(4), 1–32. https://doi.org/10.1002/widm.1305\n",
    "\n",
    "- Saroufim, M. (2019, November 13). Automatic Differentiation Step by Step. Retrieved October 15, 2020, from https://marksaroufim.medium.com/automatic-differentiation-step-by-step-24240f97a6e6\n",
    "\n",
    "- Wang, C. (2019, March 3). Automatic Differentiation, Explained. Retrieved October15, 2020,from https://towardsdatascience.com/automatic-differentiation-explained-\tb4ba8e60c2ad\n",
    "\n",
    "- Automatic Differentiation Background. Retrived October 20, 2020, from https://www.mathworks.com/help/deeplearning/ug/deep-learning-with-automatic-differentiation-in-matlab.html"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_id": "00011-e5919a5d-ff6f-483c-aef9-539d081df705",
    "deepnote_cell_type": "markdown",
    "output_cleared": false,
    "tags": []
   },
   "source": []
  }
 ],
 "metadata": {
  "deepnote_execution_queue": [],
  "deepnote_notebook_id": "da029247-3b8f-49cf-aa79-500300952868",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
