{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_id": "00002-beeaf662-2ede-4583-a414-f25e148ac07d",
    "output_cleared": false,
    "tags": []
   },
   "source": [
    "# Introduction \n",
    "\n",
    "Automatic differentiation (AD) is a set of techniques to numerically evaluate the derivative of a function specified by a computer program. In mathematics and computer fields, a large number of numerical methods require derivatives to be calculated(Margossian, 2019). For example, in machine learning, gradient plays an essential role. Neural networks are able to gradually increase accuracy with every training session through the process of gradient descent(Wang, 2019).  In physics, differentiation is needed to quantify the rate of change in the equations of motion. However, for many problems, calculating derivatives analytically is both time consuming and computationally impractical. Automatic differentiation can gives exact answers in constant time(Saroufim, 2019), which is a powerful tool to automate the calculation of derivatives, especially when differentiating complex algorithms and mathematical functions(Margossian, 2019).  \n",
    "\n",
    "In this library, we will describe the mathematical background and concepts in the Background section, explain the software organization, and discuss the implementation of the forward mode of automatic differentiation.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_id": "00001-5c4d9fa4-b525-4847-87f0-287a8cc62139",
    "output_cleared": false,
    "tags": []
   },
   "source": [
    "# Background\n",
    "\n",
    "Automatic differentiation assumes that we are working with a differentiable function composed of a finite number of elementary functions and operations with known symbolic derivatives. The table below shows some examples of elementary functions and their respective derivatives:\n",
    "\n",
    "| Elementary Function    | Derivative              |\n",
    "| :--------------------: | :----------------------:|\n",
    "| $x^3$                  | $3 x^{2}$               | \n",
    "| $e^x$                  | $e^x$                   |\n",
    "| $\\sin(x)$              | $\\cos(x)$               |\n",
    "| $\\ln(x)$               | $\\frac{1}{x}$           |\n",
    "\n",
    "Given a list of elementary functions and their corresponding derivatives, the automatic differentiation process involves the evaluations of the derivatives of complex compositions of these elementary functions through repeated applications of the chain rule:\n",
    "\n",
    "$ \\frac{\\partial}{\\partial x}\\left[f_1 \\circ (f_2 \\circ \\ldots (f_{n-1} \\circ f_n)) \\right] = \n",
    "\\frac{\\partial f_1}{\\partial f_2} \\frac{\\partial f_2}{\\partial f_3} \\ldots \\frac{\\partial f_{n-1}}{\\partial f_n}\\frac{\\partial f_n}{\\partial x}$\n",
    "\n",
    "This process can be applied to the evaluation of partial derivatives as well thus allowing for computing derivatives of multivariate and vector-valued functions.\n",
    "\n",
    "## Computational Graph\n",
    "\n",
    "The forward mode automatic differentiation process described above can be visualized in a computational graph, a directed graph with each node corresponding to the result of an elementary operation.\n",
    "\n",
    "For example, consider the simple problem of evaluating the following function and its derivative at $x=2$:\n",
    "$$\n",
    "f(x) = x^3 +2x^2\n",
    "$$\n",
    "The evaluation of this function can be represented by the computational graph below where the numbers in orange are the function values and the numbers in blue are the derivatives evaluated after applying each elementary operation:\n",
    "\n",
    "<img src=\"https://raw.githubusercontent.com/anita76/playground/master/src/ex_comp_graph.png\" width=\"75%\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_id": "00002-5de4aa89-c8f5-4293-99d9-a8f69d00475b",
    "output_cleared": false,
    "tags": []
   },
   "source": [
    "# How to Use West Coast AD\n",
    "\n",
    "Having installed the package, users can evaluate derivatives of functions written in terms of WestCoastAD variables. For instance, the derivative of $f(x) = x^3+ 2x^2$ can be evaluated as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "cell_id": "00003-9843d2cd-dbe0-411e-8488-04cfe04f6352",
    "execution_millis": 237,
    "execution_start": 1603314710130,
    "output_cleared": true,
    "source_hash": null,
    "tags": []
   },
   "outputs": [],
   "source": [
    "# 1) import WestCoastAD module\n",
    "import WestCoastAD as ad\n",
    "# 2) define the variables of your function as WestCoastAD variables \n",
    "#    with the value at which you want to evaluate the derivative and \n",
    "#    a derivative seed value\n",
    "x = ad.Variable(value=2, derivative_seed=1) \n",
    "# 3) define your function in terms of the WestCoastAD variable objects\n",
    "f = x**3 + 2* x**2\n",
    "# 4) access the computed derivative and the value of the function \n",
    "f.value\n",
    "f.derivative"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_id": "00003-9920462b-029e-4b6d-8c42-5f8a17f26175",
    "output_cleared": false,
    "tags": []
   },
   "source": [
    "# Software Organization\n",
    "## Directory Structure\n",
    "```bash\n",
    "WestCoastAD/\n",
    "             __init__.py  #Initialize the main package\n",
    "             LICENSE\n",
    "             README.MD\n",
    "             docs/         #Subpackage for package docs\n",
    "                     __init__.py\n",
    "                     docs.ipynb\n",
    "                     ...\n",
    "             \n",
    "             installation/  #Subpackage for install mode\n",
    "                     __init__.py\n",
    "                     setup.py\n",
    "                     ...             \n",
    "             src/  \n",
    "                             forward_mode/  #Subpackage for forward mode\n",
    "                                     __init__.py\n",
    "                                     forward_mode_module.py\n",
    "                                     ...\n",
    "\n",
    "                             extension/  #Subpackage for future extention\n",
    "                                     __init__.py\n",
    "                                     extension_module.py\n",
    "                                     ...\n",
    "             \n",
    "             tests/         #Subpackage for test\n",
    "                             __init__.py\n",
    "                             forward_mode_test/  #Subpackage for forward mode\n",
    "                                     __init__.py\n",
    "                                     forward_mode_test.py                \n",
    "                                     ...\n",
    "\n",
    "                             extension_test/  #test for future extension\n",
    "                                     __init__.py\n",
    "                                     extension_test.py\n",
    "                                     ...\n",
    "             \n",
    "                     ...\n",
    "```\n",
    "## Modules and Basic Functionality\n",
    "\n",
    "#### Docs Module\n",
    "* This module will contain the documentation for the package. The Docs will include the information about the package including the purpose, the organization of the software, how to use the package and how to install the package.\n",
    "\n",
    "#### Setup Module\n",
    "* This module will contain configurations to allow users to install our package. \n",
    "\n",
    "#### Forward Mode Module\n",
    "* This module will have all the code for implementing the forward mode of automatic differentiation. \n",
    "The module will contain a Variable Class that will take the value to be evaluated and the derivative seed value. \n",
    "It will return the derivative of a function that is defined in terms of the variable class.\n",
    "\n",
    "#### Extension Mode Module\n",
    "* The extension mode will contain our future expansion for the AD package.\n",
    "\n",
    "#### Test Modules\n",
    "* This module will contain all the test that will be performed before the build. It will contain unit test and acceptance test. \n",
    "We will use pytest in our package and check our coverage with CodeCov.\n",
    "\n",
    "\n",
    "## Test Suite\n",
    "* As we are using PyScaffold our test will have their own directory and we will utilize pytest to perform our test. \n",
    "Also, we use CodeCov to give us a percentage of code coverage\n",
    "\n",
    "## Travis CI\n",
    "* Travis CI allows the build to be automated so it does not require a constant approval of updates by team members. \n",
    "It requires that all pull request be tested and if all the test is passed it will update the build automatically.  \n",
    "Travis CI also contains a badge that will inform the user if the package is working or if it is down.\n",
    "\n",
    "## CodeCov\n",
    "* CodeCov will inform the how much of the lines in our package are covered by test. \n",
    "This allows us to identify where test need to be added and will help discover possible bugs in the code. \n",
    "CodeCov also contains a badge that will inform the user a percentage of how much of the code is covered by test.\n",
    "\n",
    "## Distribution\n",
    "* For this package, we will use Github to distribute our package. This will require the user to clone our repo and follow our instructions to use the package. We will look into registering with conda-forge and dockerize.\n",
    "\n",
    "## Packaging\n",
    "* We will use the packaging framework PyScaffold as it is a streamlined way to setup our packaging. PyScaffold will setup our folder structure and make the configuration simplified. We will have to update the config once we want to package our library. As per the requirements, we will use the Spinx module to setup our documentation and pytest for the testing.\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_id": "00005-367bc068-4991-481d-9118-d379e4653924",
    "output_cleared": false,
    "tags": []
   },
   "source": [
    "# Software Implementation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_id": "00006-b22de42d-90cc-452a-b4a5-9be698938663",
    "output_cleared": false,
    "tags": []
   },
   "source": [
    "#### Core Data structures\n",
    "\n",
    "As per our current design, we plan to use the following data structures -\n",
    "\n",
    "1. Numpy arrays\n",
    "\n",
    "These will be used to save the values and derivatives in our Variable class. If the array has length 1, we are dealing with scalar values and if it has a longer length we are dealing with vectors (e.g. when dealing with vector valued functions)\n",
    "\n",
    "2. `Variable` objects\n",
    "\n",
    "`Variable` class can be used to instantiate gradient objects.\n",
    "These objects should be used to express the scalar or vector function to be differentiated. We overload the primitive mathematical operations to evaluate the derivatives in addition to function operation values. This class has the following attributes and methods:\n",
    "\n",
    "\n",
    "| Class Attributes        | Usage         |\n",
    "| ----------------------- |:-------------|\n",
    "| `value`                 | Value of the variable at which the derivative has to be evaluated (an array).|\n",
    "| `derivative_seed`      | Seed value of the derivative for differentation. Defaults to 1 (an array).             |\n",
    "| `derivative`            | Returns the derivative value  (an array)        |\n",
    "\n",
    "<br>\n",
    "\n",
    "| Class Methods           | Usage|\n",
    "| ----------------------- |:-------------|\n",
    "| `mul`                   | Operator overloaded `*` with product rule in differentiation     |\n",
    "| `pow`                   | Operator overloaded `**` with power rule in differentiation. For instance x`**`3 returns 2`*`x`*`2      |\n",
    "| `sqrt`                  | Square root is a special case of pow and therefore can be implemented by calling __pow__ with exponent `1\\2`    |\n",
    "| `sin`                   | evaluates `sin` and `cos` (the derivative of sine function)    |\n",
    "| `cos`                   | evaluates `cos` and its derivative -`sin`    |\n",
    "| `exp`                   | evaluates `exp` and its derivative `exp`  |\n",
    "| `log`                   | evaluates `log` and its derivative `1/arg`     |\n",
    "| `jacobian_matrix`       | Returns the computed jacobian matrix             |\n",
    "| `differentiate`         | Specify the mode (defaulting to forward mode ) and returns the derivative vector/scalar            |\n",
    "\n",
    "\n",
    "Note: other mathematical operations (e.g. summation, division, etc) will have corresponding function definitions as well\n",
    "\n",
    "#### External Dependencies\n",
    "In addition to the built-in libraries, we will use `numpy` package for implementation."
   ]
  },
  {
   "cell_type": "markdown",
   "source": [
    "# Feedback\n",
    "\n",
    "## Milestone 1\n",
    "\n",
    "Introduction\n",
    "- How does AD compare and improve from other similar methods?\n",
    "\n",
    "Background\n",
    "- The flow could be enhanced by presenting an evaluation trace along with the computational graph.\n",
    "\n",
    "How to use\n",
    "- How do you install the package? By cloning your repo? Add more info on this here.\n",
    "- Add the expected output from your example.\n",
    "\n",
    "Software Organization\n",
    "- Looks good!\n",
    "\n",
    "Implementation\n",
    "- So you have one class to override all the mathematical functions and to take the derivative? Or is the Variable class separate to the class methods? Elaborate further.\n"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_id": "00007-b9bd821f-125c-4f52-8c79-7d2944f5dc5a",
    "output_cleared": false,
    "tags": []
   },
   "source": [
    "# References\n",
    "-  Margossian, C. C. (2019). A review of automatic differentiation and its efficient implementation. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 9(4), 1–32. https://doi.org/10.1002/widm.1305\n",
    "\n",
    "- Saroufim, M. (2019, November 13). Automatic Differentiation Step by Step. Retrieved October 15, 2020, from https://marksaroufim.medium.com/automatic-differentiation-step-by-step-24240f97a6e6\n",
    "\n",
    "- Wang, C. (2019, March 3). Automatic Differentiation, Explained. Retrieved October15, 2020,from https://towardsdatascience.com/automatic-differentiation-explained-\tb4ba8e60c2ad\n",
    "\n",
    "- Automatic Differentiation Background. Retrived October 20, 2020, from https://www.mathworks.com/help/deeplearning/ug/deep-learning-with-automatic-differentiation-in-matlab.html"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_id": "00008-4fe2d889-942c-4d49-af55-8b17b1c4e883",
    "output_cleared": false,
    "tags": []
   },
   "source": []
  }
 ],
 "metadata": {
  "deepnote_execution_queue": [],
  "deepnote_notebook_id": "5c3e70e6-1603-4167-afb0-30ce30d2cf4a",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}