{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# default_exp models.tf.widedeep"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Wide & Deep Learning (W&D)\n",
    "\n",
    "Wide and Deep Learning Model, proposed by Google, 2016, is a DNN-Linear mixed model, which combines the strength of memorization and generalization. It's useful for generic large-scale regression and classification problems with sparse input features (e.g., categorical features with a large number of possible feature values). It has been used for Google App Store for their app recommendation.\n",
    "\n",
    "![Untitled](https://github.com/RecoHut-Stanzas/S021355/raw/main/images/img5.png)\n",
    "\n",
    "To understand the concept of deep & wide recommendations, it’s best to think of it as two separate, but collaborating, engines. The wide model, often referred to in the literature as the linear model, memorizes users and their past product choices. Its inputs may consist simply of a user identifier and a product identifier, though other attributes relevant to the pattern (such as time of day) may also be incorporated.\n",
    "\n",
    "![https://github.com/RecoHut-Stanzas/S021355/raw/main/images/img6.png](https://github.com/RecoHut-Stanzas/S021355/raw/main/images/img6.png)\n",
    "\n",
    "The deep portion of the model, so named as it is a deep neural network, examines the generalizable attributes of a user and their product choices. From these, the model learns the broader characteristics that tend to favor users’ product selections.\n",
    "\n",
    "Together, the wide and deep submodels are trained on historical product selections by individual users to predict future product selections. The end result is a single model capable of calculating the probability with which a user will purchase a given item, given both memorized past choices and generalizations about a user’s preferences. These probabilities form the basis for user-specific product rankings, which can be used for making recommendations.\n",
    "\n",
    "The goal with wide and deep recommenders is to provide the same level of customer intimacy that, for example, our favorite barista does. This model uses explicit and implicit feedback to expand the considerations set for customers. Wide and deep recommenders go beyond simple weighted averaging of customer feedback found in some collaborative filters to balance what is understood about the individual with what is known about similar customers. If done properly, the recommendations make the customer feel understood and this should translate into greater value for both the customer and the business.\n",
    "\n",
    "The intuitive logic of the wide-and-deep recommender belies the complexity of its actual construction. Inputs must be defined separately for each of the wide and deep portions of the model and each must be trained in a coordinated manner to arrive at a single output, but tuned using optimizers specific to the nature of each submodel. Thankfully, the **[Tensorflow DNNLinearCombinedClassifier estimator](https://www.tensorflow.org/api_docs/python/tf/estimator/DNNLinearCombinedClassifier)** provides a pre-packaged architecture, greatly simplifying the assembly of the overall model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#hide\n",
    "from nbdev.showdoc import *"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#export\n",
    "import numpy as np\n",
    "\n",
    "import tensorflow as tf\n",
    "from tensorflow.keras import Model\n",
    "from tensorflow.keras.layers import Layer, Input\n",
    "from tensorflow.keras.layers import Dense, Embedding, Dropout\n",
    "from tensorflow.keras.regularizers import l2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#export\n",
    "class Linear(Layer):\n",
    "    def __init__(self, feature_length, w_reg=1e-6):\n",
    "        \"\"\"\n",
    "        Linear Part\n",
    "        :param feature_length: A scalar. The length of features.\n",
    "        :param w_reg: A scalar. The regularization coefficient of parameter w.\n",
    "        \"\"\"\n",
    "        super(Linear, self).__init__()\n",
    "        self.feature_length = feature_length\n",
    "        self.w_reg = w_reg\n",
    "\n",
    "    def build(self, input_shape):\n",
    "        self.w = self.add_weight(name=\"w\",\n",
    "                                 shape=(self.feature_length, 1),\n",
    "                                 regularizer=l2(self.w_reg),\n",
    "                                 trainable=True)\n",
    "\n",
    "    def call(self, inputs, **kwargs):\n",
    "        result = tf.reduce_sum(tf.nn.embedding_lookup(self.w, inputs), axis=1)  # (batch_size, 1)\n",
    "        return result\n",
    "\n",
    "\n",
    "class DNN(Layer):\n",
    "    def __init__(self, hidden_units, activation='relu', dropout=0.):\n",
    "        \"\"\"Deep Neural Network\n",
    "\t\t:param hidden_units: A list. Neural network hidden units.\n",
    "\t\t:param activation: A string. Activation function of dnn.\n",
    "\t\t:param dropout: A scalar. Dropout number.\n",
    "\t\t\"\"\"\n",
    "        super(DNN, self).__init__()\n",
    "        self.dnn_network = [Dense(units=unit, activation=activation) for unit in hidden_units]\n",
    "        self.dropout = Dropout(dropout)\n",
    "\n",
    "    def call(self, inputs, **kwargs):\n",
    "        x = inputs\n",
    "        for dnn in self.dnn_network:\n",
    "            x = dnn(x)\n",
    "        x = self.dropout(x)\n",
    "        return x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#export\n",
    "class WideDeep(Model):\n",
    "    def __init__(self, feature_columns, hidden_units, activation='relu',\n",
    "                 dnn_dropout=0., embed_reg=1e-6, w_reg=1e-6):\n",
    "        \"\"\"\n",
    "        Wide&Deep\n",
    "        :param feature_columns: A list. sparse column feature information.\n",
    "        :param hidden_units: A list. Neural network hidden units.\n",
    "        :param activation: A string. Activation function of dnn.\n",
    "        :param dnn_dropout: A scalar. Dropout of dnn.\n",
    "        :param embed_reg: A scalar. The regularizer of embedding.\n",
    "        :param w_reg: A scalar. The regularizer of Linear.\n",
    "        \"\"\"\n",
    "        super(WideDeep, self).__init__()\n",
    "        self.sparse_feature_columns = feature_columns\n",
    "        self.embed_layers = {\n",
    "            'embed_' + str(i): Embedding(input_dim=feat['feat_num'],\n",
    "                                         input_length=1,\n",
    "                                         output_dim=feat['embed_dim'],\n",
    "                                         embeddings_initializer='random_uniform',\n",
    "                                         embeddings_regularizer=l2(embed_reg))\n",
    "            for i, feat in enumerate(self.sparse_feature_columns)\n",
    "        }\n",
    "        self.index_mapping = []\n",
    "        self.feature_length = 0\n",
    "        for feat in self.sparse_feature_columns:\n",
    "            self.index_mapping.append(self.feature_length)\n",
    "            self.feature_length += feat['feat_num']\n",
    "        self.dnn_network = DNN(hidden_units, activation, dnn_dropout)\n",
    "        self.linear = Linear(self.feature_length, w_reg=w_reg)\n",
    "        self.final_dense = Dense(1, activation=None)\n",
    "\n",
    "    def call(self, inputs, **kwargs):\n",
    "        sparse_embed = tf.concat([self.embed_layers['embed_{}'.format(i)](inputs[:, i])\n",
    "                                  for i in range(inputs.shape[1])], axis=-1)\n",
    "        x = sparse_embed  # (batch_size, field * embed_dim)\n",
    "        # Wide\n",
    "        wide_inputs = inputs + tf.convert_to_tensor(self.index_mapping)\n",
    "        wide_out = self.linear(wide_inputs)\n",
    "        # Deep\n",
    "        deep_out = self.dnn_network(x)\n",
    "        deep_out = self.final_dense(deep_out)\n",
    "        # out\n",
    "        outputs = tf.nn.sigmoid(0.5 * wide_out + 0.5 * deep_out)\n",
    "        return outputs\n",
    "\n",
    "    def summary(self, **kwargs):\n",
    "        sparse_inputs = Input(shape=(len(self.sparse_feature_columns),), dtype=tf.int32)\n",
    "        Model(inputs=sparse_inputs, outputs=self.call(sparse_inputs)).summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def test_model():\n",
    "    user_features = {'feat': 'user_id', 'feat_num': 100, 'embed_dim': 8}\n",
    "    seq_features = {'feat': 'item_id', 'feat_num': 100, 'embed_dim': 8}\n",
    "    features = [user_features, seq_features]\n",
    "    model = WideDeep(features, hidden_units=[8, 4, 2], dnn_dropout=0.5)\n",
    "    model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"model\"\n",
      "__________________________________________________________________________________________________\n",
      " Layer (type)                   Output Shape         Param #     Connected to                     \n",
      "==================================================================================================\n",
      " input_1 (InputLayer)           [(None, 2)]          0           []                               \n",
      "                                                                                                  \n",
      " tf.__operators__.getitem (Slic  (None,)             0           ['input_1[0][0]']                \n",
      " ingOpLambda)                                                                                     \n",
      "                                                                                                  \n",
      " tf.__operators__.getitem_1 (Sl  (None,)             0           ['input_1[0][0]']                \n",
      " icingOpLambda)                                                                                   \n",
      "                                                                                                  \n",
      " embedding (Embedding)          (None, 8)            800         ['tf.__operators__.getitem[0][0]'\n",
      "                                                                 ]                                \n",
      "                                                                                                  \n",
      " embedding_1 (Embedding)        (None, 8)            800         ['tf.__operators__.getitem_1[0][0\n",
      "                                                                 ]']                              \n",
      "                                                                                                  \n",
      " tf.concat (TFOpLambda)         (None, 16)           0           ['embedding[0][0]',              \n",
      "                                                                  'embedding_1[0][0]']            \n",
      "                                                                                                  \n",
      " tf.__operators__.add (TFOpLamb  (None, 2)           0           ['input_1[0][0]']                \n",
      " da)                                                                                              \n",
      "                                                                                                  \n",
      " dnn (DNN)                      (None, 2)            182         ['tf.concat[0][0]']              \n",
      "                                                                                                  \n",
      " linear (Linear)                (None, 1)            200         ['tf.__operators__.add[0][0]']   \n",
      "                                                                                                  \n",
      " dense_3 (Dense)                (None, 1)            3           ['dnn[0][0]']                    \n",
      "                                                                                                  \n",
      " tf.math.multiply (TFOpLambda)  (None, 1)            0           ['linear[0][0]']                 \n",
      "                                                                                                  \n",
      " tf.math.multiply_1 (TFOpLambda  (None, 1)           0           ['dense_3[0][0]']                \n",
      " )                                                                                                \n",
      "                                                                                                  \n",
      " tf.__operators__.add_1 (TFOpLa  (None, 1)           0           ['tf.math.multiply[0][0]',       \n",
      " mbda)                                                            'tf.math.multiply_1[0][0]']     \n",
      "                                                                                                  \n",
      " tf.math.sigmoid (TFOpLambda)   (None, 1)            0           ['tf.__operators__.add_1[0][0]'] \n",
      "                                                                                                  \n",
      "==================================================================================================\n",
      "Total params: 1,985\n",
      "Trainable params: 1,985\n",
      "Non-trainable params: 0\n",
      "__________________________________________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "test_model()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Author: Sparsh A.\n",
      "\n",
      "Last updated: 2021-12-20 10:45:29\n",
      "\n",
      "Compiler    : GCC 7.5.0\n",
      "OS          : Linux\n",
      "Release     : 5.4.104+\n",
      "Machine     : x86_64\n",
      "Processor   : x86_64\n",
      "CPU cores   : 2\n",
      "Architecture: 64bit\n",
      "\n",
      "tensorflow: 2.7.0\n",
      "numpy     : 1.19.5\n",
      "IPython   : 5.5.0\n",
      "\n"
     ]
    }
   ],
   "source": [
    "#hide\n",
    "!pip install -q watermark\n",
    "%reload_ext watermark\n",
    "%watermark -a \"Sparsh A.\" -m -iv -u -t -d"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "name": "python3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}
