{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# default_exp models.tf.fm"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Factorization Machines\n",
    "> Implementation of factorization machine models like FM, DeepFM, AFM, NCF, xDeepFM"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#hide\n",
    "from nbdev.showdoc import *"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#export\n",
    "import numpy as np\n",
    "import itertools\n",
    "\n",
    "import tensorflow as tf\n",
    "from tensorflow.keras import Model\n",
    "from tensorflow.keras.layers import Layer, Input, ReLU, Flatten\n",
    "from tensorflow.keras.layers import Dense, Embedding, Dropout\n",
    "from tensorflow.keras.layers import BatchNormalization\n",
    "from tensorflow.keras.regularizers import l2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Vanilla FM\n",
    "\n",
    "Factorization Machines (FMs) are a supervised learning approach that enhances the linear regression model by incorporating the second-order feature interactions. Factorization Machine type algorithms are a combination of linear regression and matrix factorization, the cool idea behind this type of algorithm is it aims model interactions between features (a.k.a attributes, explanatory variables) using factorized parameters. By doing so it has the ability to estimate all interactions between features even with extremely sparse data.\n",
    "\n",
    "Factorization machines (FM) [Rendle, 2010], proposed by Steffen Rendle in 2010, is a supervised algorithm that can be used for classification, regression, and ranking tasks. It quickly took notice and became a popular and impactful method for making predictions and recommendations. Particularly, it is a generalization of the linear regression model and the matrix factorization model. Moreover, it is reminiscent of support vector machines with a polynomial kernel. The strengths of factorization machines over the linear regression and matrix factorization are: (1) it can model χ -way variable interactions, where χ is the number of polynomial order and is usually set to two. (2) A fast optimization algorithm associated with factorization machines can reduce the polynomial computation time to linear complexity, making it extremely efficient especially for high dimensional sparse inputs. For these reasons, factorization machines are widely employed in modern advertisement and products recommendations.\n",
    "\n",
    "Most recommendation problems assume that we have a consumption/rating dataset formed by a collection of *(user, item, rating*) tuples. This is the starting point for most variations of Collaborative Filtering algorithms and they have proven to yield nice results; however, in many applications, we have plenty of item metadata (tags, categories, genres) that can be used to make better predictions. This is one of the benefits of using Factorization Machines with feature-rich datasets, for which there is a natural way in which extra features can be included in the model and higher-order interactions can be modeled using the dimensionality parameter d. For sparse datasets, a second-order FM model suffices, since there is not enough information to estimate more complex interactions.\n",
    "\n",
    "![https://github.com/RecoHut-Stanzas/S021355/raw/main/images/img4.png](https://github.com/RecoHut-Stanzas/S021355/raw/main/images/img4.png)\n",
    "\n",
    "$$f(x) = w_0 + \\sum_{p=1}^Pw_px_p + \\sum_{p=1}^{P-1}\\sum_{q=p+1}^Pw_{p,q}x_px_q$$\n",
    "\n",
    "This model formulation may look familiar — it's simply a quadratic linear regression. However, unlike polynomial linear models which estimate each interaction term separately, FMs instead use factorized interaction parameters: feature interaction weights are represented as the inner product of the two features' latent factor space embeddings:\n",
    "\n",
    "$$f(x) = w_0 + \\sum_{p=1}^Pw_px_p + \\sum_{p=1}^{P-1}\\sum_{q=p+1}^P\\langle v_p,v_q \\rangle x_px_q$$\n",
    "\n",
    "This greatly decreases the number of parameters to estimate while at the same time facilitating more accurate estimation by breaking the strict independence criteria between interaction terms. Consider a realistic recommendation data set with 1,000,000 users and 10,000 items. A quadratic linear model would need to estimate U + I + UI ~ 10 billion parameters. A FM model of dimension F=10 would need only U + I + F(U + I) ~ 11 million parameters. Additionally, many common MF algorithms (including SVD++, ALS) can be re-formulated as special cases of the more general/flexible FM model class.\n",
    "\n",
    "The above equation can be rewritten as:\n",
    "\n",
    "$$\\begin{align*}\n",
    "\\hat{y}(\\textbf{x}) = w_{0} + \\sum_{i=1}^{n} w_{i} x_{i} +  \\sum_{i=1}^n \\sum_{j=i+1}^n \\hat{w}_{ij} x_{i} x_{j}\n",
    "\\end{align*}$$\n",
    "\n",
    "where,\n",
    "\n",
    "- $w_0$ is the global bias\n",
    "- $w_i$ denotes the weight of the i-th feature,\n",
    "- $\\hat{w}_{ij} = v_i^Tv_j$ denotes the weight of the cross feature $x_ix_j$\n",
    "- $v_i \\in \\mathcal{R}^k$ denotes the embedding vector for feature $i$\n",
    "- $k$ denotes the size of embedding vector\n",
    "\n",
    "<aside>\n",
    "💡 For large, sparse datasets...FM and FFM is good. But for small, dense datasets...try to avoid.\n",
    "\n",
    "</aside>\n",
    "\n",
    "Factorization machines appeared to be the method which answered the challenge!\n",
    "\n",
    "|  | Accuracy | Speed | Sparsity |\n",
    "| --- | --- | --- | --- |\n",
    "| Collaborative Filtering | Too Accurate | Suitable | Suitable |\n",
    "| SVM | Too Accurate | Suitable | Unsuitable |\n",
    "| Random Forest/CART | General Accuracy | Unsuitable | Unsuitable |\n",
    "| Factorization Machines (FM) | General Accuracy | Quick | Designed for it |\n",
    "\n",
    "To learn the FM model, we can use the MSE loss for regression task, the cross entropy loss for classification tasks, and the BPR loss for ranking task. Standard optimizers such as SGD and Adam are viable for optimization."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#export\n",
    "class FM_Layer(Layer):\n",
    "    def __init__(self, feature_columns, k, w_reg=1e-6, v_reg=1e-6):\n",
    "        \"\"\"\n",
    "        Factorization Machines\n",
    "        :param feature_columns: A list. sparse column feature information.\n",
    "        :param k: the latent vector\n",
    "        :param w_reg: the regularization coefficient of parameter w\n",
    "        :param v_reg: the regularization coefficient of parameter v\n",
    "        \"\"\"\n",
    "        super(FM_Layer, self).__init__()\n",
    "        self.sparse_feature_columns = feature_columns\n",
    "        self.index_mapping = []\n",
    "        self.feature_length = 0\n",
    "        for feat in self.sparse_feature_columns:\n",
    "            self.index_mapping.append(self.feature_length)\n",
    "            self.feature_length += feat['feat_num']\n",
    "        self.k = k\n",
    "        self.w_reg = w_reg\n",
    "        self.v_reg = v_reg\n",
    "\n",
    "    def build(self, input_shape):\n",
    "        self.w0 = self.add_weight(name='w0', shape=(1,),\n",
    "                                  initializer=tf.zeros_initializer(),\n",
    "                                  trainable=True)\n",
    "        self.w = self.add_weight(name='w', shape=(self.feature_length, 1),\n",
    "                                 initializer=tf.random_normal_initializer(),\n",
    "                                 regularizer=l2(self.w_reg),\n",
    "                                 trainable=True)\n",
    "        self.V = self.add_weight(name='V', shape=(self.feature_length, self.k),\n",
    "                                 initializer=tf.random_normal_initializer(),\n",
    "                                 regularizer=l2(self.v_reg),\n",
    "                                 trainable=True)\n",
    "\n",
    "    def call(self, inputs, **kwargs):\n",
    "        # mapping\n",
    "        inputs = inputs + tf.convert_to_tensor(self.index_mapping)\n",
    "        # first order\n",
    "        first_order = self.w0 + tf.reduce_sum(tf.nn.embedding_lookup(self.w, inputs), axis=1)  # (batch_size, 1)\n",
    "        # second order\n",
    "        second_inputs = tf.nn.embedding_lookup(self.V, inputs)  # (batch_size, fields, embed_dim)\n",
    "        square_sum = tf.square(tf.reduce_sum(second_inputs, axis=1, keepdims=True))  # (batch_size, 1, embed_dim)\n",
    "        sum_square = tf.reduce_sum(tf.square(second_inputs), axis=1, keepdims=True)  # (batch_size, 1, embed_dim)\n",
    "        second_order = 0.5 * tf.reduce_sum(square_sum - sum_square, axis=2)  # (batch_size, 1)\n",
    "        # outputs\n",
    "        outputs = first_order + second_order\n",
    "        return outputs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#export\n",
    "class FM_Layer_v2(Layer):\n",
    "    \"\"\"\n",
    "    Wide part\n",
    "    \"\"\"\n",
    "    def __init__(self, feature_length, w_reg=1e-6):\n",
    "        \"\"\"\n",
    "        Factorization Machine\n",
    "        In DeepFM, only the first order feature and second order feature intersect are included.\n",
    "        :param feature_length: A scalar. The length of features.\n",
    "        :param w_reg: A scalar. The regularization coefficient of parameter w.\n",
    "        \"\"\"\n",
    "        super(FM_Layer_v2, self).__init__()\n",
    "        self.feature_length = feature_length\n",
    "        self.w_reg = w_reg\n",
    "\n",
    "    def build(self, input_shape):\n",
    "        self.w = self.add_weight(name='w', shape=(self.feature_length, 1),\n",
    "                                 initializer='random_normal',\n",
    "                                 regularizer=l2(self.w_reg),\n",
    "                                 trainable=True)\n",
    "\n",
    "    def call(self, inputs, **kwargs):\n",
    "        \"\"\"\n",
    "        :param inputs: A dict with shape `(batch_size, {'sparse_inputs', 'embed_inputs'})`:\n",
    "          sparse_inputs is 2D tensor with shape `(batch_size, sum(field_num))`\n",
    "          embed_inputs is 3D tensor with shape `(batch_size, fields, embed_dim)`\n",
    "        \"\"\"\n",
    "        sparse_inputs, embed_inputs = inputs['sparse_inputs'], inputs['embed_inputs']\n",
    "        # first order\n",
    "        first_order = tf.reduce_sum(tf.nn.embedding_lookup(self.w, sparse_inputs), axis=1)  # (batch_size, 1)\n",
    "        # second order\n",
    "        square_sum = tf.square(tf.reduce_sum(embed_inputs, axis=1, keepdims=True))  # (batch_size, 1, embed_dim)\n",
    "        sum_square = tf.reduce_sum(tf.square(embed_inputs), axis=1, keepdims=True)  # (batch_size, 1, embed_dim)\n",
    "        second_order = 0.5 * tf.reduce_sum(square_sum - sum_square, axis=2)  # (batch_size, 1)\n",
    "        return first_order + second_order"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#export\n",
    "class FM(Model):\n",
    "    def __init__(self, feature_columns, k, w_reg=1e-6, v_reg=1e-6):\n",
    "        \"\"\"\n",
    "        Factorization Machines\n",
    "        :param feature_columns: A list. sparse column feature information.\n",
    "        :param k: the latent vector\n",
    "        :param w_reg: the regularization coefficient of parameter w\n",
    "\t\t:param v_reg: the regularization coefficient of parameter v\n",
    "        \"\"\"\n",
    "        super(FM, self).__init__()\n",
    "        self.sparse_feature_columns = feature_columns\n",
    "        self.fm = FM_Layer(feature_columns, k, w_reg, v_reg)\n",
    "\n",
    "    def call(self, inputs, **kwargs):\n",
    "        fm_outputs = self.fm(inputs)\n",
    "        outputs = tf.nn.sigmoid(fm_outputs)\n",
    "        return outputs\n",
    "\n",
    "    def summary(self, **kwargs):\n",
    "        sparse_inputs = Input(shape=(len(self.sparse_feature_columns),), dtype=tf.int32)\n",
    "        Model(inputs=sparse_inputs, outputs=self.call(sparse_inputs)).summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def test_model():\n",
    "    user_features = {'feat': 'user_id', 'feat_num': 100, 'embed_dim': 8}\n",
    "    seq_features = {'feat': 'item_id', 'feat_num': 100, 'embed_dim': 8}\n",
    "    features = [user_features, seq_features]\n",
    "    model = FM(features, k=8)\n",
    "    model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"model_1\"\n",
      "_________________________________________________________________\n",
      " Layer (type)                Output Shape              Param #   \n",
      "=================================================================\n",
      " input_5 (InputLayer)        [(None, 2)]               0         \n",
      "                                                                 \n",
      " fm__layer (FM_Layer)        (None, 1)                 1801      \n",
      "                                                                 \n",
      " tf.math.sigmoid (TFOpLambda  (None, 1)                0         \n",
      " )                                                               \n",
      "                                                                 \n",
      "=================================================================\n",
      "Total params: 1,801\n",
      "Trainable params: 1,801\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "test_model()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Field-aware Factorization Machines (FFM)\n",
    "\n",
    "Despite effectiveness, FM can be hindered by its modelling of all feature interactions with the same weight, as not all feature interactions are equally useful and predictive. For example, the interactions with useless features may even introduce noises and adversely degrade the performance.\n",
    "\n",
    "|  | For each | Learn |\n",
    "| --- | --- | --- |\n",
    "| Linear | feature | a weight |\n",
    "| Poly | feature pair | a weight |\n",
    "| FM | feature | a latent vector |\n",
    "| FFM | feature | multiple latent vectors |\n",
    "\n",
    "Field-aware factorization machine (FFM) is an extension to FM. It was originally introduced in [2]. The advantage of FFM over FM is that it uses different factorized latent factors for different groups of features. The \"group\" is called \"field\" in the context of FFM. Putting features into fields resolves the issue that the latent factors shared by features that intuitively represent different categories of information may not well generalize the correlation.\n",
    "\n",
    "FFM addresses this issue by splitting the original latent space into smaller latent spaces specific to the fields of the features.\n",
    "\n",
    "$$\\phi(\\pmb{w}, \\pmb{x}) = w_0 + \\sum\\limits_{i=1}^n w_i x_i + \\sum\\limits_{i=1}^n \\sum\\limits_{j=i + 1}^n \\langle \\mathbf{v}_{i, f_{2}} \\cdot \\mathbf{v}_{j, f_{1}} \\rangle x_i x_j$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#export\n",
    "class FFM_Layer(Layer):\n",
    "    def __init__(self, sparse_feature_columns, k, w_reg=1e-6, v_reg=1e-6):\n",
    "        \"\"\"\n",
    "        :param dense_feature_columns: A list. sparse column feature information.\n",
    "        :param k: A scalar. The latent vector\n",
    "        :param w_reg: A scalar. The regularization coefficient of parameter w\n",
    "\t\t:param v_reg: A scalar. The regularization coefficient of parameter v\n",
    "        \"\"\"\n",
    "        super(FFM_Layer, self).__init__()\n",
    "        self.sparse_feature_columns = sparse_feature_columns\n",
    "        self.k = k\n",
    "        self.w_reg = w_reg\n",
    "        self.v_reg = v_reg\n",
    "        self.index_mapping = []\n",
    "        self.feature_length = 0\n",
    "        for feat in self.sparse_feature_columns:\n",
    "            self.index_mapping.append(self.feature_length)\n",
    "            self.feature_length += feat['feat_num']\n",
    "        self.field_num = len(self.sparse_feature_columns)\n",
    "\n",
    "    def build(self, input_shape):\n",
    "        self.w0 = self.add_weight(name='w0', shape=(1,),\n",
    "                                  initializer=tf.zeros_initializer(),\n",
    "                                  trainable=True)\n",
    "        self.w = self.add_weight(name='w', shape=(self.feature_length, 1),\n",
    "                                 initializer='random_normal',\n",
    "                                 regularizer=l2(self.w_reg),\n",
    "                                 trainable=True)\n",
    "        self.v = self.add_weight(name='v',\n",
    "                                 shape=(self.feature_length, self.field_num, self.k),\n",
    "                                 initializer='random_normal',\n",
    "                                 regularizer=l2(self.v_reg),\n",
    "                                 trainable=True)\n",
    "\n",
    "    def call(self, inputs, **kwargs):\n",
    "        inputs = inputs + tf.convert_to_tensor(self.index_mapping)\n",
    "        # first order\n",
    "        first_order = self.w0 + tf.reduce_sum(tf.nn.embedding_lookup(self.w, inputs), axis=1)  # (batch_size, 1)\n",
    "        # field second order\n",
    "        second_order = 0\n",
    "        latent_vector = tf.reduce_sum(tf.nn.embedding_lookup(self.v, inputs), axis=1)  # (batch_size, field_num, k)\n",
    "        for i in range(self.field_num):\n",
    "            for j in range(i+1, self.field_num):\n",
    "                second_order += tf.reduce_sum(latent_vector[:, i] * latent_vector[:, j], axis=1, keepdims=True)\n",
    "        return first_order + second_order"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#export\n",
    "class FFM(Model):\n",
    "    def __init__(self, feature_columns, k, w_reg=1e-6, v_reg=1e-6):\n",
    "        \"\"\"\n",
    "        FFM architecture\n",
    "        :param feature_columns: A list. sparse column feature information.\n",
    "        :param k: the latent vector\n",
    "        :param w_reg: the regularization coefficient of parameter w\n",
    "\t\t:param field_reg_reg: the regularization coefficient of parameter v\n",
    "        \"\"\"\n",
    "        super(FFM, self).__init__()\n",
    "        self.sparse_feature_columns = feature_columns\n",
    "        self.ffm = FFM_Layer(self.sparse_feature_columns, k, w_reg, v_reg)\n",
    "\n",
    "    def call(self, inputs, **kwargs):\n",
    "        ffm_out = self.ffm(inputs)\n",
    "        outputs = tf.nn.sigmoid(ffm_out)\n",
    "        return outputs\n",
    "\n",
    "    def summary(self, **kwargs):\n",
    "        sparse_inputs = Input(shape=(len(self.sparse_feature_columns),), dtype=tf.int32)\n",
    "        tf.keras.Model(inputs=sparse_inputs, outputs=self.call(sparse_inputs)).summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def test_model():\n",
    "    user_features = {'feat': 'user_id', 'feat_num': 100, 'embed_dim': 8}\n",
    "    seq_features = {'feat': 'item_id', 'feat_num': 100, 'embed_dim': 8}\n",
    "    features = [user_features, seq_features]\n",
    "    model = FFM(features, k=8)\n",
    "    model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"model_2\"\n",
      "_________________________________________________________________\n",
      " Layer (type)                Output Shape              Param #   \n",
      "=================================================================\n",
      " input_6 (InputLayer)        [(None, 2)]               0         \n",
      "                                                                 \n",
      " ffm__layer (FFM_Layer)      (None, 1)                 3401      \n",
      "                                                                 \n",
      " tf.math.sigmoid_1 (TFOpLamb  (None, 1)                0         \n",
      " da)                                                             \n",
      "                                                                 \n",
      "=================================================================\n",
      "Total params: 3,401\n",
      "Trainable params: 3,401\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "test_model()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Neural Factorization Machine (NFM)\n",
    "\n",
    "NFM seamlessly combines the linearity of FM in modelling second-order feature interactions and the non-linearity of neural network in modelling higher-order feature interactions. Conceptually, NFM is more expressive than FM since FM can be seen as a special case of NFM without hidden layers.\n",
    "\n",
    "![Untitled](https://github.com/RecoHut-Stanzas/S021355/raw/main/images/img10.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#export\n",
    "class DNN(Layer):\n",
    "    def __init__(self, hidden_units, activation='relu', dropout=0.):\n",
    "        \"\"\"Deep Neural Network\n",
    "\t\t:param hidden_units: A list. Neural network hidden units.\n",
    "\t\t:param activation: A string. Activation function of dnn.\n",
    "\t\t:param dropout: A scalar. Dropout number.\n",
    "\t\t\"\"\"\n",
    "        super(DNN, self).__init__()\n",
    "        self.dnn_network = [Dense(units=unit, activation=activation) for unit in hidden_units]\n",
    "        self.dropout = Dropout(dropout)\n",
    "\n",
    "    def call(self, inputs, **kwargs):\n",
    "        x = inputs\n",
    "        for dnn in self.dnn_network:\n",
    "            x = dnn(x)\n",
    "        x = self.dropout(x)\n",
    "        return x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#export\n",
    "class NFM(Model):\n",
    "    def __init__(self, feature_columns, hidden_units, dnn_dropout=0., activation='relu', bn_use=True, embed_reg=1e-6):\n",
    "        \"\"\"\n",
    "        NFM architecture\n",
    "        :param feature_columns: A list. sparse column feature information.\n",
    "        :param hidden_units: A list. Neural network hidden units.\n",
    "        :param activation: A string. Activation function of dnn.\n",
    "        :param dnn_dropout: A scalar. Dropout of dnn.\n",
    "        :param bn_use: A Boolean. Use BatchNormalization or not.\n",
    "        :param embed_reg: A scalar. The regularizer of embedding.\n",
    "        \"\"\"\n",
    "        super(NFM, self).__init__()\n",
    "        self.sparse_feature_columns = feature_columns\n",
    "        self.embed_layers = {\n",
    "            'embed_' + str(i): Embedding(input_dim=feat['feat_num'],\n",
    "                                         input_length=1,\n",
    "                                         output_dim=feat['embed_dim'],\n",
    "                                         embeddings_initializer='random_normal',\n",
    "                                         embeddings_regularizer=l2(embed_reg))\n",
    "            for i, feat in enumerate(self.sparse_feature_columns)\n",
    "        }\n",
    "        self.bn = BatchNormalization()\n",
    "        self.bn_use = bn_use\n",
    "        self.dnn_network = DNN(hidden_units, activation, dnn_dropout)\n",
    "        self.dense = Dense(1, activation=None)\n",
    "\n",
    "    def call(self, inputs):\n",
    "        # Inputs layer\n",
    "        sparse_inputs = inputs\n",
    "        # Embedding layer\n",
    "        sparse_embed = [self.embed_layers['embed_{}'.format(i)](sparse_inputs[:, i])\n",
    "                 for i in range(sparse_inputs.shape[1])]\n",
    "        sparse_embed = tf.transpose(tf.convert_to_tensor(sparse_embed), [1, 0, 2])  # (None, filed_num, embed_dim)\n",
    "        # Bi-Interaction Layer\n",
    "        sparse_embed = 0.5 * (tf.pow(tf.reduce_sum(sparse_embed, axis=1), 2) -\n",
    "                       tf.reduce_sum(tf.pow(sparse_embed, 2), axis=1))  # (None, embed_dim)\n",
    "        # Concat\n",
    "        x = sparse_embed\n",
    "        # BatchNormalization\n",
    "        x = self.bn(x, training=self.bn_use)\n",
    "        # Hidden Layers\n",
    "        x = self.dnn_network(x)\n",
    "        outputs = tf.nn.sigmoid(self.dense(x))\n",
    "        return outputs\n",
    "\n",
    "    def summary(self):\n",
    "        sparse_inputs = Input(shape=(len(self.sparse_feature_columns),), dtype=tf.int32)\n",
    "        tf.keras.Model(inputs=sparse_inputs, outputs=self.call(sparse_inputs)).summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def test_model():\n",
    "    user_features = {'feat': 'user_id', 'feat_num': 100, 'embed_dim': 8}\n",
    "    seq_features = {'feat': 'item_id', 'feat_num': 100, 'embed_dim': 8}\n",
    "    features = [user_features, seq_features]\n",
    "    model = NFM(features, hidden_units=[8, 4, 2], dnn_dropout=0.5)\n",
    "    model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"model_3\"\n",
      "__________________________________________________________________________________________________\n",
      " Layer (type)                   Output Shape         Param #     Connected to                     \n",
      "==================================================================================================\n",
      " input_7 (InputLayer)           [(None, 2)]          0           []                               \n",
      "                                                                                                  \n",
      " tf.__operators__.getitem (Slic  (None,)             0           ['input_7[0][0]']                \n",
      " ingOpLambda)                                                                                     \n",
      "                                                                                                  \n",
      " tf.__operators__.getitem_1 (Sl  (None,)             0           ['input_7[0][0]']                \n",
      " icingOpLambda)                                                                                   \n",
      "                                                                                                  \n",
      " embedding_3 (Embedding)        (None, 8)            800         ['tf.__operators__.getitem[0][0]'\n",
      "                                                                 ]                                \n",
      "                                                                                                  \n",
      " embedding_4 (Embedding)        (None, 8)            800         ['tf.__operators__.getitem_1[0][0\n",
      "                                                                 ]']                              \n",
      "                                                                                                  \n",
      " tf.convert_to_tensor (TFOpLamb  (2, None, 8)        0           ['embedding_3[0][0]',            \n",
      " da)                                                              'embedding_4[0][0]']            \n",
      "                                                                                                  \n",
      " tf.compat.v1.transpose (TFOpLa  (None, 2, 8)        0           ['tf.convert_to_tensor[0][0]']   \n",
      " mbda)                                                                                            \n",
      "                                                                                                  \n",
      " tf.math.reduce_sum_5 (TFOpLamb  (None, 8)           0           ['tf.compat.v1.transpose[0][0]'] \n",
      " da)                                                                                              \n",
      "                                                                                                  \n",
      " tf.math.pow_1 (TFOpLambda)     (None, 2, 8)         0           ['tf.compat.v1.transpose[0][0]'] \n",
      "                                                                                                  \n",
      " tf.math.pow (TFOpLambda)       (None, 8)            0           ['tf.math.reduce_sum_5[0][0]']   \n",
      "                                                                                                  \n",
      " tf.math.reduce_sum_6 (TFOpLamb  (None, 8)           0           ['tf.math.pow_1[0][0]']          \n",
      " da)                                                                                              \n",
      "                                                                                                  \n",
      " tf.math.subtract_5 (TFOpLambda  (None, 8)           0           ['tf.math.pow[0][0]',            \n",
      " )                                                                'tf.math.reduce_sum_6[0][0]']   \n",
      "                                                                                                  \n",
      " tf.math.multiply_4 (TFOpLambda  (None, 8)           0           ['tf.math.subtract_5[0][0]']     \n",
      " )                                                                                                \n",
      "                                                                                                  \n",
      " batch_normalization (BatchNorm  (None, 8)           32          ['tf.math.multiply_4[0][0]']     \n",
      " alization)                                                                                       \n",
      "                                                                                                  \n",
      " dnn (DNN)                      (None, 2)            118         ['batch_normalization[0][0]']    \n",
      "                                                                                                  \n",
      " dense_3 (Dense)                (None, 1)            3           ['dnn[0][0]']                    \n",
      "                                                                                                  \n",
      " tf.math.sigmoid_2 (TFOpLambda)  (None, 1)           0           ['dense_3[0][0]']                \n",
      "                                                                                                  \n",
      "==================================================================================================\n",
      "Total params: 1,753\n",
      "Trainable params: 1,737\n",
      "Non-trainable params: 16\n",
      "__________________________________________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "test_model()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Attentional Factorization Machines (AFM)\n",
    "\n",
    "Improves FM by discriminating the importance of different feature interactions. It learns the importance of each feature interaction from data via a neural attention network. Empirically, it is shown on regression task AFM betters FM with a 8.6% relative improvement, and consistently outperforms the state-of-the-art deep learning methods Wide&Deep and DeepCross with a much simpler structure and fewer model parameters.\n",
    "\n",
    "![Untitled](https://github.com/RecoHut-Stanzas/S021355/raw/main/images/img11.png)\n",
    "\n",
    "Formally, the AFM model can be defined as:\n",
    "\n",
    "$$\\hat{y}_{AFM} (x) = w_0 + \\sum_{i=1}^nw_ix_i + p^T\\sum_{i=1}^n\\sum_{j=i+1}^na_{ij}(v_i\\odot v_j)x_ix_j$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#export\n",
    "class AFM(Model):\n",
    "    def __init__(self, feature_columns, mode, att_vector=8, activation='relu', dropout=0.5, embed_reg=1e-6):\n",
    "        \"\"\"\n",
    "        AFM \n",
    "        :param feature_columns: A list. sparse column feature information.\n",
    "        :param mode: A string. 'max'(MAX Pooling) or 'avg'(Average Pooling) or 'att'(Attention)\n",
    "        :param att_vector: A scalar. attention vector.\n",
    "        :param activation: A string. Activation function of attention.\n",
    "        :param dropout: A scalar. Dropout.\n",
    "        :param embed_reg: A scalar. the regularizer of embedding\n",
    "        \"\"\"\n",
    "        super(AFM, self).__init__()\n",
    "        self.sparse_feature_columns = feature_columns\n",
    "        self.mode = mode\n",
    "        self.embed_layers = {\n",
    "            'embed_' + str(i): Embedding(input_dim=feat['feat_num'],\n",
    "                                         input_length=1,\n",
    "                                         output_dim=feat['embed_dim'],\n",
    "                                         embeddings_initializer='random_uniform',\n",
    "                                         embeddings_regularizer=l2(embed_reg))\n",
    "            for i, feat in enumerate(self.sparse_feature_columns)\n",
    "        }\n",
    "        if self.mode == 'att':\n",
    "            self.attention_W = Dense(units=att_vector, activation=activation, use_bias=True)\n",
    "            self.attention_dense = Dense(units=1, activation=None)\n",
    "        self.dropout = Dropout(dropout)\n",
    "        self.dense = Dense(units=1, activation=None)\n",
    "\n",
    "    def call(self, inputs):\n",
    "        # Input Layer\n",
    "        sparse_inputs = inputs\n",
    "        # Embedding Layer \n",
    "        embed = [self.embed_layers['embed_{}'.format(i)](sparse_inputs[:, i]) for i in range(sparse_inputs.shape[1])]\n",
    "        embed = tf.transpose(tf.convert_to_tensor(embed), perm=[1, 0, 2])  # (None, len(sparse_inputs), embed_dim)\n",
    "        # Pair-wise Interaction Layer\n",
    "        row = []\n",
    "        col = []\n",
    "        for r, c in itertools.combinations(range(len(self.sparse_feature_columns)), 2):\n",
    "            row.append(r)\n",
    "            col.append(c)\n",
    "        p = tf.gather(embed, row, axis=1)  # (None, (len(sparse) * len(sparse) - 1) / 2, k)\n",
    "        q = tf.gather(embed, col, axis=1)  # (None, (len(sparse) * len(sparse) - 1) / 2, k)\n",
    "        bi_interaction = p * q  # (None, (len(sparse) * len(sparse) - 1) / 2, k)\n",
    "        # mode\n",
    "        if self.mode == 'max':\n",
    "            # MaxPooling Layer\n",
    "            x = tf.reduce_sum(bi_interaction, axis=1)   # (None, k)\n",
    "        elif self.mode == 'avg':\n",
    "            # AvgPooling Layer\n",
    "            x = tf.reduce_mean(bi_interaction, axis=1)  # (None, k)\n",
    "        else:\n",
    "            # Attention Layer\n",
    "            x = self.attention(bi_interaction)  # (None, k)\n",
    "        # Output Layer\n",
    "        outputs = tf.nn.sigmoid(self.dense(x))\n",
    "\n",
    "        return outputs\n",
    "\n",
    "    def summary(self):\n",
    "        sparse_inputs = Input(shape=(len(self.sparse_feature_columns),), dtype=tf.int32)\n",
    "        Model(inputs=sparse_inputs, outputs=self.call(sparse_inputs)).summary()\n",
    "\n",
    "    def attention(self, bi_interaction):\n",
    "        a = self.attention_W(bi_interaction)  # (None, (len(sparse) * len(sparse) - 1) / 2, t)\n",
    "        a = self.attention_dense(a)  # (None, (len(sparse) * len(sparse) - 1) / 2, 1)\n",
    "        a_score = tf.nn.softmax(a, axis=1)  # (None, (len(sparse) * len(sparse) - 1) / 2, 1)\n",
    "        outputs = tf.reduce_sum(bi_interaction * a_score, axis=1)  # (None, embed_dim)\n",
    "        return outputs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def test_model():\n",
    "    user_features = {'feat': 'user_id', 'feat_num': 100, 'embed_dim': 8}\n",
    "    seq_features = {'feat': 'item_id', 'feat_num': 100, 'embed_dim': 8}\n",
    "    features = [user_features, seq_features]\n",
    "    model = AFM(features, mode='att', att_vector=8, \n",
    "                activation='relu', dropout=0.5, embed_reg=1e-5)\n",
    "    model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"model_4\"\n",
      "__________________________________________________________________________________________________\n",
      " Layer (type)                   Output Shape         Param #     Connected to                     \n",
      "==================================================================================================\n",
      " input_8 (InputLayer)           [(None, 2)]          0           []                               \n",
      "                                                                                                  \n",
      " tf.__operators__.getitem_2 (Sl  (None,)             0           ['input_8[0][0]']                \n",
      " icingOpLambda)                                                                                   \n",
      "                                                                                                  \n",
      " tf.__operators__.getitem_3 (Sl  (None,)             0           ['input_8[0][0]']                \n",
      " icingOpLambda)                                                                                   \n",
      "                                                                                                  \n",
      " embedding_5 (Embedding)        (None, 8)            800         ['tf.__operators__.getitem_2[0][0\n",
      "                                                                 ]']                              \n",
      "                                                                                                  \n",
      " embedding_6 (Embedding)        (None, 8)            800         ['tf.__operators__.getitem_3[0][0\n",
      "                                                                 ]']                              \n",
      "                                                                                                  \n",
      " tf.convert_to_tensor_1 (TFOpLa  (2, None, 8)        0           ['embedding_5[0][0]',            \n",
      " mbda)                                                            'embedding_6[0][0]']            \n",
      "                                                                                                  \n",
      " tf.compat.v1.transpose_1 (TFOp  (None, 2, 8)        0           ['tf.convert_to_tensor_1[0][0]'] \n",
      " Lambda)                                                                                          \n",
      "                                                                                                  \n",
      " tf.compat.v1.gather (TFOpLambd  (None, 1, 8)        0           ['tf.compat.v1.transpose_1[0][0]'\n",
      " a)                                                              ]                                \n",
      "                                                                                                  \n",
      " tf.compat.v1.gather_1 (TFOpLam  (None, 1, 8)        0           ['tf.compat.v1.transpose_1[0][0]'\n",
      " bda)                                                            ]                                \n",
      "                                                                                                  \n",
      " tf.math.multiply_5 (TFOpLambda  (None, 1, 8)        0           ['tf.compat.v1.gather[0][0]',    \n",
      " )                                                                'tf.compat.v1.gather_1[0][0]']  \n",
      "                                                                                                  \n",
      " dense_4 (Dense)                (None, 1, 8)         72          ['tf.math.multiply_5[0][0]']     \n",
      "                                                                                                  \n",
      " dense_5 (Dense)                (None, 1, 1)         9           ['dense_4[0][0]']                \n",
      "                                                                                                  \n",
      " tf.nn.softmax (TFOpLambda)     (None, 1, 1)         0           ['dense_5[0][0]']                \n",
      "                                                                                                  \n",
      " tf.math.multiply_6 (TFOpLambda  (None, 1, 8)        0           ['tf.math.multiply_5[0][0]',     \n",
      " )                                                                'tf.nn.softmax[0][0]']          \n",
      "                                                                                                  \n",
      " tf.math.reduce_sum_7 (TFOpLamb  (None, 8)           0           ['tf.math.multiply_6[0][0]']     \n",
      " da)                                                                                              \n",
      "                                                                                                  \n",
      " dense_6 (Dense)                (None, 1)            9           ['tf.math.reduce_sum_7[0][0]']   \n",
      "                                                                                                  \n",
      " tf.math.sigmoid_3 (TFOpLambda)  (None, 1)           0           ['dense_6[0][0]']                \n",
      "                                                                                                  \n",
      "==================================================================================================\n",
      "Total params: 1,690\n",
      "Trainable params: 1,690\n",
      "Non-trainable params: 0\n",
      "__________________________________________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "test_model()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Deep Factorization Machines (DeepFM)\n",
    "\n",
    "DeepFM consists of an FM component and a deep component which are integrated in a parallel structure. The FM component is the same as the 2-way factorization machines which is used to model the low-order feature interactions. The deep component is a multi-layered perceptron that is used to capture high-order feature interactions and nonlinearities. These two components share the same inputs/embeddings and their outputs are summed up as the final prediction. It is worth pointing out that the spirit of DeepFM resembles that of the Wide & Deep architecture which can capture both memorization and generalization. The advantages of DeepFM over the Wide & Deep model is that it reduces the effort of hand-crafted feature engineering by identifying feature combinations automatically.\n",
    "\n",
    "![https://github.com/RecoHut-Stanzas/S021355/raw/main/images/img12.png](https://github.com/RecoHut-Stanzas/S021355/raw/main/images/img12.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#export\n",
    "class DeepFM(Model):\n",
    "\tdef __init__(self, feature_columns, hidden_units=(200, 200, 200), dnn_dropout=0.,\n",
    "\t\t\t\t activation='relu', fm_w_reg=1e-6, embed_reg=1e-6):\n",
    "\t\t\"\"\"\n",
    "\t\tDeepFM\n",
    "\t\t:param feature_columns: A list. sparse column feature information.\n",
    "\t\t:param hidden_units: A list. A list of dnn hidden units.\n",
    "\t\t:param dnn_dropout: A scalar. Dropout of dnn.\n",
    "\t\t:param activation: A string. Activation function of dnn.\n",
    "\t\t:param fm_w_reg: A scalar. The regularizer of w in fm.\n",
    "\t\t:param embed_reg: A scalar. The regularizer of embedding.\n",
    "\t\t\"\"\"\n",
    "\t\tsuper(DeepFM, self).__init__()\n",
    "\t\tself.sparse_feature_columns = feature_columns\n",
    "\t\tself.embed_layers = {\n",
    "\t\t\t'embed_' + str(i): Embedding(input_dim=feat['feat_num'],\n",
    "\t\t\t\t\t\t\t\t\t\t input_length=1,\n",
    "\t\t\t\t\t\t\t\t\t\t output_dim=feat['embed_dim'],\n",
    "\t\t\t\t\t\t\t\t\t\t embeddings_initializer='random_normal',\n",
    "\t\t\t\t\t\t\t\t\t\t embeddings_regularizer=l2(embed_reg))\n",
    "\t\t\tfor i, feat in enumerate(self.sparse_feature_columns)\n",
    "\t\t}\n",
    "\t\tself.index_mapping = []\n",
    "\t\tself.feature_length = 0\n",
    "\t\tfor feat in self.sparse_feature_columns:\n",
    "\t\t\tself.index_mapping.append(self.feature_length)\n",
    "\t\t\tself.feature_length += feat['feat_num']\n",
    "\t\tself.embed_dim = self.sparse_feature_columns[0]['embed_dim']  # all sparse features have the same embed_dim\n",
    "\t\tself.fm = FM_Layer_v2(self.feature_length, fm_w_reg)\n",
    "\t\tself.dnn = DNN(hidden_units, activation, dnn_dropout)\n",
    "\t\tself.dense = Dense(1, activation=None)\n",
    "\n",
    "\tdef call(self, inputs, **kwargs):\n",
    "\t\tsparse_inputs = inputs\n",
    "\t\t# embedding\n",
    "\t\tsparse_embed = tf.concat([self.embed_layers['embed_{}'.format(i)](sparse_inputs[:, i])\n",
    "                                  for i in range(sparse_inputs.shape[1])], axis=-1)  # (batch_size, embed_dim * fields)\n",
    "\t\t# wide\n",
    "\t\tsparse_inputs = sparse_inputs + tf.convert_to_tensor(self.index_mapping)\n",
    "\t\twide_inputs = {'sparse_inputs': sparse_inputs,\n",
    "\t\t\t\t\t   'embed_inputs': tf.reshape(sparse_embed, shape=(-1, sparse_inputs.shape[1], self.embed_dim))}\n",
    "\t\twide_outputs = self.fm(wide_inputs)  # (batch_size, 1)\n",
    "\t\t# deep\n",
    "\t\tdeep_outputs = self.dnn(sparse_embed)\n",
    "\t\tdeep_outputs = self.dense(deep_outputs)  # (batch_size, 1)\n",
    "\t\t# outputs\n",
    "\t\toutputs = tf.nn.sigmoid(tf.add(wide_outputs, deep_outputs))\n",
    "\t\treturn outputs\n",
    "\n",
    "\tdef summary(self):\n",
    "\t\tsparse_inputs = Input(shape=(len(self.sparse_feature_columns),), dtype=tf.int32)\n",
    "\t\tModel(inputs=sparse_inputs, outputs=self.call(sparse_inputs)).summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def test_model():\n",
    "    user_features = {'feat': 'user_id', 'feat_num': 100, 'embed_dim': 8}\n",
    "    seq_features = {'feat': 'item_id', 'feat_num': 100, 'embed_dim': 8}\n",
    "    features = [user_features, seq_features]\n",
    "    model = DeepFM(features, hidden_units=[8, 4, 2], dnn_dropout=0.5)\n",
    "    model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"model_5\"\n",
      "__________________________________________________________________________________________________\n",
      " Layer (type)                   Output Shape         Param #     Connected to                     \n",
      "==================================================================================================\n",
      " input_9 (InputLayer)           [(None, 2)]          0           []                               \n",
      "                                                                                                  \n",
      " tf.__operators__.getitem_4 (Sl  (None,)             0           ['input_9[0][0]']                \n",
      " icingOpLambda)                                                                                   \n",
      "                                                                                                  \n",
      " tf.__operators__.getitem_5 (Sl  (None,)             0           ['input_9[0][0]']                \n",
      " icingOpLambda)                                                                                   \n",
      "                                                                                                  \n",
      " embedding_13 (Embedding)       (None, 8)            800         ['tf.__operators__.getitem_4[0][0\n",
      "                                                                 ]']                              \n",
      "                                                                                                  \n",
      " embedding_14 (Embedding)       (None, 8)            800         ['tf.__operators__.getitem_5[0][0\n",
      "                                                                 ]']                              \n",
      "                                                                                                  \n",
      " tf.concat (TFOpLambda)         (None, 16)           0           ['embedding_13[0][0]',           \n",
      "                                                                  'embedding_14[0][0]']           \n",
      "                                                                                                  \n",
      " tf.reshape (TFOpLambda)        (None, 2, 8)         0           ['tf.concat[0][0]']              \n",
      "                                                                                                  \n",
      " tf.__operators__.add_3 (TFOpLa  (None, 2)           0           ['input_9[0][0]']                \n",
      " mbda)                                                                                            \n",
      "                                                                                                  \n",
      " dnn_1 (DNN)                    (None, 2)            182         ['tf.concat[0][0]']              \n",
      "                                                                                                  \n",
      " fm__layer_v2 (FM_Layer_v2)     (None, 1)            200         ['tf.reshape[0][0]',             \n",
      "                                                                  'tf.__operators__.add_3[0][0]'] \n",
      "                                                                                                  \n",
      " dense_10 (Dense)               (None, 1)            3           ['dnn_1[0][0]']                  \n",
      "                                                                                                  \n",
      " tf.math.add (TFOpLambda)       (None, 1)            0           ['fm__layer_v2[0][0]',           \n",
      "                                                                  'dense_10[0][0]']               \n",
      "                                                                                                  \n",
      " tf.math.sigmoid_4 (TFOpLambda)  (None, 1)           0           ['tf.math.add[0][0]']            \n",
      "                                                                                                  \n",
      "==================================================================================================\n",
      "Total params: 1,985\n",
      "Trainable params: 1,985\n",
      "Non-trainable params: 0\n",
      "__________________________________________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "test_model()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Extreme Deep Factorization Machines (xDeepFM)\n",
    "\n",
    "xDeepFM combines the CIN and a classical DNN into one unified model. xDeepFM is able to learn certain bounded-degree feature interactions explicitly; on the other hand, it can learn arbitrary low- and high-order feature interactions implicitly.\n",
    "\n",
    "![The architecture of xDeepFM.](https://github.com/RecoHut-Stanzas/S021355/raw/main/images/img13.png)\n",
    "\n",
    "The architecture of xDeepFM.\n",
    "\n",
    "Compressed Interaction Network (CIN) aims to generate feature interactions in an explicit fashion and at the vector-wise level. CIN share some functionalities with convolutional neural networks (CNNs) and recurrent neural networks (RNNs).\n",
    "\n",
    "![Components and architecture of the Compressed Interaction Network (CIN).](https://github.com/RecoHut-Stanzas/S021355/raw/main/images/img14.png)\n",
    "\n",
    "Components and architecture of the Compressed Interaction Network (CIN)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#export\n",
    "class DNN_v2(Layer):\n",
    "    def __init__(self, hidden_units, dnn_dropout=0., dnn_activation='relu'):\n",
    "        \"\"\"DNN\n",
    "        :param hidden_units: A list. list of hidden layer units's numbers.\n",
    "        :param dnn_dropout: A scalar. dropout number.\n",
    "        :param dnn_activation: A string. activation function.\n",
    "        \"\"\"\n",
    "        super(DNN_v2, self).__init__()\n",
    "        self.dnn_network = [Dense(units=unit, activation=dnn_activation) for unit in hidden_units]\n",
    "        self.dropout = Dropout(dnn_dropout)\n",
    "\n",
    "    def call(self, inputs, **kwargs):\n",
    "        x = inputs\n",
    "        for dnn in self.dnn_network:\n",
    "            x = dnn(x)\n",
    "        x = self.dropout(x)\n",
    "        return x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#export\n",
    "class Linear(Layer):\n",
    "    def __init__(self, feature_length, w_reg=1e-6):\n",
    "        \"\"\"\n",
    "        Linear Part\n",
    "        :param feature_length: A scalar. The length of features.\n",
    "        :param w_reg: A scalar. The regularization coefficient of parameter w.\n",
    "        \"\"\"\n",
    "        super(Linear, self).__init__()\n",
    "        self.feature_length = feature_length\n",
    "        self.w_reg = w_reg\n",
    "\n",
    "    def build(self, input_shape):\n",
    "        self.w = self.add_weight(name=\"w\",\n",
    "                                 shape=(self.feature_length, 1),\n",
    "                                 regularizer=l2(self.w_reg),\n",
    "                                 trainable=True)\n",
    "\n",
    "    def call(self, inputs, **kwargs):\n",
    "        result = tf.reduce_sum(tf.nn.embedding_lookup(self.w, inputs), axis=1)  # (batch_size, 1)\n",
    "        return result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#export\n",
    "class CIN(Layer):\n",
    "    def __init__(self, cin_size, l2_reg=1e-4):\n",
    "        \"\"\"CIN\n",
    "        :param cin_size: A list. [H_1, H_2 ,..., H_k], a list of the number of layers\n",
    "        :param l2_reg: A scalar. L2 regularization.\n",
    "        \"\"\"\n",
    "        super(CIN, self).__init__()\n",
    "        self.cin_size = cin_size\n",
    "        self.l2_reg = l2_reg\n",
    "\n",
    "    def build(self, input_shape):\n",
    "        # get the number of embedding fields\n",
    "        self.embedding_nums = input_shape[1]\n",
    "        # a list of the number of CIN\n",
    "        self.field_nums = [self.embedding_nums] + self.cin_size\n",
    "        # filters\n",
    "        self.cin_W = {\n",
    "            'CIN_W_' + str(i): self.add_weight(\n",
    "                name='CIN_W_' + str(i),\n",
    "                shape=(1, self.field_nums[0] * self.field_nums[i], self.field_nums[i + 1]),\n",
    "                initializer='random_normal',\n",
    "                regularizer=l2(self.l2_reg),\n",
    "                trainable=True)\n",
    "            for i in range(len(self.field_nums) - 1)\n",
    "        }\n",
    "\n",
    "    def call(self, inputs, **kwargs):\n",
    "        dim = inputs.shape[-1]\n",
    "        hidden_layers_results = [inputs]\n",
    "        # split dimension 2 for convenient calculation\n",
    "        split_X_0 = tf.split(hidden_layers_results[0], dim, 2)  # dim * (None, field_nums[0], 1)\n",
    "        for idx, size in enumerate(self.cin_size):\n",
    "            split_X_K = tf.split(hidden_layers_results[-1], dim, 2)  # dim * (None, filed_nums[i], 1)\n",
    "\n",
    "            result_1 = tf.matmul(split_X_0, split_X_K, transpose_b=True)  # (dim, None, field_nums[0], field_nums[i])\n",
    "\n",
    "            result_2 = tf.reshape(result_1, shape=[dim, -1, self.embedding_nums * self.field_nums[idx]])\n",
    "\n",
    "            result_3 = tf.transpose(result_2, perm=[1, 0, 2])  # (None, dim, field_nums[0] * field_nums[i])\n",
    "\n",
    "            result_4 = tf.nn.conv1d(input=result_3, filters=self.cin_W['CIN_W_' + str(idx)], stride=1,\n",
    "                                    padding='VALID')\n",
    "\n",
    "            result_5 = tf.transpose(result_4, perm=[0, 2, 1])  # (None, field_num[i+1], dim)\n",
    "\n",
    "            hidden_layers_results.append(result_5)\n",
    "\n",
    "        final_results = hidden_layers_results[1:]\n",
    "        result = tf.concat(final_results, axis=1)  # (None, H_1 + ... + H_K, dim)\n",
    "        result = tf.reduce_sum(result,  axis=-1)  # (None, dim)\n",
    "\n",
    "        return result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#export\n",
    "class xDeepFM(Model):\n",
    "    def __init__(self, feature_columns, hidden_units, cin_size, dnn_dropout=0, dnn_activation='relu',\n",
    "                 embed_reg=1e-6, cin_reg=1e-6, w_reg=1e-6):\n",
    "        \"\"\"\n",
    "        xDeepFM\n",
    "        :param feature_columns: A list. sparse column feature information.\n",
    "        :param hidden_units: A list. a list of dnn hidden units.\n",
    "        :param cin_size: A list. a list of the number of CIN layers.\n",
    "        :param dnn_dropout: A scalar. dropout of dnn.\n",
    "        :param dnn_activation: A string. activation function of dnn.\n",
    "        :param embed_reg: A scalar. The regularizer of embedding.\n",
    "        :param cin_reg: A scalar. The regularizer of cin.\n",
    "        :param w_reg: A scalar. The regularizer of Linear.\n",
    "        \"\"\"\n",
    "        super(xDeepFM, self).__init__()\n",
    "        self.sparse_feature_columns = feature_columns\n",
    "        self.embed_dim = self.sparse_feature_columns[0]['embed_dim']\n",
    "        self.embed_layers = {\n",
    "            'embed_' + str(i): Embedding(input_dim=feat['feat_num'],\n",
    "                                         input_length=1,\n",
    "                                         output_dim=feat['embed_dim'],\n",
    "                                         embeddings_initializer='random_normal',\n",
    "                                         embeddings_regularizer=l2(embed_reg))\n",
    "            for i, feat in enumerate(self.sparse_feature_columns)\n",
    "        }\n",
    "        self.index_mapping = []\n",
    "        self.feature_length = 0\n",
    "        for feat in self.sparse_feature_columns:\n",
    "            self.index_mapping.append(self.feature_length)\n",
    "            self.feature_length += feat['feat_num']\n",
    "        self.linear = Linear(self.feature_length, w_reg)\n",
    "        self.cin = CIN(cin_size=cin_size, l2_reg=cin_reg)\n",
    "        self.dnn = DNN_v2(hidden_units=hidden_units, dnn_dropout=dnn_dropout, dnn_activation=dnn_activation)\n",
    "        self.cin_dense = Dense(1)\n",
    "        self.dnn_dense = Dense(1)\n",
    "        self.bias = self.add_weight(name='bias', shape=(1, ), initializer=tf.zeros_initializer())\n",
    "\n",
    "    def call(self, inputs, **kwargs):\n",
    "        # Linear\n",
    "        linear_inputs = inputs + tf.convert_to_tensor(self.index_mapping)\n",
    "        linear_out = self.linear(linear_inputs)  # (batch_size, 1)\n",
    "        # cin\n",
    "        embed = [self.embed_layers['embed_{}'.format(i)](inputs[:, i]) for i in range(inputs.shape[1])]\n",
    "        embed_matrix = tf.transpose(tf.convert_to_tensor(embed), [1, 0, 2])\n",
    "        cin_out = self.cin(embed_matrix)  # (batch_size, dim)\n",
    "        cin_out = self.cin_dense(cin_out)  # (batch_size, 1)\n",
    "        # dnn\n",
    "        embed_vector = tf.reshape(embed_matrix, shape=(-1, embed_matrix.shape[1] * embed_matrix.shape[2]))\n",
    "        dnn_out = self.dnn(embed_vector)\n",
    "        dnn_out = self.dnn_dense(dnn_out)  # (batch_size, 1))\n",
    "        # output\n",
    "        output = tf.nn.sigmoid(linear_out + cin_out + dnn_out + self.bias)\n",
    "        return output\n",
    "\n",
    "    def summary(self):\n",
    "        sparse_inputs = Input(shape=(len(self.sparse_feature_columns),), dtype=tf.int32)\n",
    "        Model(inputs=sparse_inputs, outputs=self.call(sparse_inputs)).summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def test_model():\n",
    "    user_features = {'feat': 'user_id', 'feat_num': 100, 'embed_dim': 8}\n",
    "    seq_features = {'feat': 'item_id', 'feat_num': 100, 'embed_dim': 8}\n",
    "    features = [user_features, seq_features]\n",
    "    model = xDeepFM(features, hidden_units=[8, 4, 2], cin_size=[4, 4])\n",
    "    model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:The following Variables were used in a Lambda layer's call (tf.__operators__.add_7), but are not present in its tracked objects:   <tf.Variable 'bias:0' shape=(1,) dtype=float32>. This is a strong indication that the Lambda layer should be rewritten as a subclassed Layer.\n",
      "Model: \"model_6\"\n",
      "__________________________________________________________________________________________________\n",
      " Layer (type)                   Output Shape         Param #     Connected to                     \n",
      "==================================================================================================\n",
      " input_10 (InputLayer)          [(None, 2)]          0           []                               \n",
      "                                                                                                  \n",
      " tf.__operators__.getitem_6 (Sl  (None,)             0           ['input_10[0][0]']               \n",
      " icingOpLambda)                                                                                   \n",
      "                                                                                                  \n",
      " tf.__operators__.getitem_7 (Sl  (None,)             0           ['input_10[0][0]']               \n",
      " icingOpLambda)                                                                                   \n",
      "                                                                                                  \n",
      " embedding_19 (Embedding)       (None, 8)            800         ['tf.__operators__.getitem_6[0][0\n",
      "                                                                 ]']                              \n",
      "                                                                                                  \n",
      " embedding_20 (Embedding)       (None, 8)            800         ['tf.__operators__.getitem_7[0][0\n",
      "                                                                 ]']                              \n",
      "                                                                                                  \n",
      " tf.convert_to_tensor_2 (TFOpLa  (2, None, 8)        0           ['embedding_19[0][0]',           \n",
      " mbda)                                                            'embedding_20[0][0]']           \n",
      "                                                                                                  \n",
      " tf.compat.v1.transpose_2 (TFOp  (None, 2, 8)        0           ['tf.convert_to_tensor_2[0][0]'] \n",
      " Lambda)                                                                                          \n",
      "                                                                                                  \n",
      " tf.__operators__.add_4 (TFOpLa  (None, 2)           0           ['input_10[0][0]']               \n",
      " mbda)                                                                                            \n",
      "                                                                                                  \n",
      " cin_2 (CIN)                    (None, 8)            48          ['tf.compat.v1.transpose_2[0][0]'\n",
      "                                                                 ]                                \n",
      "                                                                                                  \n",
      " tf.reshape_1 (TFOpLambda)      (None, 16)           0           ['tf.compat.v1.transpose_2[0][0]'\n",
      "                                                                 ]                                \n",
      "                                                                                                  \n",
      " linear_2 (Linear)              (None, 1)            200         ['tf.__operators__.add_4[0][0]'] \n",
      "                                                                                                  \n",
      " dense_14 (Dense)               (None, 1)            9           ['cin_2[0][0]']                  \n",
      "                                                                                                  \n",
      " dnn_v2 (DNN_v2)                (None, 2)            182         ['tf.reshape_1[0][0]']           \n",
      "                                                                                                  \n",
      " tf.__operators__.add_5 (TFOpLa  (None, 1)           0           ['linear_2[0][0]',               \n",
      " mbda)                                                            'dense_14[0][0]']               \n",
      "                                                                                                  \n",
      " dense_15 (Dense)               (None, 1)            3           ['dnn_v2[0][0]']                 \n",
      "                                                                                                  \n",
      " tf.__operators__.add_6 (TFOpLa  (None, 1)           0           ['tf.__operators__.add_5[0][0]', \n",
      " mbda)                                                            'dense_15[0][0]']               \n",
      "                                                                                                  \n",
      " tf.__operators__.add_7 (TFOpLa  (None, 1)           0           ['tf.__operators__.add_6[0][0]'] \n",
      " mbda)                                                                                            \n",
      "                                                                                                  \n",
      " tf.math.sigmoid_5 (TFOpLambda)  (None, 1)           0           ['tf.__operators__.add_7[0][0]'] \n",
      "                                                                                                  \n",
      "==================================================================================================\n",
      "Total params: 2,042\n",
      "Trainable params: 2,042\n",
      "Non-trainable params: 0\n",
      "__________________________________________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "test_model()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Author: Sparsh A.\n",
      "\n",
      "Last updated: 2021-12-20 10:10:04\n",
      "\n",
      "Compiler    : GCC 7.5.0\n",
      "OS          : Linux\n",
      "Release     : 5.4.104+\n",
      "Machine     : x86_64\n",
      "Processor   : x86_64\n",
      "CPU cores   : 2\n",
      "Architecture: 64bit\n",
      "\n",
      "tensorflow: 2.7.0\n",
      "IPython   : 5.5.0\n",
      "numpy     : 1.19.5\n",
      "\n"
     ]
    }
   ],
   "source": [
    "#hide\n",
    "!pip install -q watermark\n",
    "%reload_ext watermark\n",
    "%watermark -a \"Sparsh A.\" -m -iv -u -t -d"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "name": "python3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}
