{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a href=\"https://colab.research.google.com/github/RecoHut-Projects/recohut/blob/master/tutorials/modeling/group_rec_ddpg_ml1m_pytorch.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Deep Reinforcement Learning based Group Recommender System\n",
    "\n",
    "`ActorCriticNetwork`, `DDPG`, `GroupRecommendation`, `MovieLens1M`, `PyTorch`, `ReplayMemory`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- **Problem:** Group recommendation problem is challenging because recommending some item that satisfies all the members of the group is rare. So it often involves some compromises that the model has to make in order to maximize the overall satisfaction of the group.\n",
    "- **Hypothesis:** RL agent can learn the required behavior that could the maximize the group's overall satisfaction.\n",
    "- **Benefits:** Meaningful to those people who want to get recommendations for their groups, such as entertainments with families and travels with friends. This model consider the influences of each group member by one self-attention mechanism.\n",
    "- **Solution:** A recommender agent is trained with actor-critic network and is optimized with DDPG algorithm, where the experience replay and target networks are used. Matrix factorization based simulator is built to simulate the MDP environment. It is an extended version of LIRD model for group recommendations. The group recommendation is viewed as a classification task. When one item is recommended to a group, if the group chooses the item, this case is marked as a positive sample. Otherwise, it will be a negative sample.\n",
    "- **Dataset:** MovieLens-1m\n",
    "- **Preprocessing:** Randomly generate groups with 2-5 users. Then, for each group, if every member gives 4-5 stars to one movie, we assume that this movie is adopted by this group with rating 1. If all members give ratings to one movie, but not all in 4-5 stars, we consider the group gives rating 0 to this movie. For other cases, the group movie ratings are missed. Finally, to ensure each group has enough interactions with items, we require each group has at least 20 ratings. Also, for each rating, 100 rating-missed items are randomly sampled. Both user and group rating data are split into training, validation, and testing datasets with the ratio of 70%, 10%, and 20% respectively by the temporal order.\n",
    "- **Metrics:** Recall, nDCG\n",
    "- **Cluster:** PyTorch 1.10 cpu"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src='https://github.com/sparsh-ai/stanza/raw/S758139/images/group_recommender_actorcritic_1.svg'>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> Note: For theoretical understanding, refer to this [this](https://recohut.notion.site/Deep-Reinforcement-Learning-based-Group-Recommender-System-6399cf01102b485897578d1bccbe3467) report."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from typing import Tuple, List, Dict\n",
    "import os\n",
    "import random\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "from scipy.sparse import coo_matrix, csr_matrix\n",
    "from collections import deque, defaultdict\n",
    "import shutil\n",
    "import zipfile\n",
    "import scipy.sparse as sp\n",
    "from collections import Counter\n",
    "\n",
    "import torch\n",
    "import torch.nn.functional as functional\n",
    "from torch import optim, nn\n",
    "\n",
    "import gym\n",
    "from sklearn.decomposition import NMF"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Config(object):\n",
    "    \"\"\"\n",
    "    Configurations\n",
    "    \"\"\"\n",
    "\n",
    "    def __init__(self):\n",
    "        # Data\n",
    "        self.data_folder_path = os.path.join('data', 'MovieLens-Rand')\n",
    "        self.item_path = os.path.join(self.data_folder_path, 'movies.dat')\n",
    "        self.user_path = os.path.join(self.data_folder_path, 'users.dat')\n",
    "        self.group_path = os.path.join(self.data_folder_path, 'groupMember.dat')\n",
    "        self.saves_folder_path = os.path.join('saves')\n",
    "\n",
    "        # Recommendation system\n",
    "        self.history_length = 5\n",
    "        self.top_K_list = [5, 10, 20]\n",
    "        self.rewards = [0, 1]\n",
    "\n",
    "        # Reinforcement learning\n",
    "        self.embedding_size = 32\n",
    "        self.state_size = self.history_length + 1\n",
    "        self.action_size = 1\n",
    "        self.embedded_state_size = self.state_size * self.embedding_size\n",
    "        self.embedded_action_size = self.action_size * self.embedding_size\n",
    "\n",
    "        # Numbers\n",
    "        self.item_num = None\n",
    "        self.user_num = None\n",
    "        self.group_num = None\n",
    "        self.total_group_num = None\n",
    "\n",
    "        # Environment\n",
    "        self.env_n_components = self.embedding_size\n",
    "        self.env_tol = 1e-4\n",
    "        self.env_max_iter = 1000\n",
    "        self.env_alpha = 0.001\n",
    "\n",
    "        # Actor-Critic network\n",
    "        self.actor_hidden_sizes = (128, 64)\n",
    "        self.critic_hidden_sizes = (32, 16)\n",
    "\n",
    "        # DDPG algorithm\n",
    "        self.tau = 1e-3\n",
    "        self.gamma = 0.9\n",
    "\n",
    "        # Optimizer\n",
    "        self.batch_size = 64\n",
    "        self.buffer_size = 100000\n",
    "        self.num_episodes = 10 # recommended = 1000\n",
    "        self.num_steps = 5 # recommended = 100\n",
    "        self.embedding_weight_decay = 1e-6\n",
    "        self.actor_weight_decay = 1e-6\n",
    "        self.critic_weight_decay = 1e-6\n",
    "        self.embedding_learning_rate = 1e-4\n",
    "        self.actor_learning_rate = 1e-4\n",
    "        self.critic_learning_rate = 1e-4\n",
    "        self.eval_per_iter = 10\n",
    "\n",
    "        # OU noise\n",
    "        self.ou_mu = 0.0\n",
    "        self.ou_theta = 0.15\n",
    "        self.ou_sigma = 0.2\n",
    "        self.ou_epsilon = 1.0\n",
    "\n",
    "        # GPU\n",
    "        if torch.cuda.is_available():\n",
    "            self.device = torch.device(\"cuda:0\")\n",
    "        else:\n",
    "            self.device = torch.device(\"cpu\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Utils"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class OUNoise(object):\n",
    "    \"\"\"\n",
    "    Ornstein-Uhlenbeck Noise\n",
    "    \"\"\"\n",
    "\n",
    "    def __init__(self, config: Config):\n",
    "        \"\"\"\n",
    "        Initialize OUNoise\n",
    "        :param config: configurations\n",
    "        \"\"\"\n",
    "        self.embedded_action_size = config.embedded_action_size\n",
    "        self.ou_mu = config.ou_mu\n",
    "        self.ou_theta = config.ou_theta\n",
    "        self.ou_sigma = config.ou_sigma\n",
    "        self.ou_epsilon = config.ou_epsilon\n",
    "        self.ou_state = None\n",
    "        self.reset()\n",
    "\n",
    "    def reset(self):\n",
    "        \"\"\"\n",
    "        Reset the OU process state\n",
    "        \"\"\"\n",
    "        self.ou_state = torch.ones(self.embedded_action_size) * self.ou_mu\n",
    "\n",
    "    def evolve_state(self):\n",
    "        \"\"\"\n",
    "        Evolve the OU process state\n",
    "        \"\"\"\n",
    "        self.ou_state += self.ou_theta * (self.ou_mu - self.ou_state) \\\n",
    "            + self.ou_sigma * torch.randn(self.embedded_action_size)\n",
    "\n",
    "    def get_ou_noise(self):\n",
    "        \"\"\"\n",
    "        Get the OU noise for one action\n",
    "        :return OU noise\n",
    "        \"\"\"\n",
    "        self.evolve_state()\n",
    "        return self.ou_state.copy()\n",
    "\n",
    "\n",
    "class ReplayMemory(object):\n",
    "    \"\"\"\n",
    "    Replay Memory\n",
    "    \"\"\"\n",
    "\n",
    "    def __init__(self, buffer_size: int):\n",
    "        \"\"\"\n",
    "        Initialize ReplayMemory\n",
    "        :param buffer_size: size of the buffer\n",
    "        \"\"\"\n",
    "        self.buffer_size = buffer_size\n",
    "        self.buffer = deque(maxlen=buffer_size)\n",
    "\n",
    "    def __len__(self):\n",
    "        return len(self.buffer)\n",
    "\n",
    "    def push(self, experience: tuple):\n",
    "        \"\"\"\n",
    "        Push one experience into the buffer\n",
    "        :param experience: (state, action, reward, new_state)\n",
    "        \"\"\"\n",
    "        self.buffer.append(experience)\n",
    "\n",
    "    def sample(self, batch_size: int):\n",
    "        \"\"\"\n",
    "        Sample one batch from the buffer\n",
    "        :param batch_size: number of experiences in the batch\n",
    "        :return: batch\n",
    "        \"\"\"\n",
    "        batch = random.sample(self.buffer, batch_size)\n",
    "        return batch"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src='https://github.com/sparsh-ai/stanza/raw/S758139/images/group_recommender_actorcritic_2.svg'>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!wget -q --show-progress https://files.grouplens.org/datasets/movielens/ml-1m.zip"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class GroupGenerator(object):\n",
    "    \"\"\"\n",
    "    Group Data Generator\n",
    "    \"\"\"\n",
    "    def __init__(self, data_path, output_path, rating_threshold, num_groups,\n",
    "                 group_sizes, min_num_ratings, train_ratio, val_ratio,\n",
    "                 negative_sample_size, verbose=False):\n",
    "        self.rating_threshold = rating_threshold\n",
    "        self.negative_sample_size = negative_sample_size\n",
    "        users_path = os.path.join(data_path, 'users.dat')\n",
    "        items_path = os.path.join(data_path, 'movies.dat')\n",
    "        ratings_path = os.path.join(data_path, 'ratings.dat')\n",
    "\n",
    "        users = self.load_users_file(users_path)\n",
    "        items = self.load_items_file(items_path)\n",
    "        rating_mat, timestamp_mat = \\\n",
    "            self.load_ratings_file(ratings_path, max(users), max(items))\n",
    "\n",
    "        groups, group_ratings, groups_rated_items_dict, groups_rated_items_set = \\\n",
    "            self.generate_group_ratings(users, rating_mat, timestamp_mat,\n",
    "                                        num_groups=num_groups,\n",
    "                                        group_sizes=group_sizes,\n",
    "                                        min_num_ratings=min_num_ratings)\n",
    "        members, group_ratings_train, group_ratings_val, group_ratings_test, \\\n",
    "            group_negative_items_val, group_negative_items_test, \\\n",
    "            user_ratings_train, user_ratings_val, user_ratings_test, \\\n",
    "            user_negative_items_val, user_negative_items_test = \\\n",
    "            self.split_ratings(group_ratings, rating_mat, timestamp_mat,\n",
    "                               groups, groups_rated_items_dict, groups_rated_items_set,\n",
    "                               train_ratio=train_ratio, val_ratio=val_ratio)\n",
    "\n",
    "        groups_path = os.path.join(output_path, 'groupMember.dat')\n",
    "        group_ratings_train_path = os.path.join(output_path, 'groupRatingTrain.dat')\n",
    "        group_ratings_val_path = os.path.join(output_path, 'groupRatingVal.dat')\n",
    "        group_ratings_test_path = os.path.join(output_path, 'groupRatingTest.dat')\n",
    "        group_negative_items_val_path = os.path.join(output_path, 'groupRatingValNegative.dat')\n",
    "        group_negative_items_test_path = os.path.join(output_path, 'groupRatingTestNegative.dat')\n",
    "        user_ratings_train_path = os.path.join(output_path, 'userRatingTrain.dat')\n",
    "        user_ratings_val_path = os.path.join(output_path, 'userRatingVal.dat')\n",
    "        user_ratings_test_path = os.path.join(output_path, 'userRatingTest.dat')\n",
    "        user_negative_items_val_path = os.path.join(output_path, 'userRatingValNegative.dat')\n",
    "        user_negative_items_test_path = os.path.join(output_path, 'userRatingTestNegative.dat')\n",
    "\n",
    "        self.save_groups(groups_path, groups)\n",
    "        self.save_ratings(group_ratings_train, group_ratings_train_path)\n",
    "        self.save_ratings(group_ratings_val, group_ratings_val_path)\n",
    "        self.save_ratings(group_ratings_test, group_ratings_test_path)\n",
    "        self.save_negative_samples(group_negative_items_val, group_negative_items_val_path)\n",
    "        self.save_negative_samples(group_negative_items_test, group_negative_items_test_path)\n",
    "        self.save_ratings(user_ratings_train, user_ratings_train_path)\n",
    "        self.save_ratings(user_ratings_val, user_ratings_val_path)\n",
    "        self.save_ratings(user_ratings_test, user_ratings_test_path)\n",
    "        self.save_negative_samples(user_negative_items_val, user_negative_items_val_path)\n",
    "        self.save_negative_samples(user_negative_items_test, user_negative_items_test_path)\n",
    "        shutil.copyfile(src=os.path.join(data_path, 'movies.dat'), dst=os.path.join(output_path, 'movies.dat'))\n",
    "        shutil.copyfile(src=os.path.join(data_path, 'users.dat'), dst=os.path.join(output_path, 'users.dat'))\n",
    "\n",
    "        if verbose:\n",
    "            num_group_ratings = len(group_ratings)\n",
    "            num_user_ratings = len(user_ratings_train) + len(user_ratings_val) + len(user_ratings_test)\n",
    "            num_rated_items = len(groups_rated_items_set)\n",
    "\n",
    "            print('Save data: ' + output_path)\n",
    "            print('# Users: ' + str(len(members)))\n",
    "            print('# Items: ' + str(num_rated_items))\n",
    "            print('# Groups: ' + str(len(groups)))\n",
    "            print('# U-I ratings: ' + str(num_user_ratings))\n",
    "            print('# G-I ratings: ' + str(num_group_ratings))\n",
    "            print('Avg. # ratings / user: {:.2f}'.format(num_user_ratings / len(members)))\n",
    "            print('Avg. # ratings / group: {:.2f}'.format(num_group_ratings / len(groups)))\n",
    "            print('Avg. group size: {:.2f}'.format(np.mean(list(map(len, groups)))))\n",
    "\n",
    "    def load_users_file(self, users_path):\n",
    "        users = []\n",
    "\n",
    "        with open(users_path, 'r') as file:\n",
    "            for line in file.readlines():\n",
    "                users.append(int(line.split('::')[0]))\n",
    "\n",
    "        return users\n",
    "\n",
    "    def load_items_file(self, items_path):\n",
    "        items = []\n",
    "\n",
    "        with open(items_path, 'r', encoding='iso-8859-1') as file:\n",
    "            for line in file.readlines():\n",
    "                items.append(int(line.split('::')[0]))\n",
    "\n",
    "        return items\n",
    "\n",
    "    def load_ratings_file(self, ratings_path, max_num_users, max_num_items):\n",
    "        rating_mat = sp.dok_matrix((max_num_users + 1, max_num_items + 1),\n",
    "                                   dtype=np.int)\n",
    "        timestamp_mat = rating_mat.copy()\n",
    "\n",
    "        with open(ratings_path, 'r') as file:\n",
    "            for line in file.readlines():\n",
    "                arr = line.replace('\\n', '').split('::')\n",
    "                user, item, rating, timestamp = \\\n",
    "                    int(arr[0]), int(arr[1]), int(arr[2]), int(arr[3])\n",
    "                rating_mat[user, item] = rating\n",
    "                timestamp_mat[user, item] = timestamp\n",
    "\n",
    "        return rating_mat, timestamp_mat\n",
    "\n",
    "    def generate_group_ratings(self, users, rating_mat, timestamp_mat,\n",
    "                               num_groups, group_sizes, min_num_ratings):\n",
    "        np.random.seed(0)\n",
    "        groups = set()\n",
    "        groups_ratings = []\n",
    "        groups_rated_items_dict = {}\n",
    "        groups_rated_items_set = set()\n",
    "\n",
    "        while len(groups) < num_groups:\n",
    "            group_id = len(groups) + 1\n",
    "\n",
    "            while True:\n",
    "                group = tuple(np.sort(\n",
    "                    np.random.choice(users, np.random.choice(group_sizes),\n",
    "                                     replace=False)))\n",
    "                if group not in groups:\n",
    "                    break\n",
    "\n",
    "            pos_group_rating_counter = Counter()\n",
    "            neg_group_rating_counter = Counter()\n",
    "            group_rating_list = []\n",
    "            group_rated_items = set()\n",
    "\n",
    "            for member in group:\n",
    "                _, items = rating_mat[member, :].nonzero()\n",
    "                pos_items = [item for item in items\n",
    "                             if rating_mat[member, item] >= self.rating_threshold]\n",
    "                neg_items = [item for item in items\n",
    "                             if rating_mat[member, item] < self.rating_threshold]\n",
    "                pos_group_rating_counter.update(pos_items)\n",
    "                neg_group_rating_counter.update(neg_items)\n",
    "\n",
    "            for item, num_ratings in pos_group_rating_counter.items():\n",
    "                if num_ratings == len(group):\n",
    "                    timestamp = max([timestamp_mat[member, item]\n",
    "                                     for member in group])\n",
    "                    group_rated_items.add(item)\n",
    "                    group_rating_list.append((group_id, item, 1, timestamp))\n",
    "\n",
    "            for item, num_ratings in neg_group_rating_counter.items():\n",
    "                if (num_ratings == len(group)) \\\n",
    "                        or (num_ratings + pos_group_rating_counter[item] == len(group)):\n",
    "                    timestamp = max([timestamp_mat[member, item]\n",
    "                                     for member in group])\n",
    "                    group_rated_items.add(item)\n",
    "                    group_rating_list.append((group_id, item, 0, timestamp))\n",
    "\n",
    "            if len(group_rating_list) >= min_num_ratings:\n",
    "                groups.add(group)\n",
    "                groups_rated_items_dict[group_id] = group_rated_items\n",
    "                groups_rated_items_set.update(group_rated_items)\n",
    "                for group_rating in group_rating_list:\n",
    "                    groups_ratings.append(group_rating)\n",
    "\n",
    "        return list(groups), groups_ratings, groups_rated_items_dict, groups_rated_items_set\n",
    "\n",
    "    def split_ratings(self, group_ratings, rating_mat, timestamp_mat,\n",
    "                      groups, groups_rated_items_dict, groups_rated_items_set, train_ratio, val_ratio):\n",
    "        num_group_ratings = len(group_ratings)\n",
    "        num_train = int(num_group_ratings * train_ratio)\n",
    "        num_test = int(num_group_ratings * (1 - train_ratio - val_ratio))\n",
    "\n",
    "        group_ratings = \\\n",
    "            sorted(group_ratings, key=lambda group_rating: group_rating[-1])\n",
    "        group_ratings_train = group_ratings[:num_train]\n",
    "        group_ratings_val = group_ratings[num_train:-num_test]\n",
    "        group_ratings_test = group_ratings[-num_test:]\n",
    "\n",
    "        timestamp_split_train = group_ratings_train[-1][-1]\n",
    "        timestamp_split_val = group_ratings_val[-1][-1]\n",
    "\n",
    "        user_ratings_train = []\n",
    "        user_ratings_val = []\n",
    "        user_ratings_test = []\n",
    "\n",
    "        members = set()\n",
    "        users_rated_items_dict = {}\n",
    "\n",
    "        for group in groups:\n",
    "            for member in group:\n",
    "                if member in members:\n",
    "                    continue\n",
    "                members.add(member)\n",
    "                user_rated_items = set()\n",
    "                _, items = rating_mat[member, :].nonzero()\n",
    "                for item in items:\n",
    "                    if item not in groups_rated_items_set:\n",
    "                        continue\n",
    "                    user_rated_items.add(item)\n",
    "                    if rating_mat[member, item] >= self.rating_threshold:\n",
    "                        rating_tuple = (member, item, 1,\n",
    "                                        timestamp_mat[member, item])\n",
    "                    else:\n",
    "                        rating_tuple = (member, item, 0,\n",
    "                                        timestamp_mat[member, item])\n",
    "                    if timestamp_mat[member, item] <= timestamp_split_train:\n",
    "                        user_ratings_train.append(rating_tuple)\n",
    "                    elif timestamp_split_train < timestamp_mat[member, item] <= timestamp_split_val:\n",
    "                        user_ratings_val.append(rating_tuple)\n",
    "                    else:\n",
    "                        user_ratings_test.append(rating_tuple)\n",
    "\n",
    "                users_rated_items_dict[member] = user_rated_items\n",
    "\n",
    "        np.random.seed(0)\n",
    "\n",
    "        user_negative_items_val = self.get_negative_samples(\n",
    "            user_ratings_val, groups_rated_items_set, users_rated_items_dict)\n",
    "        user_negative_items_test = self.get_negative_samples(\n",
    "            user_ratings_test, groups_rated_items_set, users_rated_items_dict)\n",
    "        group_negative_items_val = self.get_negative_samples(\n",
    "            group_ratings_val, groups_rated_items_set, groups_rated_items_dict)\n",
    "        group_negative_items_test = self.get_negative_samples(\n",
    "            group_ratings_test, groups_rated_items_set, groups_rated_items_dict)\n",
    "\n",
    "        return members, group_ratings_train, group_ratings_val, group_ratings_test, \\\n",
    "            group_negative_items_val, group_negative_items_test, \\\n",
    "            user_ratings_train, user_ratings_val, user_ratings_test, \\\n",
    "            user_negative_items_val, user_negative_items_test\n",
    "\n",
    "    def get_negative_samples(self, ratings, groups_rated_items_set, rated_items_dict):\n",
    "        negative_items_list = []\n",
    "        for sample in ratings:\n",
    "            sample_id, item, _, _ = sample\n",
    "            missed_items = groups_rated_items_set - rated_items_dict[sample_id]\n",
    "            negative_items = \\\n",
    "                np.random.choice(list(missed_items), self.negative_sample_size,\n",
    "                                 replace=(len(missed_items) < self.negative_sample_size))\n",
    "            negative_items_list.append((sample_id, item, negative_items))\n",
    "        return negative_items_list\n",
    "\n",
    "    def save_groups(self, groups_path, groups):\n",
    "        with open(groups_path, 'w') as file:\n",
    "            for i, group in enumerate(groups):\n",
    "                file.write(str(i + 1) + ' '\n",
    "                           + ','.join(map(str, list(group))) + '\\n')\n",
    "\n",
    "    def save_ratings(self, ratings, ratings_path):\n",
    "        with open(ratings_path, 'w') as file:\n",
    "            for rating in ratings:\n",
    "                file.write(' '.join(map(str, list(rating))) + '\\n')\n",
    "\n",
    "    def save_negative_samples(self, negative_items, negative_items_path):\n",
    "        with open(negative_items_path, 'w') as file:\n",
    "            for samples in negative_items:\n",
    "                user, item, negative_items = samples\n",
    "                file.write('({},{}) '.format(user, item)\n",
    "                           + ' '.join(map(str, list(negative_items))) + '\\n')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print('Takes approx. 5 mins...')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Save data: ./MovieLens-Rand\n",
      "# Users: 1626\n",
      "# Items: 1998\n",
      "# Groups: 1000\n",
      "# U-I ratings: 438129\n",
      "# G-I ratings: 53248\n",
      "Avg. # ratings / user: 269.45\n",
      "Avg. # ratings / group: 53.25\n",
      "Avg. group size: 2.19\n"
     ]
    }
   ],
   "source": [
    "data_folder_path = '.'\n",
    "data_path = os.path.join(data_folder_path, 'ml-1m')\n",
    "data_zip_path = os.path.join(data_folder_path, 'ml-1m.zip')\n",
    "output_path = os.path.join(data_folder_path, 'MovieLens-Rand')\n",
    "\n",
    "if not os.path.exists(data_path):\n",
    "    with zipfile.ZipFile(data_zip_path, 'r') as data_zip:\n",
    "        data_zip.extractall(data_folder_path)\n",
    "        print('Unzip file: ' + data_zip_path)\n",
    "\n",
    "if not os.path.exists(output_path):\n",
    "    os.mkdir(output_path)\n",
    "\n",
    "group_generator = GroupGenerator(data_path, output_path,\n",
    "                                    rating_threshold=4,\n",
    "                                    num_groups=1000,\n",
    "                                    group_sizes=[2, 3, 4, 5],\n",
    "                                    min_num_ratings=20,\n",
    "                                    train_ratio=0.7,\n",
    "                                    val_ratio=0.1,\n",
    "                                    negative_sample_size=100,\n",
    "                                    verbose=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dataloader"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src='https://github.com/sparsh-ai/stanza/raw/S758139/images/group_recommender_actorcritic_3.svg'>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class DataLoader(object):\n",
    "    \"\"\"\n",
    "    Data Loader\n",
    "    \"\"\"\n",
    "\n",
    "    def __init__(self, config: Config):\n",
    "        \"\"\"\n",
    "        Initialize DataLoader\n",
    "        :param config: configurations\n",
    "        \"\"\"\n",
    "        self.config = config\n",
    "        self.history_length = config.history_length\n",
    "        self.item_num = self.get_item_num()\n",
    "        self.user_num = self.get_user_num()\n",
    "        self.group_num, self.total_group_num, self.group2members_dict, self.user2group_dict = self.get_groups()\n",
    "\n",
    "        if not os.path.exists(self.config.saves_folder_path):\n",
    "            os.mkdir(self.config.saves_folder_path)\n",
    "\n",
    "    def get_item_num(self) -> int:\n",
    "        \"\"\"\n",
    "        Get number of items\n",
    "        :return: number of items\n",
    "        \"\"\"\n",
    "        df_item = pd.read_csv(self.config.item_path, sep='::', index_col=0, engine='python')\n",
    "        self.config.item_num = df_item.index.max()\n",
    "        return self.config.item_num\n",
    "\n",
    "    def get_user_num(self) -> int:\n",
    "        \"\"\"\n",
    "        Get number of users\n",
    "        :return: number of users\n",
    "        \"\"\"\n",
    "        df_user = pd.read_csv(self.config.user_path, sep='::', index_col=0, engine='python')\n",
    "        self.config.user_num = df_user.index.max()\n",
    "        return self.config.user_num\n",
    "\n",
    "    def get_groups(self):\n",
    "        \"\"\"\n",
    "        Get number of groups and group members\n",
    "        :return: group_num, total_group_num, group2members_dict, user2group_dict\n",
    "        \"\"\"\n",
    "        df_group = pd.read_csv(self.config.group_path, sep=' ', header=None, index_col=None,\n",
    "                               names=['GroupID', 'Members'])\n",
    "        df_group['Members'] = df_group['Members']. \\\n",
    "            apply(lambda group_members: tuple(map(int, group_members.split(','))))\n",
    "        group_num = df_group['GroupID'].max()\n",
    "\n",
    "        users = set()\n",
    "        for members in df_group['Members']:\n",
    "            users.update(members)\n",
    "        users = sorted(users)\n",
    "        total_group_num = group_num + len(users)\n",
    "\n",
    "        df_user_group = pd.DataFrame()\n",
    "        df_user_group['GroupID'] = list(range(group_num + 1, total_group_num + 1))\n",
    "        df_user_group['Members'] = [(user,) for user in users]\n",
    "        df_group = df_group.append(df_user_group, ignore_index=True)\n",
    "        group2members_dict = {row['GroupID']: row['Members'] for _, row in df_group.iterrows()}\n",
    "        user2group_dict = {user: group_num + user_index + 1 for user_index, user in enumerate(users)}\n",
    "\n",
    "        self.config.group_num = group_num\n",
    "        self.config.total_group_num = total_group_num\n",
    "        return group_num, total_group_num, group2members_dict, user2group_dict\n",
    "\n",
    "    def load_rating_data(self, mode: str, dataset_name: str, is_appended=True) -> pd.DataFrame():\n",
    "        \"\"\"\n",
    "        Load rating data\n",
    "        :param mode: in ['user', 'group']\n",
    "        :param dataset_name: name of the dataset in ['train', 'val', 'test']\n",
    "        :param is_appended: True to append all datasets before this dataset\n",
    "        :return: df_rating\n",
    "        \"\"\"\n",
    "        assert (mode in ['user', 'group']) and (dataset_name in ['train', 'val', 'test'])\n",
    "        rating_path = os.path.join(self.config.data_folder_path, mode + 'Rating' + dataset_name.capitalize() + '.dat')\n",
    "        df_rating_append = pd.read_csv(rating_path, sep=' ', header=None, index_col=None,\n",
    "                                       names=['GroupID', 'MovieID', 'Rating', 'Timestamp'])\n",
    "        print('Read data:', rating_path)\n",
    "\n",
    "        if is_appended:\n",
    "            if dataset_name == 'train':\n",
    "                df_rating = df_rating_append\n",
    "            elif dataset_name == 'val':\n",
    "                df_rating = self.load_rating_data(mode=mode, dataset_name='train')\n",
    "                df_rating = df_rating.append(df_rating_append, ignore_index=True)\n",
    "            else:\n",
    "                df_rating = self.load_rating_data(mode=mode, dataset_name='val')\n",
    "                df_rating = df_rating.append(df_rating_append, ignore_index=True)\n",
    "        else:\n",
    "            df_rating = df_rating_append\n",
    "\n",
    "        return df_rating\n",
    "\n",
    "    def _load_rating_matrix(self, df_rating: pd.DataFrame()):\n",
    "        \"\"\"\n",
    "        Load rating matrix\n",
    "        :param df_rating: rating data\n",
    "        :return: rating_matrix\n",
    "        \"\"\"\n",
    "        group_ids = df_rating['GroupID']\n",
    "        item_ids = df_rating['MovieID']\n",
    "        ratings = df_rating['Rating']\n",
    "        rating_matrix = coo_matrix((ratings, (group_ids, item_ids)),\n",
    "                                   shape=(self.total_group_num + 1, self.config.item_num + 1)).tocsr()\n",
    "        return rating_matrix\n",
    "\n",
    "    def load_rating_matrix(self, dataset_name: str):\n",
    "        \"\"\"\n",
    "        Load group rating matrix\n",
    "        :param dataset_name: name of the dataset in ['train', 'val', 'test']\n",
    "        :return: rating_matrix\n",
    "        \"\"\"\n",
    "        assert dataset_name in ['train', 'val', 'test']\n",
    "\n",
    "        df_user_rating = self.user2group(self.load_rating_data(mode='user', dataset_name=dataset_name))\n",
    "        df_group_rating = self.load_rating_data(mode='group', dataset_name=dataset_name)\n",
    "        df_group_rating = df_group_rating.append(df_user_rating, ignore_index=True)\n",
    "        rating_matrix = self._load_rating_matrix(df_group_rating)\n",
    "\n",
    "        return rating_matrix\n",
    "\n",
    "    def user2group(self, df_user_rating):\n",
    "        \"\"\"\n",
    "        Change user ids to group ids\n",
    "        :param df_user_rating: user rating\n",
    "        :return: df_user_rating\n",
    "        \"\"\"\n",
    "        df_user_rating['GroupID'] = df_user_rating['GroupID'].apply(lambda user_id: self.user2group_dict[user_id])\n",
    "        return df_user_rating\n",
    "\n",
    "    def _load_eval_data(self, df_data_train: pd.DataFrame(), df_data_eval: pd.DataFrame(),\n",
    "                        negative_samples_dict: Dict[tuple, list]) -> pd.DataFrame():\n",
    "        \"\"\"\n",
    "        Write evaluation data\n",
    "        :param df_data_train: train data\n",
    "        :param df_data_eval: evaluation data\n",
    "        :param negative_samples_dict: one dictionary mapping (group_id, item_id) to negative samples\n",
    "        :return: data for evaluation\n",
    "        \"\"\"\n",
    "        df_eval = pd.DataFrame()\n",
    "        last_state_dict = defaultdict(list)\n",
    "        groups = []\n",
    "        histories = []\n",
    "        actions = []\n",
    "        negative_samples = []\n",
    "\n",
    "        for group_id, rating_group in df_data_train.groupby(['GroupID']):\n",
    "            rating_group.sort_values(by=['Timestamp'], ascending=True, ignore_index=True, inplace=True)\n",
    "            state = rating_group[rating_group['Rating'] == 1]['MovieID'].values.tolist()\n",
    "            last_state_dict[group_id] = state[-self.config.history_length:]\n",
    "\n",
    "        for group_id, rating_group in df_data_eval.groupby(['GroupID']):\n",
    "            rating_group.sort_values(by=['Timestamp'], ascending=True, ignore_index=True, inplace=True)\n",
    "            action = rating_group[rating_group['Rating'] == 1]['MovieID'].values.tolist()\n",
    "            state = deque(maxlen=self.history_length)\n",
    "            state.extend(last_state_dict[group_id])\n",
    "            for item_id in action:\n",
    "                if len(state) == self.config.history_length:\n",
    "                    groups.append(group_id)\n",
    "                    histories.append(list(state))\n",
    "                    actions.append(item_id)\n",
    "                    negative_samples.append(negative_samples_dict[(group_id, item_id)])\n",
    "                state.append(item_id)\n",
    "\n",
    "        df_eval['group'] = groups\n",
    "        df_eval['history'] = histories\n",
    "        df_eval['action'] = actions\n",
    "        df_eval['negative samples'] = negative_samples\n",
    "\n",
    "        return df_eval\n",
    "\n",
    "    def load_negative_samples(self, mode: str, dataset_name: str):\n",
    "        \"\"\"\n",
    "        Load negative samples\n",
    "        :param mode: in ['user', 'group']\n",
    "        :param dataset_name: name of the dataset in ['val', 'test']\n",
    "        :return: negative_samples_dict\n",
    "        \"\"\"\n",
    "        assert (mode in ['user', 'group']) and (dataset_name in ['val', 'test'])\n",
    "        negative_samples_path = os.path.join(self.config.data_folder_path, mode + 'Rating'\n",
    "                                             + dataset_name.capitalize() + 'Negative.dat')\n",
    "        negative_samples_dict = {}\n",
    "\n",
    "        with open(negative_samples_path, 'r') as negative_samples_file:\n",
    "            for line in negative_samples_file.readlines():\n",
    "                negative_samples = line.split()\n",
    "                ids = negative_samples[0][1:-1].split(',')\n",
    "                group_id = int(ids[0])\n",
    "                if mode == 'user':\n",
    "                    group_id = self.user2group_dict[group_id]\n",
    "                item_id = int(ids[1])\n",
    "                negative_samples = list(map(int, negative_samples[1:]))\n",
    "                negative_samples_dict[(group_id, item_id)] = negative_samples\n",
    "\n",
    "        return negative_samples_dict\n",
    "\n",
    "    def load_eval_data(self, mode: str, dataset_name: str, reload=False):\n",
    "        \"\"\"\n",
    "        Load evaluation data\n",
    "        :param mode: in ['user', 'group']\n",
    "        :param dataset_name: in ['val', 'test']\n",
    "        :param reload: True to reload the dataset file\n",
    "        :return: data for evaluation\n",
    "        \"\"\"\n",
    "        assert (mode in ['user', 'group']) and (dataset_name in ['val', 'test'])\n",
    "        exp_eval_path = os.path.join(self.config.saves_folder_path, 'eval_' + mode + '_' + dataset_name + '_'\n",
    "                                     + str(self.config.history_length) + '.pkl')\n",
    "\n",
    "        if reload or not os.path.exists(exp_eval_path):\n",
    "            if dataset_name == 'val':\n",
    "                df_rating_train = self.load_rating_data(mode=mode, dataset_name='train')\n",
    "            else:\n",
    "                df_rating_train = self.load_rating_data(mode=mode, dataset_name='val')\n",
    "            df_rating_eval = self.load_rating_data(mode=mode, dataset_name=dataset_name, is_appended=False)\n",
    "\n",
    "            if mode == 'user':\n",
    "                df_rating_train = self.user2group(df_rating_train)\n",
    "                df_rating_eval = self.user2group(df_rating_eval)\n",
    "\n",
    "            negative_samples_dict = self.load_negative_samples(mode=mode, dataset_name=dataset_name)\n",
    "            df_eval = self._load_eval_data(df_rating_train, df_rating_eval, negative_samples_dict)\n",
    "            df_eval.to_pickle(exp_eval_path)\n",
    "            print('Save data:', exp_eval_path)\n",
    "        else:\n",
    "            df_eval = pd.read_pickle(exp_eval_path)\n",
    "            print('Load data:', exp_eval_path)\n",
    "\n",
    "        return df_eval"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src='https://github.com/sparsh-ai/stanza/raw/S758139/images/group_recommender_actorcritic_4.svg'>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Actor(nn.Module):\n",
    "    \"\"\"\n",
    "    Actor Network\n",
    "    \"\"\"\n",
    "\n",
    "    def __init__(self, embedded_state_size: int, action_weight_size: int, hidden_sizes: Tuple[int]):\n",
    "        \"\"\"\n",
    "        Initialize Actor\n",
    "        :param embedded_state_size: embedded state size\n",
    "        :param action_weight_size: embedded action size\n",
    "        :param hidden_sizes: hidden sizes\n",
    "        \"\"\"\n",
    "        super(Actor, self).__init__()\n",
    "\n",
    "        self.net = nn.Sequential(\n",
    "            nn.Linear(embedded_state_size, hidden_sizes[0]),\n",
    "            nn.ReLU(),\n",
    "            nn.Linear(hidden_sizes[0], hidden_sizes[1]),\n",
    "            nn.ReLU(),\n",
    "            nn.Linear(hidden_sizes[1], action_weight_size),\n",
    "        )\n",
    "\n",
    "    def forward(self, embedded_state):\n",
    "        \"\"\"\n",
    "        Forward\n",
    "        :param embedded_state: embedded state\n",
    "        :return: action weight\n",
    "        \"\"\"\n",
    "        return self.net(embedded_state)\n",
    "\n",
    "\n",
    "class Critic(nn.Module):\n",
    "    \"\"\"\n",
    "    Critic Network\n",
    "    \"\"\"\n",
    "\n",
    "    def __init__(self, embedded_state_size: int, embedded_action_size: int, hidden_sizes: Tuple[int]):\n",
    "        \"\"\"\n",
    "        Initialize Critic\n",
    "        :param embedded_state_size: embedded state size\n",
    "        :param embedded_action_size: embedded action size\n",
    "        :param hidden_sizes: hidden sizes\n",
    "        \"\"\"\n",
    "        super(Critic, self).__init__()\n",
    "\n",
    "        self.net = nn.Sequential(\n",
    "            nn.Linear(embedded_state_size + embedded_action_size, hidden_sizes[0]),\n",
    "            nn.ReLU(),\n",
    "            nn.Linear(hidden_sizes[0], hidden_sizes[1]),\n",
    "            nn.ReLU(),\n",
    "            nn.Linear(hidden_sizes[1], 1)\n",
    "        )\n",
    "\n",
    "    def forward(self, embedded_state, embedded_action):\n",
    "        \"\"\"\n",
    "        Forward\n",
    "        :param embedded_state: embedded state\n",
    "        :param embedded_action: embedded action\n",
    "        :return: Q value\n",
    "        \"\"\"\n",
    "        return self.net(torch.cat([embedded_state, embedded_action], dim=-1))\n",
    "\n",
    "\n",
    "class Embedding(nn.Module):\n",
    "    \"\"\"\n",
    "    Embedding Network\n",
    "    \"\"\"\n",
    "\n",
    "    def __init__(self, embedding_size: int, user_num: int, item_num: int):\n",
    "        \"\"\"\n",
    "        Initialize Embedding\n",
    "        :param embedding_size: embedding size\n",
    "        :param user_num: number of users\n",
    "        :param item_num: number of items\n",
    "        \"\"\"\n",
    "        super(Embedding, self).__init__()\n",
    "        self.user_embedding = nn.Embedding(user_num + 1, embedding_size)\n",
    "        self.item_embedding = nn.Embedding(item_num + 1, embedding_size)\n",
    "        self.user_attention = nn.Sequential(\n",
    "            nn.Linear(embedding_size, embedding_size),\n",
    "            nn.ReLU(),\n",
    "            nn.Linear(embedding_size, 1)\n",
    "        )\n",
    "        self.user_softmax = nn.Softmax(dim=-1)\n",
    "\n",
    "    def forward(self, group_members, history):\n",
    "        \"\"\"\n",
    "        Forward\n",
    "        :param group_members: group members\n",
    "        :param history: browsing history of items\n",
    "        :return: embedded state\n",
    "        \"\"\"\n",
    "        embedded_group_members = self.user_embedding(group_members)\n",
    "        group_member_attentions = self.user_softmax(self.user_attention(embedded_group_members))\n",
    "        embedded_group = torch.squeeze(torch.inner(group_member_attentions.T, embedded_group_members.T))\n",
    "        embedded_history = torch.flatten(self.item_embedding(history), start_dim=-2)\n",
    "        embedded_state = torch.cat([embedded_group, embedded_history], dim=-1)\n",
    "        return embedded_state"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Agent"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src='https://github.com/sparsh-ai/stanza/raw/S758139/images/group_recommender_actorcritic_5.svg'>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class DDPGAgent(object):\n",
    "    \"\"\"\n",
    "    DDPG (Deep Deterministic Policy Gradient) Agent\n",
    "    \"\"\"\n",
    "\n",
    "    def __init__(self, config: Config, noise: OUNoise, group2members_dict: dict, verbose=False):\n",
    "        \"\"\"\n",
    "        Initialize DDPGAgent\n",
    "        :param config: configurations\n",
    "        :param group2members_dict: group members data\n",
    "        :param verbose: True to print networks\n",
    "        \"\"\"\n",
    "        self.config = config\n",
    "        self.noise = noise\n",
    "        self.group2members_dict = group2members_dict\n",
    "        self.tau = config.tau\n",
    "        self.gamma = config.gamma\n",
    "        self.device = config.device\n",
    "\n",
    "        self.embedding = Embedding(embedding_size=config.embedding_size,\n",
    "                                         user_num=config.user_num,\n",
    "                                         item_num=config.item_num).to(config.device)\n",
    "        self.actor = Actor(embedded_state_size=config.embedded_state_size,\n",
    "                                 action_weight_size=config.embedded_action_size,\n",
    "                                 hidden_sizes=config.actor_hidden_sizes).to(config.device)\n",
    "        self.actor_target = Actor(embedded_state_size=config.embedded_state_size,\n",
    "                                        action_weight_size=config.embedded_action_size,\n",
    "                                        hidden_sizes=config.actor_hidden_sizes).to(config.device)\n",
    "        self.critic = Critic(embedded_state_size=config.embedded_state_size,\n",
    "                                   embedded_action_size=config.embedded_action_size,\n",
    "                                   hidden_sizes=config.critic_hidden_sizes).to(config.device)\n",
    "        self.critic_target = Critic(embedded_state_size=config.embedded_state_size,\n",
    "                                          embedded_action_size=config.embedded_action_size,\n",
    "                                          hidden_sizes=config.critic_hidden_sizes).to(config.device)\n",
    "\n",
    "        if verbose:\n",
    "            print(self.embedding)\n",
    "            print(self.actor)\n",
    "            print(self.critic)\n",
    "\n",
    "        self.copy_network(self.actor, self.actor_target)\n",
    "        self.copy_network(self.critic, self.critic_target)\n",
    "\n",
    "        self.replay_memory = ReplayMemory(buffer_size=config.buffer_size)\n",
    "        self.critic_criterion = nn.MSELoss()\n",
    "        self.embedding_optimizer = optim.Adam(self.embedding.parameters(), lr=config.embedding_learning_rate,\n",
    "                                              weight_decay=config.embedding_weight_decay)\n",
    "        self.actor_optimizer = optim.Adam(self.actor.parameters(), lr=config.actor_learning_rate,\n",
    "                                          weight_decay=config.actor_weight_decay)\n",
    "        self.critic_optimizer = optim.Adam(self.critic.parameters(), lr=config.critic_learning_rate,\n",
    "                                           weight_decay=config.critic_weight_decay)\n",
    "\n",
    "    def copy_network(self, network: nn.Module, network_target: nn.Module):\n",
    "        \"\"\"\n",
    "        Copy one network to its target network\n",
    "        :param network: the original network to be copied\n",
    "        :param network_target: the target network\n",
    "        \"\"\"\n",
    "        for parameters, target_parameters in zip(network.parameters(), network_target.parameters()):\n",
    "            target_parameters.data.copy_(parameters.data)\n",
    "\n",
    "    def sync_network(self, network: nn.Module, network_target: nn.Module):\n",
    "        \"\"\"\n",
    "        Synchronize one network to its target network\n",
    "        :param network: the original network to be synchronized\n",
    "        :param network_target: the target network\n",
    "        :return:\n",
    "        \"\"\"\n",
    "        for parameters, target_parameters in zip(network.parameters(), network_target.parameters()):\n",
    "            target_parameters.data.copy_(parameters.data * self.tau + target_parameters.data * (1 - self.tau))\n",
    "\n",
    "    def get_action(self, state: list, item_candidates: list = None, top_K: int = 1, with_noise=False):\n",
    "        \"\"\"\n",
    "        Get one action\n",
    "        :param state: one environment state\n",
    "        :param item_candidates: item candidates\n",
    "        :param top_K: top K items\n",
    "        :param with_noise: True to with noise\n",
    "        :return: action\n",
    "        \"\"\"\n",
    "        with torch.no_grad():\n",
    "            states = [state]\n",
    "            embedded_states = self.embed_states(states)\n",
    "            action_weights = self.actor(embedded_states)\n",
    "            action_weight = torch.squeeze(action_weights)\n",
    "            if with_noise:\n",
    "                action_weight += self.noise.get_ou_noise()\n",
    "\n",
    "            if item_candidates is None:\n",
    "                item_embedding_weight = self.embedding.item_embedding.weight.clone()\n",
    "            else:\n",
    "                item_candidates = np.array(item_candidates)\n",
    "                item_candidates_tensor = torch.tensor(item_candidates, dtype=torch.int).to(self.device)\n",
    "                item_embedding_weight = self.embedding.item_embedding(item_candidates_tensor)\n",
    "\n",
    "            scores = torch.inner(action_weight, item_embedding_weight).detach().cpu().numpy()\n",
    "            sorted_score_indices = np.argsort(scores)[:top_K]\n",
    "\n",
    "            if item_candidates is None:\n",
    "                action = sorted_score_indices\n",
    "            else:\n",
    "                action = item_candidates[sorted_score_indices]\n",
    "            action = np.squeeze(action)\n",
    "            if top_K == 1:\n",
    "                action = action.item()\n",
    "        return action\n",
    "\n",
    "    def get_embedded_actions(self, embedded_states: torch.Tensor, target=False):\n",
    "        \"\"\"\n",
    "        Get embedded actions\n",
    "        :param embedded_states: embedded states\n",
    "        :param target: True for target network\n",
    "        :return: embedded_actions (, actions)\n",
    "        \"\"\"\n",
    "        if not target:\n",
    "            action_weights = self.actor(embedded_states)\n",
    "        else:\n",
    "            action_weights = self.actor_target(embedded_states)\n",
    "\n",
    "        item_embedding_weight = self.embedding.item_embedding.weight.clone()\n",
    "        scores = torch.inner(action_weights, item_embedding_weight)\n",
    "        embedded_actions = torch.inner(functional.gumbel_softmax(scores, hard=True), item_embedding_weight.t())\n",
    "        return embedded_actions\n",
    "\n",
    "    def embed_state(self, state: list):\n",
    "        \"\"\"\n",
    "        Embed one state\n",
    "        :param state: state\n",
    "        :return: embedded_state\n",
    "        \"\"\"\n",
    "        group_id = state[0]\n",
    "        group_members = torch.tensor(self.group2members_dict[group_id], dtype=torch.int).to(self.device)\n",
    "        history = torch.tensor(state[1:], dtype=torch.int).to(self.device)\n",
    "        embedded_state = self.embedding(group_members, history)\n",
    "        return embedded_state\n",
    "\n",
    "    def embed_states(self, states: List[list]):\n",
    "        \"\"\"\n",
    "        Embed states\n",
    "        :param states: states\n",
    "        :return: embedded_states\n",
    "        \"\"\"\n",
    "        embedded_states = torch.stack([self.embed_state(state) for state in states], dim=0)\n",
    "        return embedded_states\n",
    "\n",
    "    def embed_actions(self, actions: list):\n",
    "        \"\"\"\n",
    "        Embed actions\n",
    "        :param actions: actions\n",
    "        :return: embedded_actions\n",
    "        \"\"\"\n",
    "        actions = torch.tensor(actions, dtype=torch.int).to(self.device)\n",
    "        embedded_actions = self.embedding.item_embedding(actions)\n",
    "        return embedded_actions\n",
    "\n",
    "    def update(self):\n",
    "        \"\"\"\n",
    "        Update the networks\n",
    "        :return: actor loss and critic loss\n",
    "        \"\"\"\n",
    "        batch = self.replay_memory.sample(self.config.batch_size)\n",
    "        states, actions, rewards, next_states = list(zip(*batch))\n",
    "\n",
    "        self.embedding_optimizer.zero_grad()\n",
    "        self.critic_optimizer.zero_grad()\n",
    "        embedded_states = self.embed_states(states)\n",
    "        embedded_actions = self.embed_actions(actions)\n",
    "        rewards = torch.unsqueeze(torch.tensor(rewards, dtype=torch.int).to(self.device), dim=-1)\n",
    "        embedded_next_states = self.embed_states(next_states)\n",
    "        q_values = self.critic(embedded_states, embedded_actions)\n",
    "\n",
    "        with torch.no_grad():\n",
    "            embedded_next_actions = self.get_embedded_actions(embedded_next_states, target=True)\n",
    "            next_q_values = self.critic_target(embedded_next_states, embedded_next_actions)\n",
    "            q_values_target = rewards + self.gamma * next_q_values\n",
    "\n",
    "        critic_loss = self.critic_criterion(q_values, q_values_target)\n",
    "        critic_loss.backward()\n",
    "        self.critic_optimizer.step()\n",
    "\n",
    "        self.actor_optimizer.zero_grad()\n",
    "        embedded_states = self.embed_states(states)\n",
    "        actor_loss = -self.critic(embedded_states, self.get_embedded_actions(embedded_states)).mean()\n",
    "        actor_loss.backward()\n",
    "        self.actor_optimizer.step()\n",
    "        self.embedding_optimizer.step()\n",
    "\n",
    "        self.sync_network(self.actor, self.actor_target)\n",
    "        self.sync_network(self.critic, self.critic_target)\n",
    "\n",
    "        return actor_loss.detach().cpu().numpy(), critic_loss.detach().cpu().numpy()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Environment"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src='https://github.com/sparsh-ai/stanza/raw/S758139/images/group_recommender_actorcritic_6.svg'>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Env(gym.Env):\n",
    "    \"\"\"\n",
    "    Environment for the recommender system\n",
    "    https://github.com/openai/gym/blob/master/gym/core.py\n",
    "    \"\"\"\n",
    "    metadata = {'render.modes': ['human']}\n",
    "    reward_range = (0, 1)\n",
    "\n",
    "    def __init__(self, config: Config, rating_matrix: csr_matrix, dataset_name: str):\n",
    "        \"\"\"\n",
    "        Initialize Env\n",
    "        :param config: configurations\n",
    "        :param rating_matrix: rating matrix\n",
    "        :param dataset_name: dataset name\n",
    "        \"\"\"\n",
    "        assert dataset_name in ['train', 'val', 'test']\n",
    "        self.config = config\n",
    "        self.action_space = gym.spaces.Box(low=-np.inf, high=np.inf, shape=(config.action_size,))\n",
    "        self.observation_space = gym.spaces.Box(low=-np.inf, high=np.inf, shape=(config.state_size,))\n",
    "\n",
    "        self.rating_matrix = rating_matrix\n",
    "        rating_matrix_coo = rating_matrix.tocoo()\n",
    "        rating_matrix_rows = rating_matrix_coo.row\n",
    "        rating_matrix_columns = rating_matrix_coo.col\n",
    "        self.rating_matrix_index_set = set(zip(*(rating_matrix_rows, rating_matrix_columns)))\n",
    "        self.env_name = 'env_' + dataset_name + '_' + str(self.config.env_n_components) + '.npy'\n",
    "        self.env_path = os.path.join(config.saves_folder_path, self.env_name)\n",
    "\n",
    "        self.rating_matrix_pred = None\n",
    "        self.load_env()\n",
    "\n",
    "        self.state = None\n",
    "        self.reset()\n",
    "\n",
    "    def load_env(self):\n",
    "        \"\"\"\n",
    "        Load environment\n",
    "        \"\"\"\n",
    "        if not os.path.exists(self.env_path):\n",
    "            env_model = NMF(n_components=self.config.env_n_components, init='random', tol=self.config.env_tol,\n",
    "                            max_iter=self.config.env_max_iter, alpha=self.config.env_alpha, verbose=True,\n",
    "                            random_state=0)\n",
    "            print('-' * 50)\n",
    "            print('Train environment:')\n",
    "            W = env_model.fit_transform(X=self.rating_matrix)\n",
    "            H = env_model.components_\n",
    "            self.rating_matrix_pred = W @ H\n",
    "            print('-' * 50)\n",
    "            np.save(self.env_path, self.rating_matrix_pred)\n",
    "            print('Save environment:', self.env_path)\n",
    "        else:\n",
    "            self.rating_matrix_pred = np.load(self.env_path)\n",
    "            print('Load environment:', self.env_path)\n",
    "\n",
    "    def reset(self):\n",
    "        \"\"\"\n",
    "        Reset the environment\n",
    "        :return: state\n",
    "        \"\"\"\n",
    "        while True:\n",
    "            group_id = np.random.choice(range(1, self.config.total_group_num + 1))\n",
    "            nonzero_row, nonzero_col = self.rating_matrix[group_id, :].nonzero()\n",
    "            if len(nonzero_col) >= self.config.history_length:\n",
    "                break\n",
    "        history = np.random.choice(nonzero_col, size=self.config.history_length, replace=False).tolist()\n",
    "        self.state = [group_id] + history\n",
    "        return self.state\n",
    "\n",
    "    def step(self, action: int):\n",
    "        \"\"\"\n",
    "        Take one action to the environment\n",
    "        :param action: action\n",
    "        :return: new_state, reward, done, info\n",
    "        \"\"\"\n",
    "        group_id = self.state[0]\n",
    "        history = self.state[1:]\n",
    "\n",
    "        if (group_id, action) in self.rating_matrix_index_set:\n",
    "            reward = self.rating_matrix[group_id, action]\n",
    "        else:\n",
    "            reward_probability = self.rating_matrix_pred[group_id, action]\n",
    "            reward = np.random.choice(self.config.rewards, p=[1 - reward_probability, reward_probability])\n",
    "\n",
    "        if reward > 0:\n",
    "            history = history[1:] + [action]\n",
    "\n",
    "        new_state = [group_id] + history\n",
    "        self.state = new_state\n",
    "        done = False\n",
    "        info = {}\n",
    "\n",
    "        return new_state, reward, done, info\n",
    "\n",
    "    def render(self, mode='human'):\n",
    "        \"\"\"\n",
    "        Render the environment\n",
    "        :param mode: mode\n",
    "        \"\"\"\n",
    "        pass"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluator"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src='https://github.com/sparsh-ai/stanza/raw/S758139/images/group_recommender_actorcritic_7.svg'>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Evaluator(object):\n",
    "    \"\"\"\n",
    "    Evaluator\n",
    "    \"\"\"\n",
    "\n",
    "    def __init__(self, config: Config):\n",
    "        \"\"\"\n",
    "        Initialize Evaluator\n",
    "        :param config: configurations\n",
    "        \"\"\"\n",
    "        self.config = config\n",
    "\n",
    "    def evaluate(self, agent: DDPGAgent, df_eval: pd.DataFrame(), mode: str, top_K=5):\n",
    "        \"\"\"\n",
    "        Evaluate the agent\n",
    "        :param agent: agent\n",
    "        :param df_eval: evaluation data\n",
    "        :param mode: in ['user', 'group']\n",
    "        :param top_K: length of the recommendation list\n",
    "        :return: avg_recall_score, avg_ndcg_score\n",
    "        \"\"\"\n",
    "        recall_scores = []\n",
    "        ndcg_scores = []\n",
    "\n",
    "        for _, row in df_eval.iterrows():\n",
    "            group = row['group']\n",
    "            history = row['history']\n",
    "            item_true = row['action']\n",
    "            item_candidates = row['negative samples'] + [item_true]\n",
    "            np.random.shuffle(item_candidates)\n",
    "\n",
    "            state = [group] + history\n",
    "            items_pred = agent.get_action(state=state, item_candidates=item_candidates, top_K=top_K)\n",
    "\n",
    "            recall_score = 0\n",
    "            ndcg_score = 0\n",
    "\n",
    "            for k, item in enumerate(items_pred):\n",
    "                if item == item_true:\n",
    "                    recall_score = 1\n",
    "                    ndcg_score = np.log2(2) / np.log2(k + 2)\n",
    "                    break\n",
    "\n",
    "            recall_scores.append(recall_score)\n",
    "            ndcg_scores.append(ndcg_score)\n",
    "\n",
    "        avg_recall_score = float(np.mean(recall_scores))\n",
    "        avg_ndcg_score = float(np.mean(ndcg_scores))\n",
    "        print('%s: Recall@%d = %.4f, NDCG@%d = %.4f' % (mode.capitalize(), top_K, avg_recall_score,\n",
    "                                                        top_K, avg_ndcg_score))\n",
    "        return avg_recall_score, avg_ndcg_score"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Trainer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src='https://github.com/sparsh-ai/stanza/raw/S758139/images/group_recommender_actorcritic_8.svg'>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def train(config: Config, env: Env, agent: DDPGAgent, evaluator: Evaluator,\n",
    "          df_eval_user: pd.DataFrame(), df_eval_group: pd.DataFrame()):\n",
    "    \"\"\"\n",
    "    Train the agent with the environment\n",
    "    :param config: configurations\n",
    "    :param env: environment\n",
    "    :param agent: agent\n",
    "    :param evaluator: evaluator\n",
    "    :param df_eval_user: user evaluation data\n",
    "    :param df_eval_group: group evaluation data\n",
    "    :return:\n",
    "    \"\"\"\n",
    "    rewards = []\n",
    "    for episode in range(config.num_episodes):\n",
    "        state = env.reset()\n",
    "        agent.noise.reset()\n",
    "        episode_reward = 0\n",
    "\n",
    "        for step in range(config.num_steps):\n",
    "            action = agent.get_action(state)\n",
    "            new_state, reward, _, _ = env.step(action)\n",
    "            agent.replay_memory.push((state, action, reward, new_state))\n",
    "            state = new_state\n",
    "            episode_reward += reward\n",
    "\n",
    "            if len(agent.replay_memory) >= config.batch_size:\n",
    "                agent.update()\n",
    "\n",
    "        rewards.append(episode_reward / config.num_steps)\n",
    "        print('Episode = %d, average reward = %.4f' % (episode, episode_reward / config.num_steps))\n",
    "        if (episode + 1) % config.eval_per_iter == 0:\n",
    "            for top_K in config.top_K_list:\n",
    "                evaluator.evaluate(agent=agent, df_eval=df_eval_user, mode='user', top_K=top_K)\n",
    "            for top_K in config.top_K_list:\n",
    "                evaluator.evaluate(agent=agent, df_eval=df_eval_group, mode='group', top_K=top_K)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Main"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "config = Config()\n",
    "dataloader = DataLoader(config)\n",
    "rating_matrix_train = dataloader.load_rating_matrix(dataset_name='val')\n",
    "df_eval_user_test = dataloader.load_eval_data(mode='user', dataset_name='test')\n",
    "df_eval_group_test = dataloader.load_eval_data(mode='group', dataset_name='test')\n",
    "env = Env(config=config, rating_matrix=rating_matrix_train, dataset_name='val')\n",
    "noise = OUNoise(config=config)\n",
    "agent = DDPGAgent(config=config, noise=noise, group2members_dict=dataloader.group2members_dict, verbose=True)\n",
    "evaluator = Evaluator(config=config)\n",
    "train(config=config, env=env, agent=agent, evaluator=evaluator,\n",
    "        df_eval_user=df_eval_user_test, df_eval_group=df_eval_group_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```\n",
    "User: Recall@5 = 0.1304, NDCG@5 = 0.0796\n",
    "User: Recall@10 = 0.2349, NDCG@10 = 0.1132\n",
    "User: Recall@20 = 0.3470, NDCG@20 = 0.1416\n",
    "Group: Recall@5 = 0.1856, NDCG@5 = 0.1149\n",
    "Group: Recall@10 = 0.3153, NDCG@10 = 0.1568\n",
    "Group: Recall@20 = 0.4448, NDCG@20 = 0.1901\n",
    "Episode = 220, average reward = 0.0200\n",
    "Episode = 221, average reward = 0.0000\n",
    "Episode = 222, average reward = 0.0000\n",
    "Episode = 223, average reward = 0.0100\n",
    "Episode = 224, average reward = 0.0000\n",
    "Episode = 225, average reward = 0.0000\n",
    "Episode = 226, average reward = 0.0000\n",
    "Episode = 227, average reward = 0.0000\n",
    "Episode = 228, average reward = 0.1300\n",
    "Episode = 229, average reward = 0.0000\n",
    "User: Recall@5 = 0.1328, NDCG@5 = 0.0810\n",
    "User: Recall@10 = 0.2405, NDCG@10 = 0.1156\n",
    "User: Recall@20 = 0.3595, NDCG@20 = 0.1458\n",
    "Group: Recall@5 = 0.1876, NDCG@5 = 0.1170\n",
    "Group: Recall@10 = 0.3249, NDCG@10 = 0.1613\n",
    "Group: Recall@20 = 0.4619, NDCG@20 = 0.1964\n",
    "Episode = 230, average reward = 0.3000\n",
    "Episode = 231, average reward = 0.0000\n",
    "Episode = 232, average reward = 0.0200\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[31mERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.\n",
      "markdown 3.3.6 requires importlib-metadata>=4.4; python_version < \"3.10\", but you have importlib-metadata 2.1.2 which is incompatible.\u001b[0m\n",
      "Author: Sparsh A.\n",
      "\n",
      "Last updated: 2021-11-26 13:20:10\n",
      "\n",
      "Compiler    : GCC 7.5.0\n",
      "OS          : Linux\n",
      "Release     : 5.4.104+\n",
      "Machine     : x86_64\n",
      "Processor   : x86_64\n",
      "CPU cores   : 2\n",
      "Architecture: 64bit\n",
      "\n",
      "torch  : 1.10.0+cu111\n",
      "IPython: 5.5.0\n",
      "pandas : 1.1.5\n",
      "gym    : 0.17.3\n",
      "numpy  : 1.19.5\n",
      "\n"
     ]
    }
   ],
   "source": [
    "!pip install -q watermark\n",
    "%reload_ext watermark\n",
    "%watermark -a \"Sparsh A.\" -m -iv -u -t -d"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## References\n",
    "1. [DRGR: Deep Reinforcement Learning based Group Recommender System](https://arxiv.org/abs/2106.06900v1)\n",
    "2. [Deep Reinforcement Learning based Group Recommender System](https://recohut.notion.site/Deep-Reinforcement-Learning-based-Group-Recommender-System-6399cf01102b485897578d1bccbe3467)\n",
    "3. Source code:\n",
    " - https://github.com/zefang-liu/group-recommender\n",
    " - https://github.com/sparsh-ai/stanza/tree/S758139 \n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**END**"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "name": "python3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}
