{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Existence Of Node Clusters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we demonstrate that in random forest that has been trained on some set of data, the nodes can be reasonably organized into clusters.\n",
    "\n",
    "First, we must train or load a forest:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import scanpy as sc\n",
    "\n",
    "import sys\n",
    "sys.path.append('../')\n",
    "import tree_reader as tr \n",
    "import lumberjack\n",
    "\n",
    "data_location = \"/Users/bbrener1/battle/rusty_forest_4/data/aging_brain/\"\n",
    "!ls {data_location}\n",
    "# data_location = \"../data/aging_brain/\"\n",
    "\n",
    "forest = tr.Forest.load(data_location + 'cv_forest_trimmed_extra')\n",
    "forest.arguments\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# print(len(forest.output_features))\n",
    "# print(len(forest.split_clusters))\n",
    "print(len(forest.nodes()))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A Random Forest is a collection of decision trees, and a decision tree is a collection of individual decision points, commonly known as \"Nodes\"\n",
    "\n",
    "To understand Random Forests and Decision Trees, it is important to understand how Nodes work. Each individual node is a (very crappy) regressor, eg. each Node makess a prediction based on a rule like \"If Gene 1 has expression > 10, Gene 2 will have expression < 5\", or \"If a house is < 5 miles from a school, it will cost > $100,000\". A very important property of each node, however, is that it can also have children, which are other nodes. When a node makes a prediction like \"If Gene 1 has expression > 10 then Gene 2 has expression < 5\", it can pass all the samples for which Gene 1 is > 10 to one of its children, and all the samples for which Gene 1 < 10 to the other child. After that, each one of its children can make a different prediction, which results in compound rules.\n",
    "\n",
    "This is how a decision tree is formed. A decision tree with a depth of 2 might contain a rule like \"If Gene 1 > 10 AND Gene 3 > 10, THEN Gene 2 and Gene 4 are both < 2, which would represent one of the \"Leaf\" nodes that it has. Leaf nodes are nodes with no children. \n",
    "\n",
    "Individual decision trees, then, are somewhat crappy predictors, but they're better than individual nodes. In order to improve the performance of decision trees, we can construct a Random Forest. To construct a random forest, we can train many decision trees on bootstraps of a dataset\n",
    "\n",
    "If many decision trees are combined and their predictions averaged together, you have a Random Forest, which is a pretty good kind of regressor. \n",
    "\n",
    "A practical demonstration might help:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "forest.reset_split_clusters()\n",
    "forest.interpret_splits(depth=4,mode='sample',metric='cosine',pca=100,relatives=False,k=10,resolution=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So now that we know that random forests are collections of ordered nodes, we can examine a more interesting question: do certain nodes occur repeatedly in the forest, despite operating on bootstrapped samples? \n",
    "\n",
    "In order to examine this question first we must understand different ways of describing a node. I think generally there are three helpful ways of looking at a node:\n",
    "\n",
    "* **Node Sample Encoding**: A binary vector the length of the number of samples you are considering. 0 or false means the sample is absent from the node. A 1 or true means the sample is present in the node. \n",
    "\n",
    "* **Node Mean Encoding**: A float vector the length of the number of targets you are considering. Each value is the mean of the target values for all samples in this node. This is the node's prediction for samples that occur in it.\n",
    "\n",
    "* **Node Additive Encoding**: A float vector the length of the number of targets you are considering. Each value is THE DIFFERENCE between the mean value for that target in THIS NODE and the mean value for that target IN THE PARENT of this node. For root nodes, which have no parents, the additive encoding is simply th mean value across the entire dataset. (As if the mean of a hypothetical parent would have been 0). This encoding represents the marginal effect of each node.\n",
    "\n",
    "We should examine if there are any common patterns that appear if we encode many nodes from a forest using each of these representations:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Here we plot the sample representations of nodes. \n",
    "# This generates a set of figures demonstrating the existence of node clusters\n",
    "\n",
    "from sklearn.decomposition import PCA\n",
    "\n",
    "# For ease of processing we have to construct dimensionally reduced representations of the encodings. \n",
    "\n",
    "nodes = forest.nodes(root=False,depth=3)\n",
    "\n",
    "sample_encoding = forest.node_representation(nodes,mode='sample')\n",
    "reduced_sample = PCA(n_components=100).fit_transform(sample_encoding.T)\n",
    "reduced_sample_node = PCA(n_components=100).fit_transform(sample_encoding)\n",
    "\n",
    "print(sample_encoding.shape)\n",
    "print(reduced_sample.shape)\n",
    "print(reduced_sample_node.shape)\n",
    "\n",
    "from scipy.cluster.hierarchy import linkage,dendrogram\n",
    "\n",
    "\n",
    "# sample_agglomeration = dendrogram(linkage(reduced_sample, metric='cosine', method='average'), no_plot=True)['leaves']\n",
    "# node_sample_agglomeration = dendrogram(linkage(reduced_node, metric='cosine', method='average'), no_plot=True)['leaves']\n",
    "\n",
    "# plt.figure()\n",
    "# plt.title(\"Cell Presence in Node (Clustered)\")\n",
    "# plt.imshow(sample_encoding[node_sample_agglomeration].T[sample_agglomeration].T,cmap='binary',aspect='auto',interpolation='none')\n",
    "# plt.xlabel(\"Cells\")\n",
    "# plt.ylabel(\"Nodes\")\n",
    "# plt.colorbar()\n",
    "# plt.tight_layout()\n",
    "# plt.show()\n",
    "\n",
    "# # # And here we sort the nodes after they have been clustered (more on the clustering procedure in a bit)\n",
    "\n",
    "# node_cluster_sort = np.argsort([n.split_cluster for n in nodes])\n",
    "\n",
    "# plt.figure()\n",
    "# plt.title(\"Cell Presence in Node (Clustered)\")\n",
    "# plt.imshow(sample_encoding[node_cluster_sort].T[sample_agglomeration].T,cmap='binary',aspect='auto',interpolation='none')\n",
    "# plt.xlabel(\"Cells\")\n",
    "# plt.ylabel(\"Nodes\")\n",
    "# plt.colorbar()\n",
    "# plt.tight_layout()\n",
    "# plt.show()\n",
    "\n",
    "plt.figure()\n",
    "plt.suptitle(\"Cell Presence in Node (Two-Way Agglomerated)\")\n",
    "ax1 = plt.axes([0,.7,.8,.2])\n",
    "node_sample_agglomeration = dendrogram(linkage(reduced_sample_node, metric='cosine', method='average'),orientation='top', no_plot=False)['leaves']\n",
    "plt.xticks([])\n",
    "plt.yticks([])\n",
    "ax2 = plt.axes([.8,0,.2,.7])\n",
    "sample_agglomeration = dendrogram(linkage(reduced_sample, metric='cosine', method='average'),orientation='right', no_plot=False)['leaves']\n",
    "plt.xticks([])\n",
    "plt.yticks([])\n",
    "ax3 = plt.axes([0,0,.8,.7])\n",
    "im = plt.imshow(sample_encoding[node_sample_agglomeration].T[sample_agglomeration[::-1]],cmap='binary',aspect='auto',interpolation='none')\n",
    "plt.xlabel(\"Nodes\")\n",
    "plt.ylabel(\"Cells\")\n",
    "plt.tight_layout()\n",
    "plt.show()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "## from sklearn.decomposition import PCA\n",
    "\n",
    "sister_encoding = forest.node_representation(nodes,mode='sister')\n",
    "reduced_sister = PCA(n_components=100).fit_transform(sister_encoding.T)\n",
    "reduced_sister_node = PCA(n_components=100).fit_transform(sister_encoding)\n",
    "\n",
    "print(sister_encoding.shape)\n",
    "print(reduced_sister.shape)\n",
    "print(reduced_sister_node.shape)\n",
    "\n",
    "# from scipy.cluster.hierarchy import linkage,dendrogram\n",
    "\n",
    "# sister_agglomeration = dendrogram(linkage(reduced_sister, metric='cosine', method='average'), no_plot=True)['leaves']\n",
    "# node_sister_agglomeration = dendrogram(linkage(reduced_sister_node, metric='cosine', method='average'), no_plot=True)['leaves']\n",
    "\n",
    "# plt.figure()\n",
    "# plt.title(\"Sample Presence in Node vs Sister (Two-Way Agglomerated)\")\n",
    "# plt.imshow(sister_encoding[node_sister_agglomeration].T[sister_agglomeration].T,cmap='bwr',aspect='auto',interpolation='none')\n",
    "# plt.xlabel(\"Samples\")\n",
    "# plt.ylabel(\"Nodes\")\n",
    "# plt.colorbar()\n",
    "# plt.tight_layout()\n",
    "# plt.show()\n",
    "\n",
    "# plt.figure()\n",
    "# plt.title(\"Sample Presence in Node vs Sister (Clustered By Gain)\")\n",
    "# plt.imshow(sister_encoding[node_cluster_sort].T[sister_agglomeration].T,cmap='bwr',aspect='auto',interpolation='none')\n",
    "# plt.xlabel(\"Samples\")\n",
    "# plt.ylabel(\"Nodes\")\n",
    "# plt.colorbar()\n",
    "# plt.tight_layout()\n",
    "# plt.show()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Here we plot the construct and agglomerate the additive gain representation \n",
    "\n",
    "\n",
    "feature_encoding = forest.node_representation(nodes,mode='partial_absolute')\n",
    "reduced_feature = PCA(n_components=100).fit_transform(feature_encoding.T)\n",
    "reduced_feature_node = PCA(n_components=100).fit_transform(feature_encoding)\n",
    "\n",
    "minimax = np.max(np.abs(feature_encoding))\n",
    "\n",
    "# feature_agglomeration = dendrogram(linkage(reduced_feature, metric='cosine', method='average'), no_plot=True)['leaves']\n",
    "# node_feature_agglomeration = dendrogram(linkage(reduced_node, metric='cosine', method='average'), no_plot=True)['leaves']\n",
    "\n",
    "# node_cluster_sort = np.argsort([n.split_cluster for n in nodes])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Here we plot the additive gain representation \n",
    "\n",
    "# print(feature_encoding.shape)\n",
    "\n",
    "# plt.figure()\n",
    "# plt.title(\"Target Gain in Node (Double-Agglomerated)\")\n",
    "# plt.imshow(feature_encoding[node_feature_agglomeration].T[feature_agglomeration].T,cmap='bwr',interpolation='none',aspect='auto',vmin=-2,vmax=2)\n",
    "# plt.xlabel(\"Genes\")\n",
    "# plt.ylabel(\"Nodes\")\n",
    "# plt.colorbar(label=\"Parent Mean - Node Mean (Log TPM)\")\n",
    "# plt.tight_layout()\n",
    "# plt.show()\n",
    "\n",
    "# plt.figure()\n",
    "# plt.title(\"Target Gain in Node (Clustered)\")\n",
    "# plt.imshow(feature_encoding[node_cluster_sort].T[feature_agglomeration].T,cmap='bwr',interpolation='none',aspect='auto',vmin=-2,vmax=2)\n",
    "# plt.xlabel(\"Genes\")\n",
    "# plt.ylabel(\"Nodes\")\n",
    "# plt.colorbar(label=\"Parent Mean - Node Mean (Log TPM)\")\n",
    "# plt.tight_layout()\n",
    "# plt.show()\n",
    "\n",
    "plt.figure()\n",
    "plt.suptitle(\"Gene Variance Explained by Node (Two-Way Agglomerated)\")\n",
    "ax1 = plt.axes([0,.7,.8,.2])\n",
    "node_feature_agglomeration = dendrogram(linkage(reduced_feature_node, metric='cosine', method='average'),orientation='top', no_plot=False)['leaves']\n",
    "plt.xticks([])\n",
    "plt.yticks([])\n",
    "ax2 = plt.axes([.8,0,.2,.7])\n",
    "feature_agglomeration = dendrogram(linkage(reduced_feature, metric='cosine', method='average'),orientation='right', no_plot=False)['leaves']\n",
    "plt.xticks([])\n",
    "plt.yticks([])\n",
    "ax3 = plt.axes([0,0,.8,.7])\n",
    "im = plt.imshow(feature_encoding[node_sample_agglomeration].T[feature_agglomeration[::-1]],cmap='bwr',vmin=-minimax,vmax=minimax,aspect='auto',interpolation='none')\n",
    "plt.xlabel(\"Nodes\")\n",
    "plt.ylabel(\"Genes\")\n",
    "plt.tight_layout()\n",
    "plt.show()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "\n",
    "plt.figure(figsize=(4,4))\n",
    "plt.suptitle(\"Node Encodings (Two-Way Agglomerated)\")\n",
    "\n",
    "# ax1 = plt.axes([0,.7,.8,.2])\n",
    "# node_sister_agglomeration = dendrogram(linkage(reduced_sister_node, metric='cosine', method='average'), orientation='top', no_plot=False)['leaves']\n",
    "# plt.xticks([])\n",
    "# plt.yticks([])\n",
    "# ax2 = plt.axes([.8,.35,.2,.33])\n",
    "# sister_agglomeration = dendrogram(linkage(reduced_sister, metric='cosine', method='average'), orientation='right', no_plot=False)['leaves']\n",
    "# plt.xticks([])\n",
    "# plt.yticks([])\n",
    "# ax3 = plt.axes([0,.35,.8,.33])\n",
    "# im1 = plt.imshow(sister_encoding[node_sister_agglomeration].T[sister_agglomeration[::-1]],cmap='bwr',aspect='auto',interpolation='none')\n",
    "# plt.xticks([])\n",
    "# plt.yticks([])\n",
    "\n",
    "ax1 = plt.axes([0,.7,.8,.2])\n",
    "node_sample_agglomeration = dendrogram(linkage(reduced_sample_node, metric='cosine', method='average'),orientation='top', color_threshold=0, no_plot=False)['leaves']\n",
    "plt.xticks([])\n",
    "plt.yticks([])\n",
    "ax2 = plt.axes([.8,.35,.2,.33])\n",
    "sample_agglomeration = dendrogram(linkage(reduced_sample, metric='cosine', method='average'),orientation='right', color_threshold=0, no_plot=False)['leaves']\n",
    "plt.xticks([])\n",
    "plt.yticks([])\n",
    "ax3 = plt.axes([0,.35,.8,.33])\n",
    "im1 = plt.imshow(sample_encoding[node_sample_agglomeration].T[sample_agglomeration[::-1]],cmap='binary',aspect='auto',interpolation='none')\n",
    "plt.xticks([])\n",
    "plt.yticks([])\n",
    "plt.ylabel(\"Cells\")\n",
    "ax4 = plt.axes([.8,0,.2,.33])\n",
    "feature_agglomeration = dendrogram(linkage(reduced_feature, metric='cosine', method='average'),orientation='right', color_threshold=0, no_plot=False)['leaves']\n",
    "plt.xticks([])\n",
    "plt.yticks([])\n",
    "ax5 = plt.axes([0,0,.8,.33])\n",
    "# im2 = plt.imshow(feature_encoding[node_sister_agglomeration].T[feature_agglomeration[::-1]],cmap='bwr',vmin=-minimax,vmax=minimax,aspect='auto',interpolation='none')\n",
    "im2 = plt.imshow(feature_encoding[node_sample_agglomeration].T[feature_agglomeration[::-1]],cmap='seismic',vmin=-minimax,vmax=minimax,aspect='auto',interpolation='none')\n",
    "plt.yticks([])\n",
    "plt.xlabel(\"Nodes\")\n",
    "plt.ylabel(\"Genes\")\n",
    "plt.colorbar(im1,ax=ax2,orientation='vertical',label=\"Cell in Node\",shrink=.7)\n",
    "plt.colorbar(im2,ax=ax4,orientation='vertical',label=\"Δ Mean Expression\",shrink=.7)\n",
    "plt.text(-.03,.65,\"A.\",ha='right',transform=plt.gcf().transFigure)\n",
    "plt.text(-.03,.3,\"B.\",ha='right',transform=plt.gcf().transFigure)\n",
    "plt.tight_layout()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure()\n",
    "plt.hist(feature_encoding.flatten())\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Let's try to look at some distance metrics for the sample encodings\n",
    "\n",
    "# from scipy.spatial.distance import cdist,pdist,squareform\n",
    "\n",
    "# cosine_sample = 1 - squareform(pdist(sample_encoding,metric='cosine'))\n",
    "\n",
    "# plt.figure()\n",
    "# plt.imshow(cosine_sample[node_sample_agglomeration].T[node_sample_agglomeration],aspect='auto',cmap='bwr',vmin=-1,vmax=1)\n",
    "# plt.colorbar()\n",
    "# plt.show()\n",
    "\n",
    "# plt.figure()\n",
    "# plt.imshow(cosine_sample[node_feature_agglomeration].T[node_feature_agglomeration],aspect='auto',cmap='bwr',vmin=-1,vmax=1)\n",
    "# plt.colorbar()\n",
    "# plt.show()\n",
    "\n",
    "plt.figure()\n",
    "plt.imshow(cosine_sample[node_cluster_sort].T[node_cluster_sort],aspect='auto',cmap='bwr',vmin=-1,vmax=1)\n",
    "plt.colorbar()\n",
    "plt.show()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# cosine_feature = 1 - squareform(pdist(feature_encoding,metric='cosine'))\n",
    "\n",
    "# plt.figure()\n",
    "# plt.imshow(cosine_feature[node_sample_agglomeration].T[node_sample_agglomeration],aspect='auto',cmap='bwr',vmin=-1,vmax=1)\n",
    "# plt.colorbar()\n",
    "# plt.show()\n",
    "\n",
    "# plt.figure()\n",
    "# plt.imshow(cosine_feature[node_feature_agglomeration].T[node_feature_agglomeration],aspect='auto',cmap='bwr',vmin=-1,vmax=1)\n",
    "# plt.colorbar()\n",
    "# plt.show()\n",
    "\n",
    "plt.figure()\n",
    "plt.imshow(cosine_feature[node_cluster_sort].T[node_cluster_sort],aspect='auto',cmap='bwr',vmin=-1,vmax=1)\n",
    "plt.colorbar()\n",
    "plt.show()\n",
    "\n",
    "# euclidean_feature = squareform(pdist(feature_encoding,metric='euclidean'))\n",
    "\n",
    "# plt.figure()\n",
    "# plt.imshow(euclidean_feature[node_cluster_sort].T[node_cluster_sort],aspect='auto',)\n",
    "# plt.colorbar()\n",
    "# plt.show()\n",
    "\n",
    "# plt.figure()\n",
    "# plt.imshow(euclidean_feature[node_feature_agglomeration].T[node_feature_agglomeration],aspect='auto',)\n",
    "# plt.colorbar()\n",
    "# plt.show()\n",
    "\n",
    "# correlation_feature = 1 - squareform(pdist(feature_encoding,metric='correlation'))\n",
    "\n",
    "# plt.figure()\n",
    "# plt.imshow(correlation_feature[node_feature_agglomeration].T[node_feature_agglomeration],aspect='auto',cmap='bwr',vmin=-1,vmax=1)\n",
    "# plt.colorbar()\n",
    "# plt.show()\n",
    "\n",
    "plt.figure()\n",
    "plt.imshow(correlation_feature[node_cluster_sort].T[node_cluster_sort],aspect='auto',cmap='bwr',vmin=-1,vmax=1)\n",
    "plt.colorbar()\n",
    "plt.show()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally we can look at silhouette plots scores for various node encodings in order to get a feel for whether or not we are adequately clustering them and whether or not the clusters meaningfully exist. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Silhouette Plots For Node Clusters \n",
    "\n",
    "from sklearn.metrics import silhouette_samples, silhouette_score\n",
    "\n",
    "node_labels = np.array([n.split_cluster for n in forest.nodes(root=False)])\n",
    "\n",
    "# silhouette_scores = silhouette_samples(reduced_node,node_labels,metric='cosine')\n",
    "silhouette_scores = silhouette_samples(feature_encoding,node_labels,metric='euclidean')\n",
    "# silhouette_scores = silhouette_samples(sample_encoding,node_labels,metric='cosine')\n",
    "# silhouette_scores = silhouette_samples(sister_encoding,node_labels,metric='cosine')\n",
    "\n",
    "sorted_silhouette = np.zeros(silhouette_scores.shape)\n",
    "sorted_colors = np.zeros(silhouette_scores.shape)\n",
    "\n",
    "current_index = 0\n",
    "next_index = 0\n",
    "for i in sorted(set(node_labels)):\n",
    "    mask = node_labels == i\n",
    "    selected_values = sorted(silhouette_scores[mask])    \n",
    "    next_index = current_index + np.sum(mask)\n",
    "    sorted_silhouette[current_index:next_index] = selected_values\n",
    "    sorted_colors[current_index:next_index] = i\n",
    "    current_index = next_index"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.cm as cm\n",
    "\n",
    "plt.figure()\n",
    "plt.title(\"Silhouette Plots For Nodes Clustered By Gain\")\n",
    "for i,node in enumerate(sorted_silhouette):\n",
    "    plt.plot([0,node],[i,i],color=cm.nipy_spectral(sorted_colors[i] / len(forest.split_clusters)),linewidth=0.5)\n",
    "# plt.scatter(sorted_silhouette,np.arange(len(sorted_silhouette)),s=1)\n",
    "plt.plot([0,0],[0,len(sorted_silhouette)],color='red')\n",
    "plt.xlabel(\"Silhouette Score\")\n",
    "plt.ylabel(\"Nodes\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "node_populations = np.array([n.pop() for n in forest.nodes(root=False)])\n",
    "mask = node_populations > 100\n",
    "\n",
    "\n",
    "feature_encoding = forest.node_representation(forest.nodes(root=False),mode='partial')[mask]\n",
    "reduced_feature = PCA(n_components=100).fit_transform(feature_encoding.T)\n",
    "reduced_node = PCA(n_components=100).fit_transform(feature_encoding)\n",
    "\n",
    "\n",
    "feature_agglomeration = dendrogram(linkage(reduced_feature, metric='cosine', method='average'), no_plot=True)['leaves']\n",
    "node_agglomeration = dendrogram(linkage(reduced_node, metric='cosine', method='average'), no_plot=True)['leaves']\n",
    "\n",
    "node_cluster_sort = np.argsort(np.array([n.split_cluster for n in forest.nodes(root=False)])[mask])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "print(feature_encoding.shape)\n",
    "\n",
    "plt.figure()\n",
    "plt.title(\"Target Gain in Node (Double-Agglomerated)\")\n",
    "plt.imshow(feature_encoding[node_agglomeration].T[feature_agglomeration].T,cmap='bwr',interpolation='none',aspect='auto',vmin=-2,vmax=2)\n",
    "plt.xlabel(\"Genes\")\n",
    "plt.ylabel(\"Nodes\")\n",
    "plt.colorbar(label=\"Parent Mean - Node Mean (Log TPM)\")\n",
    "plt.tight_layout()\n",
    "plt.show()\n",
    "\n",
    "\n",
    "plt.figure()\n",
    "plt.title(\"Target Gain in Node (Clustered)\")\n",
    "plt.imshow(feature_encoding[node_cluster_sort].T[feature_agglomeration].T,cmap='bwr',interpolation='none',aspect='auto',vmin=-2,vmax=2)\n",
    "plt.xlabel(\"Genes\")\n",
    "plt.ylabel(\"Nodes\")\n",
    "plt.colorbar(label=\"Parent Mean - Node Mean (Log TPM)\")\n",
    "plt.tight_layout()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import silhouette_samples, silhouette_score\n",
    "\n",
    "node_labels = np.array([n.split_cluster for n in forest.nodes(root=False)])[mask]\n",
    "\n",
    "# silhouette_scores = silhouette_samples(reduced_node,node_labels,metric='cosine')\n",
    "silhouette_scores = silhouette_samples(feature_encoding,node_labels,metric='euclidean')\n",
    "# silhouette_scores = silhouette_samples(sample_encoding,node_labels,metric='cosine')\n",
    "# silhouette_scores = silhouette_samples(sister_encoding,node_labels,metric='cosine')\n",
    "\n",
    "sorted_silhouette = np.zeros(silhouette_scores.shape)\n",
    "sorted_colors = np.zeros(silhouette_scores.shape)\n",
    "\n",
    "current_index = 0\n",
    "next_index = 0\n",
    "for i in sorted(set(node_labels)):\n",
    "    mask = node_labels == i\n",
    "    selected_values = sorted(silhouette_scores[mask])    \n",
    "    next_index = current_index + np.sum(mask)\n",
    "    sorted_silhouette[current_index:next_index] = selected_values\n",
    "    sorted_colors[current_index:next_index] = i\n",
    "    current_index = next_index"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.cm as cm\n",
    "\n",
    "plt.figure()\n",
    "plt.title(\"Silhouette Plots For Nodes Clustered By Gain\")\n",
    "for i,node in enumerate(sorted_silhouette):\n",
    "    plt.plot([0,node],[i,i],color=cm.nipy_spectral(sorted_colors[i] / len(forest.split_clusters)),linewidth=0.5)\n",
    "# plt.scatter(sorted_silhouette,np.arange(len(sorted_silhouette)),s=1)\n",
    "plt.plot([0,0],[0,len(sorted_silhouette)],color='red')\n",
    "plt.xlabel(\"Silhouette Score\")\n",
    "plt.ylabel(\"Nodes\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.manifold import TSNE"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "reduced_feature.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "trans_node = TSNE(n_components=2,metric='cosine').fit_transform(reduced_node)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure()\n",
    "plt.imshow(reduced_node[node_agglomeration][:,:20],aspect='auto',cmap='bwr',vmin=-20,vmax=20,interpolation='none')\n",
    "plt.colorbar()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure()\n",
    "plt.scatter(*trans_node.T,s=2,c=node_labels,cmap='rainbow')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "from sklearn.cluster import KMeans\n",
    "\n",
    "for i in range(5,50,5):\n",
    "\n",
    "    node_labels = KMeans(i).fit_predict(reduced_node[:,:20])\n",
    "        \n",
    "    silhouette_scores = silhouette_samples(feature_encoding,node_labels,metric='cosine')\n",
    "\n",
    "    sorted_silhouette = np.zeros(silhouette_scores.shape)\n",
    "    sorted_colors = np.zeros(silhouette_scores.shape)\n",
    "\n",
    "    current_index = 0\n",
    "    next_index = 0\n",
    "    for i in sorted(set(node_labels)):\n",
    "        mask = node_labels == i\n",
    "        selected_values = sorted(silhouette_scores[mask])    \n",
    "        next_index = current_index + np.sum(mask)\n",
    "        sorted_silhouette[current_index:next_index] = selected_values\n",
    "        sorted_colors[current_index:next_index] = i\n",
    "        current_index = next_index\n",
    "        \n",
    "        \n",
    "    plt.figure()\n",
    "    plt.title(\"Silhouette Plots For Nodes Clustered By Gain\")\n",
    "    for i,node in enumerate(sorted_silhouette):\n",
    "        plt.plot([0,node],[i,i],color=cm.nipy_spectral(sorted_colors[i] / len(forest.split_clusters)),linewidth=0.5)\n",
    "    # plt.scatter(sorted_silhouette,np.arange(len(sorted_silhouette)),s=1)\n",
    "    plt.plot([0,0],[0,len(sorted_silhouette)],color='red')\n",
    "    plt.xlabel(\"Silhouette Score\")\n",
    "    plt.ylabel(\"Nodes\")\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "len(set(clustered))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "len(clustered)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "optics_sort = np.argsort(clustered)\n",
    "\n",
    "plt.figure()\n",
    "plt.imshow(reduced_node[optics_sort][:,:20],aspect='auto',cmap='bwr',vmin=-20,vmax=20,interpolation='none')\n",
    "plt.colorbar()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Clustering Explanatory Power"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "feature_encoding = forest.node_representation(forest.nodes(root=False),mode='additive_mean')\n",
    "labels = np.array([n.split_cluster for n in forest.nodes(root=False)])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "feature_encoding.shape\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "remaining = 0\n",
    "\n",
    "for cluster in sorted(list(set(labels))):\n",
    "    mask = labels == cluster\n",
    "    means = np.mean(feature_encoding[mask],axis=0)\n",
    "    residuals = feature_encoding[mask] - means\n",
    "    mse = np.sum(np.power(residuals,2)) / (np.sum(mask) * feature_encoding.shape[1])\n",
    "    remaining += (np.sum(mask) / feature_encoding.shape[0]) * mse\n",
    "    print((cluster,mse))\n",
    "    \n",
    "means = np.mean(feature_encoding,axis=0)\n",
    "residuals = feature_encoding - means\n",
    "mse = np.sum(np.power(residuals,2)) / (feature_encoding.shape[0] * feature_encoding.shape[1])\n",
    "print(f\"Remaining: {remaining}\")\n",
    "print(f\"All:{mse}\")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tree_reader_utils import fast_knn,hacked_louvain\n",
    "\n",
    "shuffled = feature_encoding.copy()\n",
    "\n",
    "for f in shuffled.T:\n",
    "    np.random.shuffle(f)\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "relabel = hacked_louvain(fast_knn(shuffled,50))\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "remaining_sims = []\n",
    "\n",
    "for i in range(20):\n",
    "    print(i)\n",
    "    \n",
    "    shuffled = feature_encoding.copy()\n",
    "\n",
    "    for f in shuffled.T:\n",
    "        np.random.shuffle(f)\n",
    "\n",
    "    relabel = hacked_louvain(fast_knn(shuffled,50))\n",
    "\n",
    "    remaining = 0\n",
    "    for cluster in sorted(list(set(relabel))):\n",
    "        mask = relabel == cluster\n",
    "        means = np.mean(shuffled[mask],axis=0)\n",
    "        residuals = shuffled[mask] - means\n",
    "        mse = np.sum(np.power(residuals,2)) / (np.sum(mask) * shuffled.shape[1])\n",
    "        remaining += (np.sum(mask) / shuffled.shape[0]) * mse\n",
    "#         print((cluster,mse))\n",
    "    remaining_sims.append(remaining)\n",
    "    \n",
    "means = np.mean(shuffled,axis=0)\n",
    "residuals = shuffled - means\n",
    "mse = np.sum(np.power(residuals,2)) / (shuffled.shape[0] * shuffled.shape[1])\n",
    "print(f\"Remaining: {remaining}\")\n",
    "print(f\"All:{mse}\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Now for sample encoding"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "nodes = forest.nodes(root=False)\n",
    "\n",
    "sample_encoding = forest.node_representation(nodes,mode='sample')\n",
    "labels = np.array([n.split_cluster for n in nodes])\n",
    "\n",
    "sample_encoding.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "remaining = 0\n",
    "\n",
    "for cluster in sorted(list(set(labels))):\n",
    "    mask = labels == cluster\n",
    "    means = np.mean(sample_encoding[mask],axis=0)\n",
    "    residuals = sample_encoding[mask] - means\n",
    "    mse = np.sum(np.power(residuals,2)) / (np.sum(mask) * sample_encoding.shape[1])\n",
    "    remaining += (np.sum(mask) / sample_encoding.shape[0]) * mse\n",
    "    print((cluster,mse))\n",
    "    print( np.sum(mask))\n",
    "    \n",
    "means = np.mean(sample_encoding,axis=0)\n",
    "residuals = sample_encoding - means\n",
    "mse = np.sum(np.power(residuals,2)) / (sample_encoding.shape[0] * sample_encoding.shape[1])\n",
    "print(f\"Remaining: {remaining}\")\n",
    "print(f\"All:{mse}\")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tree_reader_utils import fast_knn,hacked_louvain\n",
    "\n",
    "shuffled = feature_encoding.copy()\n",
    "\n",
    "for f in shuffled.T:\n",
    "    np.random.shuffle(f)\n",
    "\n",
    "relabel = hacked_louvain(fast_knn(shuffled,50))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "remaining = 0\n",
    "for cluster in sorted(list(set(relabel))):\n",
    "    mask = relabel == cluster\n",
    "    means = np.mean(shuffled[mask],axis=0)\n",
    "    residuals = shuffled[mask] - means\n",
    "    mse = np.sum(np.power(residuals,2)) / (np.sum(mask) * shuffled.shape[1])\n",
    "    remaining += (np.sum(mask) / shuffled.shape[0]) * mse\n",
    "    print((cluster,mse))\n",
    "remaining_sims.append(remaining)\n",
    "\n",
    "means = np.mean(shuffled,axis=0)\n",
    "residuals = shuffled - means\n",
    "mse = np.sum(np.power(residuals,2)) / (shuffled.shape[0] * shuffled.shape[1])\n",
    "print(f\"Remaining: {remaining}\")\n",
    "print(f\"All:{mse}\")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.var(remaining_sims)\n",
    "# np.mean(remaining_sims)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "remaining_sims"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tn = forest.trees[0].root.nodes()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "len(tn)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "additive = forest.node_representation(tn,mode='additive_mean')\n",
    "sample = forest.node_representation(tn,mode='sample')\n",
    "\n",
    "pops = np.sum(sample,axis=1)\n",
    "pops.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure()\n",
    "plt.imshow(additive,aspect='auto',cmap='bwr')\n",
    "plt.show()\n",
    "\n",
    "plt.figure()\n",
    "plt.imshow(sample,aspect='auto',cmap='binary')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.dot(additive.T,pops) # equals zero, law of total expectation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "explained = np.dot(np.power(additive.T,2),pops) / 16027"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "total = np.var(forest.output,axis=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "explained / total"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "nse = forest.node_representation(forest.nodes(),mode='sample')\n",
    "nge = forest.node_representation(forest.nodes(),mode='additive_mean')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "nse.shape\n",
    "nge.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ncv = np.cov(nse.T,nge.T)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ncv.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}

