{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pwd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Existence Of Node Clusters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we demonstrate that in random forest that has been trained on some set of data, the nodes can be reasonably organized into clusters.\n",
    "\n",
    "First, we must train or load a forest:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import scanpy as sc\n",
    "from scipy.stats import multivariate_normal\n",
    "\n",
    "import sys\n",
    "# sys.path.append('/localscratch/bbrener1/rusty_forest_v3/src')\n",
    "sys.path.append('../../')\n",
    "import rusty_axe.lumberjack as lumberjack\n",
    "\n",
    "data_location = \"/home/boris/battle/rf_5/data/iris/\"\n",
    "\n",
    "iris = np.loadtxt(data_location + \"iris.tsv\")\n",
    "iris_features = np.loadtxt(data_location + \"header.txt\",dtype=str)\n",
    "\n",
    "iris_forest = lumberjack.fit(\n",
    "    iris,\n",
    "    trees=100,\n",
    "    ifs=3,\n",
    "    ofs=3,\n",
    "    ss=100,\n",
    "    leaves=10,\n",
    "    depth=3,\n",
    "    dispersion_mode='ssme',\n",
    "    sfr=0,\n",
    "    norm='l1',\n",
    "    reduction = 4,\n",
    "    standardize='true',\n",
    "    reduce_input='true',\n",
    "    reduce_output='true',\n",
    ")\n",
    "\n",
    "shuffled = iris.copy()\n",
    "\n",
    "for column in shuffled.T:\n",
    "    np.random.shuffle(column)\n",
    "\n",
    "shuffled_forest = lumberjack.fit(\n",
    "    shuffled,\n",
    "    trees=100,\n",
    "    ifs=3,\n",
    "    ofs=3,\n",
    "    ss=100,\n",
    "    leaves=10,\n",
    "    depth=3,\n",
    "    dispersion_mode='ssme',\n",
    "    sfr=0,\n",
    "    norm='l1',\n",
    "    reduction = 4,\n",
    "    standardize='true',\n",
    "    reduce_input='true',\n",
    "    reduce_output='true',\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "iris_class = np.loadtxt(data_location + \"class.txt\",dtype='str')\n",
    "iris_header = np.zeros(150,dtype=int)\n",
    "iris_header[iris_class == \"Iris-setosa\"] = 0\n",
    "iris_header[iris_class == \"Iris-versicolor\"] = 1\n",
    "iris_header[iris_class == \"Iris-virginica\"] = 2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A Random Forest is a collection of decision trees, and a decision tree is a collection of individual decision points, commonly known as \"Nodes\"\n",
    "\n",
    "To understand Random Forests and Decision Trees, it is important to understand how Nodes work. Each individual node is a (very crappy) regressor, eg. each Node makess a prediction based on a rule like \"If Gene 1 has expression > 10, Gene 2 will have expression < 5\", or \"If a house is < 5 miles from a school, it will cost > $100,000\". A very important property of each node, however, is that it can also have children, which are other nodes. When a node makes a prediction like \"If Gene 1 has expression > 10 then Gene 2 has expression < 5\", it can pass all the samples for which Gene 1 is > 10 to one of its children, and all the samples for which Gene 1 < 10 to the other child. After that, each one of its children can make a different prediction, which results in compound rules.\n",
    "\n",
    "This is how a decision tree is formed. A decision tree with a depth of 2 might contain a rule like \"If Gene 1 > 10 AND Gene 3 > 10, THEN Gene 2 and Gene 4 are both < 2, which would represent one of the \"Leaf\" nodes that it has. Leaf nodes are nodes with no children. \n",
    "\n",
    "Individual decision trees, then, are somewhat crappy predictors, but they're better than individual nodes. In order to improve the performance of decision trees, we can construct a Random Forest. To construct a random forest, we can train many decision trees on bootstraps of a dataset\n",
    "\n",
    "If many decision trees are combined and their predictions averaged together, you have a Random Forest, which is a pretty good kind of regressor. \n",
    "\n",
    "A practical demonstration might help:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "iris_forest.reset_split_clusters()\n",
    "iris_forest.interpret_splits(depth=3,mode='partial',metric='cosine',relatives=True,k=20)\n",
    "\n",
    "\n",
    "shuffled_forest.reset_split_clusters()\n",
    "shuffled_forest.interpret_splits(depth=3,mode='partial',metric='cosine',relatives=True,k=20)\n",
    "\n",
    "iris_forest.maximum_spanning_tree(mode='samples')\n",
    "\n",
    "iris_forest.html_tree_summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So now that we know that random forests are collections of ordered nodes, we can examine a more interesting question: do certain nodes occur repeatedly in the forest, despite operating on bootstrapped samples? \n",
    "\n",
    "In order to examine this question first we must understand different ways of describing a node. I think generally there are three helpful ways of looking at a node:\n",
    "\n",
    "* **Node Sample Encoding**: A binary vector the length of the number of samples you are considering. 0 or false means the sample is absent from the node. A 1 or true means the sample is present in the node. \n",
    "\n",
    "* **Node Mean Encoding**: A float vector the length of the number of targets you are considering. Each value is the mean of the target values for all samples in this node. This is the node's prediction for samples that occur in it.\n",
    "\n",
    "* **Node Additive Encoding**: A float vector the length of the number of targets you are considering. Each value is THE DIFFERENCE between the mean value for that target in THIS NODE and the mean value for that target IN THE PARENT of this node. For root nodes, which have no parents, the additive encoding is simply th mean value across the entire dataset. (As if the mean of a hypothetical parent would have been 0). This encoding represents the marginal effect of each node.\n",
    "\n",
    "We should examine if there are any common patterns that appear if we encode many nodes from a forest using each of these representations:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "# Here we plot the sample representations of nodes for the iris forest. \n",
    "# This generates a set of figures demonstrating the existence of node clusters\n",
    "\n",
    "# from sklearn.decomposition import PCA\n",
    "\n",
    "# sample_encoding = iris_forest.node_representation(iris_forest.nodes(depth=3,root=False),mode='sample')\n",
    "# sister_encoding = iris_forest.node_representation(iris_forest.nodes(depth=3,root=False),mode='sister')\n",
    "# reduced_sample = PCA(n_components=100).fit_transform(sample_encoding.T)\n",
    "# reduced_sister = PCA(n_components=100).fit_transform(sister_encoding.T)\n",
    "# # reduced_node = PCA(n_components=100).fit_transform(sample_encoding)\n",
    "# reduced_node = PCA(n_components=100).fit_transform(sister_encoding)\n",
    "\n",
    "# print(sample_encoding.shape)\n",
    "# print(reduced_sample.shape)\n",
    "# print(reduced_node.shape)\n",
    "\n",
    "# from scipy.cluster.hierarchy import linkage,dendrogram\n",
    "\n",
    "# sample_agglomeration = dendrogram(linkage(reduced_sample, metric='cosine', method='average'), no_plot=True)['leaves']\n",
    "# sister_agglomeration = dendrogram(linkage(reduced_sister, metric='cosine', method='average'), no_plot=True)['leaves']\n",
    "# node_agglomeration = dendrogram(linkage(reduced_node, metric='cosine', method='average'), no_plot=True)['leaves']\n",
    "\n",
    "# plt.figure()\n",
    "# plt.title(\"Iris Sample Presence in Node (Two-Way Agglomerated)\")\n",
    "# plt.imshow(sample_encoding[node_agglomeration].T[sample_agglomeration].T,cmap='binary',aspect='auto',interpolation='none')\n",
    "# plt.xlabel(\"Samples\")\n",
    "# plt.ylabel(\"Nodes\")\n",
    "# plt.colorbar(label=\"Presence In Node\")\n",
    "# plt.tight_layout()\n",
    "# plt.show()\n",
    "\n",
    "# plt.figure()\n",
    "# plt.title(\"Iris Sample Presence in Node (Two-Way Agglomerated)\")\n",
    "# plt.imshow(sister_encoding[node_agglomeration].T[sister_agglomeration].T,cmap='bwr',aspect='auto',interpolation='none')\n",
    "# plt.xlabel(\"Samples\")\n",
    "# plt.ylabel(\"Nodes\")\n",
    "# plt.colorbar(label=\"Presence In Node\")\n",
    "# plt.tight_layout()\n",
    "# plt.show()\n",
    "\n",
    "\n",
    "# plt.figure()\n",
    "# plt.title(\"Iris Sample Presence in Node (Agglomerated)\")\n",
    "# plt.imshow(sample_encoding[node_agglomeration],cmap='binary',aspect='auto',interpolation='none')\n",
    "# plt.xlabel(\"Samples\")\n",
    "# plt.ylabel(\"Nodes\")\n",
    "# plt.colorbar(label=\"Presence In Node\")\n",
    "# plt.tight_layout()\n",
    "# plt.show()\n",
    "\n",
    "# plt.figure()\n",
    "# plt.title(\"Iris Sample Presence in Node (Agglomerated)\")\n",
    "# plt.imshow(sister_encoding[node_agglomeration],cmap='bwr',aspect='auto',interpolation='none')\n",
    "# plt.xlabel(\"Samples\")\n",
    "# plt.ylabel(\"Nodes\")\n",
    "# plt.colorbar(label=\"Presence In Node\")\n",
    "# plt.tight_layout()\n",
    "# plt.show()\n",
    "\n",
    "plt.figure()\n",
    "# plt.suptitle(\"Iris Sample Presence in Node (Agglomerated)\")\n",
    "ax1 = plt.axes([0,.1,.9,.9])\n",
    "im1 = plt.imshow(sister_encoding[node_agglomeration],cmap='bwr',aspect='auto',interpolation='none')\n",
    "plt.xlabel(\"Samples\")\n",
    "plt.ylabel(\"Nodes\")\n",
    "plt.xticks([])\n",
    "ax2 = plt.axes([0,0,.9,.1])\n",
    "plt.imshow(np.array([iris_header,]),aspect='auto',interpolation='none',cmap='Set3')\n",
    "plt.ylabel(\"Species\")\n",
    "plt.xlabel(\"Samples\")\n",
    "plt.yticks([])\n",
    "ax3 = plt.axes([.92,.1,.06,.9])\n",
    "plt.colorbar(im1,cax=ax3,label=\"Presence In Node\")\n",
    "# plt.tight_layout()\n",
    "plt.show()\n",
    "\n",
    "# And here we sort the nodes after they have been clustered (more on the clustering procedure in a bit)\n",
    "\n",
    "# node_cluster_sort = np.argsort([n.split_cluster for n in iris_forest.nodes(depth=3,root=False)])\n",
    "\n",
    "# plt.figure()\n",
    "# plt.title(\"Sample Presence in Node (Clustered by Gain)\")\n",
    "# plt.imshow(sample_encoding[node_cluster_sort],cmap='binary',aspect='auto',interpolation='none')\n",
    "# plt.xlabel(\"Samples\")\n",
    "# plt.ylabel(\"Nodes\")\n",
    "# plt.colorbar(label=\"Presence In Node\")\n",
    "# plt.tight_layout()\n",
    "# plt.show()\n",
    "\n",
    "# plt.figure()\n",
    "# plt.title(\"Sample Presence in Node (Clustered by Gain)\")\n",
    "# plt.imshow(sister_encoding[node_cluster_sort],cmap='bwr',aspect='auto',interpolation='none')\n",
    "# plt.xlabel(\"Samples\")\n",
    "# plt.ylabel(\"Nodes\")\n",
    "# plt.colorbar(label=\"Presence In Node\")\n",
    "# plt.tight_layout()\n",
    "# plt.show()\n",
    "\n",
    "# plt.figure(figsize=(4,1))\n",
    "# plt.imshow(np.array([iris_header,]),aspect='auto',interpolation='none',cmap='rainbow')\n",
    "# plt.xlabel(\"Samples\")\n",
    "# plt.yticks([])\n",
    "# plt.title(\"Iris Species By Color\")\n",
    "# plt.show()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "# Here we plot the sample representations of nodes for the iris forest. \n",
    "# This generates a set of figures demonstrating the existence of node clusters\n",
    "\n",
    "from sklearn.decomposition import PCA\n",
    "\n",
    "sample_encoding = spherical_forest.node_representation(spherical_forest.nodes(depth=3,root=False),mode='sample')\n",
    "sister_encoding = spherical_forest.node_representation(spherical_forest.nodes(depth=3,root=False),mode='sister')\n",
    "reduced_sample = PCA(n_components=100).fit_transform(sample_encoding.T)\n",
    "reduced_sister = PCA(n_components=100).fit_transform(sister_encoding.T)\n",
    "# reduced_node = PCA(n_components=100).fit_transform(sample_encoding)\n",
    "reduced_node = PCA(n_components=100).fit_transform(sister_encoding)\n",
    "\n",
    "print(sample_encoding.shape)\n",
    "print(reduced_sample.shape)\n",
    "print(reduced_node.shape)\n",
    "\n",
    "from scipy.cluster.hierarchy import linkage,dendrogram\n",
    "\n",
    "sample_agglomeration = dendrogram(linkage(reduced_sample, metric='cosine', method='average'), no_plot=True)['leaves']\n",
    "sister_agglomeration = dendrogram(linkage(reduced_sister, metric='cosine', method='average'), no_plot=True)['leaves']\n",
    "node_agglomeration = dendrogram(linkage(reduced_node, metric='cosine', method='average'), no_plot=True)['leaves']\n",
    "node_cluster_sort = np.argsort([n.split_cluster for n in spherical_forest.nodes(depth=3,root=False)])\n",
    "\n",
    "plt.figure()\n",
    "plt.title(\"Gaussian Noise Sample Presence in Node \\n(Two-Way Agglomerated)\")\n",
    "plt.imshow(sample_encoding[node_agglomeration].T[sample_agglomeration].T,cmap='binary',aspect='auto',interpolation='none')\n",
    "plt.xlabel(\"Samples\")\n",
    "plt.ylabel(\"Nodes\")\n",
    "plt.colorbar(label=\"Presence In Node\")\n",
    "plt.tight_layout()\n",
    "plt.show()\n",
    "\n",
    "plt.figure()\n",
    "plt.title(\"Gaussian Noise Sample Presence in Node vs Sister \\n(Two-Way Agglomerated)\")\n",
    "plt.imshow(sister_encoding[node_agglomeration].T[sister_agglomeration].T,cmap='bwr',aspect='auto',interpolation='none')\n",
    "plt.xlabel(\"Samples\")\n",
    "plt.ylabel(\"Nodes\")\n",
    "plt.colorbar(label=\"Presence In Node\")\n",
    "plt.tight_layout()\n",
    "plt.show()\n",
    "\n",
    "# And here we sort the nodes after they have been clustered (more on the clustering procedure in a bit)\n",
    "\n",
    "\n",
    "plt.figure()\n",
    "plt.title(\"Gaussian Noise Sample Presence in Node (Clustered by Gain)\")\n",
    "plt.imshow(sample_encoding[node_cluster_sort].T[sample_agglomeration].T,cmap='binary',aspect='auto',interpolation='none')\n",
    "plt.xlabel(\"Samples\")\n",
    "plt.ylabel(\"Nodes\")\n",
    "plt.colorbar(label=\"Presence In Node\")\n",
    "plt.tight_layout()\n",
    "plt.show()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "from sklearn.decomposition import PCA\n",
    "\n",
    "sample_encoding = iris_forest.node_representation(iris_forest.nodes(depth=3,root=False),mode='sister')\n",
    "reduced_sample = PCA(n_components=100).fit_transform(sample_encoding.T)\n",
    "reduced_node = PCA(n_components=100).fit_transform(sample_encoding)\n",
    "\n",
    "print(sample_encoding.shape)\n",
    "print(reduced_sample.shape)\n",
    "print(reduced_node.shape)\n",
    "\n",
    "from scipy.cluster.hierarchy import linkage,dendrogram\n",
    "\n",
    "sample_agglomeration = dendrogram(linkage(reduced_sample, metric='cosine', method='average'), no_plot=True)['leaves']\n",
    "node_agglomeration = dendrogram(linkage(reduced_node, metric='cosine', method='average'), no_plot=True)['leaves']\n",
    "\n",
    "node_cluster_sort = np.argsort([n.split_cluster for n in iris_forest.nodes(depth=3,root=False)])\n",
    "\n",
    "plt.figure()\n",
    "plt.title(\"Iris Sample Presence in Node vs Sister (Two-Way Agglomerated)\")\n",
    "plt.imshow(sample_encoding[node_agglomeration].T[sample_agglomeration].T,cmap='bwr',aspect='auto',interpolation='none')\n",
    "plt.xlabel(\"Samples\")\n",
    "plt.ylabel(\"Nodes\")\n",
    "plt.colorbar()\n",
    "plt.tight_layout()\n",
    "plt.show()\n",
    "\n",
    "plt.figure()\n",
    "plt.title(\"Iris Sample Presence in Node vs Sister \\n(Clustered By Gain)\")\n",
    "plt.imshow(sample_encoding[node_cluster_sort].T[sample_agglomeration].T,cmap='bwr',aspect='auto',interpolation='none')\n",
    "plt.xlabel(\"Samples\")\n",
    "plt.ylabel(\"Nodes\")\n",
    "plt.colorbar()\n",
    "plt.tight_layout()\n",
    "plt.show()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Here we plot the construct and agglomerate the additive gain representation \n",
    "\n",
    "\n",
    "sample_encoding = iris_forest.node_representation(iris_forest.nodes(depth=3,root=False),mode='sample')\n",
    "reduced_sample = PCA(n_components=100).fit_transform(sample_encoding.T)\n",
    "reduced_node = PCA(n_components=100).fit_transform(sample_encoding)\n",
    "\n",
    "sample_agglomeration = dendrogram(linkage(reduced_sample, metric='cosine', method='average'), no_plot=True)['leaves']\n",
    "node_agglomeration = dendrogram(linkage(reduced_node, metric='cosine', method='average'), no_plot=True)['leaves']\n",
    "\n",
    "feature_encoding = iris_forest.node_representation(iris_forest.nodes(depth=3,root=False),mode='additive_mean')\n",
    "reduced_feature = PCA().fit_transform(feature_encoding.T)\n",
    "reduced_node = PCA().fit_transform(feature_encoding)\n",
    "\n",
    "feature_agglomeration = dendrogram(linkage(reduced_feature, metric='cosine', method='average'), no_plot=True)['leaves']\n",
    "node_agglomeration = dendrogram(linkage(reduced_node, metric='cosine', method='average'), no_plot=True)['leaves']\n",
    "\n",
    "node_cluster_sort = np.argsort([n.split_cluster for n in iris_forest.nodes(depth=3,root=False)])\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Here we plot the additive gain representation \n",
    "\n",
    "# print(feature_encoding.shape)\n",
    "\n",
    "# plt.figure()\n",
    "# plt.title(\"Target Gain in Node (Double-Agglomerated)\")\n",
    "# plt.imshow(feature_encoding[node_agglomeration].T[feature_agglomeration].T,cmap='bwr',interpolation='none',aspect='auto',vmin=-2,vmax=2)\n",
    "# plt.xlabel(\"Features\")\n",
    "# plt.ylabel(\"Nodes\")\n",
    "# plt.colorbar(label=\"Parent Target Mean - Node Target Mean\")\n",
    "# plt.tight_layout()\n",
    "# plt.show()\n",
    "\n",
    "\n",
    "plt.figure(figsize=(5,3))\n",
    "plt.title(\"Target Gain in Node (Clustered)\")\n",
    "plt.imshow(feature_encoding[node_cluster_sort].T[feature_agglomeration].T,cmap='bwr',interpolation='none',aspect='auto',vmin=-2,vmax=2)\n",
    "plt.xlabel(\"Features\")\n",
    "plt.ylabel(\"Nodes\")\n",
    "plt.colorbar(label=\"Parent Mean - Node Mean\")\n",
    "plt.xticks(np.arange(4),labels=iris_features,rotation=20)\n",
    "plt.tight_layout()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# iris_forest.html_tree_summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally we can look at silhouette plots scores for various node encodings in order to get a feel for whether or not we are adequately clustering them and whether or not the clusters meaningfully exist. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.decomposition import PCA\n",
    "from scipy.cluster.hierarchy import linkage,dendrogram\n",
    "\n",
    "\n",
    "sample_encoding = spherical_forest.node_representation(spherical_forest.nodes(depth=3,root=False),mode='sample')\n",
    "reduced_sample = PCA(n_components=100).fit_transform(sample_encoding.T)\n",
    "reduced_node = PCA(n_components=100).fit_transform(sample_encoding)\n",
    "\n",
    "sample_agglomeration = dendrogram(linkage(reduced_sample, metric='cosine', method='average'), no_plot=True)['leaves']\n",
    "node_agglomeration = dendrogram(linkage(reduced_node, metric='cosine', method='average'), no_plot=True)['leaves']\n",
    "\n",
    "feature_encoding = spherical_forest.node_representation(spherical_forest.nodes(depth=3,root=False),mode='additive_mean')\n",
    "reduced_feature = PCA().fit_transform(feature_encoding.T)\n",
    "reduced_node = PCA().fit_transform(feature_encoding)\n",
    "\n",
    "feature_agglomeration = dendrogram(linkage(reduced_feature, metric='cosine', method='average'), no_plot=True)['leaves']\n",
    "node_agglomeration = dendrogram(linkage(reduced_node, metric='cosine', method='average'), no_plot=True)['leaves']\n",
    "\n",
    "node_cluster_sort = np.argsort([n.split_cluster for n in spherical_forest.nodes(depth=3,root=False)])\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# plt.figure()\n",
    "# plt.title(\"Target Gain in Node (Gaussian Noise, Double-Agglomerated)\")\n",
    "# plt.imshow(feature_encoding[node_agglomeration].T[feature_agglomeration].T,cmap='bwr',interpolation='none',aspect='auto',vmin=-2,vmax=2)\n",
    "# plt.xlabel(\"Features\")\n",
    "# plt.ylabel(\"Nodes\")\n",
    "# plt.colorbar(label=\"Parent Target Mean - Node Target Mean\")\n",
    "# plt.tight_layout()\n",
    "# plt.show()\n",
    "\n",
    "original_clusters = np.array([n.split_cluster for n in spherical_forest.nodes(depth=3,root=False)])\n",
    "renumbered_clusters = original_clusters.copy()\n",
    "renumbered_clusters[original_clusters == 1] = 3\n",
    "renumbered_clusters[original_clusters == 2] = 4\n",
    "renumbered_clusters[original_clusters == 3] = 1\n",
    "renumbered_clusters[original_clusters == 4] = 2\n",
    "sort_renumbered = np.argsort(renumbered_clusters)\n",
    "\n",
    "plt.figure()\n",
    "plt.title(\"Target Gain in Node (Gaussian Noise, Clustered)\")\n",
    "plt.imshow(feature_encoding[sort_renumbered].T[feature_agglomeration].T,cmap='bwr',interpolation='none',aspect='auto',vmin=-2,vmax=2)\n",
    "plt.xlabel(\"Features\")\n",
    "plt.ylabel(\"Nodes\")\n",
    "plt.colorbar(label=\"Parent Target Mean - Node Target Mean\")\n",
    "plt.xticks([0,1,2,3],[1,2,3,4])\n",
    "plt.tight_layout()\n",
    "plt.show()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Node mean encoding\n",
    "\n",
    "mean_encoding = iris_forest.node_representation(iris_forest.nodes(depth=3,root=False),mode='mean')\n",
    "\n",
    "node_agglomeration = dendrogram(linkage(mean_encoding, metric='cosine', method='average'), no_plot=True)['leaves']\n",
    "mean_agglomeration = dendrogram(linkage(mean_encoding.T, metric='cosine', method='average'), no_plot=True)['leaves']\n",
    "\n",
    "plt.figure()\n",
    "plt.title(\"Figure S2 a: Target Mean in Node (Gaussian Noise, Double-Agglomerated)\")\n",
    "plt.imshow(mean_encoding[node_agglomeration].T[mean_agglomeration].T,cmap='viridis',interpolation='none',aspect='auto')\n",
    "plt.xlabel(\"Features\")\n",
    "plt.ylabel(\"Nodes\")\n",
    "plt.colorbar(label=\"Parent Target Mean - Node Target Mean\")\n",
    "plt.tight_layout()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Silhouette Plots For Node Clusters \n",
    "\n",
    "from sklearn.metrics import silhouette_samples, silhouette_score\n",
    "\n",
    "# feature_encoding = iris_forest.node_representation(iris_forest.nodes(depth=3,root=False),mode='additive_mean')\n",
    "# node_labels = np.array([n.split_cluster for n in iris_forest.nodes(depth=3,root=False)])\n",
    "\n",
    "feature_encoding = spherical_forest.node_representation(spherical_forest.nodes(depth=3,root=False),mode='additive_mean')\n",
    "node_labels = np.array([n.split_cluster for n in spherical_forest.nodes(depth=3,root=False)])\n",
    "\n",
    "silhouette_scores = silhouette_samples(feature_encoding,node_labels,metric='cosine')\n",
    "\n",
    "sorted_silhouette = np.zeros(silhouette_scores.shape)\n",
    "sorted_colors = np.zeros(silhouette_scores.shape)\n",
    "sorted_indices = []\n",
    "\n",
    "current_index = 0\n",
    "next_index = 0\n",
    "for i in sorted(set(node_labels)):\n",
    "    mask = node_labels == i\n",
    "#     selected_values = sorted(silhouette_scores[mask])    \n",
    "    value_sort = np.argsort(silhouette_scores[mask])\n",
    "    selected_values = silhouette_scores[mask][value_sort]\n",
    "    sorted_local_indices = np.arange(len(silhouette_scores))[mask][value_sort]\n",
    "    sorted_indices.extend(sorted_local_indices)\n",
    "    next_index = current_index + np.sum(mask)\n",
    "    sorted_silhouette[current_index:next_index] = selected_values\n",
    "    sorted_colors[current_index:next_index] = i\n",
    "    current_index = next_index"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.cm as cm\n",
    "\n",
    "plt.figure()\n",
    "plt.title(\"Silhouette Plots For Node-Gain Encodings \\n Clustered By Gain\")\n",
    "for i,node in enumerate(sorted_silhouette):\n",
    "    plt.plot([0,node],[i,i],color=cm.nipy_spectral(sorted_colors[i] / len(iris_forest.split_clusters)))\n",
    "# plt.scatter(sorted_silhouette,np.arange(len(sorted_silhouette)),s=1)\n",
    "plt.plot([0,0],[0,len(sorted_silhouette)],color='black')\n",
    "plt.xlabel(\"Silhouette Score\")\n",
    "plt.ylabel(\"Nodes\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure()\n",
    "plt.title(\"Figure S2 b: Target Gain in Node (Clustered)\")\n",
    "plt.imshow(feature_encoding[sorted_indices].T[feature_agglomeration].T,cmap='bwr',interpolation='none',aspect='auto',vmin=-2,vmax=2)\n",
    "plt.xlabel(\"Features\")\n",
    "plt.ylabel(\"Nodes\")\n",
    "plt.colorbar(label=\"Parent Target Mean - Node Target Mean\")\n",
    "plt.tight_layout()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Sample silhouettes \n",
    "\n",
    "\n",
    "from sklearn.metrics import silhouette_samples, silhouette_score\n",
    "\n",
    "sample_encoding = iris_forest.node_representation(iris_forest.nodes(depth=3,root=False),mode='sister')\n",
    "node_labels = np.array([n.split_cluster for n in iris_forest.nodes(depth=3,root=False)])\n",
    "\n",
    "# sample_encoding = spherical_forest.node_representation(spherical_forest.nodes(depth=3,root=False),mode='sister')\n",
    "# node_labels = np.array([n.split_cluster for n in spherical_forest.nodes(depth=3,root=False)])\n",
    "\n",
    "silhouette_scores = silhouette_samples(sample_encoding,node_labels,metric='cosine')\n",
    "\n",
    "sorted_silhouette = np.zeros(silhouette_scores.shape)\n",
    "sorted_colors = np.zeros(silhouette_scores.shape)\n",
    "sorted_indices = []\n",
    "\n",
    "current_index = 0\n",
    "next_index = 0\n",
    "for i in sorted(set(node_labels)):\n",
    "    mask = node_labels == i\n",
    "    value_sort = np.argsort(silhouette_scores[mask])\n",
    "    selected_values = silhouette_scores[mask][value_sort]\n",
    "    sorted_local_indices = np.arange(len(silhouette_scores))[mask][value_sort]\n",
    "    sorted_indices.extend(sorted_local_indices)\n",
    "    next_index = current_index + np.sum(mask)\n",
    "    sorted_silhouette[current_index:next_index] = selected_values\n",
    "    sorted_colors[current_index:next_index] = i\n",
    "    current_index = next_index"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.cm as cm\n",
    "\n",
    "plt.figure()\n",
    "plt.title(\"Silhouette Plots For Node-Sister Encodings \\n Clustered By Gain\")\n",
    "for i,node in enumerate(sorted_silhouette):\n",
    "    plt.plot([0,node],[i,i],color=cm.nipy_spectral(sorted_colors[i] / len(iris_forest.split_clusters)))\n",
    "# plt.scatter(sorted_silhouette,np.arange(len(sorted_silhouette)),s=1)\n",
    "plt.plot([0,0],[0,len(sorted_silhouette)],color='black')\n",
    "plt.ylim(1180,0)\n",
    "plt.xlabel(\"Silhouette Score\")\n",
    "plt.ylabel(\"Nodes\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure()\n",
    "plt.title(\"Figure S2 b: Target Gain in Node (Clustered)\")\n",
    "plt.imshow(sample_encoding[sorted_indices].T[sample_agglomeration].T,cmap='bwr',interpolation='none',aspect='auto')\n",
    "plt.xlabel(\"Features\")\n",
    "plt.ylabel(\"Nodes\")\n",
    "plt.colorbar(label=\"Parent Target Mean - Node Target Mean\")\n",
    "plt.tight_layout()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sample_encoding.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sorted_indices"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "list(silhouette_scores[sorted_indices])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pa = iris_forest.node_representation(iris_forest.nodes(),mode='partial_absolute')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.sum(np.abs(pa),axis=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "[r.index for r in iris_forest.roots()]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pa[899]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.around(iris_forest.node_representation(iris_forest.trees[0].nodes(),mode='partial_absolute'),3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "iris_forest.trees[0].root.nodes()[-1].level"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
