{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Simulations\n",
    "\n",
    "In this notebook we wish to run a simulation demonstrating some of the basic claims we make regarding the random \n",
    "forest. \n",
    "\n",
    "The key claims we would like to demonstrate are thus:\n",
    "\n",
    "- A dataset can have heirarchal behavior\n",
    "    - an RF will identify such hierarchal structure \n",
    "    - an RF will capture local changes in covariance etc\n",
    "    \n",
    "    - A PCA CANNOT capture some of the effects that we will identify as local in distinct PCs.\n",
    "\n",
    "- When a dataset undergoes changes in population prevalence, we identify this as a shift in factor values\n",
    "\n",
    "- When a dataset undergoes a change in population behavior we identify this as a shift in predictive power"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To reflect a hierarchal structure with meaningful local behavior, we will need several features that have different means among different clusters, but importantly also interact with each other, especially in different ways within different clusters. \n",
    "\n",
    "Let's operate on 10 features total. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## On The Basis of Component Vectors"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "from scipy.stats import multivariate_normal,norm,beta\n",
    "from sklearn.datasets import make_blobs\n",
    "from sklearn.manifold import TSNE\n",
    "from sklearn.preprocessing import scale\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "true_factor_scores = np.zeros((1000,3))\n",
    "coordinates = np.zeros((1000,4))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# First we will generate the macro-structure. We will generate an eigenvector that applies globally, has a \n",
    "# multivariate normal set of loadings and a bimodal normal distribution of values\n",
    "\n",
    "def generate_global(true_factor_scores,coordinates,noise_multiplier=1):\n",
    "\n",
    "    global_noise = [\n",
    "        1,1,1,1\n",
    "    ]\n",
    "\n",
    "    loading_means_global = [\n",
    "        5,0,0,0\n",
    "        ]\n",
    "    \n",
    "    \n",
    "    noise = multivariate_normal(global_noise,np.identity(4)*noise_multiplier).rvs(1000)    \n",
    "    loadings = multivariate_normal(loading_means_global,np.identity(4)/3).rvs(1000)\n",
    "\n",
    "    score_draws = norm().rvs(1000) / 3\n",
    "\n",
    "    score_draws[:200] += 2\n",
    "    score_draws[200:] += 5\n",
    "\n",
    "    true_factor_scores[:,0] = score_draws\n",
    "\n",
    "    coordinates += (loadings * np.tile(true_factor_scores[:,0],(4,1)).T) + (noise * noise_multiplier)\n",
    "\n",
    "    return true_factor_scores,coordinates\n",
    "\n",
    "true_factor_scores,coordinates = generate_global(true_factor_scores,coordinates,noise_multiplier=1)\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Let's take a look at the data we have generated. \n",
    "# We should have two broad clusters which are easily distinguished \n",
    "\n",
    "plt.figure()\n",
    "plt.title(\"Example Feature Values\")\n",
    "plt.imshow(coordinates,aspect='auto',interpolation='none')\n",
    "plt.colorbar()\n",
    "plt.show()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# We would now like to generate two local factors\n",
    "\n",
    "# A situation that causes difficulties during PCA analysis is when two \n",
    "# features are associatedin opposite directions in different parts of the \n",
    "# dataset\n",
    "\n",
    "# Therefore in one part of the dataset, feature 2 (at index 1) will be\n",
    "# positively associated with features 6-8, and in another part, it will \n",
    "# be negatively associated. \n",
    "\n",
    "def generate_local(true_factor_scores,coordinates):\n",
    "\n",
    "    local_loading_means_1 = [\n",
    "        0,0,5,0,\n",
    "        \n",
    "    ]\n",
    "\n",
    "    local_loading_means_2 = [\n",
    "        0,0,0,5,\n",
    "    ]\n",
    "\n",
    "    local_loadings_1 = multivariate_normal(local_loading_means_1,np.identity(4)/10).rvs(200) \n",
    "    local_loadings_2 = multivariate_normal(local_loading_means_2,np.identity(4)/10).rvs(800)\n",
    "\n",
    "#     true_factor_scores[:200,1] = np.array(sorted(beta(.1,.1).rvs(200))) - .5 \n",
    "#     true_factor_scores[200:,2] = np.array(sorted(beta(.3,.3).rvs(800))) - .5\n",
    "    true_factor_scores[:200,1] = np.array(sorted(norm().rvs(200)))\n",
    "    true_factor_scores[200:,2] = np.array(sorted(norm().rvs(800)))\n",
    "    \n",
    "\n",
    "    local_coordinates_1 = np.tile(true_factor_scores[:200,1],(4,1)).T * local_loadings_1\n",
    "    local_coordinates_2 = np.tile(true_factor_scores[200:,2],(4,1)).T * local_loadings_2\n",
    "\n",
    "    coordinates[:200] += local_coordinates_1\n",
    "    coordinates[200:] += local_coordinates_2\n",
    "    \n",
    "    return true_factor_scores,coordinates\n",
    "\n",
    "true_factor_scores,coordinates = generate_local(true_factor_scores,coordinates)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Optionally we may scale the features for later analysis\n",
    "\n",
    "# coordinates = scale(coordinates,axis=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Now we may have a generative procedure we can repeat quickly:\n",
    "\n",
    "true_factor_scores = np.zeros((1000,3))\n",
    "coordinates = np.zeros((1000,4))\n",
    "\n",
    "true_factor_scores,coordinates = generate_global(\n",
    "    true_factor_scores,\n",
    "    coordinates,\n",
    "    noise_multiplier=.1,\n",
    ")\n",
    "\n",
    "true_factor_scores,coordinates = generate_local(\n",
    "    true_factor_scores,\n",
    "    coordinates,\n",
    ")\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from scipy.cluster.hierarchy import linkage,dendrogram\n",
    "\n",
    "# sample_agglomeration = dendrogram(linkage(coordinates, metric='cosine', method='average'), no_plot=True)['leaves']\n",
    "\n",
    "plt.figure()\n",
    "plt.title(\"Individual Feature Values\")\n",
    "plt.imshow(coordinates,aspect='auto',interpolation='none')\n",
    "plt.colorbar()\n",
    "plt.xlabel(\"Features\")\n",
    "plt.ylabel(\"Samples\")\n",
    "plt.show()\n",
    "\n",
    "plt.figure()\n",
    "plt.title(\"True Factor Scores\")\n",
    "plt.imshow(true_factor_scores,aspect='auto',interpolation='none')\n",
    "plt.colorbar()\n",
    "plt.xlabel(\"Factors\")\n",
    "plt.ylabel(\"Samples\")\n",
    "plt.show()\n",
    "\n",
    "# plt.figure()\n",
    "# plt.imshow(coordinates[sample_agglomeration],aspect='auto',interpolation='none')\n",
    "# plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Now we will produce an embedding of the newly generated dataset for \n",
    "# easier visualization. \n",
    "\n",
    "t_coordinates = TSNE().fit_transform(coordinates)\n",
    "\n",
    "plt.figure()\n",
    "plt.title(\"TSNE Embedded Simulated Samples\")\n",
    "plt.scatter(*t_coordinates.T)\n",
    "plt.show()\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "# First we can visualize the true factor values in order to understand \n",
    "# which clusters are which\n",
    "\n",
    "plt.figure()\n",
    "plt.title(\"True Factor 1 Scores\")\n",
    "plt.scatter(*t_coordinates.T,c=true_factor_scores[:,0],cmap='bwr')\n",
    "plt.colorbar()\n",
    "plt.show()\n",
    "\n",
    "plt.figure()\n",
    "plt.title(\"True Factor 2 Scores\")\n",
    "plt.scatter(*t_coordinates.T,c=true_factor_scores[:,1],cmap='bwr')\n",
    "plt.colorbar()\n",
    "plt.show()\n",
    "\n",
    "plt.figure()\n",
    "plt.title(\"True Factor 3 Scores\")\n",
    "plt.scatter(*t_coordinates.T,c=true_factor_scores[:,2],cmap='bwr')\n",
    "plt.colorbar()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Now we can perform a PCA analysis to see if we can recover the \n",
    "# global and local factors accurately. \n",
    "\n",
    "from sklearn.decomposition import PCA\n",
    "\n",
    "model = PCA().fit(coordinates)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# First we observe that PCA DOES explain most of the variance present in the datset,\n",
    "# however it does so after using 4 components.\n",
    "\n",
    "plt.figure()\n",
    "plt.title(\"PC Explanatory Power (Ratio)\")\n",
    "plt.plot(model.explained_variance_ratio_)\n",
    "plt.xlabel(\"PCs\")\n",
    "plt.ylabel(\"Variance Fraction Explained\")\n",
    "plt.show()\n",
    "\n",
    "model.explained_variance_ratio_\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# However, we would be interested to see if it's possible to understand reover local \n",
    "# feature relationships in the PC loadings. After all, in an ideal case, we would recover\n",
    "# our loadings exactly.\n",
    "\n",
    "print(model.components_)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Here we see that the loadings of the PCs discovered do NOT contain a negative association \n",
    "# between 2 and the 6-7 pair. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pct = model.transform(coordinates)\n",
    "pct.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from scipy.cluster.hierarchy import linkage,dendrogram\n",
    "\n",
    "\n",
    "plt.figure()\n",
    "plt.imshow(pct,cmap='bwr',aspect='auto',interpolation='none')\n",
    "plt.colorbar()\n",
    "plt.show()\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "# We can observe directly that while the PCs recover the overall struture somewhat correctly,\n",
    "# they make inappropriately global inferences about individual PCs.\n",
    "\n",
    "# This occurs despite the fact that it is in principle perfectly possible to represent the \n",
    "# data structure corretly using 3 PCs with non-standardly distributed scores. \n",
    "# (As this was the generative process)\n",
    "\n",
    "plt.figure()\n",
    "plt.title(\"PC1 Scores\")\n",
    "plt.scatter(*t_coordinates.T,c=pct[:,0],cmap='bwr')\n",
    "plt.colorbar()\n",
    "plt.show()\n",
    "\n",
    "plt.figure()\n",
    "plt.title(\"PC2 Scores\")\n",
    "plt.scatter(*t_coordinates.T,c=pct[:,1],cmap='bwr')\n",
    "plt.colorbar()\n",
    "plt.show()\n",
    "\n",
    "plt.figure()\n",
    "plt.title(\"PC3 Scores\")\n",
    "plt.scatter(*t_coordinates.T,c=pct[:,2],cmap='bwr')\n",
    "plt.colorbar()\n",
    "plt.show()\n",
    "\n",
    "plt.figure()\n",
    "plt.title(\"PC4 Scores\")\n",
    "plt.scatter(*t_coordinates.T,c=pct[:,3],cmap='bwr')\n",
    "plt.colorbar()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure()\n",
    "plt.title(\"Principal Component Loadings\")\n",
    "plt.imshow(pct[:,:4],interpolation='none',aspect='auto',cmap='bwr',vmin=-20,vmax=20)\n",
    "plt.xlabel(\"PCs\")\n",
    "plt.ylabel(\"Samples\")\n",
    "plt.colorbar()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "# plt.figure()\n",
    "# plt.scatter(true_factor_scores[:,1],pct[:,1])\n",
    "# plt.show()\n",
    "\n",
    "# plt.figure()\n",
    "# plt.scatter(true_factor_scores[:,1],pct[:,2])\n",
    "# plt.show()\n",
    "\n",
    "# plt.figure()\n",
    "# plt.scatter(true_factor_scores[:,1],pct[:,3])\n",
    "# plt.show()\n",
    "\n",
    "# plt.figure()\n",
    "# plt.scatter(true_factor_scores[:,2],pct[:,1])\n",
    "# plt.show()\n",
    "\n",
    "# plt.figure()\n",
    "# plt.scatter(true_factor_scores[:,2],pct[:,2])\n",
    "# plt.show()\n",
    "\n",
    "# plt.figure()\n",
    "# plt.scatter(true_factor_scores[:,2],pct[:,3])\n",
    "# plt.show()\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# model = PCA(n_components=3).fit(coordinates)\n",
    "# pct = model.transform(coordinates)\n",
    "\n",
    "# recovered = model.inverse_transform(pct)\n",
    "\n",
    "# null_residuals = coordinates - np.mean(coordinates,axis=0)\n",
    "# recovered_residuals = coordinates - recovered\n",
    "\n",
    "# null_error = np.sum(np.power(null_residuals,2))\n",
    "# recovered_error = np.sum(np.power(recovered_residuals,2))\n",
    "\n",
    "# print(null_error)\n",
    "# print(recovered_error)\n",
    "\n",
    "# recovered_error / null_error"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Now we can examine whether a Random Forest can capture the structure that eluded a PCA:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys\n",
    "sys.path.append('../src/')\n",
    "import tree_reader as tr \n",
    "import lumberjack"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# We will train a relatively shallow forest, since the dataset is not complex. \n",
    "# We have relatively few features and would like a reliable structure, so will bootstrap a \n",
    "# large number of features per node (80% bootstrap)\n",
    "\n",
    "# The rest of the parameters aren't deeply important, and so are left without comment. \n",
    "\n",
    "forest = lumberjack.fit(\n",
    "    coordinates,\n",
    "    trees=100,\n",
    "    ifs=8,\n",
    "    ofs=8,\n",
    "    braids=1,\n",
    "    ss=1000,\n",
    "    leaves=10,\n",
    "    depth=5,\n",
    "    norm='l1',\n",
    "    sfr=0,\n",
    "    reduce_input='true',\n",
    "#     reduce_output='true',\n",
    "#     reduce_input='false',\n",
    "    reduce_output='false',\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "forest.tsne_coordinates = t_coordinates\n",
    "forest.reset_split_clusters()\n",
    "forest.interpret_splits(mode='additive_mean',metric='cosine',depth=4,k=200,relatives=True)\n",
    "forest.maximum_spanning_tree(mode='samples')\n",
    "# forest.most_likely_tree(depth=4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "forest.html_tree_summary(n=5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# The resulting tree is available at:\n",
    "\n",
    "# https://bx.bio.jhu.edu/track-hubs/bc/sc_summary/simulation/tree_template.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "plt.figure()\n",
    "plt.imshow(forest.factor_matrix(),cmap='bwr',aspect='auto',interpolation='none',vmin=-1,vmax=1)\n",
    "plt.colorbar()\n",
    "plt.show()\n",
    "\n",
    "# selected_factors = [3,10,5,8]\n",
    "\n",
    "# plt.figure()\n",
    "# plt.imshow(forest.factor_matrix().T[selected_factors].T,aspect='auto',interpolation='none',cmap='bwr',vmin=-1,vmax=1)\n",
    "# plt.colorbar()\n",
    "# plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "plt.figure()\n",
    "plt.scatter(true_factor_scores[:,1],forest.factor_matrix()[:,10])\n",
    "plt.show()\n",
    "\n",
    "plt.figure()\n",
    "plt.scatter(true_factor_scores[:,1],forest.factor_matrix()[:,4])\n",
    "plt.show()\n",
    "\n",
    "plt.figure()\n",
    "plt.scatter(true_factor_scores[:,2],forest.factor_matrix()[:,10])\n",
    "plt.show()\n",
    "\n",
    "plt.figure()\n",
    "plt.scatter(true_factor_scores[:,2],forest.factor_matrix()[:,4])\n",
    "plt.show()\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Now we would like to range over a variety of noise levels\n",
    "\n",
    "# noise_range = np.zeros((5,10000,10))\n",
    "# true_range = np.zeros((5,10000,3))\n",
    "\n",
    "# for i,noise in enumerate(range(-2,3)):\n",
    "    \n",
    "#     true_factor_scores = np.zeros((10000,3))\n",
    "#     coordinates = np.zeros((10000,10))\n",
    "\n",
    "#     true_factor_scores,coordinates = generate_global(\n",
    "#         true_factor_scores,\n",
    "#         coordinates,\n",
    "#         noise_multiplier=2**noise,\n",
    "#     )\n",
    "\n",
    "#     true_factor_scores,coordinates = generate_local(\n",
    "#         true_factor_scores,\n",
    "#         coordinates,\n",
    "#     )\n",
    "\n",
    "\n",
    "\n",
    "#     plt.figure()\n",
    "#     plt.title(\"Individual Feature Values\")\n",
    "#     plt.imshow(coordinates,aspect='auto',interpolation='none')\n",
    "#     plt.colorbar()\n",
    "#     plt.xlabel(\"Features\")\n",
    "#     plt.ylabel(\"Samples\")\n",
    "#     plt.show()\n",
    "    \n",
    "#     noise_range[i] = coordinates\n",
    "#     true_range[i] = true_factor_scores\n",
    "\n",
    "\n",
    "# plt.figure()\n",
    "# plt.title(\"Individual Feature Values\")\n",
    "# plt.imshow(noise_range.reshape((50000,10)),aspect='auto',interpolation='none')\n",
    "# plt.colorbar()\n",
    "# plt.xlabel(\"Features\")\n",
    "# plt.ylabel(\"Samples\")\n",
    "# plt.show()\n",
    "\n",
    "    \n",
    "# t_coordinates = TSNE().fit_transform(noise_range.reshape((50000,10)))\n",
    "\n",
    "# plt.figure()\n",
    "# plt.title(\"TSNE Embedded Simulated Samples\")\n",
    "# plt.scatter(*t_coordinates.T)\n",
    "# plt.show()\n",
    "\n",
    "for i in range(5):\n",
    "    plt.figure()\n",
    "    plt.title(\"TSNE Embedded Simulated Samples\")\n",
    "    plt.scatter(*t_coordinates[i*10000:(i+1)*10000].T,c=np.arange(10000),cmap='rainbow')\n",
    "    plt.show()\n",
    "    \n",
    "    \n",
    "# for i in range(5):\n",
    "    \n",
    "#     forest = lumberjack.fit(\n",
    "#         noise_range[i],\n",
    "#         trees=300,\n",
    "#         ifs=8,\n",
    "#         ofs=8,\n",
    "#         braids=1,\n",
    "#         ss=1000,\n",
    "#         leaves=10,\n",
    "#         depth=4,\n",
    "#         norm='l1',\n",
    "#         sfr=0,\n",
    "#     #     reduce_input='true',\n",
    "#     #     reduce_output='true',\n",
    "#         reduce_input='false',\n",
    "#         reduce_output='false',\n",
    "#     )\n",
    "    \n",
    "#     forest.tsne_coordinates = t_coordinates[i*10000:(i+1)*10000]\n",
    "#     forest.reset_split_clusters()\n",
    "#     forest.interpret_splits(mode='additive_mean',metric='cosine',depth=4,pca=3,k=500,relatives=True)\n",
    "#     forest.maximum_spanning_tree(mode='samples')\n",
    "    \n",
    "#     plt.figure()\n",
    "#     plt.imshow(forest.factor_matrix(),aspect='auto',interpolation='none',cmap='bwr')\n",
    "#     plt.show()\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys\n",
    "sys.path.append('../src/')\n",
    "import tree_reader as tr \n",
    "import lumberjack\n",
    "\n",
    "# We also want to quantify the best ability of both forest and PCA to capture Local Factor 2.\n",
    "# We will quantify that by plotting the best-correlated factor of both the forest and the PCA\n",
    "# over various ranges of noise value.\n",
    "\n",
    "from scipy.spatial.distance import cdist\n",
    "\n",
    "best_forest_correlations = []\n",
    "best_pca_correlations = []\n",
    "\n",
    "for i in range(5):\n",
    "\n",
    "    forest = lumberjack.fit(\n",
    "        noise_range[i],\n",
    "        trees=300,\n",
    "        ifs=8,\n",
    "        ofs=8,\n",
    "        braids=1,\n",
    "        ss=1000,\n",
    "        leaves=10,\n",
    "        depth=4,\n",
    "        norm='l1',\n",
    "        sfr=0,\n",
    "    #     reduce_input='true',\n",
    "    #     reduce_output='true',\n",
    "        reduce_input='false',\n",
    "        reduce_output='false',\n",
    "    )\n",
    "\n",
    "    forest.interpret_splits(mode='additive_mean',metric='cosine',depth=4,pca=3,k=500,relatives=True)\n",
    "    forest_factor_matrix = forest.factor_matrix()\n",
    "    forest_correlations = cdist(true_range[i].T,forest_factor_matrix.T[1:],metric='correlation')\n",
    "    best_forest_correlations.append(np.max(np.abs(forest_correlations-1),axis=1))\n",
    "    \n",
    "    print(f\"{forest_correlations}\")\n",
    "    \n",
    "    pca_model = PCA().fit(noise_range[i])\n",
    "    pca_coordinates = pca_model.transform(noise_range[i])    \n",
    "    pca_correlations = cdist(true_range[i].T,pca_coordinates.T,metric='correlation')\n",
    "    best_pca_correlations.append(np.max(np.abs(pca_correlations-1),axis=1))\n",
    "    \n",
    "    print(f\"{pca_correlations}\")\n",
    "    \n",
    "print(best_forest_correlations)\n",
    "print(best_pca_correlations)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "best_forest_correlations"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "best_pca_correlations"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Now that we have trained a forest, we should examine two ways in which an RFR can detet changes in the underlying \n",
    "# structure of a dataset: changes in behavior or changes in population composition.\n",
    "\n",
    "# For this we will construct two new datasets, one with a shifted population, and one with shifted behavior.\n",
    "\n",
    "# We will leave the global structure identical:\n",
    "\n",
    "global_noise = [\n",
    "    1,1,1,1,1,\n",
    "    1,1,1,\n",
    "    1,1,\n",
    "]\n",
    "\n",
    "loading_means_global = [\n",
    "        1,0,-2,3,5,\n",
    "        0,0,2,\n",
    "        3,3\n",
    "    ]\n",
    "    \n",
    "true_factor_scores = np.zeros((10000,3))\n",
    "    \n",
    "noise = multivariate_normal(global_noise,np.identity(10)/10).rvs(10000)    \n",
    "loadings = multivariate_normal(loading_means_global,np.identity(10)/3).rvs(10000)\n",
    "\n",
    "score_draws = norm().rvs(10000) / 3\n",
    "\n",
    "score_draws[:3000] += 2\n",
    "score_draws[3000:] += 5\n",
    "\n",
    "true_factor_scores[:,0] = score_draws\n",
    "\n",
    "coordinates = (loadings * np.tile(true_factor_scores[:,0],(10,1)).T) + noise\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# We will now copy that structure and create the population shifted local effects.\n",
    "\n",
    "# The mean loadings are the same:\n",
    "\n",
    "local_loading_means_1 = [\n",
    "    0,2,0,0,0,\n",
    "    1,3,1,\n",
    "    0,2,\n",
    "]\n",
    "\n",
    "local_loading_means_2 = [\n",
    "    0,-2,0,-2,0,\n",
    "    1,3,3,\n",
    "    3,0,\n",
    "]\n",
    "\n",
    "local_loadings_1 = multivariate_normal(local_loading_means_1,np.identity(10)/10).rvs(5000)\n",
    "local_loadings_2 = multivariate_normal(local_loading_means_2,np.identity(10)/10).rvs(2000)\n",
    "\n",
    "# However the factor scores we will draw will be different. \n",
    "\n",
    "true_factor_scores[3000:8000,1] = np.array(sorted((beta(.1,.1).rvs(5000) * 3 ) + 3))\n",
    "\n",
    "# Note the asymmetric values for the beta distribution. \n",
    "\n",
    "true_factor_scores[8000:,2] = np.array(sorted((beta(.5,.2).rvs(2000) * 3 ) + 3))\n",
    "\n",
    "\n",
    "\n",
    "local_coordinates_1 = np.tile(true_factor_scores[3000:8000,1],(10,1)).T * local_loadings_1\n",
    "local_coordinates_2 = np.tile(true_factor_scores[8000:,2],(10,1)).T * local_loadings_2\n",
    "\n",
    "coordinates[3000:8000] += local_coordinates_1\n",
    "coordinates[8000:] += local_coordinates_2\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure()\n",
    "plt.title(\"Individual Feature Values\")\n",
    "plt.imshow(coordinates,aspect='auto',interpolation='none')\n",
    "plt.colorbar()\n",
    "plt.xlabel(\"Features\")\n",
    "plt.ylabel(\"Samples\")\n",
    "plt.show()\n",
    "\n",
    "plt.figure()\n",
    "plt.title(\"True Factor Scores\")\n",
    "plt.imshow(true_factor_scores,aspect='auto',interpolation='none')\n",
    "plt.colorbar()\n",
    "plt.xlabel(\"Factors\")\n",
    "plt.ylabel(\"Samples\")\n",
    "plt.show()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "field = np.zeros((100,100))\n",
    "field += np.arange(0,1,.01)\n",
    "field = (field.T + np.arange(0,1,.01)).T\n",
    "plt.figure()\n",
    "plt.imshow(field)\n",
    "plt.colorbar()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}

