Often for visualizations in genomics, massive social networks, or sensor data visualizations, it helps to be able to plot millions rather than simply hundreds of thousands of points.
By default, pydeck sends data from Jupyter to the frontend by serializing data to JSON. However, for massive data sets, the costs to serialize and deserialize this JSON can prevent a visualization from rendering.
In order to get around this, pydeck supports binary data transfer, which significantly reduces data size. Binary transfer relies on NumPy and its typed arrays, which are converted to JavaScript typed arrays and passed to deck.gl using precalculated binary attributes.
Binary transport will only work if the following requirements are met:
pydeck.bindings.deck.Deck.show
. It relies on the socket-level
communication built into the Jupyter environment.use_binary_transport
. use_binary_transport
must be set to True
explictly on your Layer
.Layer
's data
argument must be a pandas.DataFrame
object.Layer
.pandas.DataFrame
. For example, get_position='position'
is correct, not get_position=['x', 'y']
.
For example, this data format, where x
& y
represent a position and r
, g
, and b
represent color values, like this–x | y | r | g | b |
---|---|---|---|---|
0 | 1 | 0 | 0 | 0 |
0 | 5 | 255 | 0 | 0 |
5 | 1 | 255 | 255 | 0 |
should be converted to a nested format like this–
position | color |
---|---|
[0, 1] | [0, 0, 0] |
[0, 5] | [255, 0, 0] |
[5, 1] | [255, 255, 0] |
import pydeck as pdk
import pandas as pd
NODES_URL = "https://raw.githubusercontent.com/ajduberstein/geo_datasets/master/social_nodes.csv"
def generate_graph_data(num_nodes, random_seed):
"""Generates a graph of 10k nodes with a 3D force layout
This function is unused but serves as an example of how the data in
this visualization was generated
"""
import networkx as nx # noqa
g = nx.random_internet_as_graph(num_nodes, random_seed)
node_positions = nx.fruchterman_reingold_layout(g, dim=3)
force_layout_df = pd.DataFrame.from_records(node_positions).transpose()
force_layout_df["group"] = [d[1]["type"] for d in g.nodes.data()]
force_layout_df.columns = ["x", "y", "z", "group"]
return force_layout_df
We will use deck.gl's OrbitView (example here) which can help us plot these points in 3D:
The data we'll render is a 3D position with some categorical data (group
):
nodes = pd.read_csv(NODES_URL)
nodes.head()
x | y | z | group | |
---|---|---|---|---|
0 | 0.021245 | -0.103758 | -0.007543 | T |
1 | -0.099934 | -0.057345 | 0.047415 | T |
2 | 0.007010 | 0.107698 | 0.006490 | T |
3 | 0.076356 | -0.007087 | 0.008351 | T |
4 | 0.030020 | -0.126868 | 0.022745 | T |
We need this flattened data in a nested format, so we transform it:
nodes["position"] = nodes.apply(lambda row: [row["x"], row["y"], row["z"]], axis=1)
# Remove all unused columns
del nodes["x"]
del nodes["y"]
del nodes["z"]
nodes.head()
group | position | |
---|---|---|
0 | T | [0.0212450698018074, -0.1037581041455268, -0.0... |
1 | T | [-0.0999342873692512, -0.0573454238474369, 0.0... |
2 | T | [0.0070100156590342, 0.1076982915401458, 0.006... |
3 | T | [0.0763555988669395, -0.007086653728038, 0.008... |
4 | T | [0.0300203301012516, -0.1268675178289413, 0.02... |
We'll also color the points by group and then normalize the colors since they are provided as floats–see the deck.gl documentation.
colors = pdk.data_utils.assign_random_colors(nodes["group"])
# Divide by 255 to normalize the colors
# Specify positions and colors as columns of lists
nodes["color"] = nodes.apply(
lambda row: [c / 255 for c in colors.get(row["group"])], axis=1
)
# Remove unused column
del nodes["group"]
nodes.head()
position | color | |
---|---|---|
0 | [0.0212450698018074, -0.1037581041455268, -0.0... | [0.12156862745098039, 0.8705882352941177, 1.0] |
1 | [-0.0999342873692512, -0.0573454238474369, 0.0... | [0.12156862745098039, 0.8705882352941177, 1.0] |
2 | [0.0070100156590342, 0.1076982915401458, 0.006... | [0.12156862745098039, 0.8705882352941177, 1.0] |
3 | [0.0763555988669395, -0.007086653728038, 0.008... | [0.12156862745098039, 0.8705882352941177, 1.0] |
4 | [0.0300203301012516, -0.1268675178289413, 0.02... | [0.12156862745098039, 0.8705882352941177, 1.0] |
view_state = pdk.ViewState(
offset=[0, 0],
latitude=None,
longitude=None,
bearing=None,
pitch=None,
zoom=10
)
views = [pdk.View(type="OrbitView", controller=True)]
nodes_layer = pdk.Layer(
"PointCloudLayer",
nodes,
id="binary-points",
use_binary_transport=True,
get_position="position",
get_normal=[10, 100, 10],
get_color="color",
pickable=True,
auto_highlight=True,
highlight_color=[255, 255, 0],
radius=50,
)
pdk.Deck(layers=[nodes_layer], initial_view_state=view_state, views=views, map_provider=None).show()
You can open up your brower's developer console to confirm the data is being sent via binary transfer. In the console you should see a message like:
>> transport.js:68 Delivering transport message
>> binary: {"binary-points": {…}}
>> transport: n {name: "Jupyter Transport (JavaScript <=> Jupyter Kernel)", _messageQueue: Array(0), userData: {…}, jupyterModel: n, jupyterView: n}
type: "json-with-binary"
You should also be able to see the binary payload in your developer console's Network tab.