Binary data transfer¶
Motivation¶
Often for visualizations in genomics, massive social networks, or sensor data visualizations, it helps to be able to plot millions rather than simply hundreds of thousands of points.
By default, pydeck sends data from Jupyter to the frontend by serializing data to JSON. However, for massive data sets, the costs to serialize and deserialize this JSON can prevent a visualization from rendering.
In order to get around this, pydeck supports binary data transfer, which significantly reduces data size. Binary transfer relies on NumPy and its typed arrays, which are converted to JavaScript typed arrays and passed to deck.gl using precalculated binary attributes.
Usage¶
Binary transport will only work if the following requirements are met:
- Use Jupyter. Binary transfer only works within Jupyter environments via
pydeck.bindings.deck.Deck.show
. It relies on the socket-level communication built into the Jupyter environment. - Set
use_binary_transport
.use_binary_transport
must be set toTrue
explictly on yourLayer
. - Pass data as a DataFrame. The
Layer
'sdata
argument must be apandas.DataFrame
object. - Don't pass un-used columns. Data that is not intend to be rendered should not be passed into the
Layer
. - Prefer columns of lists where possible. Accessor names must be strings representing column names within the
pandas.DataFrame
. For example,get_position='position'
is correct, notget_position=['x', 'y']
. For example, this data format, wherex
&y
represent a position andr
,g
, andb
represent color values, like this–
x | y | r | g | b |
---|---|---|---|---|
0 | 1 | 0 | 0 | 0 |
0 | 5 | 255 | 0 | 0 |
5 | 1 | 255 | 255 | 0 |
should be converted to a nested format like this–
position | color |
---|---|
[0, 1] | [0, 0, 0] |
[0, 5] | [255, 0, 0] |
[5, 1] | [255, 255, 0] |
import pydeck as pdk
import pandas as pd
NODES_URL = "https://raw.githubusercontent.com/ajduberstein/geo_datasets/master/social_nodes.csv"
def generate_graph_data(num_nodes, random_seed):
"""Generates a graph of 10k nodes with a 3D force layout
This function is unused but serves as an example of how the data in
this visualization was generated
"""
import networkx as nx # noqa
g = nx.random_internet_as_graph(num_nodes, random_seed)
node_positions = nx.fruchterman_reingold_layout(g, dim=3)
force_layout_df = pd.DataFrame.from_records(node_positions).transpose()
force_layout_df["group"] = [d[1]["type"] for d in g.nodes.data()]
force_layout_df.columns = ["x", "y", "z", "group"]
return force_layout_df
We will use deck.gl's OrbitView (example here) which can help us plot these points in 3D:
The data we'll render is a 3D position with some categorical data (group
):
nodes = pd.read_csv(NODES_URL)
nodes.head()
x | y | z | group | |
---|---|---|---|---|
0 | 0.021245 | -0.103758 | -0.007543 | T |
1 | -0.099934 | -0.057345 | 0.047415 | T |
2 | 0.007010 | 0.107698 | 0.006490 | T |
3 | 0.076356 | -0.007087 | 0.008351 | T |
4 | 0.030020 | -0.126868 | 0.022745 | T |
We need this flattened data in a nested format, so we transform it:
nodes["position"] = nodes.apply(lambda row: [row["x"], row["y"], row["z"]], axis=1)
# Remove all unused columns
del nodes["x"]
del nodes["y"]
del nodes["z"]
nodes.head()
group | position | |
---|---|---|
0 | T | [0.0212450698018074, -0.1037581041455268, -0.0... |
1 | T | [-0.0999342873692512, -0.0573454238474369, 0.0... |
2 | T | [0.0070100156590342, 0.1076982915401458, 0.006... |
3 | T | [0.0763555988669395, -0.007086653728038, 0.008... |
4 | T | [0.0300203301012516, -0.1268675178289413, 0.02... |
We'll also color the points by group and then normalize the colors since they are provided as floats–see the deck.gl documentation.
colors = pdk.data_utils.assign_random_colors(nodes["group"])
# Divide by 255 to normalize the colors
# Specify positions and colors as columns of lists
nodes["color"] = nodes.apply(
lambda row: [c / 255 for c in colors.get(row["group"])], axis=1
)
# Remove unused column
del nodes["group"]
nodes.head()
position | color | |
---|---|---|
0 | [0.0212450698018074, -0.1037581041455268, -0.0... | [0.3176470588235294, 0.4392156862745098, 0.854... |
1 | [-0.0999342873692512, -0.0573454238474369, 0.0... | [0.3176470588235294, 0.4392156862745098, 0.854... |
2 | [0.0070100156590342, 0.1076982915401458, 0.006... | [0.3176470588235294, 0.4392156862745098, 0.854... |
3 | [0.0763555988669395, -0.007086653728038, 0.008... | [0.3176470588235294, 0.4392156862745098, 0.854... |
4 | [0.0300203301012516, -0.1268675178289413, 0.02... | [0.3176470588235294, 0.4392156862745098, 0.854... |
view_state = pdk.ViewState(
offset=[0, 0],
latitude=None,
longitude=None,
bearing=None,
pitch=None,
zoom=10
)
views = [pdk.View(type="OrbitView", controller=True)]
nodes_layer = pdk.Layer(
"PointCloudLayer",
nodes,
id="binary-points",
use_binary_transport=True,
get_position="position",
get_normal=[10, 100, 10],
get_color="color",
pickable=True,
auto_highlight=True,
highlight_color=[255, 255, 0],
radius=50,
)
pdk.Deck(layers=[nodes_layer], initial_view_state=view_state, views=views, map_provider=None).show()
Confirming that data is being transferred over web sockets¶
You can open up your brower's developer console to confirm the data is being sent via binary transfer. In the console you should see a message like:
>> transport.js:68 Delivering transport message
>> binary: {"binary-points": {…}}
>> transport: n {name: "Jupyter Transport (JavaScript <=> Jupyter Kernel)", _messageQueue: Array(0), userData: {…}, jupyterModel: n, jupyterView: n}
type: "json-with-binary"
You should also be able to see the binary payload in your developer console's Network tab.