Binary data transfer¶

Motivation¶

Often for visualizations in genomics, massive social networks, or sensor data visualizations, it helps to be able to plot millions rather than simply hundreds of thousands of points.

By default, pydeck sends data from Jupyter to the frontend by serializing data to JSON. However, for massive data sets, the costs to serialize and deserialize this JSON can prevent a visualization from rendering.

In order to get around this, pydeck supports binary data transfer, which significantly reduces data size. Binary transfer relies on NumPy and its typed arrays, which are converted to JavaScript typed arrays and passed to deck.gl using precalculated binary attributes.

Usage¶

Binary transport will only work if the following requirements are met:

  • Use Jupyter. Binary transfer only works within Jupyter environments via pydeck.bindings.deck.Deck.show. It relies on the socket-level communication built into the Jupyter environment.
  • Set use_binary_transport. use_binary_transport must be set to True explictly on your Layer.
  • Pass data as a DataFrame. The Layer's data argument must be a pandas.DataFrame object.
  • Don't pass un-used columns. Data that is not intend to be rendered should not be passed into the Layer.
  • Prefer columns of lists where possible. Accessor names must be strings representing column names within the pandas.DataFrame. For example, get_position='position' is correct, not get_position=['x', 'y']. For example, this data format, where x & y represent a position and r, g, and b represent color values, like this–
x y r g b
0 1 0 0 0
0 5 255 0 0
5 1 255 255 0

should be converted to a nested format like this–

position color
[0, 1] [0, 0, 0]
[0, 5] [255, 0, 0]
[5, 1] [255, 255, 0]

Example¶

To demonstrate this, we'll plot a series of nodes generated at random from the networkx library:

In [1]:
import pydeck as pdk
import pandas as pd


NODES_URL = "https://raw.githubusercontent.com/ajduberstein/geo_datasets/master/social_nodes.csv"


def generate_graph_data(num_nodes, random_seed):
    """Generates a graph of 10k nodes with a 3D force layout

    This function is unused but serves as an example of how the data in
    this visualization was generated
    """
    import networkx as nx  # noqa

    g = nx.random_internet_as_graph(num_nodes, random_seed)
    node_positions = nx.fruchterman_reingold_layout(g, dim=3)

    force_layout_df = pd.DataFrame.from_records(node_positions).transpose()
    force_layout_df["group"] = [d[1]["type"] for d in g.nodes.data()]
    force_layout_df.columns = ["x", "y", "z", "group"]
    return force_layout_df

We will use deck.gl's OrbitView (example here) which can help us plot these points in 3D:

The data we'll render is a 3D position with some categorical data (group):

In [2]:
nodes = pd.read_csv(NODES_URL)
nodes.head()
Out[2]:
x y z group
0 0.021245 -0.103758 -0.007543 T
1 -0.099934 -0.057345 0.047415 T
2 0.007010 0.107698 0.006490 T
3 0.076356 -0.007087 0.008351 T
4 0.030020 -0.126868 0.022745 T

We need this flattened data in a nested format, so we transform it:

In [3]:
nodes["position"] = nodes.apply(lambda row: [row["x"], row["y"], row["z"]], axis=1)

# Remove all unused columns
del nodes["x"]
del nodes["y"]
del nodes["z"]
nodes.head()
Out[3]:
group position
0 T [0.0212450698018074, -0.1037581041455268, -0.0...
1 T [-0.0999342873692512, -0.0573454238474369, 0.0...
2 T [0.0070100156590342, 0.1076982915401458, 0.006...
3 T [0.0763555988669395, -0.007086653728038, 0.008...
4 T [0.0300203301012516, -0.1268675178289413, 0.02...

We'll also color the points by group and then normalize the colors since they are provided as floats–see the deck.gl documentation.

In [4]:
colors = pdk.data_utils.assign_random_colors(nodes["group"])
# Divide by 255 to normalize the colors
# Specify positions and colors as columns of lists
nodes["color"] = nodes.apply(
    lambda row: [c / 255 for c in colors.get(row["group"])], axis=1
)
# Remove unused column
del nodes["group"]
nodes.head()
Out[4]:
position color
0 [0.0212450698018074, -0.1037581041455268, -0.0... [0.3176470588235294, 0.4392156862745098, 0.854...
1 [-0.0999342873692512, -0.0573454238474369, 0.0... [0.3176470588235294, 0.4392156862745098, 0.854...
2 [0.0070100156590342, 0.1076982915401458, 0.006... [0.3176470588235294, 0.4392156862745098, 0.854...
3 [0.0763555988669395, -0.007086653728038, 0.008... [0.3176470588235294, 0.4392156862745098, 0.854...
4 [0.0300203301012516, -0.1268675178289413, 0.02... [0.3176470588235294, 0.4392156862745098, 0.854...
In [5]:
view_state = pdk.ViewState(
    offset=[0, 0],
    latitude=None,
    longitude=None,
    bearing=None,
    pitch=None,
    zoom=10
)
views = [pdk.View(type="OrbitView", controller=True)]

nodes_layer = pdk.Layer(
    "PointCloudLayer",
    nodes,
    id="binary-points",
    use_binary_transport=True,
    get_position="position",
    get_normal=[10, 100, 10],
    get_color="color",
    pickable=True,
    auto_highlight=True,
    highlight_color=[255, 255, 0],
    radius=50,
)

pdk.Deck(layers=[nodes_layer], initial_view_state=view_state, views=views, map_provider=None).show()
Out[5]:

Confirming that data is being transferred over web sockets¶

You can open up your brower's developer console to confirm the data is being sent via binary transfer. In the console you should see a message like:

>> transport.js:68 Delivering transport message 
>> binary: {"binary-points": {…}}
>> transport: n {name: "Jupyter Transport (JavaScript <=> Jupyter Kernel)", _messageQueue: Array(0), userData: {…}, jupyterModel: n, jupyterView: n}
type: "json-with-binary"

You should also be able to see the binary payload in your developer console's Network tab.