Simulating Argus II

Background

ArgusII is an epiretinal implant developed by Second Sight Medical Products, Inc. (Sylmar, CA). Now discontinued, it was the first retinal implant to get FDA approval in the US and the CE mark in Europe, and has been implanted in close to 500 patients worldwide.

A number of studies have documented how the artificial vision provided by ArgusII (Second Sight Medical Products, Inc.) differs from normal sight. Argus II contains 60 electrodes of 225 um diameter arranged in a 6 x 10 grid (575 um center-to-center separation) [Yue2020]. Some researchers therefore assumed that the stimulation of a grid of electrodes on the retina would lead to the perception of a grid of luminous dots (“phosphenes”). We refer to this as the ScoreboardModel of prosthetic vision. However, a growing body of evidence has shown that retinal implant users often report seeing distorted phosphenes and require extensive rehabilitative training to make use of their new vision [Beyeler2019], [EricksonDavis2021].

Although single-electrode phosphenes are consistent from trial to trial, they vary across electrodes and users [Luo2016], [Beyeler2019]. Phosphene shape strongly depends on stimulation parameters [Nanduri2012] as well as the retinal location of the stimulating electrode [Beyeler2019] due to inadvertent activation of passing axon fibers in the retina. The result is a rich repertoire of phosphene shape that includes blobs, arcs, wedges, and triangles [Beyeler2019]:

import matplotlib.pyplot as plt
import pulse2percept as p2p

fig, axes = plt.subplots(ncols=3, figsize=(10, 3))
for ax, subject, scale in zip(axes, ['S2', 'S3', 'S4'], [1, 1, 0.5]):
    data = p2p.datasets.fetch_beyeler2019(subjects=subject)
    p2p.viz.plot_argus_phosphenes(data, ax=ax, scale=scale)
    ax.axis('off')
    ax.set_title(subject)
fig.tight_layout()

Clipping input data to the valid range for imshow with RGB data ([0..1] for floats or [0..255] for integers). Got range [-2.400000000000001..1.0].
Clipping input data to the valid range for imshow with RGB data ([0..1] for floats or [0..255] for integers). Got range [-1.1999999999999997..1.0].
Clipping input data to the valid range for imshow with RGB data ([0..1] for floats or [0..255] for integers). Got range [-2.800000000000001..1.0].

These phosphene shapes can be simulated with the AxonMapModel, which was developed to fit behavioral data (see [Beyeler2019] for details).

Boston Train sequence

To simulate the vision provided by Argus II, we first need to set up a new axon map model. We can specify phosphene size (rho) and elongation (lam) as well as the visual field we would like to simulate (given in degrees of visual angle):

model = p2p.models.AxonMapModel(rho=400, lam=200,
                                xrange=(-12, 12), yrange=(-8, 8))
model.build()

AxonMapModel(ax_segments_range=(0, 50),
             axon_pickle='axons.pickle',
             axons_range=(-180, 180), eye='RE',
             grid_type='rectangular', ignore_pickle=False,
             lam=200, loc_od=(np.float64(15.5), 1.5),
             min_ax_sensitivity=0.001,
             min_current_spread=1e-08, n_ax_segments=500,
             n_axons=1000, n_gray=None, n_jobs=14,
             n_threads=14, ndim=[2], noise=None, rho=400,
             spatial=AxonMapSpatial, step=0.25,
             temporal=None, thresh_percept=0, verbose=True,
             vfmap=Watson2014Map(ndim=2), xrange=(-12, 12),
             yrange=(-8, 8))

We can visualize where the implant sits on the axon map as follows:

model.plot()
p2p.implants.ArgusII().plot()

<Axes: xlabel='x (microns)', ylabel='y (microns)'>

We then need to choose a stimulus to run through the model:

p2p.stimuli.BostonTrain().play()

⏮ ⏪ ▶ ⏩ ⏭ 1/94 Once Loop Reflect

In real life, the Argus II camera would capture a video just like the above, convert it to grayscale, downscale it, and then assign each pixel to an electrode in the implant.

After feeding the resulting stimulus through the axon map model, we get a pretty good idea of what this video would look like to an Argus II patient:

# The video is encoded into current first -- a model reads microamps, not gray
# levels -- at a pulse rate that keeps up with the frame rate:
video = p2p.stimuli.BostonTrain()
implant = p2p.implants.ArgusII()
implant.stim = video.encode(implant=implant, freq=30)
model.predict_percept(implant, t_percept=video.time).play()

⏮ ⏪ ▶ ⏩ ⏭ 1/94 Once Loop Reflect

The above simulation is basically equivalent to the scoreboard model, because each electrode appears as a large blob.

However, as mentioned above, every patient sees phosphenes differently. To some they appear thin and elongated, to others they appear big and arc-like.

To increase the arc length of individual phosphenes, we can choose a larger lam value:

model.lam = 600
model.build()
model.predict_percept(implant, t_percept=video.time).play()

⏮ ⏪ ▶ ⏩ ⏭ 1/94 Once Loop Reflect

On the other hand, some retinal implant recipients (e.g., 51-009) report seeing thin and elongated phosphenes. In this case, the same video may appear very different to them:

model.rho = 100
model.lam = 1000
model.build()
model.predict_percept(implant, t_percept=video.time).play()

⏮ ⏪ ▶ ⏩ ⏭ 1/94 Once Loop Reflect

Girl Pool sequence

Another video shows a girl jumping into a swimming pool:

p2p.stimuli.GirlPool().play()

⏮ ⏪ ▶ ⏩ ⏭ 1/91 Once Loop Reflect

Similar to the above video, we can convert it to grayscale and downscale it, then feed it through the axon map model:

video = p2p.stimuli.GirlPool()
implant = p2p.implants.ArgusII()
implant.stim = video.encode(implant=implant, freq=30)
model.rho = 400
model.lam = 200
model.build()
model.predict_percept(implant, t_percept=video.time).play()

⏮ ⏪ ▶ ⏩ ⏭ 1/91 Once Loop Reflect

Here is the same video with longer phosphenes:

model.lam = 600
model.build()
model.predict_percept(implant, t_percept=video.time).play()

⏮ ⏪ ▶ ⏩ ⏭ 1/91 Once Loop Reflect

Here is the same video with thin and long phosphenes:

model.rho = 100
model.lam = 1000
model.build()
model.predict_percept(implant, t_percept=video.time).play()

⏮ ⏪ ▶ ⏩ ⏭ 1/91 Once Loop Reflect

In reality, the vision provided by Argus II may be even worse for several reasons:

1. The above simulations do not consider temporal distortions (e.g., flicker, persistence, fading)

2. For some patients, individual phosphenes do not assemble into more complex percepts, or do so in a highly unpredictable way.