pulse2percept.vision
Visual scenes, residual vision, and binocular views.
Added in version 0.11.0.
- class pulse2percept.vision.BinocularScene(left, right)[source]
The left and right monocular views, side by side
A
Sceneis one eye’s visual field.BinocularSceneholds two of them, and says which eye each belongs to. The two sides are equal peers: neither is the implanted or the primary one, and they need not share a source, FOV, shape, scotoma or aperture.It does not model how the visual system combines them. There is no fusion, suppression, rivalry, stereopsis or ocular dominance here: this is what the left eye sees, and this is what the right eye sees. Plotting them is therefore an HMD-style pair of monocular views rather than one binocularly fused image.
Models remain monocular in v0.11, so a prediction names the eye it is about:
percept = model.predict_percept(binocular.left)
Added in version 0.11.0.
- Parameters:
Examples
A unilateral implant: a scotoma in the left eye, the fellow eye intact.
>>> import numpy as np >>> from pulse2percept.units import dva >>> from pulse2percept.vision import BinocularScene, Scene, Scotoma >>> picture = np.zeros((8, 8)) >>> binocular = BinocularScene( ... left=Scene(picture, fov=40 * dva, scotoma=Scotoma.circle(8 * dva)), ... right=Scene(picture, fov=40 * dva)) >>> binocular.left.scotoma is None, binocular.right.scotoma is None (False, True)
A bilateral loss that is symmetric about the vertical meridian, built with
fellow_eye():>>> left = Scene(picture, fov=40 * dva, ... scotoma=Scotoma.circle(3 * dva, center=(6, 0) * dva)) >>> binocular = BinocularScene(left, left.fellow_eye()) >>> float(binocular.right.scotoma(-6, 0)) 1.0
Imagery already packed left-right side by side, split with
from_side_by_side():>>> stereo = np.zeros((10, 40)) >>> stereo[:, 20:] = 1.0 >>> binocular = BinocularScene.from_side_by_side(stereo, fov=40 * dva) >>> binocular.left.shape, binocular.right.shape ((10, 20), (10, 20)) >>> float(binocular.left.source.data.max()) 0.0
- classmethod from_side_by_side(source, fov, **scene_kwargs)[source]
Build a binocular scene from a side-by-side stereo image.
The left half of the image becomes the left-eye scene and the right half becomes the right-eye scene. The split is exact: neither view is flipped, resampled, or interpolated.
This method unpacks existing stereo imagery; it does not create disparity or infer depth.
Added in version 0.11.0.
- Parameters:
source (ImageStimulus or image) – Side-by-side stereo image. Filenames, NumPy arrays, and other inputs accepted by
ImageStimulusare converted to one first. Metadata is preserved in both eye images. Stereo video is not supported.fov (float or
(width, height)) – Per-eye field of view in degrees of visual angle. Unlessextentis passed, each eye’s extent is inferred from one half of the image, not the packed stereo frame.**scene_kwargs – Additional arguments passed unchanged to both
Sceneconstructors. Use separate scenes when the two eyes need different scotomas or other settings.
- Returns:
binocular – The left and right monocular scenes.
- Return type:
Examples
>>> import numpy as np >>> from pulse2percept.units import dva >>> from pulse2percept.vision import BinocularScene >>> stereo = np.zeros((10, 40)) >>> binocular = BinocularScene.from_side_by_side( ... stereo, fov=(60, 40) * dva) >>> binocular.left.shape, binocular.left.fov ((10, 20), (60.0, 40.0))
- property left
What the left eye sees, as a Scene
- property right
What the right eye sees, as a Scene
- plot(left_percept=None, right_percept=None, gaze=None, frame=0, rings=False, meridians=False, vmax=None, vmin=None, axes=None, view='scene', context_alpha=0.25, **kwargs)[source]
Plot the two eyes side by side
Each panel is its own
plot(): an eye given no percept shows its native or residual scene, and an eye given one shows that percept in its own field. The two are drawn independently and never combined.- Parameters:
left_percept (
Percept, optional) – A brightness percept for that eye, or None to draw its scene.right_percept (
Percept, optional) – A brightness percept for that eye, or None to draw its scene.gaze ((x, y), optional) – Where both eyes are pointing, in dva. Defaults to the origin. One gaze is shared: vergence is not modeled.
frame (int, optional) – Which frame of a video scene to draw. Ignored for still scenes.
rings (bool, float, or sequence, optional) – Visual-field grid about each eye’s fovea, as in
plot().meridians (bool, float, or sequence, optional) – Visual-field grid about each eye’s fovea, as in
plot().vmax (float, optional) – The percept brightness that displays as white, shared by both eyes. Defaults to the maximum across both percepts.
vmin (float, optional) – The percept brightness that displays as black. Defaults to 0.
axes (sequence of two matplotlib.axes.Axes, optional) – Axes to draw the left and right eye on, in that order. If None, makes a new side-by-side pair.
view ({'scene', 'eye'}, optional) – As in
plot(). Both panels share limits: the union of bothextentvalues in the scene view (default), the largest FOV in the eye view.context_alpha (float, optional) – Opacity of the source outside each FOV in the scene view, as in
plot().**kwargs – Passed on to
plot().
- Returns:
axes – The two axes, always in left-eye, right-eye order.
- Return type:
(ax_left, ax_right)
- class pulse2percept.vision.Gaze(positions, time=None)[source]
A sparse sequence of fixations, each starting at a given time
Gaze is piecewise constant: position
ibecomes active attime[i]and is held untiltime[i + 1]; the last fixation is held indefinitely. Nothing is interpolated, so a saccade takes no time. Resolution is right-continuous, so an event landing exactly on a frame time applies to that frame; times before the first event raise.Accepted wherever
Sceneandpredict_percept()take agaze, given a clock to resolve against: a video scene’s frame times, or a timed percept’s for a still scene.Added in version 0.11.0.
- Parameters:
positions ((n, 2) or (n, 3) array_like) – Scene locations that fall on the fovea, in degrees of visual angle. Unitful values are accepted. Without
time, each row is(x, y, time): x and y in dva, time in ms, unless unitful. Each entry converts on its own, so a bare 0 needs no unit.time ((n,) array_like, optional) – When each fixation begins, in milliseconds unless given as a unitful time. Must be finite and strictly increasing. Required for
(n, 2)positions; not allowed for(x, y, time)rows.
Examples
>>> from pulse2percept.units import dva, ms, s >>> from pulse2percept.vision import Gaze >>> gaze = Gaze([(0, 0), (6, 2)] * dva, time=[0, 400] * ms) >>> gaze.positions array([[0., 0.], [6., 2.]]) >>> gaze = Gaze([(0, 0, 0), (6 * dva, 2 * dva, 0.4 * s)]) >>> gaze.time array([ 0., 400.])
- property positions
The
(n, 2)fixations, in dva; read-only
- property time
When each fixation begins, counted in
time_unit; read-only
- property time_unit
The unit
timeis counted in
- class pulse2percept.vision.Scene(source, fov, extent=None, scotoma=None, scotoma_fill=0, scotoma_blend=0.5, background=0, aperture='rectangular')[source]
What is visually present, and where native vision is lost
A scene places a picture in the world at a fixed angular
extent, and views it through a field of view (fov) centered on the fovea, optionally with a region where native vision is missing.Geometry follows one convention:
extentis the outer(left, right, bottom, top)of the source in scene coordinates, so it reaches half an angular pixel past the outermost pixel centers.Pixel coordinates address pixel centers.
Row 0 is the top of the frame and therefore the largest
y.
Two coordinate frames are used:
Scene coordinates are fixed world coordinates.
extentplaces the source in them.Eye coordinates are centered on the fovea. The FOV, aperture, scotoma, prosthetic percept, visual-field grid and an implant live here.
Gaze moves the viewing window through the scene; the source does not move or rescale:
(x_scene, y_scene) = (x_eye, y_eye) + (x_gaze, y_gaze)
plot()andplay()default toview='scene': fixed axes spanningextent, with the FOV moving with gaze and the source outside it dimmed.view='eye'shows the FOV in eye coordinates instead, spanning[-fov / 2, fov / 2], asrender()always does. There, parts of the FOV beyondextentare black. Gaze also sets what the device is given to encode, unless the implant’sscene_input_frameis'head'(a head-fixed camera the eye cannot move). Device input is sampled from the wholeextent, not only the FOV.Source,
extentandfovare fixed after construction.The scotoma is native vision’s business only. What an implant is given to encode is sampled from the source itself, inside the scotoma as well as outside it: a camera does not go blind where its wearer has.
Added in version 0.11.0.
- Parameters:
source (ImageStimulus, VideoStimulus, or image) – The background scene itself. Anything that is not already a
ImageStimulusor aVideoStimulus, such as a file name or a NumPy array, is handed toImageStimulus.fov (float or
(width, height)) – Size of the viewing window, in degrees of visual angle, centered on the fovea (e.g.40 * dva). A scalar is a square window.extent (float or
(left, right, bottom, top), optional) – Where the source sits in scene coordinates, in dva. A scalar is the span of the shorter source dimension, centered with square pixels (e.g.,45 * dvaon a 173 x 320 source is 83.2 x 45 dva). If None, the source is centered with square pixels and scaled to the smallest extent that containsfov: with a scalarfov, the shorter source dimension spansfov.extentneed not containfov.scotoma (
Scotoma, optional) – The region where native vision is lost. If None, native vision is intact everywhere and the scene is simply what is out there.scotoma_fill (float, color, or 'inpaint', optional) – Scotoma fill: a gray level in [0, 1] (default 0, black), an (r, g, b) triple in [0, 1], or any Matplotlib color string. ‘inpaint’ fills from the surrounding image using
skimage.restoration.inpaint_biharmonic()and ignores scotoma_blend. It cannot be used when composing a prosthetic percept.background (float or (r, g, b), optional) – Gray level or RGB value to use for transparent pixels. Defaults to black.
scotoma_blend (float, optional) –
Standard deviation, in degrees of visual angle, of a Gaussian blur applied to the rasterized loss map before it is drawn, softening the boundary from both sides. Defaults to 0.5 dva; 0 leaves it as sharp as the scotoma itself. Converted to pixels against whichever raster is being drawn, so the angular softness does not depend on resolution. Rendering only: the scotoma’s geometry is unchanged.
Changed in version 0.11.0: Measured in degrees of visual angle rather than in scene pixels.
aperture ({'rectangular', 'round'}, optional) – Shape of the FOV.
fovsets its size and this sets its shape: the default'rectangular'shows the whole window, while'round'inscribes an eye-centered ellipse of semi-axesfov / 2in it, so a squarefovrenders as a disc and a non-square one as an ellipse. The aperture is a display boundary:plot()clips the full-intensity FOV, percept and grid to it (scene-view context stays visible outside),render()writes black outside it, while scene sampling, device input, stimulation and the prosthetic model response are untouched.
Examples
A 40-degree window onto a logo, seen with a central 16-degree scotoma. The logo is landscape, so its height spans the window:
>>> from pulse2percept.stimuli import samples >>> from pulse2percept.units import dva >>> from pulse2percept.vision import Scene, Scotoma >>> scene = Scene(samples.logo_bvl(), fov=40 * dva, ... scotoma=Scotoma.circle(8 * dva)) >>> scene.fov, scene.extent ((40.0, 40.0), (-25.0, 25.0, -20.0, 20.0))
A 40-degree disc onto a wider world.
rendercovers the FOV at the source’s angular pixel pitch:>>> world = Scene(samples.logo_bvl(), extent=(-50, 50, -40, 40) * dva, ... fov=40 * dva, aperture='round') >>> world.render().shape (288, 288, 3, 1)
blank()gives a black field instead of a picture – darkness, not blindness:>>> blank = Scene.blank() >>> blank.plot()
- classmethod blank(fov=45, **kwargs)[source]
A uniformly black visual field
Black is scene content – a dark world – not blindness: a device sampling it is given black. Use a
Scotomafor vision that is lost.The black source sits on a fixed 512 x 512 raster, which is what
render()falls back to; askrenderforsteporshapeto get another output resolution. Neither sets the grid a prosthetic model predicts on.Added in version 0.11.0.
- Parameters:
fov (float or
(width, height), optional) – Viewing window, in dva. The default 45 dva is a 45 x 45 dva disc.**kwargs – Any other
Sceneargument.aperturedefaults to'round'rather than'rectangular'.
Examples
>>> from pulse2percept.units import dva >>> from pulse2percept.vision import Scene >>> blank = Scene.blank(fov=60 * dva) >>> blank.fov (60.0, 60.0)
- property source
The picture itself, as an ImageStimulus or a VideoStimulus
- property scotoma
Where native vision is lost, or None if it is intact
- property background
What shows through a transparent source, as
(r, g, b)
- property scotoma_fill
Scotoma fill as a gray level, (r, g, b) triple, or ‘inpaint’.
Matplotlib color strings are stored as RGB triples.
- property scotoma_blend
Gaussian sigma, in dva, softening the drawn scotoma boundary
- property aperture
'rectangular'or'round'- Type:
Shape of the field’s support
- property fov
Viewing window
(width, height), in dva, centered on the fovea
- property extent
Source bounds
(left, right, bottom, top), in scene dva
- property shape
The
(rows, cols)of one frame
- property time
Frame times of the source, or None for a still scene
- property time_unit
The unit
timeis counted in
- pixel_to_dva(col, row)[source]
Visual-field coordinates of a pixel center
- Parameters:
col (float or array_like) – Pixel coordinates, where
(0, 0)is the center of the top-left pixel. Fractional values address points between pixel centers.row (float or array_like) – Pixel coordinates, where
(0, 0)is the center of the top-left pixel. Fractional values address points between pixel centers.
- Returns:
x, y – Scene coordinates in degrees of visual angle, as set by
extent.ygrows upwards, so row 0 has the largesty.- Return type:
np.ndarray
- dva_to_pixel(x, y)[source]
Pixel coordinates of a point in the scene
The inverse of
pixel_to_dva().- Parameters:
- Returns:
col, row – Continuous pixel coordinates, where
(0, 0)is the center of the top-left pixel. They are not rounded and not clipped to the frame: a point outsideextentmaps outside the pixel grid.- Return type:
np.ndarray
- fellow_eye()[source]
The homologous scene for the other eye
Reflects eye-specific geometry (e.g., scotoma) across the vertical meridian. The scene image is not flipped.
Added in version 0.11.0.
- Returns:
scene – A new scene. The original is left unchanged.
- Return type:
Examples
A loss 6 degrees into one eye’s right hemifield sits 6 degrees into the other eye’s left hemifield:
>>> import numpy as np >>> from pulse2percept.units import dva >>> from pulse2percept.vision import Scene, Scotoma >>> left = Scene(np.zeros((8, 8)), fov=40 * dva, ... scotoma=Scotoma.circle(3 * dva, center=(6, 0) * dva)) >>> right = left.fellow_eye() >>> float(right.scotoma(-6, 0)), float(right.scotoma(6, 0)) (1.0, 0.0)
- property n_frames
How many frames the source has; 1 for a still scene
- render(percept=None, gaze=None, vmax=None, vmin=None, step=None, shape=None)[source]
Rasterize this field onto one dense RGB percept
Residual native vision, with a prosthetic
perceptcomposed into the loss where there is a scotoma, on one common raster. Use it when a single RGB image is needed, for saving or for downstream image processing;plot()draws the same content without forcing a shared resolution.The raster spans the FOV in eye-centered coordinates,
[-fov / 2, fov / 2], whatever the gaze; gaze selects which part of the source fills it, and parts beyondextentare black. Its pitch defaults to the source’s angular pixel pitch. The source is resampled unless the raster lands on its pixel centers (e.g.fovmatchingextent, at zero gaze).steporshapechooses another raster; a finestepover a wide field is expensive by construction.Added in version 0.11.0.
- Parameters:
percept (
Percept, optional) – An eye-centered brightness percept, drawn at its visual-field coordinates within the FOV. With a scotoma it is composed into the loss as(1 - loss) * native + loss * max(scotoma_fill, phosphene); with none it is rendered alone on black, because superimposing it on intact native vision would assert an unmodeled interaction.scotoma_fill='inpaint'cannot be composed with one.gaze ((x, y), (n_frames, 2), or
Gaze, optional) – Where the eye is pointing: the scene location that falls on the fovea, in dva. Defaults to the origin. AGazeis resolved against the output clock: a video scene’s frame times, or a timed percept’s for a still scene.vmax (float, optional) – The percept brightness that displays as white. Defaults to the maximum brightness across the whole
percept.vmin (float, optional) – The percept brightness that displays as black. Defaults to 0.
step (float or (dx, dy), optional) – Angular sampling of the render raster, in dva. Rounded up, so the rendered pixels are never coarser than this. Mutually exclusive with
shape.shape ((rows, cols), optional) – The render raster itself. Mutually exclusive with
step.
- Returns:
percept – An RGB percept in
[0, 1]on the render raster, black outside the aperture. The source is left unchanged.- Return type:
Examples
>>> import numpy as np >>> from pulse2percept.units import dva >>> from pulse2percept.vision import Scene >>> scene = Scene(np.zeros((60, 80)), fov=(40, 30) * dva) >>> scene.render().shape (60, 80, 3, 1) >>> scene.render(step=0.25 * dva).shape (120, 160, 3, 1)
- plot(gaze=None, frame=0, ax=None, rings=False, meridians=False, grid_color='0.5', percept=None, vmax=None, vmin=None, view='scene', context_alpha=0.25, **kwargs)[source]
Plot what is left of native vision
The scene where vision is intact and
scotoma_fillwhere it is lost, inside the FOV at the givengaze.With
view='scene'(default), the axes spanextentin scene coordinates. The source stays fixed; the FOV, aperture, scotoma, percept and visual-field grid are centered ongaze. The source outside the FOV is drawn dimmed, ascontext_alpha * source.With
view='eye', the axes span[-fov / 2, fov / 2]in eye-centered coordinates, asrender()shows them, and gaze moves the source through the fixed window. Parts of the FOV beyondextentare black.Passing a
perceptdraws it in this field as well, so its size and place can be read against the FOV. Each layer keeps its own resolution: the source at its own angular pitch, the percept as a local patch on its own visual-field grid. Neither is resampled onto a common raster; seerender()for that.Inside the patch the layers compose as
(1 - loss) * native + loss * max(scotoma_fill, phosphene), so where the percept is dark the patch is ordinary residual vision and its boundary does not show. With no scotoma the percept is drawn alone on black inside the FOV, because superimposing it on intact native vision would assert an unmodeled interaction.- Parameters:
gaze ((x, y) or
Gaze, optional) – Where the eye is pointing: the scene location that falls on the fovea, in dva. Defaults to the origin. AGazeis resolved against the output clock, and the fixation held atframeis drawn.frame (int, optional) – Which frame of a video scene to draw. Ignored for a still scene.
ax (matplotlib.axes.Axes, optional) – The axes to draw on. If None, uses the current axes.
rings (bool, float, or sequence, optional) – Eccentricity rings (dva) about the fovea, at the center of the FOV. True draws 1.25, 2.5, 5, 10, 20, … dva, a number is a spacing, and a sequence is the eccentricities themselves. Automatic rings stop at the nearest FOV edge.
meridians (bool, float, or sequence, optional) – Polar-angle meridians (geometric deg) from the fovea to the FOV edge: 0 is +x, 90 is +y, counterclockwise. True is every 45 deg, a number is a spacing from 0, and a sequence is the angles themselves. Rings and meridians are display annotations only.
grid_color (color, optional) – Matplotlib color of rings, meridians, and ring labels.
percept (
Percept, optional) – An eye-centered brightness percept, drawn at its visual-field coordinates at its own resolution over the source.vmax (float, optional) – The percept brightness that displays as white. Defaults to the maximum brightness across the whole
percept.vmin (float, optional) – The percept brightness that displays as black. Defaults to 0.
view ({'scene', 'eye'}, optional) – Scene-centered axes spanning
extent(default), or eye-centered axes spanning the FOV.context_alpha (float, optional) – Opacity in [0, 1] of the source outside the FOV, over black, in the scene view: 0 is black, 1 is the undimmed source. Display only; ignored for
view='eye'.**kwargs – Passed on to
plot().
- Returns:
ax
- Return type:
- play(gaze=None, rings=False, meridians=False, grid_color='0.5', ax=None, *, percept=None, vmax=None, vmin=None, fps=None, repeat=True, annotate_time=True, fmt='png', title=None, view='scene', context_alpha=0.25)[source]
Animate a video scene, optionally with a prosthetic percept
Shows the content
render()composes for the samepercept,gaze,vmaxandvmin, on that result’s clock.With
view='scene'(default), the axes spanextent: the source stays fixed, the FOV moves with gaze, and the source outside the FOV is dimmed tocontext_alpha * source. Withview='eye', the frames are exactly thoserenderreturns: a fixed eye-centered FOV with the source moving through it.- Parameters:
gaze ((x, y), (n_frames, 2), or
Gaze, optional) – Where the eye is pointing, in dva. One pair fixates throughout; one pair per frame moves the eye between frames. AGazeis resolved against the scene’s frame times.rings (optional) – Visual-field grid, as in
plot(), painted into the displayed frames. The scene’s own data is not touched.meridians (optional) – Visual-field grid, as in
plot(), painted into the displayed frames. The scene’s own data is not touched.grid_color (optional) – Visual-field grid, as in
plot(), painted into the displayed frames. The scene’s own data is not touched.ax (matplotlib.axes.Axes, optional) – Axes to animate on. If None, the player makes its own.
percept (
Percept, optional) – A brightness percept composed into the scene as inrender().vmax (float, optional) – The percept brightness that displays as white. Defaults to the maximum brightness across the whole
percept.vmin (float, optional) – The percept brightness that displays as black. Defaults to 0.
fps (optional) – Player options, as in
play().repeat (optional) – Player options, as in
play().annotate_time (optional) – Player options, as in
play().fmt (optional) – Player options, as in
play().title (optional) – Player options, as in
play().view ({'scene', 'eye'}, optional) – Scene-centered frames spanning
extent(default), or eye-centered frames spanning the FOV.context_alpha (float, optional) – Opacity in [0, 1] of the source outside the FOV, as in
plot().
- Returns:
ani
- Return type:
- class pulse2percept.vision.Scotoma(mask, name=None)[source]
A region of the visual field where native vision is lost
A scotoma is eye-centered: it is defined in degrees of visual angle relative to the fovea, and it does not move when gaze does. Neither does an implant, which sits on the retina; the two hold their positions relative to each other while the scene moves past them.
A scotoma says only how much vision is lost where. What lost vision looks like – black, gray, blurred, filled in – is a separate question, and one for whoever composes the final image.
Added in version 0.11.0.
- Parameters:
mask (callable) –
mask(x, y)returning the loss at eye-centered visual-field coordinatesx,y(in dva). 0 is intact native vision and 1 is complete loss; anything in between is a partial defect, which is what leaves room for a measured or graded scotoma without another API.name (str, optional) – What to call this scotoma when it is printed.
Examples
A central geographic-atrophy scotoma 10 degrees across:
>>> from pulse2percept.vision import Scotoma >>> from pulse2percept.units import dva >>> scotoma = Scotoma.circle(5 * dva) >>> float(scotoma(0, 0)), float(scotoma(9, 0)) (1.0, 0.0)
- mirror(name=None)[source]
A copy reflected across the vertical meridian
Purely geometric:
mirrored(x, y) == original(-x, y)in eye-centered visual-field coordinates.Added in version 0.11.0.
- Parameters:
name (str, optional) – Name for the mirrored scotoma (default: original name, marked as mirrored)
- Returns:
scotoma – A new scotoma. The original is left unchanged.
- Return type:
Examples
The fellow eye of a bilateral, meridian-symmetric loss:
>>> from pulse2percept.units import dva >>> from pulse2percept.vision import Scotoma >>> left_scotoma = Scotoma.circle(3 * dva, center=(6, 0) * dva) >>> right_scotoma = left_scotoma.mirror() >>> float(right_scotoma(-6, 0)), float(right_scotoma(6, 0)) (1.0, 0.0)
- classmethod ellipse(x_radius, y_radius, center=(0, 0), name=None)[source]
An elliptical scotoma, lost inside and intact outside
- Parameters:
x_radius (float or Quantity) – Semi-axes of the ellipse, in degrees of visual angle.
y_radius (float or Quantity) – Semi-axes of the ellipse, in degrees of visual angle.
center ((x, y), optional) – Where the ellipse sits relative to the fovea, in dva. Defaults to the fovea itself.
name (str, optional) – What to call this scotoma when it is printed.