pulse2percept.vision.scene

Scene

Classes

Scene(source, fov[, extent, scotoma, ...])

What is visually present, and where native vision is lost
class pulse2percept.vision.scene.Scene(source, fov, extent=None, scotoma=None, scotoma_fill=0, scotoma_blend=0.5, background=0, aperture='rectangular')[source]

What is visually present, and where native vision is lost

A scene places a picture in the world at a fixed angular extent, and views it through a field of view (fov) centered on the fovea, optionally with a region where native vision is missing.

Geometry follows one convention:

  • extent is the outer (left, right, bottom, top) of the source in scene coordinates, so it reaches half an angular pixel past the outermost pixel centers.

  • Pixel coordinates address pixel centers.

  • Row 0 is the top of the frame and therefore the largest y.

Two coordinate frames are used:

  • Scene coordinates are fixed world coordinates. extent places the source in them.

  • Eye coordinates are centered on the fovea. The FOV, aperture, scotoma, prosthetic percept, visual-field grid and an implant live here.

Gaze moves the viewing window through the scene; the source does not move or rescale:

(x_scene, y_scene) = (x_eye, y_eye) + (x_gaze, y_gaze)

plot() and play() default to view='scene': fixed axes spanning extent, with the FOV moving with gaze and the source outside it dimmed. view='eye' shows the FOV in eye coordinates instead, spanning [-fov / 2, fov / 2], as render() always does. There, parts of the FOV beyond extent are black. Gaze also sets what the device is given to encode, unless the implant’s scene_input_frame is 'head' (a head-fixed camera the eye cannot move). Device input is sampled from the whole extent, not only the FOV.

Source, extent and fov are fixed after construction.

The scotoma is native vision’s business only. What an implant is given to encode is sampled from the source itself, inside the scotoma as well as outside it: a camera does not go blind where its wearer has.

Added in version 0.11.0.

Parameters:

  • source (ImageStimulus, VideoStimulus, or image) – The background scene itself. Anything that is not already a ImageStimulus or a VideoStimulus, such as a file name or a NumPy array, is handed to ImageStimulus.

  • fov (float or (width, height)) – Size of the viewing window, in degrees of visual angle, centered on the fovea (e.g. 40 * dva). A scalar is a square window.

  • extent (float or (left, right, bottom, top), optional) – Where the source sits in scene coordinates, in dva. A scalar is the span of the shorter source dimension, centered with square pixels (e.g., 45 * dva on a 173 x 320 source is 83.2 x 45 dva). If None, the source is centered with square pixels and scaled to the smallest extent that contains fov: with a scalar fov, the shorter source dimension spans fov. extent need not contain fov.

  • scotoma (Scotoma, optional) – The region where native vision is lost. If None, native vision is intact everywhere and the scene is simply what is out there.

  • scotoma_fill (float, color, or 'inpaint', optional) – Scotoma fill: a gray level in [0, 1] (default 0, black), an (r, g, b) triple in [0, 1], or any Matplotlib color string. ‘inpaint’ fills from the surrounding image using skimage.restoration.inpaint_biharmonic() and ignores scotoma_blend. It cannot be used when composing a prosthetic percept.

  • background (float or (r, g, b), optional) – Gray level or RGB value to use for transparent pixels. Defaults to black.

  • scotoma_blend (float, optional) –

    Standard deviation, in degrees of visual angle, of a Gaussian blur applied to the rasterized loss map before it is drawn, softening the boundary from both sides. Defaults to 0.5 dva; 0 leaves it as sharp as the scotoma itself. Converted to pixels against whichever raster is being drawn, so the angular softness does not depend on resolution. Rendering only: the scotoma’s geometry is unchanged.

    Changed in version 0.11.0: Measured in degrees of visual angle rather than in scene pixels.

  • aperture ({'rectangular', 'round'}, optional) – Shape of the FOV. fov sets its size and this sets its shape: the default 'rectangular' shows the whole window, while 'round' inscribes an eye-centered ellipse of semi-axes fov / 2 in it, so a square fov renders as a disc and a non-square one as an ellipse. The aperture is a display boundary: plot() clips the full-intensity FOV, percept and grid to it (scene-view context stays visible outside), render() writes black outside it, while scene sampling, device input, stimulation and the prosthetic model response are untouched.

Examples

A 40-degree window onto a logo, seen with a central 16-degree scotoma. The logo is landscape, so its height spans the window:

>>> from pulse2percept.stimuli import samples
>>> from pulse2percept.units import dva
>>> from pulse2percept.vision import Scene, Scotoma
>>> scene = Scene(samples.logo_bvl(), fov=40 * dva,
...               scotoma=Scotoma.circle(8 * dva))
>>> scene.fov, scene.extent
((40.0, 40.0), (-25.0, 25.0, -20.0, 20.0))

A 40-degree disc onto a wider world. render covers the FOV at the source’s angular pixel pitch:

>>> world = Scene(samples.logo_bvl(), extent=(-50, 50, -40, 40) * dva,
...               fov=40 * dva, aperture='round')
>>> world.render().shape
(288, 288, 3, 1)

blank() gives a black field instead of a picture – darkness, not blindness:

>>> blank = Scene.blank()
>>> blank.plot()
classmethod blank(fov=45, **kwargs)[source]

A uniformly black visual field

Black is scene content – a dark world – not blindness: a device sampling it is given black. Use a Scotoma for vision that is lost.

The black source sits on a fixed 512 x 512 raster, which is what render() falls back to; ask render for step or shape to get another output resolution. Neither sets the grid a prosthetic model predicts on.

Added in version 0.11.0.

Parameters:

  • fov (float or (width, height), optional) – Viewing window, in dva. The default 45 dva is a 45 x 45 dva disc.

  • **kwargs – Any other Scene argument. aperture defaults to 'round' rather than 'rectangular'.

Examples

>>> from pulse2percept.units import dva
>>> from pulse2percept.vision import Scene
>>> blank = Scene.blank(fov=60 * dva)
>>> blank.fov
(60.0, 60.0)
property source

The picture itself, as an ImageStimulus or a VideoStimulus

property scotoma

Where native vision is lost, or None if it is intact

property background

What shows through a transparent source, as (r, g, b)

property scotoma_fill

Scotoma fill as a gray level, (r, g, b) triple, or ‘inpaint’.

Matplotlib color strings are stored as RGB triples.

property scotoma_blend

Gaussian sigma, in dva, softening the drawn scotoma boundary

property aperture

'rectangular' or 'round'

Type:

Shape of the field’s support

property fov

Viewing window (width, height), in dva, centered on the fovea

property extent

Source bounds (left, right, bottom, top), in scene dva

property shape

The (rows, cols) of one frame

property time

Frame times of the source, or None for a still scene

property time_unit

The unit time is counted in

pixel_to_dva(col, row)[source]

Visual-field coordinates of a pixel center

Parameters:

  • col (float or array_like) – Pixel coordinates, where (0, 0) is the center of the top-left pixel. Fractional values address points between pixel centers.

  • row (float or array_like) – Pixel coordinates, where (0, 0) is the center of the top-left pixel. Fractional values address points between pixel centers.

Returns:

x, y – Scene coordinates in degrees of visual angle, as set by extent. y grows upwards, so row 0 has the largest y.

Return type:

np.ndarray

dva_to_pixel(x, y)[source]

Pixel coordinates of a point in the scene

The inverse of pixel_to_dva().

Parameters:

  • x (float or array_like) – Scene coordinates in degrees of visual angle, as set by extent.

  • y (float or array_like) – Scene coordinates in degrees of visual angle, as set by extent.

Returns:

col, row – Continuous pixel coordinates, where (0, 0) is the center of the top-left pixel. They are not rounded and not clipped to the frame: a point outside extent maps outside the pixel grid.

Return type:

np.ndarray

fellow_eye()[source]

The homologous scene for the other eye

Reflects eye-specific geometry (e.g., scotoma) across the vertical meridian. The scene image is not flipped.

Added in version 0.11.0.

Returns:

scene – A new scene. The original is left unchanged.

Return type:

Scene

Examples

A loss 6 degrees into one eye’s right hemifield sits 6 degrees into the other eye’s left hemifield:

>>> import numpy as np
>>> from pulse2percept.units import dva
>>> from pulse2percept.vision import Scene, Scotoma
>>> left = Scene(np.zeros((8, 8)), fov=40 * dva,
...              scotoma=Scotoma.circle(3 * dva, center=(6, 0) * dva))
>>> right = left.fellow_eye()
>>> float(right.scotoma(-6, 0)), float(right.scotoma(6, 0))
(1.0, 0.0)
property n_frames

How many frames the source has; 1 for a still scene

render(percept=None, gaze=None, vmax=None, vmin=None, step=None, shape=None)[source]

Rasterize this field onto one dense RGB percept

Residual native vision, with a prosthetic percept composed into the loss where there is a scotoma, on one common raster. Use it when a single RGB image is needed, for saving or for downstream image processing; plot() draws the same content without forcing a shared resolution.

The raster spans the FOV in eye-centered coordinates, [-fov / 2, fov / 2], whatever the gaze; gaze selects which part of the source fills it, and parts beyond extent are black. Its pitch defaults to the source’s angular pixel pitch. The source is resampled unless the raster lands on its pixel centers (e.g. fov matching extent, at zero gaze). step or shape chooses another raster; a fine step over a wide field is expensive by construction.

Added in version 0.11.0.

Parameters:

  • percept (Percept, optional) – An eye-centered brightness percept, drawn at its visual-field coordinates within the FOV. With a scotoma it is composed into the loss as (1 - loss) * native + loss * max(scotoma_fill, phosphene); with none it is rendered alone on black, because superimposing it on intact native vision would assert an unmodeled interaction. scotoma_fill='inpaint' cannot be composed with one.

  • gaze ((x, y), (n_frames, 2), or Gaze, optional) – Where the eye is pointing: the scene location that falls on the fovea, in dva. Defaults to the origin. A Gaze is resolved against the output clock: a video scene’s frame times, or a timed percept’s for a still scene.

  • vmax (float, optional) – The percept brightness that displays as white. Defaults to the maximum brightness across the whole percept.

  • vmin (float, optional) – The percept brightness that displays as black. Defaults to 0.

  • step (float or (dx, dy), optional) – Angular sampling of the render raster, in dva. Rounded up, so the rendered pixels are never coarser than this. Mutually exclusive with shape.

  • shape ((rows, cols), optional) – The render raster itself. Mutually exclusive with step.

Returns:

percept – An RGB percept in [0, 1] on the render raster, black outside the aperture. The source is left unchanged.

Return type:

Percept

Examples

>>> import numpy as np
>>> from pulse2percept.units import dva
>>> from pulse2percept.vision import Scene
>>> scene = Scene(np.zeros((60, 80)), fov=(40, 30) * dva)
>>> scene.render().shape
(60, 80, 3, 1)
>>> scene.render(step=0.25 * dva).shape
(120, 160, 3, 1)
plot(gaze=None, frame=0, ax=None, rings=False, meridians=False, grid_color='0.5', percept=None, vmax=None, vmin=None, view='scene', context_alpha=0.25, **kwargs)[source]

Plot what is left of native vision

The scene where vision is intact and scotoma_fill where it is lost, inside the FOV at the given gaze.

With view='scene' (default), the axes span extent in scene coordinates. The source stays fixed; the FOV, aperture, scotoma, percept and visual-field grid are centered on gaze. The source outside the FOV is drawn dimmed, as context_alpha * source.

With view='eye', the axes span [-fov / 2, fov / 2] in eye-centered coordinates, as render() shows them, and gaze moves the source through the fixed window. Parts of the FOV beyond extent are black.

Passing a percept draws it in this field as well, so its size and place can be read against the FOV. Each layer keeps its own resolution: the source at its own angular pitch, the percept as a local patch on its own visual-field grid. Neither is resampled onto a common raster; see render() for that.

Inside the patch the layers compose as (1 - loss) * native + loss * max(scotoma_fill, phosphene), so where the percept is dark the patch is ordinary residual vision and its boundary does not show. With no scotoma the percept is drawn alone on black inside the FOV, because superimposing it on intact native vision would assert an unmodeled interaction.

Parameters:

  • gaze ((x, y) or Gaze, optional) – Where the eye is pointing: the scene location that falls on the fovea, in dva. Defaults to the origin. A Gaze is resolved against the output clock, and the fixation held at frame is drawn.

  • frame (int, optional) – Which frame of a video scene to draw. Ignored for a still scene.

  • ax (matplotlib.axes.Axes, optional) – The axes to draw on. If None, uses the current axes.

  • rings (bool, float, or sequence, optional) – Eccentricity rings (dva) about the fovea, at the center of the FOV. True draws 1.25, 2.5, 5, 10, 20, … dva, a number is a spacing, and a sequence is the eccentricities themselves. Automatic rings stop at the nearest FOV edge.

  • meridians (bool, float, or sequence, optional) – Polar-angle meridians (geometric deg) from the fovea to the FOV edge: 0 is +x, 90 is +y, counterclockwise. True is every 45 deg, a number is a spacing from 0, and a sequence is the angles themselves. Rings and meridians are display annotations only.

  • grid_color (color, optional) – Matplotlib color of rings, meridians, and ring labels.

  • percept (Percept, optional) – An eye-centered brightness percept, drawn at its visual-field coordinates at its own resolution over the source.

  • vmax (float, optional) – The percept brightness that displays as white. Defaults to the maximum brightness across the whole percept.

  • vmin (float, optional) – The percept brightness that displays as black. Defaults to 0.

  • view ({'scene', 'eye'}, optional) – Scene-centered axes spanning extent (default), or eye-centered axes spanning the FOV.

  • context_alpha (float, optional) – Opacity in [0, 1] of the source outside the FOV, over black, in the scene view: 0 is black, 1 is the undimmed source. Display only; ignored for view='eye'.

  • **kwargs – Passed on to plot().

Returns:

ax

Return type:

matplotlib.axes.Axes

play(gaze=None, rings=False, meridians=False, grid_color='0.5', ax=None, *, percept=None, vmax=None, vmin=None, fps=None, repeat=True, annotate_time=True, fmt='png', title=None, view='scene', context_alpha=0.25)[source]

Animate a video scene, optionally with a prosthetic percept

Shows the content render() composes for the same percept, gaze, vmax and vmin, on that result’s clock.

With view='scene' (default), the axes span extent: the source stays fixed, the FOV moves with gaze, and the source outside the FOV is dimmed to context_alpha * source. With view='eye', the frames are exactly those render returns: a fixed eye-centered FOV with the source moving through it.

Parameters:

  • gaze ((x, y), (n_frames, 2), or Gaze, optional) – Where the eye is pointing, in dva. One pair fixates throughout; one pair per frame moves the eye between frames. A Gaze is resolved against the scene’s frame times.

  • rings (optional) – Visual-field grid, as in plot(), painted into the displayed frames. The scene’s own data is not touched.

  • meridians (optional) – Visual-field grid, as in plot(), painted into the displayed frames. The scene’s own data is not touched.

  • grid_color (optional) – Visual-field grid, as in plot(), painted into the displayed frames. The scene’s own data is not touched.

  • ax (matplotlib.axes.Axes, optional) – Axes to animate on. If None, the player makes its own.

  • percept (Percept, optional) – A brightness percept composed into the scene as in render().

  • vmax (float, optional) – The percept brightness that displays as white. Defaults to the maximum brightness across the whole percept.

  • vmin (float, optional) – The percept brightness that displays as black. Defaults to 0.

  • fps (optional) – Player options, as in play().

  • repeat (optional) – Player options, as in play().

  • annotate_time (optional) – Player options, as in play().

  • fmt (optional) – Player options, as in play().

  • title (optional) – Player options, as in play().

  • view ({'scene', 'eye'}, optional) – Scene-centered frames spanning extent (default), or eye-centered frames spanning the FOV.

  • context_alpha (float, optional) – Opacity in [0, 1] of the source outside the FOV, as in plot().

Returns:

ani

Return type:

HTMLAnimation