anvil.camera

anvil.camera

The cameras: a UVC camera streaming one mode, or one of the robot's cameras in a sim.

A Camera opens a device at one of the modes modes() lists and keeps its newest frame on a thread of its own, which any number of FrameReaders each read at their own pace. A SimCamera is the same over one of the robot's camera links in a sim, rendered at the mode it is asked for. Cameras opens every camera a workcell states, once, keyed by its name, from the devices or from a sim. devices(), controls() and set_control() find a host's cameras and read and write their settings.

A frame is a JPEG, the camera's own or, for a camera that sends only YUYV such as a RealSense D405, its frame encoded on the capture thread, with every stamp it was given: the driver's, which is its t_ns, and the capture thread's and the reader's, all in nanoseconds on CLOCK_MONOTONIC, what time.monotonic_ns() reads and the runtime stamps its ticks on, except where FrameStamps.driver_clock says the driver's is not.

Classes:

Name	Description
DriverClock	The clock a driver stamped a frame on, which says what its FrameStamps.driver_ns can be compared with.
StampSource	The moment of a frame a driver says its stamp marks.
FrameStamps	Every stamp a frame has, each kept on its own, all in nanoseconds.
Frame	One frame of a camera: its own JPEG and every stamp it was given, as FrameReader.frame returns it.
FrameCounts	What a camera has delivered and dropped since it opened, as Camera.counts returns it.
Mode	One mode a camera streams: a frame size and a frame rate, as modes() lists a device's.
VideoDevice	One video node on this host that streams video in, as devices() lists it.
ControlKind	How a camera control is set.
MenuChoice	One value a menu control takes, and what the driver calls it.
CameraControl	One setting a camera offers, with what it holds now, as controls() lists it.
Camera	A UVC camera streaming one mode, on a thread of its own.
SimCamera	One of the robot's cameras in a sim, read as any Camera is.
SimFreeCamera	A sim's free camera, read as any Camera is: the world as MuJoCo's own window shows it.
FrameReader	One reader of a camera's frames, with its own place in them, from Camera.reader.
Cameras	Every camera a workcell states, each opened once and keyed by its name: cameras["head"] is that Camera.

Functions:

Name	Description
devices	Returns every video node this user can open that streams video in as MJPEG or YUYV, sorted by path.
modes	Returns every mode device offers, in MJPEG where it has it, else YUYV, in the order its driver lists them.
controls	Returns every setting device offers, in the order its driver lists them, each with what it holds now.
set_control	Sets the setting key of device to value now, whether or not the camera streams.

DriverClock

Bases: StrEnum

The clock a driver stamped a frame on, which says what its FrameStamps.driver_ns can be compared with.

Attributes:

Name	Type	Description
MONOTONIC		CLOCK_MONOTONIC, the clock the runtime stamps its ticks on: every UVC camera's.
SIM		Simulated time, the clock the sim stamps its state on.
OTHER		Unknown, or one the driver does not name: not comparable with the runtime's stamps.

MONOTONIC class-attribute instance-attribute

MONOTONIC

CLOCK_MONOTONIC, the clock the runtime stamps its ticks on: every UVC camera's.

SIM class-attribute instance-attribute

SIM

Simulated time, the clock the sim stamps its state on.

A sim camera's frame carries the stamp of the tick it shows.

OTHER class-attribute instance-attribute

OTHER

Unknown, or one the driver does not name: not comparable with the runtime's stamps.

StampSource

Bases: StrEnum

The moment of a frame a driver says its stamp marks.

Attributes:

Name	Type	Description
END_OF_FRAME		The end of the frame.
START_OF_EXPOSURE		The start of the exposure.
OTHER		A source the driver does not name.

END_OF_FRAME class-attribute instance-attribute

END_OF_FRAME

The end of the frame.

START_OF_EXPOSURE class-attribute instance-attribute

START_OF_EXPOSURE

The start of the exposure.

uvcvideo claims this while stamping the frame's first USB packet, which is after it; a sim camera's render is the world at one instant, so its exposure starts and ends there.

OTHER class-attribute instance-attribute

OTHER

A source the driver does not name.

FrameStamps dataclass

FrameStamps(
    driver_ns: int,
    driver_clock: DriverClock,
    driver_source: StampSource,
    queued_ns: int,
    dequeued_ns: int,
    published_ns: int,
    read_ns: int,
)

Every stamp a frame has, each kept on its own, all in nanoseconds.

driver_ns is on the clock driver_clock names; the rest are on CLOCK_MONOTONIC, what time.monotonic_ns() reads.

Attributes:

Name	Type	Description
driver_ns	int	The driver's stamp on the frame, on the clock driver_clock names.
driver_clock	DriverClock	Which clock driver_ns is on.
driver_source	StampSource	Which moment of the frame the driver says driver_ns marks.
queued_ns	int	When the buffer went back to the driver to be filled.
dequeued_ns	int	When the capture thread took the filled buffer from the driver.
published_ns	int	When the capture thread made it the newest frame for readers.
read_ns	int	When the reader that returned it took it.

driver_ns instance-attribute

driver_ns: int

The driver's stamp on the frame, on the clock driver_clock names.

A UVC camera's is taken when the frame's first USB packet arrives, after the exposure, whatever driver_source says.

driver_clock instance-attribute

driver_clock: DriverClock

Which clock driver_ns is on.

driver_source instance-attribute

driver_source: StampSource

Which moment of the frame the driver says driver_ns marks.

queued_ns instance-attribute

queued_ns: int

When the buffer went back to the driver to be filled.

The nearest thing to a request, since a UVC camera pushes frames rather than being asked for them.

dequeued_ns instance-attribute

dequeued_ns: int

When the capture thread took the filled buffer from the driver.

published_ns instance-attribute

published_ns: int

When the capture thread made it the newest frame for readers.

read_ns instance-attribute

read_ns: int

When the reader that returned it took it.

Frame dataclass

Frame(
    t_ns: int,
    stamps: FrameStamps,
    sequence: int,
    width_px: int,
    height_px: int,
    jpeg: bytes,
)

One frame of a camera: its own JPEG and every stamp it was given, as FrameReader.frame returns it.

Attributes:

Name	Type	Description
t_ns	int	The frame's timestamp: the driver's, stamps.driver_ns, on the clock stamps.driver_clock names.
stamps	FrameStamps	Every stamp the frame has, t_ns among them.
sequence	int	The driver's frame counter.
width_px	int	Width of the frame, in pixels.
height_px	int	Height of the frame, in pixels.
jpeg	bytes	The JPEG: exactly as the camera produced it, or for a camera that sends YUYV, its frame encoded on the capture

t_ns instance-attribute

t_ns: int

The frame's timestamp: the driver's, stamps.driver_ns, on the clock stamps.driver_clock names.

stamps instance-attribute

stamps: FrameStamps

Every stamp the frame has, t_ns among them.

sequence instance-attribute

sequence: int

The driver's frame counter.

Between two frames one reader returned, a gap is frames the reader skipped or the driver dropped; Camera.counts tells the two apart.

width_px instance-attribute

width_px: int

Width of the frame, in pixels.

height_px instance-attribute

height_px: int

Height of the frame, in pixels.

jpeg instance-attribute

jpeg: bytes

The JPEG: exactly as the camera produced it, or for a camera that sends YUYV, its frame encoded on the capture thread.

FrameCounts dataclass

FrameCounts(frames: int, dropped: int)

What a camera has delivered and dropped since it opened, as Camera.counts returns it.

Attributes:

Name	Type	Description
frames	int	Frames delivered, counted as each arrives whether or not anyone reads it.
dropped	int	Frames the driver dropped: the gaps in its Frame.sequence.

frames instance-attribute

frames: int

Frames delivered, counted as each arrives whether or not anyone reads it.

dropped instance-attribute

dropped: int

Frames the driver dropped: the gaps in its Frame.sequence.

Mode dataclass

Mode(width_px: int, height_px: int, fps: int)

One mode a camera streams: a frame size and a frame rate, as modes() lists a device's.

Attributes:

Name	Type	Description
width_px	int	Width of a frame, in pixels.
height_px	int	Height of a frame, in pixels.
fps	int	Frames per second.

width_px instance-attribute

width_px: int

Width of a frame, in pixels.

height_px instance-attribute

height_px: int

Height of a frame, in pixels.

fps instance-attribute

fps: int

Frames per second.

VideoDevice dataclass

VideoDevice(device: str, name: str, aliases: list[str])

One video node on this host that streams video in, as devices() lists it.

Attributes:

Name	Type	Description
device	str	Where it is: its /dev/v4l/by-path/ link, else the node itself, e.g. /dev/video0.
name	str	What the driver calls the camera behind it.
aliases	list[str]	Every other path that reaches the same node right now.

device instance-attribute

device: str

Where it is: its /dev/v4l/by-path/ link, else the node itself, e.g. /dev/video0.

The by-path link names the USB port it is plugged into, and so stays put across reboots and replugs into that port. Not its /dev/v4l/by-id/ link: two cameras of one model often report one serial, and udev then gives the link to whichever was plugged in last.

name instance-attribute

name: str

What the driver calls the camera behind it.

aliases instance-attribute

aliases: list[str]

Every other path that reaches the same node right now.

The node itself and its other by-path and by-id links, so a camera saved under any of them is still recognized.

ControlKind

Bases: StrEnum

How a camera control is set.

Attributes:

Name	Type	Description
INTEGER		A number from CameraControl.min to CameraControl.max, in steps of CameraControl.step.
BOOLEAN		On (1) or off (0).
MENU		One of the values in CameraControl.menu.

INTEGER class-attribute instance-attribute

INTEGER

A number from CameraControl.min to CameraControl.max, in steps of CameraControl.step.

BOOLEAN class-attribute instance-attribute

BOOLEAN

On (1) or off (0).

MENU class-attribute instance-attribute

MENU

One of the values in CameraControl.menu.

MenuChoice dataclass

MenuChoice(value: int, label: str)

One value a menu control takes, and what the driver calls it.

Attributes:

Name	Type	Description
value	int	What is written to set it.
label	str	What the driver calls it, e.g. "Aperture Priority Mode".

value instance-attribute

value: int

What is written to set it.

label instance-attribute

label: str

What the driver calls it, e.g. "Aperture Priority Mode".

CameraControl dataclass

CameraControl(
    key: str,
    name: str,
    kind: ControlKind,
    min: int,
    max: int,
    step: int,
    default: int,
    value: int | None,
    menu: list[MenuChoice],
    inactive: bool,
    read_only: bool,
)

One setting a camera offers, with what it holds now, as controls() lists it.

The settings are those v4l2-ctl --list-ctrls-menus prints, keyed as it keys them.

Attributes:

Name	Type	Description
key	str	Its name in snake case, as v4l2-ctl keys it, e.g. "white_balance_temperature".
name	str	The driver's name for it, e.g. "White Balance Temperature".
kind	ControlKind	How it is set.
min	int	The least value it takes.
max	int	The greatest value it takes.
step	int	The step between the values it takes.
default	int	The value the driver starts it at.
value	int | None	What it holds now; None where the driver will not say, as for a write-only control.
menu	list[MenuChoice]	A menu control's choices, in the order the driver lists them; empty for any other kind.
inactive	bool	Whether another control decides it now, as auto exposure decides the exposure time.
read_only	bool	Whether it can only be read.

key instance-attribute

key: str

Its name in snake case, as v4l2-ctl keys it, e.g. "white_balance_temperature".

What set_control takes and a workcell saves it under.

name instance-attribute

name: str

The driver's name for it, e.g. "White Balance Temperature".

kind instance-attribute

kind: ControlKind

How it is set.

min instance-attribute

min: int

The least value it takes.

max instance-attribute

max: int

The greatest value it takes.

step instance-attribute

step: int

The step between the values it takes.

default instance-attribute

default: int

The value the driver starts it at.

value instance-attribute

value: int | None

What it holds now; None where the driver will not say, as for a write-only control.

menu instance-attribute

menu: list[MenuChoice]

A menu control's choices, in the order the driver lists them; empty for any other kind.

inactive instance-attribute

inactive: bool

Whether another control decides it now, as auto exposure decides the exposure time.

Setting it then may be refused or have no effect.

read_only instance-attribute

read_only: bool

Whether it can only be read.

Camera

Camera(
    device: str,
    mode: Mode,
    controls: Mapping[str, int] | None = None,
)

Bases: Handle

A UVC camera streaming one mode, on a thread of its own.

It streams MJPEG where the device offers it, passing the camera's own JPEG on, else YUYV, each frame of which the thread encodes to a JPEG: a RealSense D405's color stream is one such.

The thread keeps the newest frame whether or not anyone reads it; read frames through reader(), one reader for each consumer that wants every frame. A camera that delivers no frame for five seconds, or whose device is unplugged or fails, stops for good: error() says why, and every reader raises it. close() stops the stream and releases the device, as garbage collection does, and with Camera(device=..., mode=...) as camera: closes it on the way out. Every method but close raises anvil.OperationError once it is closed.

Opens device at mode and starts it streaming, returning once its first frame has arrived.

A UVC camera takes about a second to send its first frame, and one that sends none in five seconds is refused. Opening also turns off the camera's exposure_dynamic_framerate, wherever it has one, whatever controls says: auto exposure would otherwise lengthen the exposure in dim light and drop the frame rate below mode's.

Parameters:

Name	Type	Description	Default
device	str	The video node to open, e.g. /dev/video0 or a VideoDevice.device.	required
mode	Mode	The mode to stream, one of those modes(device) lists; anything else is refused.	required
controls	Mapping[str, int] | None	Settings to write before it streams, each a CameraControl.key and its value; a setting not named keeps what the camera holds. None or empty writes none. A setting another one decides, such as the exposure time under auto exposure, is written again once the rest are, so their order does not matter.	None

Raises:

Type	Description
InvalidArgumentError	A field of mode is negative or over 2**32 - 1; a value in controls is not an integer.
NotFoundError	Nothing is at device: the camera is unplugged, or was saved on another USB port.
BusyError	Another program has the camera open.
TimedOutError	It started streaming but sent no first frame in five seconds.
OperationError	device cannot be opened by this user or is not a streaming camera, it does not offer mode (the message lists the modes it does), a setting in controls is one it has not got or will not take, or its settings cannot be listed while controls names any.

Methods:

Name	Description
counts	Returns what this camera has delivered and dropped since it opened: a rate without reading every frame.
error	Returns why this camera stopped delivering for good, or None while it streams.
reader	Returns a new FrameReader of this camera.
close	Stops the stream and releases the device; closing twice does nothing.

Attributes:

Name	Type	Description
device	str | None	The video node it was opened at, as given; None for a SimCamera.

device instance-attribute

device: str | None

The video node it was opened at, as given; None for a SimCamera.

counts

counts() -> FrameCounts

Returns what this camera has delivered and dropped since it opened: a rate without reading every frame.

error

error() -> str | None

Returns why this camera stopped delivering for good, or None while it streams.

It stops when its device is unplugged or fails, or delivers no frame for five seconds; a SimCamera stops when its sim goes. The fix is a new camera once the device is back.

reader

reader() -> FrameReader

Returns a new FrameReader of this camera.

close

close() -> None

Stops the stream and releases the device; closing twice does nothing.

Every reader's frame then raises anvil.OperationError saying the camera is closed.

SimCamera

SimCamera(world: World, name: str, mode: Mode, lens: Lens)

Bases: Camera

One of the robot's cameras in a sim, read as any Camera is.

It renders one of the camera links of the robot in an anvil.sim.World at the mode it is asked for. Each frame shows one tick and carries that tick's stamp, on DriverClock.SIM. The sim renders a pinhole, so a fisheye lens is a very wide rectilinear view. Its frames stop, and error() says why, when the robot it watches is replaced by another or the world goes.

Opens the camera name of the robot in world, returning once its first frame is in.

Parameters:

Name	Type	Description	Default
world	World	The world whose robot it watches.	required
name	str	The robot's camera link, e.g. "left_wrist_cam", one of anvil.workcell.RobotConfig.camera_links().	required
mode	Mode	The size and rate to render at; each must be positive.	required
lens	Lens	The lens to render through, which sets the horizontal field of view.	required

Raises:

Type	Description
InvalidArgumentError	A field of mode is negative or over 2**32 - 1.
OperationError	No sim answers at world's address, the robot has no camera name (the message names those it has), mode has a zero, the sim is from another release, or no first frame came in six seconds.

Methods:

Name	Description
close	Stops the stream and releases the device; closing twice does nothing.
counts	Returns what this camera has delivered and dropped since it opened: a rate without reading every frame.
error	Returns why this camera stopped delivering for good, or None while it streams.
reader	Returns a new FrameReader of this camera.

Attributes:

Name	Type	Description
name		The robot's camera link it renders.

name instance-attribute

name

The robot's camera link it renders.

close

close() -> None

Stops the stream and releases the device; closing twice does nothing.

Every reader's frame then raises anvil.OperationError saying the camera is closed.

counts

counts() -> FrameCounts

Returns what this camera has delivered and dropped since it opened: a rate without reading every frame.

error

error() -> str | None

Returns why this camera stopped delivering for good, or None while it streams.

It stops when its device is unplugged or fails, or delivers no frame for five seconds; a SimCamera stops when its sim goes. The fix is a new camera once the device is back.

reader

reader() -> FrameReader

Returns a new FrameReader of this camera.

SimFreeCamera

SimFreeCamera(world: World, mode: Mode)

Bases: Camera

A sim's free camera, read as any Camera is: the world as MuJoCo's own window shows it.

It renders an anvil.sim.World from wherever anvil.sim.World.set_free_camera last put the camera, at the mode it is asked for. Each frame shows one tick and carries that tick's stamp, on DriverClock.SIM. Its frames stop, and error() says why, when the robot in the world is replaced by another or the world goes.

Opens the free camera of world, returning once its first frame is in.

Parameters:

Name	Type	Description	Default
world	World	The world it watches.	required
mode	Mode	The size and rate to render at; each must be positive.	required

Raises:

Type	Description
InvalidArgumentError	A field of mode is negative or over 2**32 - 1.
OperationError	No sim answers at world's address, mode has a zero, the sim is from another release, or no first frame came in six seconds.

Methods:

Name	Description
close	Stops the stream and releases the device; closing twice does nothing.
counts	Returns what this camera has delivered and dropped since it opened: a rate without reading every frame.
error	Returns why this camera stopped delivering for good, or None while it streams.
reader	Returns a new FrameReader of this camera.

close

close() -> None

Stops the stream and releases the device; closing twice does nothing.

Every reader's frame then raises anvil.OperationError saying the camera is closed.

counts

counts() -> FrameCounts

Returns what this camera has delivered and dropped since it opened: a rate without reading every frame.

error

error() -> str | None

Returns why this camera stopped delivering for good, or None while it streams.

It stops when its device is unplugged or fails, or delivers no frame for five seconds; a SimCamera stops when its sim goes. The fix is a new camera once the device is back.

reader

reader() -> FrameReader

Returns a new FrameReader of this camera.

FrameReader

FrameReader(camera: Camera)

Bases: Handle

One reader of a camera's frames, with its own place in them, from Camera.reader.

frame() returns the newest frame newer than the last this reader returned, whatever other readers took, so each consumer that wants every frame takes a reader of its own. A reader slower than the camera skips, and what it missed shows as a gap in Frame.sequence. A reader copies each frame into a buffer of its own, so only one thread reads through it; another thread takes another reader. It keeps its camera from being collected, but not from being closed. Every method but close raises anvil.OperationError once it is closed.

Makes a reader of camera that has had no frame yet, so its first is the newest.

Parameters:

Name	Type	Description	Default
camera	Camera	The camera to read.	required

Raises:

Type	Description
OperationError	camera is closed.

Methods:

Name	Description
frame	Returns the newest frame this reader has not had, waiting up to timeout_s for one.
close	Frees the reader while the camera streams on for its others; closing twice does nothing.

Attributes:

Name	Type	Description
camera		The camera it reads.

camera instance-attribute

camera

The camera it reads.

frame

frame(timeout_s: float = 1.0) -> Frame | None

Returns the newest frame this reader has not had, waiting up to timeout_s for one.

Parameters:

Name	Type	Description	Default
timeout_s	float	How long to wait for a frame, in seconds, to the millisecond; 0 returns only one already waiting.	1.0

Returns:

Type	Description
Frame | None	The frame, stamped read_ns as it is taken, or None when none came in timeout_s.

Raises:

Type	Description
InvalidArgumentError	timeout_s is negative.
OperationError	The camera has stopped, with why: most often its device went away, and the fix is a new Camera once it is back; or the camera was closed.

close

close() -> None

Frees the reader while the camera streams on for its others; closing twice does nothing.

Cameras

Cameras(
    workcell: WorkcellConfig, world: World | None = None
)

Bases: Mapping[str, Camera]

Every camera a workcell states, each opened once and keyed by its name: cameras["head"] is that Camera.

Whatever reads one, a recorder or a viewer, takes a Camera.reader() of its own, so each gets every frame. With a sim, each is instead the SimCamera of the robot link its anvil.workcell.CameraConfig.link names, at its mode, so a workcell a bench runs on metal runs in the sim as it is. close() closes them all, as with anvil.camera.Cameras(workcell=...) as cameras: does on the way out.

Opens every camera workcell states, in turn, each waiting for its first frame.

A camera that will not open, or an interrupt while one opens, closes the ones opened before it; a camera's anvil.OperationError is raised again as the same class, naming the camera.

Parameters:

Name	Type	Description	Default
workcell	WorkcellConfig	The workcell whose cameras to open.	required
world	World | None	The sim world to render them in instead of opening their devices, which are left alone; None opens the devices.	None

Raises:

Type	Description
InvalidArgumentError	workcell fails its checks, or, with world, a camera has no link, since the sim has no view from it.
NotFoundError	A camera's device is not plugged in.
BusyError	Another program has a camera's device open.
TimedOutError	A camera's device sent no first frame in time.
OperationError	A camera did not open, for any reason Camera or SimCamera gives.

Methods:

Name	Description
close	Closes every camera, after which each one's readers raise saying so; closing twice does nothing.

close

close() -> None

Closes every camera, after which each one's readers raise saying so; closing twice does nothing.

devices

devices() -> list[VideoDevice]

Returns every video node this user can open that streams video in as MJPEG or YUYV, sorted by path.

A UVC camera's second node, which carries metadata rather than frames, is not one, nor a depth camera's depth and infrared nodes: a RealSense D405 is listed once, by its color node. A node this user may not open is left out, and a warning on the anvil.camera logger names it and how to grant access.

modes

modes(device: str) -> list[Mode]

Returns every mode device offers, in MJPEG where it has it, else YUYV, in the order its driver lists them.

Asks without streaming, so it answers while the camera is open elsewhere.

Parameters:

Name	Type	Description	Default
device	str	The video node, e.g. /dev/video0 or a VideoDevice.device.	required

Returns:

Type	Description
list[Mode]	The modes; empty for a node that streams neither MJPEG nor YUYV, such as a depth camera's depth node, or one that is not a camera at all.

Raises:

Type	Description
InvalidArgumentError	device is empty.
NotFoundError	Nothing is at device.
OperationError	device cannot be opened by this user, or does not answer the listing.

controls

controls(device: str) -> list[CameraControl]

Returns every setting device offers, in the order its driver lists them, each with what it holds now.

Settings are what a number sets, such as brightness, exposure, white balance and focus; buttons, the headings that group the rest, text settings and those the driver has disabled are left out. Reads only, and works while the camera streams.

Parameters:

Name	Type	Description	Default
device	str	The video node, e.g. /dev/video0 or a VideoDevice.device.	required

Raises:

Type	Description
InvalidArgumentError	device is empty.
NotFoundError	Nothing is at device.
OperationError	device cannot be opened by this user, or does not list its settings. Some newer webcams list a setting of a kind the SDK cannot read, such as a region of interest, and none of their settings can be listed or set.

set_control

set_control(device: str, key: str, value: int) -> None

Sets the setting key of device to value now, whether or not the camera streams.

It holds until the camera is unplugged or set again; save it in the workcell's anvil.workcell.CameraConfig.controls to have it written every time the camera opens.

Parameters:

Name	Type	Description	Default
device	str	The video node, e.g. /dev/video0 or a VideoDevice.device.	required
key	str	The setting's CameraControl.key, e.g. "brightness".	required
value	int	The value to write, from its CameraControl.min to its CameraControl.max.	required

Raises:

Type	Description
InvalidArgumentError	device is empty, or has no setting key (the message lists those it has); value is outside the setting's range; or the driver refused value as invalid. Nothing was written.
NotFoundError	Nothing is at device.
BusyError	The driver will not change the setting now, as for one it locks while the camera streams.
OperationError	device cannot be opened by this user, or its settings cannot be listed; or the driver refused the write for another reason, as it may for an inactive setting.