Metadata-Version: 2.5
Name: ubo-app
Version: 2.0.1.dev260811103975110156
Summary: Ubo main app, running on device initialization. A platform for running other apps.
Author-email: Mehrdad M <mehrdad@getubo.com>, Sassan Haradji <me@sassanh.com>
Maintainer-email: Mehrdad M <mehrdad@getubo.com>, Sassan Haradji <me@sassanh.com>
License-Expression: Apache-2.0
Keywords: home assistance,raspberry pi,rpi,ubo,ubo-pod
Requires-Python: <3.12,>=3.11
Requires-Dist: adafruit-blinka-raspberry-pi5-neopixel==1.0.0rc2; platform_machine == 'aarch64' and sys_platform == 'linux'
Requires-Dist: adafruit-blinka>=9.2.0
Requires-Dist: adafruit-circuitpython-ahtx0==1.0.28
Requires-Dist: adafruit-circuitpython-aw9523==1.1.12
Requires-Dist: adafruit-circuitpython-bh1750==1.1.14
Requires-Dist: adafruit-circuitpython-bme280==2.6.28
Requires-Dist: adafruit-circuitpython-bme680==3.7.16
Requires-Dist: adafruit-circuitpython-bmp3xx==1.3.27
Requires-Dist: adafruit-circuitpython-ens160==1.1.2
Requires-Dist: adafruit-circuitpython-irremote==5.0.4
Requires-Dist: adafruit-circuitpython-mcp9808==3.3.26
Requires-Dist: adafruit-circuitpython-neopixel==6.3.16
Requires-Dist: adafruit-circuitpython-pct2075==1.1.24
Requires-Dist: adafruit-circuitpython-pixelbuf==2.0.8
Requires-Dist: adafruit-circuitpython-pm25==2.1.24
Requires-Dist: adafruit-circuitpython-rgb-display==3.14.0
Requires-Dist: adafruit-circuitpython-scd4x==1.4.12
Requires-Dist: adafruit-circuitpython-sgp40==1.3.24
Requires-Dist: adafruit-circuitpython-sht4x==1.0.22
Requires-Dist: adafruit-circuitpython-veml7700==2.1.3
Requires-Dist: adafruit-circuitpython-vl53l1x==1.1.12
Requires-Dist: aiofiles==24.1.0
Requires-Dist: aiohttp==3.12.9
Requires-Dist: aiomqtt==2.4.0
Requires-Dist: aiostream==0.6.4
Requires-Dist: betterproto[compiler]==2.0.0b7
Requires-Dist: dill==0.4.0
Requires-Dist: docker==7.1.0
Requires-Dist: fasteners==0.19
Requires-Dist: google-cloud-aiplatform==1.96.0
Requires-Dist: google-cloud-speech==2.32.0
Requires-Dist: gpiozero==2.0.1; platform_machine != 'aarch64'
Requires-Dist: headless-kivy==0.13.0
Requires-Dist: netifaces==0.11.0
Requires-Dist: ollama==0.5.1
Requires-Dist: onnxruntime>=1.22.0; sys_platform == 'linux' and platform_machine == 'aarch64'
Requires-Dist: opencv-python==4.10.0.84
Requires-Dist: openwakeword==0.6.0
Requires-Dist: pillow>=11.3.0
Requires-Dist: platformdirs
Requires-Dist: psutil==7.0.0
Requires-Dist: pulsectl==24.12.0
Requires-Dist: pyalsaaudio==0.11.0; platform_machine == 'aarch64' and sys_platform == 'linux'
Requires-Dist: pyaudio==0.2.14; platform_machine != 'aarch64' or sys_platform != 'linux'
Requires-Dist: pymicro-wakeword==2.4.1
Requires-Dist: pypng==0.20220715.0
Requires-Dist: python-debouncer==0.1.5
Requires-Dist: python-dotenv==1.1.0
Requires-Dist: python-fake==0.2.0
Requires-Dist: python-redux==0.25.1
Requires-Dist: python-strtobool==1.0.3
Requires-Dist: pyzbar==0.1.9
Requires-Dist: quart==0.20.0
Requires-Dist: rpi-lgpio==0.6; platform_machine == 'aarch64' and sys_platform == 'linux'
Requires-Dist: rpi-ws281x==5.0.0; platform_machine == 'aarch64'
Requires-Dist: sdbus-networkmanager==2.0.0; platform_machine == 'aarch64'
Requires-Dist: sentry-sdk==2.29.1
Requires-Dist: simpleaudio==1.0.4
Requires-Dist: soxr==0.5.0.post1
Requires-Dist: speexdsp-ns==0.1.2; sys_platform == 'linux'
Requires-Dist: tenacity==9.1.2
Requires-Dist: ubo-app-raw-bindings
Requires-Dist: ubo-gui==0.13.17
Requires-Dist: vosk==0.3.44
Requires-Dist: wyoming[zeroconf]==1.10.0
Provides-Extra: test-investigation
Requires-Dist: graphviz>=0.20.3; extra == 'test-investigation'
Requires-Dist: objgraph>=3.6.2; extra == 'test-investigation'
Description-Content-Type: text/markdown

# ☯️ Ubo App

[![PyPI version](https://img.shields.io/pypi/v/ubo-app.svg)](https://pypi.python.org/pypi/ubo-app)
[![License](https://img.shields.io/pypi/l/ubo-app.svg)](https://github.com/ubopod/ubo-app/LICENSE)
[![Python version](https://img.shields.io/pypi/pyversions/ubo-app.svg)](https://pypi.python.org/pypi/ubo-app)
[![Actions status](https://github.com/ubopod/ubo-app/workflows/CI/CD/badge.svg)](https://github.com/ubopod/ubo-app/actions)
[![codecov](https://codecov.io/gh/ubopod/ubo-app/graph/badge.svg?token=KUI1KRDDY0)](https://codecov.io/gh/ubopod/ubo-app)
[![Ask DeepWiki](https://deepwiki.com/badge.svg)](https://deepwiki.com/ubopod/ubo_app)

## 📑 Table of Contents

- [🌟 Overview](#🌟-overview)
- [🚧 Disclaimer](#🚧-disclaimer)
- [⚙️ Notable Features](#⚙️-notable-features)
- [📋 Requirements](#📋-requirements)
- [🪏 Installation](#🪏-installation)
  - [Pre-packaged image](#pre-packaged-image)
  - [Install on existing OS](#install-on-existing-os)
  - [ESP32 satellites](#esp32-satellites)
  - [Mobile and wearable apps](#mobile-and-wearable-apps)
- [🤝 Contributing](#🤝-contributing)
  - [ℹ️️ Conventions](#ℹ️️-conventions)
  - [Development](#development)
- [🛠️ Hardware](#🛠️-hardware)
  - [Emulation](#emulation)
  - [Ubo Pod](#ubo-pod)
  - [DIY Path](#diy-path)
- [🏗️ Architecture](#🏗️-architecture)
- [📦 Notable dependencies](#📦-notable-dependencies)
- [🗺️ Roadmap](#🗺️-roadmap)
- [🔒 License](#🔒-license)

## 🌟 Overview

Ubo App is a Python application that provides a unified interface and tools for developing and running hardware-integrated apps. 

It offers a minimalistic, yet intuitive UI for end-users to install and interact with developer apps. It is optimized for Raspberry Pi (4 & 5) devices. 

Hardware specific capabilities such as infrared send/receive, sensing, LED ring, etc. are supported by Ubo Pod hardware. It is also possible to DIY your own hardware, see the [hardware DIY section](#diy-path) below.

The pod is not the only surface. [**ESP32 satellites**](#esp32-satellites) are companion
boards that extend a pod with an extra screen, audio and more, over WiFi or USB, and
experimental [**phone and watch apps**](#mobile-and-wearable-apps) for iOS, watchOS,
Android and Wear OS provide detached hardware remotely.

### Goals

The design is centered around the following goals:

  - Making hardware-integarted app development easier 
  - Offer no-code/no-terminal UI/UX optionsto both developers and end-users of their apps
  - Give developers tools to build apps with multi-modal UX
  - Leverage tight hardware and software co-development to unlock new potentials
  - Let users focus on their app logic while Ubo app handles the rest (hardware abstractions, UI, etc.)
  - Hot-pluggable services
  - Modular and friendly to AI tool-calling
  - Remote API access (gRPC)

⚠️ Due to limited development resources, we are not able to support every single board computer (SBC), operating system, and hardware configuration. 

If you are willing to supporting other SBCs or operating systems, please consider contributing to the project.

<b>Example GUI screenshots</b>

![GUI Overview](https://raw.githubusercontent.com/ubopod/mediakit/main/images/gui-overview.png))

## 🚧 Disclaimer

Be aware that at the moment, Ubo app sends crash reports to Sentry. Soon we will limit this to beta versions only.

## ⚙️ Notable Features

- Easy WiFi on-boarding with QR code or hotspot  
- Headless (no monitor/keyboard) remote access setup 
    - SSH
    - VS Code tunnel
    - Raspberry Pi Connect
- Install and run Dockerized apps headlessly
- One-click install for pre-configured apps
- Access and control basic Linux utilities and settings
  - User management
  - Network management
  - File system operations
- Natural language interactions for tool calling (voice AI) (experimental)
- User-defined short voice commands with regex-style phrase patterns that map
  utterances to bindable actions
- Web UI
- Infrared remote control (send/receive), including Web UI assignment of
  registered IR keys to bindable actions
- gRPC API for remote control - find sample clients [here](https://github.com/ubopod/ubo-grpc-clients)
- ESP32 satellites - companion boards
  ([Waveshare ESP32-C6-Touch-AMOLED-1.8](https://www.waveshare.com/esp32-c6-touch-amoled-1.8.htm),
  [Espressif ESP32-S3-BOX-3](https://github.com/espressif/esp-box)) that extend the pod
  over WiFi or USB: the same GUI rendered natively in C/LVGL with touch navigation,
  audio playback and capture, on-device WiFi setup via a captive portal, and — on the
  ESP32-S3-BOX-3 — far-field microphones with an on-device wake word
- Native phone and watch clients for iOS, watchOS, Android and Wear OS (including an
  Android Glance widget) - experimental/beta, [build from source](#mobile-and-wearable-apps)

Check [roadmap section](#🗺️-roadmap) below for upcoming features.

## 📋 Requirements

At minimum you need a Raspberry Pi 4 or 5 to run Ubo App. 

To run LLM models locally, we recommend a Raspberry Pi 5 with at least 8GB of RAM.

For features that require add-on hardware that is not natively supported by Raspberry Pi (such as audio, infrared rx/tx, sensors, etc), you can:

1. Purchase an Ubo Pod Development Kit 
2. DIY the hardware
3. Use only subset of hardware features emulated in the browser

For more details check out the [hardware section](#🛠️-hardware) below.

🙏 Please consider supporting this project by pre-ordering an Ubo Pod Dev Edition on [Kickstarter](https://www.kickstarter.com/projects/ubopod/ubo-pod-hackable-personal-ai-assistant). 

The sales proceeds from the hardware will be used to support continued development and maintenance of Ubo App and its open source dependencies.

<b> Note </b>: 
The app still functions even if some special hardware elements (audio, infrared rx/tx, sensors, etc) are not provided. The features that rely on these hardware components just won't function. For example, WiFi onboarding with QR code requires a camera onboard. 

## 🪏 Installation

### Pre-packaged image

Ubo Pod ships with a pre-flashed MicroSD card that has the app installed on it by default.

If you don't have it, or you just want to set up a fresh device, then:

1. Download one of the images from the release section
1. Use Raspberry Pi Images and choose `custom image` to provide the download image file.
1. Write to the image
1. Use the image to boot your Ubo Pod or Raspberry Pi

This is the fastest, easiest, and recommended way to get started with Ubo App. 

🙋‍♂️If this is the first time you are flashing an image for Raspberry Pi, I recommend following the more detailed steps [here](https://github.com/ubopod/ubo-image).

To run the app on bare Raspberry Pi, you can watch this short [demo video](https://www.youtube.com/watch?v=Rro3YLVIUx4).

### Install on existing OS

If you want to install the image on an existing operating system, then read on. Otherwise, skip this section.

---

⚠️ **Executing scripts directly from the internet with root privileges poses a significant security risk. It's generally a good practice to ensure you understand the script's content before running it. You can check the content of this particular script [here](https://raw.githubusercontent.com/ubopod/ubo-app/main/ubo_app/system/install.sh) before running it.**

---

To install ubo, run this command in a terminal shell:

```bash
curl -sSL https://raw.githubusercontent.com/ubopod/ubo-app/main/ubo_app/system/scripts/install.sh | sudo bash
```

If you don't want to install docker service you can set the `WITH_DOCKER` environment variable to `false`:

```bash
curl -sSL https://raw.githubusercontent.com/ubopod/ubo-app/main/ubo_app/system/scripts/install.sh | sudo WITHOUT_DOCKER=true bash
```

The installer also provisions the `uv`/`uvx` and Node.js/`npx` runtimes (under the `ubo` user) so the MCP gateway can launch stdio-based MCP servers. To skip either, set the `WITHOUT_UV` or `WITHOUT_NODE` environment variable to `true`:

```bash
curl -sSL https://raw.githubusercontent.com/ubopod/ubo-app/main/ubo_app/system/scripts/install.sh | sudo WITHOUT_UV=true WITHOUT_NODE=true bash
```

To install a specific version of ubo, you can set the `TARGET_VERSION` environment variable to the desired version:

```bash
curl -sSL https://raw.githubusercontent.com/ubopod/ubo-app/main/ubo_app/system/scripts/install.sh | sudo TARGET_VERSION=0.0.1 bash
```

Note that as part of the installation process, these debian packages are installed:

- accountsservice
- dhcpcd
- dnsmasq
- git
- hostapd
- i2c-tools
- ir-keytable
- libasound2-dev
- libcap-dev
- libegl1
- libgl1
- libmtdev1
- libzbar0
- python3-alsaaudio
- python3-apt
- python3-dev
- python3-gpiozero
- python3-libcamera
- python3-picamera2
- python3-pip
- python3-virtualenv
- rpi-lgpio

Also be aware that ubo-app only installs in `/opt/ubo` and it is not customizable
at the moment.

### ESP32 satellites

An ESP32 satellite is a companion board for an existing Ubo Pod (or any host running
ubo-app) — it is not a standalone install. Set up the pod first, then flash the
satellite.

Supported boards:

| Board | Hardware | Status |
| --- | --- | --- |
| Waveshare **ESP32-C6-Touch-AMOLED-1.8** | SH8601 368×448 AMOLED, FT3168 touch, ES8311 audio in/out | complete, verified on-device |
| Espressif **ESP32-S3-BOX-3** | ILI9341 320×240 LCD, GT911 touch, ES8311 out + ES7210 2-mic array, wake word | bring-up in progress |

**No toolchain required.** Every release ships a prebuilt merged firmware image:

1. Download `ubo-lvgl-esp32c6-<version>-merged.bin` from the
   [Releases](https://github.com/ubopod/ubo_app/releases) page. Pick the release matching
   your installed ubo-app version — the client's protobuf schema must match the core it
   talks to.
1. Connect the board over USB and open
   [ESPConnect](https://thelastoutpostworkshop.github.io/ESPConnect/) in Chrome or Edge
   (Web Serial), then select the board's serial port.
1. In the **Flash** tab, choose the `.bin`, set the offset to **`0x0`**, enable
   **Erase before flash**, and flash.
1. After the reboot, join the open `ubo-setup` WiFi access point from a phone; the
   captive portal asks for your network and, optionally, the ubo-core host/port. The
   device saves them and reboots onto your network.

To move the board to a different network later, hold the BOOT button for ~8 seconds to
clear the stored credentials and return to `ubo-setup`.

Boards cabled to a Ubo Pod can carry their traffic **over the USB cable itself** (PPP
over USB) instead of WiFi — the `ppp` firmware profile, which is what the pod build
ships.

Full details — pin maps, the WiFi setup journey, the USB/PPP link, wake word setup, and
per-board status — are in [`ubo_lvgl/esp32/README.md`](ubo_lvgl/esp32/README.md).

### Mobile and wearable apps

---

⚠️ **These apps are experimental and still in beta.** They are **not published on the
App Store or Google Play**, and there is no timeline for that yet. The only way to try
them today is to **build them from source** yourself, which means a working Xcode or
Android Studio setup and a developer account for on-device installs. Expect rough
edges, breaking changes, and features that only work against a matching ubo-app version.

---

Native clients that connect to a Ubo Pod over the [gRPC API](#🏗️-architecture)
(port `50051`) and render the same UI remotely — they are thin renderers, so all logic
stays on the pod. Each platform is split into a bindings package (generated from this
repo's protobuf definitions, plus hand-written wrappers) and the app itself:

| Platform | App | gRPC bindings |
| --- | --- | --- |
| iOS + watchOS (SwiftUI) | [`ubo-swift-app`](https://github.com/ubopod/ubo-swift-app) | [`ubo-swift-grpc`](https://github.com/ubopod/ubo-swift-grpc) |
| Android + Wear OS, incl. a Glance widget (Kotlin) | [`ubo-kotlin-apps`](https://github.com/ubopod/ubo-kotlin-apps) | [`ubo-kotlin-grpc`](https://github.com/ubopod/ubo-kotlin-grpc) |

The app repos pull in their bindings package as a dependency, so for a plain build you
only need the app repo — clone the bindings repo too if you want to build against local
protobuf changes. Build instructions live in each repository.

Toolchain and deployment targets: **iOS 18 / watchOS 11** (Xcode, SwiftPM) and
**Android minSdk 31 / target SDK 34** (JDK 17 + Android SDK 34, Gradle).

The apps need to reach the pod's gRPC port. Usually that means being on the same LAN as
the device, but it does not have to be — you can reach the pod remotely through a
reverse proxy or tunnel (Pangolin, Twingate and ngrok all ship as one-click Docker apps).

⚠️ **The gRPC API has no authentication layer yet.** Anything that can reach the port has
full control of the device, so exposing it to the public internet is strongly discouraged
— and if you tunnel to it, put the access control in the tunnel. Even on a LAN, treat the
port as unprotected and only run it on a network you trust.

## 🤝 Contributing

Contributions following Python best practices are welcome.

> **New contributor?** Start with [CONTRIBUTING.md](CONTRIBUTING.md) — it walks
> through the branch model, the local quality gate (`uv run poe sanity`), and
> the optional [ubo-claude](https://github.com/ubopod/ubo-claude) Claude Code
> tooling (specialized agents, `/onboard`, `/pr-preflight`).

### ℹ️️ Conventions

- Use `UBO_` prefix for environment variables.
- Use `ubo:` prefix for notification ids used in ubo core and `<service_name>:` prefix for notification ids used in services.
- Use `ubo:` prefix for icon ids used in ubo core and `<service_name>:` prefix for icon ids used in services.

### Development

#### Setting up the development environment

##### Quick start (automated)

After cloning the repository, you can set up the whole development environment with a single script. It detects your platform (macOS or Raspberry Pi/Linux), installs the required tooling (`uv`, `buf`, `git-lfs`, `node`), and bootstraps the project (virtual env, dependencies, protobuf, web app). It is safe to re-run and never requires `sudo` on the Raspberry Pi (run it as the `ubo` user):

```bash
./scripts/setup-dev.sh
```

Useful flags: `--tools-only` (install tools, skip project bootstrap), `--skip-web` (skip the web app build), `--help`. Pinned tool versions can be overridden via environment variables (e.g. `NODE_VERSION=22 ./scripts/setup-dev.sh`).

When it finishes, it prints the command to run the app in development mode.

##### Manual setup

To set up the development environment manually, you need to [have `uv` installed](https://docs.astral.sh/uv/).

First, clone the repository (you need to have [git-lfs installed](https://docs.github.com/en/repositories/working-with-files/managing-large-files/installing-git-large-file-storage)):

```bash
git clone https://github.com/ubopod/ubo_app.git
git lfs install
git lfs pull
```

In environments where some python packages are installed system-wide, like Raspberry Pi OS, you need to run the following command to create a virtual environment with system site packages enabled:

```bash
uv venv --system-site-packages
```

Then, navigate to the project directory and install the dependencies:

```bash
uv sync --dev
```

Next, you need to compile protobuf files and build the web application. You only need to do this once or whenever you update store actions/events or the web app.
Please refer to [Generating the protobuf files](#generating-the-protobuf-files) and [Building the web application](#building-the-web-application) sections for the steps.


Now you can run the app with:

```bash
HEADLESS_KIVY_DEBUG=true uv run ubo
```

#### Run the app on the physical device

Add `ubo-development-pod` host in your ssh config at `~/.ssh/config`:

```plaintext
Host ubo-development-pod
  HostName <ubopod IP here>
  User pi
```

⚠️*Note: You may want to add the ssh public key to the device's authorized keys (`~/.ssh/authorized_keys`) so that you don't need to enter the password each time you ssh into the device. If you decide to use password instead,  you need to reset the password for Pi user first using the GUI on the device by going to Hamburger Menu -> Settings -> System -> Users and select pi user*

Before you deploy the code onto the pod, you have to run the following command to generate the protobuf files and compile the web application.

##### Generating the protobuf files

Please make sure you have [buf](https://github.com/bufbuild/buf) library installed locally. If you are developing on a Mac or Linux, you can install it using Homebrew:

```bash
brew install bufbuild/buf/buf
```

Then, run the following command to generate the protobuf files whenever an action or

```bash
uv run poe proto
```

This is a shortcut for running the following commands:

```bash
uv run poe proto:generate # generate the protobuf files based on the actions/events defined in python files
uv run poe proto:compile  # compile the protobuf files to python files
```

##### Building the web application

If you are running it for the firt time, you first need to install the dependencies for the web application:

```bash
cd ubo_app/services/090-web-ui/web-app
npm install # Only needed the first time or when dependencies change
```

Then, you need to compile the protobuf files and build the web application:

```bash
cd ubo_app/services/090-web-ui/web-app
npm run proto:compile
npm run build
```

If you are modifying web-app typescript files, run `npm run build:watch` and let it stay running in a terminal. This way, whenever you modify web-app files, it will automatically update the built files in `dist` directory as long as it’s running.

If you ever add, modify or remove an action or an event you need to run `poe proto` and `npm run proto:compile` again manually.

---

Then you need to run this command once to set up the pod for development:

```bash
uv run poe device:deploy:complete
```

After that, you can deploy the app to the device with:

```bash
uv run poe device:deploy
```

To run the app on the device, you can use either of these commands:

```bash
uv run poe device:deploy:restart # gracefully restart the app with systemctl
uv run poe device:deploy:kill    # kill the process, which will be restarted by systemd if the service is not stopped
```

#### Running unit tests

Pure unit tests for store logic, navigation, and view computation can be run locally without Docker, Kivy, or Raspberry Pi hardware:

```bash
uv run poe test:unit
```

This runs all tests in `tests/store/` and `tests/navigation/` (~285 tests, takes ~3 seconds).

To run them inside Docker:

```bash
uv run poe docker:test:unit
```

#### Running tests on desktop

Easiest way to run tests is to use the provided `Dockerfile`s. To run the tests in a container, you first need to create the development images by running:

```bash
uv run poe build-docker-images
```

Then you can run the tests with:

```bash
docker run --rm -it --name ubo-app-test -v .:/ubo-app -v ubo-app-dev-uv-cache:/root/.cache/uv ubo-app-test
```

To run a specific test file or a single test, use `docker:test:raw` rather than
passing pytest args directly to `ubo-app-test` — the default entrypoint task
runs the unit and app test tiers as two separate pytest invocations (see
`scripts/run_test_tiers.py`), so extra args get appended to both tiers' fixed
directory lists instead of replacing them:

```bash
uv run poe docker:test:raw tests/reproduction/test_menu.py -v
uv run poe docker:test:raw tests/integration/test_services.py::test_all_services_register -v -x
```

If this fails with a `setuptools-scm`/version-detection error from the
bind-mounted repo, pass `PRETEND_VERSION` through from your shell:

```bash
PRETEND_VERSION=0.0.0.dev0 uv run poe docker:test:raw tests/reproduction/test_menu.py -v
```

To pass command line options to the full suite, add a double-dash before the options:

```bash
docker run --rm -it -v .:/ubo-app -v ubo-app-dev-uv-cache:/root/.cache/uv -v ubo-app-dev-uv-local:/root/.local/share/uv -v ubo-app-dev-uv-venv:/ubo-app/.venv ubo-app-test -- -svv --make-screenshots --override-store-snapshots --override-window-snapshots
```

**Useful pytest options for snapshot testing:**

- `--make-screenshots` - Generate PNG screenshot files alongside hash files. When a test fails due to snapshot mismatch, this creates `.mismatch.png` files showing the actual rendered output for debugging.
- `--override-window-snapshots` - Update window snapshot hash files to match current output (use after verifying the visual changes are correct).
- `--override-store-snapshots` - Update store snapshot files to match current state.

For example, to debug a failing snapshot test:

```bash
uv run poe docker:test:raw --make-screenshots tests/integration/
```

Then check the generated `.mismatch.png` files in `tests/integration/results/` to see what changed.

You can also run the tests in your local environment by running:

```bash
uv run poe test
```

⚠️**Note:** When running the tests in your local environment, the window snapshots produced by tests may mismatch the expected snapshots. This is because the snapshots are taken with a certain DPI and some environments may have different DPI settings. For example, we are aware that the snapshots taken in macOS have different DPI settings. If you encounter this issue, you should run the tests in a Docker container as described above.

#### Running tests on the device

You need to install dependencies with following commands once:

```bash
uv run poe device:test:copy
uv run poe device:test:deps
```

Then you can use the following command each time you want to run the tests:

```bash
uv run poe device:test
```

#### Running linter

To run the linter run the following command:

```bash
uv run poe lint
```

To automatically fix the linting issues run:

```bash
uv run poe lint --fix
```

#### Running type checker

To run the type checker run the following command on the pod:

```bash
uv run poe typecheck
```

⚠️*Note: Please note typecheck needs all packages to be present. To run the above command on the pod, you need to clone the ubo-app repository on the pod, apply your changes on it, have uv installed on the pod and install the dependencies.*

If you prefer to run typecheck on the local machine, clone [stubs repository](https://github.com/ubopod/ubo-non-rpi-stubs) (which includes typing stubs for third-party packages) and place the files under `typings` directory. Then run `poe typecheck` command.

#### Adding new services

It is not documented at the moment, but you can see examples in `ubo_app/services` directory.

⚠️*Note: To make sure your async tasks are running in your service's event loop and not in the main event loop, you should use the `create_task` function imported from `ubo_app.utils.async_` to create a new task. Using `await` inside `async` functions is always fine and doesn't need any special attention.*

⚠️*Note: Your service's setup function, if async, should finish at some point, this is needed so that ubo can know the service has finished its initialization and ready to be used. So it should not run forever, by having a loop at the end, or awaiting an ongoing async function or similar patterns. Running a never-ending async function using `create_task` imported from `ubo_app.utils.async_` is alright.

#### QR code

In development environment, the camera is probably not working, as it is relying on `picamera2`, so it may become challenging to test the flows relying on QR code input.

To address this, the camera module, in not-RPi environments, will try reading from `/tmp/qrcode_input.txt` and `/tmp/qrcode_input.png` too. So, whenever you encounter a QR code input, you can write the content of the QR code in the text file path or put the qrcode image itself in the image file path and the application will read it from there and continue the flow.

Alternatively you may be able to provide the input in the web-ui (needs refresh at the moment) or provide it by `InputProvideAction` in grpc channel.

#### LVGL GUI client and ESP32 satellite firmware

The C/LVGL renderer under [`ubo_lvgl/`](ubo_lvgl/README.md) is one codebase with three
targets: a desktop SDL window, the Raspberry Pi ST7789 SPI panel, and the ESP32
satellite boards. The same renderer and the same transport code are compiled for all
three; only the display backend and the transport's host layer differ.

First fetch the submodules (LVGL itself and nanopb):

```bash
git submodule update --init ubo_lvgl/lvgl ubo_lvgl/third_party/nanopb
```

Build the renderer and the native C client for the desktop (needs CMake ≥ 3.15 and
SDL2 — `brew install sdl2` / `apt install libsdl2-dev`):

```bash
cmake -S ubo_lvgl -B ubo_lvgl/build -DCMAKE_PREFIX_PATH=/opt/homebrew  # macOS/brew
cmake --build ubo_lvgl/build -j8
ctest --test-dir ubo_lvgl/build                                        # C unit tests
```

Run the LVGL client instead of the Kivy one — either on the desktop against a live core,
or on a device by setting the supervisor's backend env var:

```bash
uv run ubo-core                                       # core, gRPC only, no GUI
UBO_LVGL_ASSETS_DIR=ubo_lvgl/assets \
  ubo_lvgl/build/client/ubo_lvgl_client --backend sdl --web-grpc-url localhost:50054

UBO_GUI_BACKEND=lvgl UBO_LVGL_BACKEND=st7789 ubo      # on a pod, LVGL on the panel
```

The C client talks to the core over **tcp-lite**, a lightweight raw-TCP protocol served
by `ubo_app/rpc/mcu_server.py` on port `50054` — no HTTP and no Envoy, which is what
makes it fit comfortably on an MCU. gRPC-Web over Envoy is still supported as a
build-time alternative.

Build and flash the ESP32 firmware (needs ESP-IDF v6; the poe tasks resolve the
toolchain environment, per-board build directory, and sdkconfig for you):

```bash
uv run poe esp32:build   --board c6                        # or --board s3
uv run poe esp32:flash   --board s3 --port /dev/cu.usbmodem1101
uv run poe esp32:monitor --board c6 --profile wifi         # Ctrl-] to exit
```

`--board` is required (`c6` = Waveshare ESP32-C6-Touch-AMOLED-1.8, `s3` = Espressif
ESP32-S3-BOX-3). `--profile` defaults to `ppp`, the shipping USB/PPP build, which has
**no USB console** — pass `--profile wifi` for the debug build `esp32:monitor` can
actually read.

⚠️*Note: the C client uses a **curated** protobuf schema
(`ubo_lvgl/client/proto/ubo_client.proto`) whose oneof field numbers must match the
running core's bindings exactly. `uv run poe proto` includes `proto:lvgl:generate`,
which checks the tags and regenerates the nanopb output — so run it after changing any
action or event, and don't hand-edit the generated `.pb.{c,h}` files.*

Further reading:

- [`ubo_lvgl/README.md`](ubo_lvgl/README.md) — architecture, build options, transports,
  headless snapshots, and how the renderer mirrors `ubo_gui`'s layout
- [`ubo_lvgl/client/README.md`](ubo_lvgl/client/README.md) — the native C client:
  framing, nanopb decode, view translation, threading
- [`ubo_lvgl/esp32/README.md`](ubo_lvgl/esp32/README.md) — board pin maps, ESP-IDF
  toolchain setup, captive-portal provisioning, USB/PPP, FreeRTOS task and memory
  budgets, per-board status
- [`ubo_lvgl/esp32/AFE-FAR-FIELD.md`](ubo_lvgl/esp32/AFE-FAR-FIELD.md) — far-field audio
  front end and wake word on the ESP32-S3-BOX-3

#### Mobile and wearable app bindings

The Swift and Kotlin client apps (see
[mobile and wearable apps](#mobile-and-wearable-apps)) live in their own repositories,
but their gRPC bindings are generated from *this* repo's protobuf definitions. Check the
bindings repos out beside the core, then regenerate after changing any action or event:

```bash
uv run poe proto:swift     # → ../ubo-swift-grpc  (needs protobuf, swift-protobuf, grpc-swift)
uv run poe proto:kotlin    # → ../ubo-kotlin-grpc (needs JDK 17 + Android SDK 34)
uv run poe proto:complete  # Python + Swift + Kotlin in one go
```

`proto:swift:check` and `proto:kotlin:check` verify the committed generated sources still
match a fresh regen. Never hand-edit generated files.

⚠️*Note: unlike the LVGL client, these clients regenerate the whole proto, so they never
suffer field-tag drift — but a new action needs a matching branch in each client's
hand-written action mapping. Kotlin catches a missing branch at compile time; Swift does
not, and a missing case dispatches silently as a no-op.*

## 🛠️ Hardware 

This section presents different hardware or emulation options that you can use with Ubo app.

### Emulation

To remove barriers to adoption as much as possible and allow developers use Ubo app without hardware depenencies, we are currently emulating the physical GUI in the browser. 

The audio playback is also streamed through the broswer. 

We plan to emulate camera and microphone with WebRTC in the future.

![Ubo Pod photo](https://raw.githubusercontent.com/ubopod/mediakit/main/images//gui_emulation.png)

However, other specialized hardware components (sensors, infrared rx/tx, etc) cannot be emulated. 

### Ubo Pod

![Ubo Pod photo](https://raw.githubusercontent.com/ubopod/mediakit/main/images/rotating-pod.gif)

Ubo pod is an open hardware that includes the following additional hardware capabilities that is supported by Ubo app out of the box:

- A built-in minimal GUI (color LCD display and keypad)
- Stereo microphone and speakers (2W)
- Camera (5MP)
- LED ring (27 addressable RGB LEDs)
- Sensors
   - Ambient light sensor
   - Temperature sensor
   - STEMMA QT / Qwiic connector for additional sensors
- Infrared
  - Receiver (wideband)
  - Transmitter (4 high power LEDs)
- 2 full HDMI ports
- Power/reset button 
- NVMe storage (Pi 5 only)

For more information on hardware spec, see the website [getubo.com](https://getubo.com).

This is an open hardware. You can access mechanical design files [here](https://github.com/ubopod/ubo-mechanical) and electrical design files [here](https://github.com/ubopod/ubo-pcb).

### DIY path

You can also buy different HATs from different vendors to DIY the hardware. Future plans include supporting USB microphone, speakers, cameras as well with headless setup.

This however involves having to purchase multiple HATs from different vendors and the process may not be the easiest and most frictionless. You may have to dig into the code and make some small changes to certain setups and configurations.

I made the table below that shows options for audio, cameras, and other sub-components:

| Function | Options |
| --- | --- |
| Audio | [Respeaker 2-Mic Audio HAT](https://www.seeedstudio.com/ReSpeaker-2-Mics-Pi-HAT.html), [Adafruit Voice Bonnet](https://www.adafruit.com/product/4757), [Waveshare WM8960 Hat](https://www.waveshare.com/wm8960-audio-hat.htm), [Adafruit BrainCraft HAT](https://www.adafruit.com/product/4374) |
| Speakers | [1 or 2W, 8 Ohm](https://www.adafruit.com/product/1669) |
| Camera | Raspberry Pi Camera Modules V1.3, [V2](https://www.raspberrypi.com/products/camera-module-v2/), or [V3](https://www.raspberrypi.com/products/camera-module-3/) |
| LCD (also emulated in the browser) | [240x240 TFT Display](https://www.adafruit.com/product/4421), [Adafruit BrainCraft HAT](https://www.adafruit.com/product/4374) |
| Keypad | [AW9523 GPIO Expander](https://www.adafruit.com/product/4886) |
| LED ring | [Neopixel LED ring](https://www.adafruit.com/product/1586) |
| Ambient Light Sensor | [VEML7700 Lux Sensor](https://www.adafruit.com/product/4162) |
| Temperature Sensor | [PCT2075 Temperature Sensor](https://www.adafruit.com/product/4369) |

## 🏗️ Architecture

The architecture is fundamentally event-driven and reactive, built around a centralized Redux store that coordinates all system interactions through immutable state updates and event dispatching. 

Services communicate exclusively through Redux actions and events rather than direct method calls, with each service running in its own isolated thread while subscribing to relevant state changes and events. 

The system uses custom event handlers that automatically route events to the appropriate service threads, enabling reactive responses to state changes across hardware interfaces, user interactions, and system events.  

This reactive architecture allows components like the web UI to subscribe to display render events and audio playback events in real-time, creating a responsive system where changes propagate automatically through the event stream without tight coupling between components.

![Software architecture](https://raw.githubusercontent.com/ubopod/mediakit/main/images/architecture.jpg)

The following is a summary of key architecture components.

-  <b>Redux-Based State Management</b>: Central `UboStore` manages all application state through immutable state trees, with each service contributing its own state slice (audio, camera, display, docker, wifi, etc.) and communicating via actions and events.

-  <b>Modular Service Architecture</b>: 21+ core services run in isolated threads with dedicated event loops, organized by priority (`000-0xx` for hardware, `030-0xx` for networking, `080-0xx` for applications, `090-0xx` for UI), each with their own setup.py, reducer.py, and ubo_handle.py files.

-  <b>Hardware Abstraction Layer</b>: Comprehensive abstraction for Raspberry Pi components (ST7789 LCD, WM8960 audio, GPIO keypad, sensors, camera, RGB ring) with automatic environment detection and mock implementations for development on non-RPi systems.

-  <b>Multi-Interface Access</b>: Supports web browser access (port 4321 for direct bootstrap/recovery, Envoy's gRPC-web frontend port for the full live UI), gRPC API (port 50051), SSH access, and direct hardware interaction, with a web UI service providing hotspot configuration and dashboard functionality.

-  <b>System Integration</b>: Integrates with `systemd` and `d-bus` for service management, Docker for container runtime, and `NetworkManager` for network configuration, with a separate system manager process handling root-privilege operations via Unix sockets.

<b>Notes:</b>  

The application follows a structured initialization sequence through `ubo_app/main.py` and uses the `uv` package manager for dependency management. 

The architecture supports both production deployment on Raspberry Pi devices and development environments with comprehensive mocking systems, making it suitable for cross-platform development while maintaining hardware-specific capabilities.

DeepWiki pages you might want to explore:

- [Overview](https://deepwiki.com/ubopod/ubo_app/1-overview)
- [Architecture](https://deepwiki.com/ubopod/ubo_app/2-architecture)

## Notable dependencies

Here are the key dependencies organized by category:

### Core Framework & State Management

- `python-redux`: Redux-based state management system for the entire app
- `ubo-gui`: Custom GUI framework built on Kivy for the user interface
- `headless-kivy`: Headless Kivy implementation for supporting LCD display over SPI

### Hardware Control (Raspberry Pi)

- `adafruit-circuitpython-rgb-display`: ST7789 LCD display driver
- `adafruit-circuitpython-neopixel`: RGB LED ring control
- `adafruit-circuitpython-aw9523`: I2C GPIO expander for keypad
- `adafruit-circuitpython-pct2075`: Temperature sensor driver
- `adafruit-circuitpython-veml7700`: Light sensor driver
- `rpi-lgpio`: Low-level GPIO access for Raspberry Pi
- `gpiozero`: GPIO abstraction layer
- `rpi-ws281x`: WS281x LED strip control library
- `pyalsaaudio`: ALSA audio interface for Linux audio control
- `pulsectl`: PulseAudio control for audio management
- `simpleaudio`: Simple audio playback functionality

### Voice AI

- `piper-tts`: Text-to-speech synthesis engine
- `vosk`: Speech recognition library
- `pvorca`: Picovoice Text-to-speech synthesis engine
- `pipecat-ai`: framework for building real-time voice and multimodal conversational agents

### Networking & Services

- `aiohttp`: Async HTTP client/server for web services
- `quart`: Async web framework for the web UI service
- `sdbus-networkmanager`: NetworkManager D-Bus interface for WiFi
- `netifaces`: Network interface enumeration
- `docker`: Docker API client for container management

### QR Codes

- `pyzbar`: QR code and barcode scanning library

### System Utilities

- `psutil`: System and process monitoring utilities
- `platformdirs`: Platform-specific directory paths
- `tenacity`: Retry logic and error handling
- `fasteners`: File locking and synchronization

### Development Environment Abstraction

- `python-fake`: Mock hardware components for development

### gRPC Communication

- `betterproto`: Protocol buffer compiler and runtime

<b>Notes:</b>
The project uses platform-specific dependencies with markers like `platform_machine=='aarch64'` for Raspberry Pi-specific libraries and `sys_platform=='linux'` for Linux-only components. The python-fake library enables development on non-Raspberry Pi systems by providing mock implementations of hardware components.

## 🗺️ Roadmap

This is a tentative roadmap for future features. It is subject to change.

- Emulation for camera and microphone inside browser (requires SSL certificate for browser permissions)
- Allow users to pick their soundcard for play and record via GUI (e.g. USB audio)
- Allow users to pick their camera for video via GUI (e.g. USB camera)
- Option to turn Ubo pod into a voice satellite with wyoming protocol with Home Assistant
- Make all on-board sensors and infrared discoverable and accessible by Home Assistant
- Expose `pipecat-ai` preset pipeline configuration via GUI
- Support for Debian Trixie (13)

If you have any suggestions or feature requests, please open a discussion [here](https://github.com/ubopod/ubo_app/discussions).

## 🔒 License

This project is released under the Apache-2.0 License. See the [LICENSE](./LICENSE) file for more details.
