Research Findings: Architecture Corrections

Verified against Anvil, COSMIC (cosmic-comp), and niri

Critical Fixes

1. Smithay Version: 0.3 → 0.7.0
Smithay 0.3 is years old and missing most APIs we need (desktop, renderer_glow, WlrLayerShellState, X11Wm, VirtualKeyboardManagerState, SeatState). All these exist in 0.7.0 (latest, published 2025-06-24). calloop 0.14 is also only compatible with 0.7.0.
2. Tiling: BSP Tree → N-ary Tree
Neither COSMIC nor niri uses BSP. COSMIC uses id_tree (n-ary tree with Group/Mapped/Placeholder nodes). Niri uses column-based scrolling. N-ary trees are more flexible for real-world window management. We should follow COSMIC's pattern.
3. Missing Cargo Features
Must add: backend_gbm (DRM buffer allocation), backend_egl (GL context), backend_session_libseat (VT switching), renderer_multi (multi-GPU + software fallback), use_system_lib (EGL/Mesa).
4. Screencopy: Not Built Into Smithay
No ScreencopyManagerState in Smithay. Must implement manually using wayland-protocols-wlr bindings, or just do renderer pixel readback for AI capture (skip the protocol for now).

Validated Decisions (Confirmed Best Practice)

Calloop-only (no Tokio)
All 3 production compositors are calloop-only. None use Tokio. Confirmed correct.
Single central state struct
All 3 use one central state struct. Smithay's delegate macros require it. Confirmed correct.
Hybrid tiling/floating
COSMIC does exactly this — tiling with floating support. Confirmed correct.
Unix socket IPC (JSON)
Niri uses exactly this pattern — Unix socket + newline-delimited JSON, integrated with calloop via Generic::new(listener, Interest::READ, Mode::Level).
WindowElement enum for Wayland + X11
Anvil and COSMIC both wrap Wayland and X11 surfaces in an enum. Confirmed correct.
Layer Shell as separate client
Standard Wayland pattern. Compositor provides protocol; palette is its own process. Confirmed correct.

Revised Cargo.toml

compositor/Cargo.toml
# Changed: 0.3 → 0.7
smithay = { version = "0.7", features = [
  "desktop", "wayland_frontend",
  "backend_drm", "backend_udev", "backend_winit",
  "backend_gbm", "backend_egl", # NEW
  "backend_session_libseat", # NEW
  "backend_libinput",
  "renderer_glow", "renderer_multi", # NEW
  "use_system_lib", # NEW
  "xwayland",
] }
calloop = "0.14"
tokio # REMOVED
id_tree = "1" # NEW — n-ary tree for tiling (COSMIC pattern)
image = "0.25" # NEW — PNG encoding for screen capture
xkbcommon = "0.8" # NEW — keyboard mapping
base64 = "0.22" # NEW — capture encoding

Missing Protocols to Add

Essential (must have)
linux-dmabuf-v1
ext-session-lock-v1
wp-viewporter
wp-fractional-scale-v1
xdg-activation-v1
cursor-shape-v1
Important (daily driver)
wp-presentation-time
data-control (clipboard)
pointer-constraints + relative-pointer
keyboard-shortcuts-inhibit
ext-idle-notify-v1
virtual-keyboard-v1 + virtual-pointer