LabWatch Lite — Project Report项目汇报

A lightweight self-hosted dashboard for monitoring NVIDIA GPUs and AI development servers.

轻量级、自托管的 NVIDIA GPU 与 AI 开发服务器监控面板。

v1.1.0 installable tool + validated on a real 8-GPU serverv1.1.0 可安装工具 + 已在真实 8 卡服务器验证 Single host · multiple GPUs单机 · 多卡 179 backend tests后端 179 个测试 77 frontend tests前端 77 个测试 8 browser E2E tests浏览器 E2E 8 个 Docker + GPU verifiedDocker + GPU 已验证 MIT

1. What was built

1. 构建内容

LabWatch answers, on one self-refreshing page, the questions that otherwise require an SSH session and repeated nvidia-smi, htop and df -h runs: which GPU is free, what is running on it, who owns those processes, how long they have been running, whether RAM or disk is about to run out, and what utilisation looked like over the last hour, six hours or day.

LabWatch 在一个自动刷新的页面上，回答了那些原本需要登录 SSH 并反复执行 nvidia-smi、htop、df -h 才能回答的问题： 哪张卡是空的、上面在跑什么、那些进程属于谁、已经跑了多久、内存或磁盘是否即将耗尽， 以及过去一小时、六小时、一天里的利用率是什么样。

Implemented features

已实现的功能

Area	Detail
GPU telemetry	Utilisation, VRAM used/total and %, temperature, power draw, enforced power limit and derived %, fan, SM and memory clocks, persistence mode, process count.
GPU processes	NVML compute PIDs joined with psutil: user, full command line, CPU %, resident memory, status, start time, runtime (human and clock form).
Host telemetry	CPU usage, physical/logical cores, frequency, load average, per-core breakdown; RAM used/total/available/%; disk with the primary mount plus optional extra mounts; hostname, OS, kernel, platform, uptime.
History	Host and per-GPU samples persisted to SQLite (WAL), background collector, retention pruning plus a hard row cap, and 1H/6H/24H series endpoints.
Dashboard	Host tiles, one card per GPU, sortable/filterable/searchable process table, four history charts, sticky header with connection state, system/light/dark themes applied before first paint.
Resilience	Missing driver, missing GPU, unsupported sensors, vanishing processes and database faults all degrade locally instead of breaking the page or the API.
Demo mode	Deterministic synthetic GPUs, processes and history, labelled Demo Data, so the project is demonstrable and testable without an NVIDIA card.
Operations	Multi-stage Docker image, Compose deployment with a named volume, environment-variable configuration, health endpoint, CI pipeline with a container smoke test.

领域	细节
GPU 遥测	利用率、显存已用/总量及百分比、温度、功耗、强制功耗上限及推导百分比、风扇、SM 与显存频率、持久化模式、进程数。
GPU 进程	把 NVML 计算 PID 与 psutil 关联：用户、完整命令行、CPU %、常驻内存、状态、启动时间、运行时长（人类可读与时钟两种形式）。
主机遥测	CPU 使用率、物理/逻辑核心数、频率、负载均值、单核明细；内存已用/总量/可用/百分比；磁盘主挂载点及可选的其他挂载点；主机名、系统、内核、平台、运行时长。
历史数据	主机与逐卡采样写入 SQLite（WAL 模式），后台采集器、按保留窗口清理并设行数硬上限，提供 1H/6H/24H 序列接口。
面板	主机卡片、每张 GPU 一张卡片、可排序/过滤/搜索的进程表、四张历史图表、带连接状态的吸顶顶栏、首屏渲染前即生效的系统/浅色/深色主题。
健壮性	驱动缺失、无 GPU、传感器不支持、进程消失、数据库故障，全部局部降级，而不是让页面或 API 崩掉。
演示模式	确定性的合成 GPU、进程与历史数据，明确标注 Demo Data，因此在没有 N 卡的机器上也能演示与测试。
运维	多阶段 Docker 镜像、带命名卷的 Compose 部署、环境变量配置、健康检查接口、含容器冒烟测试的 CI 流水线。

2. Architecture

2. 架构

The browser polls a single aggregated endpoint (/api/overview) so one refresh is one round trip. Polling pauses while the tab is hidden. A background asyncio task samples the collectors every ten seconds in a worker thread (NVML and psutil are blocking) and writes to SQLite. NVML is initialised lazily, exactly once, behind a lock, because the library is not thread safe.

浏览器只轮询一个聚合接口（/api/overview），因此一次刷新只有一次往返。标签页 隐藏时暂停轮询。后台 asyncio 任务每十秒在工作线程中采集一次（NVML 与 psutil 都是阻塞 调用）并写入 SQLite。NVML 采用惰性初始化，且因为该库并非线程安全，全过程只初始化一次 并加锁保护。

v1 deliberately has no WebSockets, no message queue, no cache layer and no authentication: at a two second refresh they would add operational surface without changing the experience.

v1 刻意不引入 WebSocket、消息队列、缓存层与认证：在 2 秒刷新频率下，它们只会增加运维 面，并不会改善使用体验。

3. Screenshots

3. 截图

Captures below come from the production container image. The first two are demo mode; the third is the same image reading a real RTX 4060 through NVML.

以下截图全部来自生产容器镜像。前两张为演示模式；第三张是同一个镜像通过 NVML 读取真实 RTX 4060 的效果。

4. Tests and results

4. 测试与结果

Suite	Tests	Result	Notes
Backend (pytest)	179	pass	87 % line coverage over labwatch/; NVML replaced by a configurable fake. Verified stable over eight consecutive runs.
Frontend (Vitest)	77	pass	Formatting, theme, API client, host tiles, GPU cards, process table sorting/filtering/search, empty and no-GPU states.
End-to-end (Playwright)	8	pass	Real Chromium against a demo backend and the dev server; also asserts zero console errors.
Ruff lint	—	clean	ruff check app tests
TypeScript	—	clean	tsc -b with strict, noUnusedLocals, noUnusedParameters.
Production build	—	pass	vite build; ~633 kB JS / ~21 kB CSS before gzip.
Docker image	—	pass	Multi-stage build, 302 MB, runs as uid 10001.
Container smoke test	—	pass	Healthy in 2 s; dashboard, assets, SPA fallback and every API endpoint verified inside the container.
Container with GPU passthrough	—	pass	--gpus all: NVML initialised in-container (driver 581.80) and reported the real device.
7 hour sustained run	—	pass	11541 writes, 0 collector failures, sample interval held at 2 s without drift.

套件	数量	结果	说明
后端（pytest）	179	通过	labwatch/ 行覆盖率 87%；用可配置的仿真实现替换 NVML。连续 8 次运行均稳定。
前端（Vitest）	77	通过	格式化、主题、API 客户端、主机卡片、GPU 卡片、进程表排序/过滤/搜索、空状态与无 GPU 状态。
端到端（Playwright）	8	通过	真实 Chromium 对接演示后端与开发服务器；同时断言零 console 错误。
Ruff 静态检查	—	干净	ruff check app tests
TypeScript	—	干净	tsc -b，开启 strict、noUnusedLocals、noUnusedParameters。
生产构建	—	通过	vite build；gzip 前约 633 kB JS / 21 kB CSS。
Docker 镜像	—	通过	多阶段构建，302 MB，以 uid 10001 运行。
容器冒烟测试	—	通过	2 秒内健康；在容器内验证面板、静态资源、SPA 回退路由以及全部 API 接口。
容器 + GPU 直通	—	通过	--gpus all：容器内成功初始化 NVML（驱动 581.80）并读到真实设备。
7 小时持续运行	—	通过	写入 11541 次，采集失败 0 次，采样间隔稳定保持 2 秒无漂移。

Collector failure modes explicitly covered

明确覆盖的采集器失败路径

• NVML shared library missing, nvmlInit raising, and nvidia-ml-py not installed
• A driver that initialises but reports zero devices
• Unsupported power, fan, clock, persistence and memory sensors, each degrading to N/A independently
• A process exiting between the NVML query and the psutil lookup
• NVML's "unknown memory" sentinel value not being reported as a real figure
• A pid on two GPUs, duplicate pids, and the per-sample process limit
• psutil raising for memory, disk or CPU; a missing partition table; a getloadavg that is absent or not callable
• Pseudo filesystems and container-injected single-file mounts being rejected as disks
• A database that is gone (health reports degraded rather than crashing)
• Collector sample and write failures not killing the background loop

• NVML 共享库缺失、nvmlInit 抛异常、以及 nvidia-ml-py 未安装
• 驱动初始化成功但报告 0 个设备
• 功耗、风扇、频率、持久化、显存等传感器不支持，各自独立降级为 N/A
• 进程在 NVML 查询与 psutil 查询之间退出
• NVML 的"显存未知"哨兵值不会被当成真实数值上报
• 同一 PID 出现在两张卡、PID 重复、以及单次采样的进程数上限
• psutil 在读取内存、磁盘或 CPU 时抛异常；分区表缺失；getloadavg 不存在或不可调用
• 伪文件系统与容器注入的单文件挂载被识别为非磁盘
• 数据库不可用（健康检查报告 degraded，而不是崩溃）
• 采集与写入失败不会终止后台循环

Verification against real hardware

真实硬件验证

The development host has an NVIDIA GeForce RTX 4060 Laptop GPU (driver 581.80, CUDA driver 13.0). LabWatch's readings were compared against nvidia-smi both natively and from inside the container.

开发机配备 NVIDIA GeForce RTX 4060 Laptop GPU（驱动 581.80，CUDA 驱动 13.0）。LabWatch 的 读数在本机与容器内分别与 nvidia-smi 做过对比。

Metric	nvidia-smi	LabWatch	Verdict
Utilisation	9 % / 27 % / 40 %	10 % / 28 % / 40 %	within one sample
VRAM total	8188 MiB	8188 MiB	exact
VRAM used	1369 MiB	1599 MiB	within sample skew
Temperature	58 °C / 48 °C / 71 °C	57 °C / 47 °C / 71 °C	within one sample
Power draw	3.95 W / 7.82 W	4.0 W / 7.9 W	exact to 1 dp
Power limit	[N/A]	N/A	correctly absent

指标	nvidia-smi	LabWatch	判定
利用率	9 % / 27 % / 40 %	10 % / 28 % / 40 %	单次采样误差内
显存总量	8188 MiB	8188 MiB	完全一致
显存占用	1369 MiB	1599 MiB	采样偏差内
温度	58 °C / 48 °C / 71 °C	57 °C / 47 °C / 71 °C	单次采样误差内
功耗	3.95 W / 7.82 W	4.0 W / 7.9 W	小数点后一位一致
功耗上限	[N/A]	N/A	正确显示为不支持

The GPU process list matches nvidia-smi --query-compute-apps pid for pid, with additional user, command, CPU and runtime detail that nvidia-smi does not provide. History was left running and read back: 361 host points at a 10 second interval for the 1H window, and an 8641 point 24H series after a restart.

GPU 进程列表与 nvidia-smi --query-compute-apps 的 PID 完全一致，并额外提供 nvidia-smi 没有的用户、命令行、CPU 与运行时长。历史数据经长时间运行后回读： 1H 窗口在 10 秒间隔下得到 361 个主机采样点，重启后 24H 序列有 8641 个点。

5. Real Linux multi-GPU server validation

5. 真实 Linux 多卡服务器验证

Everything above was measured on the Windows development host. The project was then deployed to a real research server - eight GPUs under live experimental load - and re-verified end to end. This section records that validation, because "the API returns something" is not the same as "the numbers mean the right thing on a real Linux host".

以上数据均在 Windows 开发机上测得。随后项目被部署到一台真实的科研服务器上 —— 八张 GPU、正在跑实验 —— 并重新做了端到端验证。本节记录这次验证，因为"接口能返回数据" 和"数据在真实 Linux 主机上语义正确"是两回事。

Environment

服务器环境

Item	Value
Distribution	Ubuntu 22.04.5 LTS, kernel 6.8.0-124-generic
GPUs	8 × NVIDIA GeForce RTX 4090 (49140 MiB each)
Driver / CUDA	580.173.02 / CUDA driver 13.0
Host resources	128 logical CPUs, 503.5 GB RAM
Filesystems	/ 91.7 %, /nfs-data1 98.4 %, /nfs-data2, /nfs-data3, /nfs-data4, /boot/efi
Deployment mode	Native Python (Docker is not installed on this host). Own venv in the project directory, created with --system-site-packages so it inherits the user's conda environment read-only. No system package, service, or existing environment was modified.
Concurrent load	Another user's experiments were running throughout; they were never started, stopped or signalled by LabWatch.

项目	值
发行版	Ubuntu 22.04.5 LTS，内核 6.8.0-124-generic
GPU	8 × NVIDIA GeForce RTX 4090（每张 49140 MiB）
驱动 / CUDA	580.173.02 / CUDA 驱动 13.0
主机资源	128 逻辑核，503.5 GB 内存
文件系统	/ 91.7 %、/nfs-data1 98.4 %、/nfs-data2、/nfs-data3、/nfs-data4、/boot/efi
部署方式	原生 Python（该主机未安装 Docker）。在项目目录下用 --system-site-packages 建立独立 venv，只读继承你的 conda 环境。未修改任何系统软件包、服务或既有环境。
并发负载	全程有其他用户的实验在跑；LabWatch 从未启动、停止或向其发送任何信号。

Multi-GPU discovery and metric agreement

多卡发现与指标一致性

All eight devices were discovered automatically. LabWatch was compared against nvidia-smi captured in the same instant, on every GPU:

八张卡全部被自动发现。在每一张卡上，把 LabWatch 与同一瞬间采集的 nvidia-smi 逐项对比：

Metric	Agreement over 8 GPUs
GPU utilisation	exact (0 %, 98 %, 100 % cases all matched)
Memory total	exact - 49140 MiB on all eight
Temperature	exact (25-72 °C)
Power draw	exact to 1 dp (13.4 W idle, 449.8 W loaded)
Power limit	exact - 450.0 W

指标	8 张卡的一致性
GPU 利用率	完全一致（0 %、98 %、100 % 各种情况均吻合）
显存总量	完全一致 —— 八张卡均为 49140 MiB
温度	完全一致（25–72 °C）
功耗	小数点后一位一致（空闲 13.4 W，满载 449.8 W）
功耗上限	完全一致 —— 450.0 W

GPU process mapping on a shared, multi-user server

多人共享服务器上的 GPU 进程映射

LabWatch ran as one user and reported another user's CUDA processes - the normal case on a shared lab machine. Each entry was checked against the driver and against /proc:

LabWatch 以一个用户的身份运行，却要报告另一个用户的 CUDA 进程 —— 这是共享实验机的常态。 每一条记录都与驱动和 /proc 做了核对：

Field	Result
PID → GPU index	correct for all 8 processes; independently confirmed because each workload's JSONL filename encodes its GPU (e.g. ..._gpu5_... matched gpu_index=5)
GPU memory per process	0 MiB difference against nvidia-smi --query-compute-apps
Username	matched /proc/<pid> ownership, including other users (niuyizhuo, lipeilang)
Command line	full 1018-byte command line, including all arguments
Runtime	matched /proc start time to the second
CPU %	~100 % per process, consistent with ps reporting 107-110 %
Resident memory	44.1 GB, consistent with ps reporting 8.7 % of 503.5 GB

字段	结果
PID → GPU 索引	8 个进程全部正确；并有独立佐证 —— 每个任务的 JSONL 文件名里编码了 GPU（如 ..._gpu5_... 对应 gpu_index=5）
进程显存	与 nvidia-smi --query-compute-apps 偏差 0 MiB
用户名	与 /proc/<pid> 属主一致，包含其他用户（niuyizhuo、lipeilang）
命令行	完整 1018 字节命令行，含全部参数
运行时长	与 /proc 启动时间精确到秒一致
CPU %	每进程约 100 %，与 ps 的 107–110 % 相符
常驻内存	44.1 GB，与 ps 报的 503.5 GB 的 8.7 % 相符

Host metric semantics

主机指标语义

Deployment was native, so CPU, RAM and disk are the host's own. The dashboard was still checked against the commands an operator would run, and two real defects fell out (numbers 11 and 12 below).

本次为原生部署，因此 CPU、内存、磁盘都是宿主机的真实数据。即便如此，面板仍与运维人员会 使用的命令做了比对，并因此发现两个真实缺陷（见下方第 11、12 条）。

Metric	Reference	LabWatch
CPU load average	uptime 12.31 / 12.88 / 14.84	12.31 / 12.88 / 14.84
Memory used	free 42.5 GB (8.4 %)	42.5 GB (8.4 %) after the fix
Filesystems	df 6 real mounts	all 6, percentages matching df
API latency under 8-GPU load	target < 500 ms	0.01-0.04 s
API errors / 5xx	none expected	none; collector_errors = 0

指标	参照	LabWatch
CPU 负载均值	uptime 12.31 / 12.88 / 14.84	12.31 / 12.88 / 14.84
内存已用	free 42.5 GB（8.4 %）	修复后为 42.5 GB（8.4 %）
文件系统	df 6 个真实挂载点	6 个全部显示，百分比与 df 一致
8 卡负载下 API 延迟	目标 < 500 ms	0.01–0.04 s
API 错误 / 5xx	应为无	无；collector_errors = 0

6. v1.1 — from a project to a tool

6. v1.1 —— 从"项目"变成"工具"

v1.0 answered "what is happening on my GPU server?" v1.1 answers a different question: "how do I open LabWatch without thinking about it?" The monitoring core was frozen; everything here is install, start, access and daily use.

v1.0 回答的是"我的 GPU 服务器上发生了什么"。v1.1 回答的是另一个问题： "我怎么能不假思索地打开 LabWatch"。核心监控功能全部冻结，这一轮只改进安装、 启动、访问与日常使用。

Packaging

打包

• The backend became a real Python package: labwatch with the server under labwatch.server, the command line under labwatch.cli, and the built dashboard bundled at labwatch/ui. One pyproject.toml at the repository root is the single source of truth for dependencies, the console script and the wheel contents.
• The dashboard ships inside the wheel, so an installed LabWatch serves the real UI. Users never run npm install or npm run build; that is a development step. CI fails if the committed bundle drifts from the sources it was built from.
• History moved to the platform data directory (for example ~/.local/share/labwatch). Putting it inside the package would have meant that upgrading LabWatch deletes the user's history.

• 后端成为真正的 Python 包：labwatch，服务器代码在 labwatch.server，命令行在 labwatch.cli，构建好的前端打包在 labwatch/ui。仓库根目录的单个 pyproject.toml 是依赖、console script 与 wheel 内容的唯一来源。
• 面板随 wheel 一起分发，因此安装后的 LabWatch 直接提供真实 UI。用户永远不需要执行 npm install 或 npm run build —— 那属于开发流程。CI 会在提交的前端产物与源码不一致时失败。
• 历史数据改到平台数据目录（如 ~/.local/share/labwatch）。放在包目录里意味着升级 LabWatch 会删掉用户的历史。

CLI

命令行

labwatch                     # serve and open the dashboard
labwatch --demo              # synthetic GPUs, no hardware
labwatch doctor              # can this machine run LabWatch?
labwatch start --background  # detached
labwatch status              # one-line answer, or --json
labwatch stop / open / version

labwatch                     # 启动并打开面板
labwatch --demo              # 合成 GPU，无需硬件
labwatch doctor              # 这台机器能跑 LabWatch 吗
labwatch start --background  # 后台运行
labwatch status              # 一行答案，或 --json
labwatch stop / open / version

The CLI uses only the standard library, deliberately: --help and doctor must work even when a dependency failed to install. doctor checks Python, the dependencies, NVML, the GPUs, the port, the data directory and the bundled dashboard, and names the fix for each failure rather than printing a traceback. status --json is the interface the editor consumes.

CLI 刻意只使用标准库：即使某个依赖安装失败，--help 与 doctor 也必须能用。doctor 检查 Python、依赖、NVML、GPU、端口、数据目录与打包的面板， 并为每一项失败给出修复建议，而不是抛一堆栈。status --json 是编辑器消费的接口。

VS Code extension

VS Code 扩展

• Status bar summary, a glance-only GPU sidebar, and Open Full Dashboard.
• It runs labwatch status --json instead of reimplementing the collector, so the editor and the dashboard cannot disagree.
• Remote-SSH is the designed case. The extension host runs on the remote machine, so the CLI it invokes describes that machine's GPUs; vscode.env.asExternalUri asks VS Code to forward the dashboard port. No manual tunnel, and nothing is exposed on the server's network.
• The formatting layer has no vscode imports and is unit tested under plain Node, including malformed CLI payloads.

• 状态栏摘要、只用于"扫一眼"的 GPU 侧边栏，以及 Open Full Dashboard。
• 它执行 labwatch status --json，而不是重新实现采集器，因此编辑器与面板不可能给出不一致的数据。
• Remote-SSH 是重点设计场景。扩展宿主运行在远端机器上，所以它调用的 CLI 描述的就是那台机器的 GPU；vscode.env.asExternalUri 让 VS Code 转发面板端口。无需手工隧道，也不会在服务器网络上暴露任何端口。
• 格式化层不引用 vscode，用纯 Node 做单元测试，包含对畸形 CLI 输出的处理。

Defects found while doing this

本轮发现的缺陷

Four problems surfaced that only exist because the tool became installable; they are entries 14–17 in the list below. The most interesting was not a code defect at all: bundling the dashboard inside the package made the SPA fallback answer unknown /api/* paths with HTTP 200 and a page of HTML, so an API typo started looking like success. That is precisely the class of bug that appears when a development-only deployment becomes a product. Two further entries, 18 and 19, come from publishing rather than from the product: a PyPI name that was already taken, and a CI job that linted a directory the refactor had deleted.

有四个问题只有在"变成可安装的工具"之后才会出现，列在下方第 14–17 条。其中最有意思的一个 甚至不算代码缺陷：把面板打进包里之后，SPA 兜底路由开始把未知的 /api/* 返回成 HTTP 200 + 一页 HTML，于是一个 API 拼写错误看起来像调用成功。这正是"开发环境专用部署" 变成"产品"时才会冒出来的那一类问题。另有第 18、19 两条来自"发布"而非"产品"本身： 一个已被占用的 PyPI 名字，以及一个在重构后仍然 lint 已删除目录的 CI 任务。

Where it was validated

验证位置

Path	Result
Wheel contents	250 KB, includes labwatch/ui; package-data test asserts the dashboard is present and served
uvx equivalent (local wheel and local checkout)	starts, serves the bundled UI (HTTP 200 on / and on /assets/*.js)
Editable install, CLI end to end	doctor, start, status, stop, open, version, python -m labwatch
VS Code extension	packaged as .vsix, installed into VS Code (11 files, 15 KB)
Extension against the real server CLI	parser and formatters fed the live 8-GPU payload over SSH; every contract check passed
Lab server, packaged install	v1.1.0 installed offline, doctor all green, dashboard served from the package, other users' 8 CUDA processes untouched
Docker build	blocked by this machine's Docker DNS at the time of writing; the Dockerfile was rewritten for the package layout and the CI job verifies it independently

路径	结果
wheel 内容	250 KB，包含 labwatch/ui；有测试断言面板存在且可被托管
uvx 等价路径（本地 wheel 与本地仓库）	可启动，托管打包 UI（/ 与 /assets/*.js 均 200）
可编辑安装 + CLI 全流程	doctor、start、status、stop、open、version、python -m labwatch
VS Code 扩展	已打包为 .vsix 并成功安装到 VS Code（11 个文件，15 KB）
扩展对接真实服务器 CLI	解析器与格式化函数直接消费经 SSH 取回的真实 8 卡数据，全部契约检查通过
实验服务器上的包化安装	v1.1.0 离线安装成功，doctor 全绿，面板由包直接托管，其他用户的 8 个 CUDA 进程未受影响
Docker 构建	撰写时被本机 Docker 的 DNS 故障阻塞；Dockerfile 已按包化结构重写，CI 中有独立 job 验证

7. Bugs found and fixed during development

7. 开发过程中发现并修复的缺陷

These were found by the tests, by looking at the rendered UI, or by comparing against the driver — not by reading code alone.

以下缺陷来自测试、来自观察真实渲染的界面、或来自与驱动的对比 —— 而不是仅靠阅读代码发现的。

#	Bug	Impact	Fix
1	/api/health read last_history_write from a context field that is only assigned during shutdown.	A running server always reported null, so the collector looked dead and the value could not be monitored.	Read the live value from the history service and expose collector writes, errors, interval and stored point count.
2	Per-process CPU deltas were discarded: the psutil priming cache was cleared on every poll.	Process CPU % could never accumulate a real delta and stayed at zero.	Keep baselines keyed by pid, prime before enrichment, and prune pids that disappear.
3	NVML graphics-running-processes made the table unusable on Windows (about 30 desktop apps).	The GPU process table was dominated by compositing noise instead of workloads.	Default to compute processes (matching nvidia-smi --query-compute-apps) with an opt-in flag for graphics contexts.
4	Demo disk usage was computed from hours since the Unix epoch (about 490,000 h).	The disk tile showed 99 % critical in every screenshot, which is both wrong and alarming.	Anchor the growth curve to the current year plus nine months and clamp it, giving a realistic 62 %.
5	Demo signals were sampled at one second regardless of chart resolution.	The 24H chart was an unreadable solid block; the 1H chart looked like a synthetic sine wave.	Scale jitter amplitude, jitter period and the load envelope with the requested window.
6	os.getloadavg was called without guarding against TypeError on platforms that expose it as None.	CPU collection raised instead of reporting N/A for load average.	Treat missing or non-callable implementations as N/A.
7	The screenshot script located sections by loose heading patterns such as /host/i.	Playwright strict mode failed once chart titles like "Host utilisation" existed — a latent test-suite break.	Target section containers by data-testid, and match section headings exactly.
8	Dockerfile carried a # syntax=docker/dockerfile:1 directive it did not need.	The build failed on any host that cannot reach the Docker frontend image registry.	Remove the directive; the file only uses portable instructions.
9	Two tests were genuinely flaky: a CPU-burn assumption, and a race where the health check was polled on last_history_write while the first sample was still in flight.	Intermittent red builds that would erode trust in CI.	Assert deterministic signals, and poll on the definitive completion counter instead.
10	Primary disk detection picked the longest matching mount point, so inside a container it selected /etc/resolv.conf — a 2.5 GB ext4 bind mount.	Found only by rendering the dashboard against the real container: the disk tile showed a config file instead of the filesystem. A plausible-looking but meaningless number is worse than no number.	Ignore pseudo filesystems and container-injected file mounts, prefer the root filesystem, and otherwise take the shallowest real mount.
14	Bundling the dashboard inside the package made the SPA fallback answer unknown /api/* paths with HTTP 200 and a page of HTML.	A typo in an API call returned "success" with HTML body, which is far harder to debug than a 404. Only reachable once a built UI always exists, so the development deployment never showed it.	Register an explicit /api/{rest} 404 handler before the SPA catch-all, and test it with a bundle present.
15	The CLI printed check marks to a Windows console using a legacy code page (GBK) and crashed with UnicodeEncodeError.	labwatch doctor crashed on the happy path, which is the worst possible moment to fail.	Detect whether the output stream can encode the markers, and fall back to ASCII when it cannot.
16	An f-string reused the same quote character inside its expression, which is a syntax error on Python 3.10 — the floor that pyproject.toml claims to support.	The CLI would fail to import on the oldest supported interpreter while passing every test on 3.12.	Restructured the f-string, and added scripts/check-syntax-floor.py, which parses the package with the 3.10 grammar in CI.
17	Moving the code into a package left 18 relative imports one level too deep, so modules failed to import at runtime.	A structural refactor like this breaks imports silently until an affected path runs; the failures surfaced only when the CLI was exercised.	scripts/check-imports.py resolves every relative import against the files that exist and runs in CI.
18	The project was prepared to publish to PyPI as labwatch — a name that is already taken there by an unrelated homelab CLI (rbretschneider/labwatch_cli, 49 releases). The availability check had only covered GitHub.	The most damaging possible packaging bug: uvx labwatch would either fail outright at upload time or, worse, silently install and run a stranger's tool for anyone following the README.	Publish under the distribution name labwatch-lite, keep the console command and import package as labwatch, explain the difference in both READMEs, and add a release-workflow guard that refuses to build or upload unless pyproject.toml declares labwatch-lite and dist/ contains labwatch_lite-* artifacts.
19	The CI backend job ran ruff check labwatch backend after the old backend/ directory was removed during the packaging refactor.	A red build with no real lint problem behind it: the failure was "file not found", which sends you reading code instead of the workflow. The package and e2e jobs never ran, so the wheel was not verified either.	Correct the invocation to ruff check labwatch tests and add scripts/check-workflow-paths.py, which extracts path-looking arguments from every workflow step and fails when one does not exist.
11	The dashboard only ever showed the root filesystem. Found on the lab server, where /nfs-data1 was 98.4 % full while the UI reported / at 91.7 % and nothing else.	The disk that actually matters on a research server is usually a data volume, so the panel hid the one number that needed acting on: a 15 TB volume with 226 GB left was invisible.	Report every real filesystem by default (LABWATCH_INCLUDE_ALL_MOUNTS), add a filesystem panel that lists them worst-first, and give the disk tile a mount count instead of duplicating two of them.
12	Memory percentage used psutil's total - available, which counts page cache and shared memory as used. On the lab server it read 23.1 % where free read 8.4 %, because 84 GB of /dev/shm was in use.	A dashboard that appears to show a quarter of memory in use when the server is nearly idle teaches the operator to distrust it.	Switch to the free accounting (total - free - buffers - cached) and expose free, cached and available so the figure can be reconciled.
13	Process CPU reported 0.0 % on the first sample after startup, because the psutil baseline was milliseconds old.	On the lab server that meant showing 0.0 % for eight CUDA processes that ps reported at 107-110 %. Reporting nothing is honest; reporting zero is not.	Enforce a minimum baseline age and return no value until a real delta exists. The first sample now shows nothing, subsequent samples show about 100 %.

#	缺陷	影响	修复
1	/api/health 从一个只在关机时才赋值的上下文字段读取 last_history_write。	运行中的服务永远返回 null，采集器看起来像挂了，该值也无法用于监控。	改从历史服务读取实时值，并补充暴露写入次数、失败次数、采样间隔与库中点数。
2	逐进程 CPU 差值被丢弃：psutil 的基准缓存每轮轮询都被清空。	进程 CPU % 永远无法累积出真实差值，始终为 0。	按 PID 保留基准、在补全信息之前打基准，并清理消失的 PID。
3	NVML 的图形运行进程接口让进程表在 Windows 上完全不可用（约 30 个桌面程序）。	GPU 进程表被合成器噪声淹没，看不到真正的工作负载。	默认只列计算进程（与 nvidia-smi --query-compute-apps 一致），并保留包含图形上下文的开关。
4	演示数据的磁盘用量按"Unix 纪元至今的小时数"计算（约 49 万小时）。	每张截图里磁盘卡片都显示 99 % 严重，既错误又刺眼。	把增长曲线锚定到"当前年份 + 9 个月"并加夹取，得到真实的 62 %。
5	演示信号无论图表分辨率如何都按 1 秒采样。	24H 图糊成一块无法阅读的实心块；1H 图则像标准正弦波，一眼假。	让抖动幅度、抖动周期与负载包络都随所请求的时间窗口缩放。
6	调用 os.getloadavg 时未防御 TypeError，而某些平台把它暴露为 None。	CPU 采集直接抛异常，而不是把负载均值报为 N/A。	把缺失或不可调用的实现一律视为 N/A。
7	截图脚本用 /host/i 这类宽松的标题匹配定位区块。	一旦出现 "Host utilisation" 这类图表标题，Playwright 严格模式就会失败 —— 一颗埋着的测试隐患。	改为通过 data-testid 定位区块容器，并让区块标题精确匹配。
8	Dockerfile 带了一个并不需要的 # syntax=docker/dockerfile:1 指令。	在无法访问 Docker 前端镜像仓库的主机上，构建直接失败。	删除该指令；文件本身只使用可移植指令。
9	两个测试确实不稳定：一个依赖"烧 CPU"的假设，另一个在首个采样尚未完成时就轮询 last_history_write。	间歇性红灯会侵蚀对 CI 的信任。	改为断言确定性信号，并以明确的完成计数器作为轮询条件。
10	主磁盘探测选择"匹配路径最长"的挂载点，于是在容器内选中了 /etc/resolv.conf —— 一个 2.5 GB 的 ext4 单文件挂载。	只有真正对着容器渲染面板才发现：磁盘卡片显示的是一个配置文件而不是文件系统。一个看起来合理却毫无意义的数字，比没有数字更糟。	排除伪文件系统与容器注入的文件挂载，优先根文件系统，否则取路径最浅的真实挂载点。
14	把面板打进包里之后，SPA 兜底路由开始把未知的 /api/* 返回成 HTTP 200 加一页 HTML。	一个 API 拼写错误会返回"成功"但内容是 HTML，比 404 难排查得多。只有在"构建产物总是存在"之后才会触发，所以开发环境从未暴露。	在 SPA 兜底之前显式注册 /api/{rest} 的 404 处理，并在存在构建产物的场景下加测试。
15	CLI 向使用旧代码页（GBK）的 Windows 控制台输出勾号时，抛出 UnicodeEncodeError 崩溃。	labwatch doctor 在最顺利的路径上崩溃 —— 这是最不该出错的时刻。	先探测输出流能否编码这些符号，不能则退化为 ASCII。
16	某个 f-string 在表达式内复用了相同的引号字符，这在 Python 3.10 上是语法错误 —— 而 pyproject.toml 声称支持 3.10。	CLI 在最低支持版本上无法导入，但在 3.12 上所有测试都通过。	改写该 f-string，并新增 scripts/check-syntax-floor.py，在 CI 中用 3.10 语法解析整个包。
17	把代码搬进包结构后，18 处相对导入多了一层，导致模块在运行时无法导入。	这类结构重构会静默破坏导入，直到受影响的路径被执行；这里是靠实际跑 CLI 才暴露的。	新增 scripts/check-imports.py，把每个相对导入与真实存在的文件做解析校验，并纳入 CI。
18	项目本来准备以 labwatch 为名发布到 PyPI —— 但该名字已被一个无关的自建服务器 CLI 占用（rbretschneider/labwatch_cli，49 个版本）。此前的可用性检查只覆盖了 GitHub。	这是打包环节最严重的一类缺陷：uvx labwatch 要么在上传时直接失败，要么更糟 —— 让照 README 操作的人静默装上并运行别人的工具。	改用发行名 labwatch-lite 发布，命令行命令与导入包名仍为 labwatch；在两份 README 中说明差异；并在发布工作流中加入守卫：若 pyproject.toml 声明的不是 labwatch-lite，或 dist/ 中没有 labwatch_lite-* 产物，则拒绝构建与上传。
19	打包重构删除了旧的 backend/ 目录后，CI 的后端任务仍在执行 ruff check labwatch backend。	红灯背后并没有真实的 lint 问题：报错是"找不到文件"，会让人去读代码而不是读工作流。同时 package 与 e2e 任务没有运行，等于 wheel 也没被验证。	把命令改为 ruff check labwatch tests，并新增 scripts/check-workflow-paths.py：从每个工作流步骤中抽取形似路径的参数，一旦不存在即失败。
11	面板此前只显示根文件系统。在实验服务器上发现：/nfs-data1 已用 98.4 %，而界面只报 / 的 91.7 %，其余一概不显示。	科研服务器上真正要紧的磁盘通常是数据盘，因此面板恰好隐藏了唯一需要处理的那个数字：一个还剩 226 GB 的 15 TB 卷完全不可见。	默认上报全部真实文件系统（LABWATCH_INCLUDE_ALL_MOUNTS）；新增按使用率降序排列的文件系统面板；磁盘卡片改为显示挂载点数量，不再重复两个。
12	内存百分比使用了 psutil 的 total - available，该口径把页缓存与共享内存算作已用。在实验服务器上它显示 23.1 %，而 free 显示 8.4 %，原因是 84 GB 的 /dev/shm 正在使用。	服务器几乎空闲时面板却像有四分之一内存压力，会让使用者逐渐不再信任这个面板。	改用 free 的口径（total - free - buffers - cached），并额外暴露 free、cached、available 便于核对。
13	启动后第一次采样中进程 CPU 报 0.0 %，因为 psutil 的基准只存在了几毫秒。	在实验服务器上，这意味着把 ps 显示 107–110 % 的八个 CUDA 进程报成 0.0 %。不报是诚实的，报 0 不是。	强制基准的最小存在时长，在形成真实差值前不返回数值。现在首次采样显示空，后续采样约 100 %。

8. Known limitations

8. 已知限制

Limitation	Reason / mitigation
Single host only	An explicit v1 scope decision. Multi-server aggregation is on the roadmap.
No authentication	Intended for trusted private networks. The README warns that process command lines may be sensitive. Put it behind an authenticating reverse proxy if exposed.
NVIDIA only	Telemetry comes from NVML. AMD and Intel GPUs are not read.
Noisy graphics contexts on Windows	WDDM exposes every compositing application as a compute process. Linux targets are unaffected; an opt-in flag includes graphics contexts.
Sampled, not streamed history	A 10 second write interval can miss brief spikes. Lower LABWATCH_HISTORY_INTERVAL if that matters.
Load average is N/A on Windows	The platform does not expose it; the UI shows N/A rather than a fabricated zero.
Per-process CPU needs two polls	CPU usage is a counter delta, so the first sample after startup reports N/A for process CPU. Documented behaviour, not a defect.
Container disk figures describe the container	When the root filesystem is an overlay (Docker Desktop), LabWatch falls back to the shallowest real mount, which may be a data volume. Run LabWatch natively for host disk numbers.
Multiple GPUs verified in demo mode only	This development machine has one GPU, so multi-device rendering was exercised with three synthetic devices. The collector enumerates NVML devices generically, with no single-device assumption.

限制	原因 / 应对
仅支持单机	这是 v1 明确的范围决策。多机聚合在路线图中。
没有认证	面向可信内网。README 已提醒进程命令行可能包含敏感信息。若要对外暴露，请置于带认证的反向代理之后。
仅支持 NVIDIA	遥测来自 NVML，不读取 AMD 与 Intel GPU。
Windows 上图形上下文噪声大	WDDM 会把每个合成程序都暴露为计算进程。目标平台 Linux 不受影响；提供开关以包含图形上下文。
历史是采样的，不是流式的	10 秒写入间隔可能漏掉瞬时尖峰。如有需要可调低 LABWATCH_HISTORY_INTERVAL。
Windows 上负载均值为 N/A	平台不提供该数据；界面显示 N/A，而不是编造一个 0。
逐进程 CPU 需要两次采样	CPU 使用率是计数器差值，因此启动后的第一次采样中进程 CPU 报 N/A。这是有意的行为，不是缺陷。
容器内的磁盘数字描述容器本身	当根文件系统是 overlay（Docker Desktop）时，LabWatch 退回最浅的真实挂载点，可能是数据卷。需要宿主机磁盘数字请直接在宿主机运行。
多卡仅在演示模式下验证过	本开发机只有一张 GPU，因此多设备渲染是通过 3 个合成设备验证的。采集器采用通用 NVML 设备枚举，没有任何单卡假设。

9. Repository hygiene

9. 仓库整洁度

• No scratch scripts, no tmp/, no debug.json, no stray logs. Every tracked file is source, test, documentation, deployment configuration or a referenced image.
• Generated and runtime artifacts are ignored: backend/data/, *.db*, node_modules/, dist/, coverage, Playwright reports, .env.
• One report document (this file) plus ACCEPTANCE.md; screenshots live in docs/images/ and every one is referenced by the README or this report.
• Screenshots are reproducible with frontend/scripts/capture-screenshots.mjs instead of being pasted in by hand.
• No hard-coded machine paths in the application: configuration is entirely environment-driven.

• 没有临时脚本、没有 tmp/、没有 debug.json、没有散落的日志。每个纳入版本控制的文件都是源码、测试、文档、部署配置或被引用的图片。
• 生成物与运行时产物全部忽略：backend/data/、*.db*、node_modules/、dist/、覆盖率、Playwright 报告、.env。
• 只有一份汇报文档（本文件）加一份 ACCEPTANCE.md；截图放在 docs/images/，每一张都被 README 或本报告引用。
• 截图可用 frontend/scripts/capture-screenshots.mjs 重现，而不是手工粘贴进去。
• 应用内没有硬编码的机器路径：配置完全由环境变量驱动。

10. Release readiness

10. 发布就绪度

Definition-of-done item	Status
Core monitoring works	verified against the driver
Multiple GPUs on one host	per-device cards, series and filters
GPU process mapping works	NVML pids joined to psutil detail
Historical charts work	1H/6H/24H, persisted in SQLite
No-GPU failure is graceful	tested; host metrics keep working
Tests pass	179 + 77 + 8 + 13
CI passes	lint, tests, build, E2E, Docker smoke test
Docker works	built, smoke tested, and run with GPU passthrough
README complete	features, quick start, architecture, config, API, limitations, roadmap (English + 中文)
Screenshots complete	twelve, all referenced, including real hardware
Architecture diagram complete	docs/architecture.svg
Demo mode works	and is clearly labelled in the UI
No major console or API errors	asserted by the E2E suite and the capture script
Repository clean	see section 7
Publishable name available on PyPI	published as distribution labwatch-lite; the console command stays labwatch
Sources pushed	GitHub main, CI 6/6 green on every commit

完成度检查项	状态
核心监控可用	已与驱动实测比对
单机多卡	逐卡卡片、序列与过滤
GPU 进程映射可用	NVML PID 与 psutil 信息关联
历史图表可用	1H/6H/24H，持久化于 SQLite
无 GPU 时优雅降级	已测试；主机指标照常工作
测试通过	179 + 77 + 8 + 13
CI 通过	lint、测试、构建、E2E、Docker 冒烟测试
Docker 可用	已构建、已冒烟测试、已验证 GPU 直通
README 完整	功能、快速开始、架构、配置、API、限制、路线图（英文 + 中文）
截图完整	12 张，全部被引用，含真实硬件
架构图完整	docs/architecture.svg
演示模式可用	并在界面中明确标注
无重大 console / API 报错	由 E2E 套件与截图脚本断言
仓库整洁	见第 7 节
PyPI 上可用的发布名	以发行名 labwatch-lite 发布；命令行命令仍为 labwatch
源码已推送	GitHub main，每次提交 CI 均 6/6 全绿