Metadata-Version: 2.4
Name: racklint
Version: 0.1.0
Summary: Check bare-metal and GPU servers against NetBox before they go into service.
Author: Reza Iskandar Achmad
License-Expression: MIT
Project-URL: Repository, https://github.com/riskand/racklint
Keywords: redfish,bmc,netbox,commissioning,gpu,data-centre
Classifier: Development Status :: 3 - Alpha
Classifier: Environment :: Console
Classifier: Intended Audience :: System Administrators
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: System :: Hardware
Classifier: Topic :: System :: Systems Administration
Classifier: Typing :: Typed
Requires-Python: >=3.12
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: httpx>=0.28
Requires-Dist: pydantic>=2.9
Requires-Dist: pydantic-settings>=2.6
Requires-Dist: pyyaml>=6.0
Requires-Dist: rich>=13.9
Requires-Dist: typer>=0.15
Provides-Extra: dev
Requires-Dist: mypy>=1.13; extra == "dev"
Requires-Dist: pytest>=8.3; extra == "dev"
Requires-Dist: pytest-asyncio>=0.25; extra == "dev"
Requires-Dist: pytest-cov>=6.0; extra == "dev"
Requires-Dist: respx>=0.22; extra == "dev"
Requires-Dist: ruff>=0.8; extra == "dev"
Requires-Dist: types-pyyaml>=6.0; extra == "dev"
Dynamic: license-file

# racklint

**Check servers against the inventory before they go into service.**

racklint is a commissioning validator for bare-metal and GPU clusters. It reads what each server reports through
its BMC over [Redfish](https://www.dmtf.org/standards/redfish) and compares that with what
[NetBox](https://netbox.dev) says should be there. Every mismatch is flagged: a wrong GPU count, a wrong serial
number, firmware not on the approved baseline, a missing NIC.

You can use it in two ways: as the `racklint` command, or as a Python library.

## Why

When a rack is brought up, the question is not "what hardware is in it?" but "is the hardware in it what we
ordered and planned for?" Existing tools such as [netbox-sync](https://github.com/bb-Ricardo/netbox-sync) and
[netbox-agent](https://github.com/Solvik/netbox-agent) answer the first question: they read the hardware and
write it into NetBox, so NetBox ends up describing whatever was found.

racklint works the other way. NetBox holds the planned state and stays the source of truth. racklint checks the
hardware against it, changes nothing, and fails loudly on any difference. That makes it useful as a gate: a
node only moves to the next stage of bring-up once racklint passes it.

It is read-only. It sends only GET requests to NetBox and to the BMCs. The host OS does not need to be running.

## Install

```bash
pip install racklint
```

racklint needs Python 3.12 or later.

## Command line

### Settings

The command reads its settings from `RACKLINT_*` environment variables, or from a `.env` file in the working
directory.

| Setting | Default | Meaning |
|---|---|---|
| `RACKLINT_NETBOX_URL` | required | Base URL of NetBox, for example `https://netbox.example.net` |
| `RACKLINT_NETBOX_TOKEN` | required | NetBox API token |
| `RACKLINT_BMC_USERNAME` | required | User name for every BMC |
| `RACKLINT_BMC_PASSWORD` | required | Password for every BMC |
| `RACKLINT_BMC_SCHEME` | `https` | `https` or `http` |
| `RACKLINT_BMC_VERIFY_TLS` | `true` | Verify BMC TLS certificates |
| `RACKLINT_BMC_CA_BUNDLE` | unset | CA bundle file used to verify BMC certificates |
| `RACKLINT_CONCURRENCY` | `16` | BMCs read at once |
| `RACKLINT_BMC_CONCURRENCY` | `4` | Requests in flight to one BMC |

A token that starts with `nbt_` (NetBox 4.5 and later) is sent as a bearer token. Any other token is sent as a
legacy `Token`.

### Run a check

```bash
racklint check --rack R01 --baseline baseline.yaml
racklint check --device gpu-01 --device gpu-02 --baseline baseline.yaml
```

Pass exactly one of `--rack` and `--device`.

| Option | Meaning |
|---|---|
| `--rack NAME` | Check every device in this NetBox rack |
| `--site SLUG` | Site slug. Needed only if the rack name exists in several sites |
| `--device NAME` | Check this device. Repeat for more |
| `--baseline PATH` | Firmware baseline file. Required |
| `--format table\|json` | Output format. Default `table` |
| `--concurrency N` | BMCs to read at once. Overrides `RACKLINT_CONCURRENCY` |
| `--ca-bundle PATH` | CA bundle for BMC certificates. Overrides `RACKLINT_BMC_CA_BUNDLE` |
| `--insecure` | Do not verify BMC certificates. Overrides `RACKLINT_BMC_VERIFY_TLS` |

The baseline is a YAML file with the approved firmware versions for each NetBox device type. Each `match` is a
case-sensitive glob on the `Id` of a member of the BMC's firmware inventory. Versions must be quoted strings.

```yaml
device_types:
  gpu-node-h100:             # NetBox device-type slug
    - match: "BMC"
      approved: ["1.4.0"]
    - match: "GPU_*"
      approved: ["96.00.89.00.01"]
```

### Exit codes

| Code | Meaning |
|---|---|
| `0` | Every node passed |
| `1` | At least one node failed, or its BMC could not be read |
| `2` | Usage or configuration error: bad arguments, invalid settings or baseline, NetBox unreachable or returning an error, rack or device not found |

Use the exit code to gate a pipeline.

### Output

The table has one row per node and one line per reason, then a summary:

```console
┏━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Node ┃ BMC        ┃ Status ┃ Reasons                                         ┃
┡━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ n01  │ 192.0.2.11 │ PASS   │                                                 │
│ n02  │ 192.0.2.12 │ PASS   │                                                 │
│ n03  │ 192.0.2.13 │ FAIL   │ gpu: 7 found, 8 planned                         │
│ n04  │ 192.0.2.14 │ PASS   │                                                 │
│ n05  │ 192.0.2.15 │ FAIL   │ memory: 31 DIMMs (1984 GiB), 32 planned (2048   │
│      │            │        │ GiB)                                            │
│      │            │        │ firmware: BMC 1.3.2 not in approved [1.4.0]     │
│ n06  │ 192.0.2.16 │ PASS   │                                                 │
└──────┴────────────┴────────┴─────────────────────────────────────────────────┘
4 passed, 2 failed, 0 errored
```

A node has status `PASS`, `FAIL` or `ERROR`. `ERROR` means racklint could not read the BMC (a connection
failure after retries, a timeout, a rejected login or an invalid reply). No checks run for that node, and the
reason is the error.

With `--format json` the output is the report as JSON. This is one failing node:

```json
{
  "schema_version": 1,
  "results": [
    {
      "node": "n03",
      "bmc_address": "192.0.2.13",
      "status": "fail",
      "findings": [
        {
          "check": "gpu",
          "message": "7 found, 8 planned"
        }
      ],
      "error": null
    }
  ]
}
```

`status` is `pass`, `fail` or `error`. `error` is set only when the status is `error`.

## Python library

The names exported from `racklint` (its `__all__`) are the public API. Anything else may change before 1.0. The
package ships type hints (`py.typed`). The names fall into these groups:

| Group | Names |
|---|---|
| State | `ExpectedNode`, `PlannedInterface`, `HardwareSpec`, `GpuSpec`, `MemorySpec`, `ActualNode`, `Gpu`, `Dimm`, `NetworkPort`, `FirmwareItem` |
| Verdict | `Report`, `NodeResult`, `Finding`, `Status` |
| Running checks | `check_nodes`, `iter_results`, `ActualReader`, `Check`, `default_checks`, `SerialCheck`, `GpuCheck`, `MemoryCheck`, `NicCheck`, `FirmwareCheck` |
| Sources | `NetBoxSource`, `netbox_client`, `RedfishReader`, `BmcAccess`, `FirmwareBaseline`, `FirmwareRule` |
| Errors | `RacklintError`, `BaselineError`, `NetBoxError`, `InventoryError`, `RedfishError` |

The library is async and does not read `RACKLINT_*` settings: you pass it explicit objects. This example reads
the plan for rack R01 from NetBox, reads each BMC and prints a verdict per node. It takes its connection details
from the environment.

```python
import asyncio
import os
from pathlib import Path

from racklint import (
    BmcAccess,
    FirmwareBaseline,
    NetBoxSource,
    RedfishReader,
    check_nodes,
    default_checks,
    netbox_client,
)


async def main() -> None:
    async with netbox_client(os.environ["NETBOX_URL"], os.environ["NETBOX_TOKEN"]) as client:
        nodes = await NetBoxSource(client).rack("R01")

    access = BmcAccess(username=os.environ["BMC_USERNAME"], password=os.environ["BMC_PASSWORD"])
    baseline = FirmwareBaseline.load(Path("baseline.yaml"))
    report = await check_nodes(nodes, RedfishReader(access), default_checks(baseline), concurrency=16)

    for result in report.results:
        print(result.node, result.status, [finding.message for finding in result.findings])
    print("all passed" if report.passed else "not all passed")


asyncio.run(main())
```

`check_nodes` returns a `Report` with the results in the order of `nodes`. To act on each result as soon as it
is ready, for live progress, loop over `iter_results` instead. It takes the same arguments and yields
`NodeResult`s in the order they finish:

```python
import asyncio
import os
from pathlib import Path

from racklint import (
    BmcAccess,
    FirmwareBaseline,
    NetBoxSource,
    RedfishReader,
    default_checks,
    iter_results,
    netbox_client,
)


async def main() -> None:
    async with netbox_client(os.environ["NETBOX_URL"], os.environ["NETBOX_TOKEN"]) as client:
        nodes = await NetBoxSource(client).devices(["gpu-01", "gpu-02"])

    access = BmcAccess(username=os.environ["BMC_USERNAME"], password=os.environ["BMC_PASSWORD"])
    checks = default_checks(FirmwareBaseline.load(Path("baseline.yaml")))
    async for result in iter_results(nodes, RedfishReader(access), checks, concurrency=16):
        print(result.node, result.status)


asyncio.run(main())
```

`NetBoxSource` raises `NetBoxError` and `FirmwareBaseline.load` raises `BaselineError`; both are subclasses of
`RacklintError`. A BMC that cannot be read does not raise: its node gets status `error`.

A `FirmwareBaseline` can also be built from a dict with `FirmwareBaseline.model_validate`, so an application
can keep the baseline in its own database.

The NetBox source is optional. A NetBox plugin runs inside NetBox, so it can build `ExpectedNode`s from the
Django ORM instead of the REST API, then call `asyncio.run(check_nodes(...))` from a background job, which is
synchronous code.

## What it checks

| Check | Fails when | Example reason |
|---|---|---|
| Serial | The device's serial matches the serial of no system on the BMC | `serial: expected SN-R01-03, BMC reports SN-R01-30` |
| GPU | The GPU count differs, or a GPU's model differs from the planned model | `gpu: 7 found, 8 planned` |
| Memory | The enabled DIMM count or the total capacity differs | `memory: 31 DIMMs (1984 GiB), 32 planned (2048 GiB)` |
| NIC | A planned MAC address is not reported, or fewer ports are reported than interfaces are planned | `nic: eth2 (a0:88:c2:00:03:03) not found` |
| Firmware | The device type has no baseline, a baseline entry matches no firmware item, or a matched item's version is not approved | `firmware: BMC 1.3.2 not in approved [1.4.0]` |

A node passes only if every check passes. Firmware items that match no baseline entry are ignored, and so are
reported ports beyond the plan.

If NetBox lacks a value a check needs, the node fails with `not planned in NetBox: <item>`. A device with no
BMC address fails with that reason alone.

### What NetBox must hold

| Item | Where racklint reads it |
|---|---|
| Serial | The device's `serial` field |
| BMC address | The device's out-of-band IP (`oob_ip`). The prefix length is dropped |
| Device type | The device type's slug. It selects the firmware baseline |
| NICs | The device's physical interfaces that are not management-only. A MAC address is compared if the interface has a primary MAC; without one, racklint only counts the port |
| GPU and memory | The `racklint` key of the device's config context |

Assign the config context to the device type, so every device of that type gets it:

```yaml
racklint:
  gpu:    {count: 8, model: "NVIDIA H100 80GB HBM3"}
  memory: {dimm_count: 32, total_gib: 2048}
```

An invalid `racklint` config context is a configuration error and stops the run with exit code `2`.

On the BMC side, racklint starts at `/redfish/v1` and follows the links the service returns. It hardcodes no
vendor paths. It reads systems, processors, memory, network adapters and the firmware inventory. It uses HTTP
Basic authentication on every request and opens no Redfish sessions. It makes up to three attempts at a
request that fails with a connection error, a timeout, HTTP 5xx or 429, and uses a connect timeout of 5
seconds and a read timeout of 15 seconds.

## Try it without hardware

[`examples/no-hardware/`](examples/no-hardware) is a Docker Compose stack with a real NetBox and six mock BMCs,
two of them built wrong on purpose. One command checks the rack, and the summary line is
`4 passed, 2 failed, 0 errored`. See its README for the steps and what is planted.

## Limitations

- GPUs are found only as Redfish `Processors` with `ProcessorType == "GPU"`. A BMC that exposes GPUs another way
  shows zero GPUs.
- One set of BMC credentials applies to every node in a run.
- Read-only by design. racklint reports differences and never fixes them in NetBox or on a BMC.

## Related work

| Project | What it does |
|---|---|
| [bb-Ricardo/netbox-sync](https://github.com/bb-Ricardo/netbox-sync) | Syncs VMware and Redfish sources into NetBox |
| [Solvik/netbox-agent](https://github.com/Solvik/netbox-agent) | Agent on each server that pushes hardware into NetBox |
| [bb-Ricardo/check_redfish](https://github.com/bb-Ricardo/check_redfish) | Monitoring plugin for component health over Redfish |
| OpenStack Ironic, Metal3, MAAS, Warewulf | Full provisioning platforms with BMC inspection |

racklint is smaller than any of these and does one thing: it compares the hardware with NetBox's plan and
reports the differences.

## License

[MIT](LICENSE)
