One query embedding,
many independent labels.

EoMT in one breath

Before the attributes make sense, the host model: EoMT is a DINOv2-with-registers ViT whose last few transformer blocks are augmented with a fixed set of learnable queries (the Mask2Former idea). Each query is one "slot" that learns to latch onto one object instance.

After the encoder runs, every query produces a single vector — the per-query embedding, shape [B, Q, hidden] (batch × queries × width). The whole model is essentially "turn that embedding into predictions":

• the class head (class_predictor) turns it into a primary label,
• the mask head (or a box head, in the detect family) turns it into geometry.

It's NMS-free: two overlapping cars stay two distinct queries instead of being merged by non-max suppression. That property is what lets attributes stay attached to the right instance.

Two families share all of the above and differ only in the geometry head: instance → masks, detect → boxes. Everything about attributes applies identically to both.

The label space explodes

A normal detector has one flat list of classes. So when an object has a type and a colour and a state, the only way to express it is the Cartesian product: type × colour × state. That blows up combinatorially, starves each leaf class of examples, and multiplies the targets the Hungarian matcher has to assign.

attr-eomt factorizes instead. Keep the primary taxonomy small and general (car, apple, person); push orthogonal distinctions into independent heads that add, not multiply. Drag the sliders — this is the entire argument for the feature:

The heads tap the same embedding

The key implementation idea: the attribute heads don't get a new feature extractor. They read the exact same per-query embedding that feeds class_predictor — captured non-invasively with a PyTorch forward hook. The detector doesn't even know they're there.

In code it's about as small as it sounds. Each head is built by _build_aux_head — a bare linear probe, or a tiny MLP if you ask for more layers:

# model.py — one head per attribute spec, sharing one arch
self.aux_heads = nn.ModuleDict({
    s.name: _build_aux_head(hidden, s.num_classes, arch)  # Linear, or small MLP
    for s in self.aux_specs
})

# non-invasive: grab the embedding feeding the class head
self.eomt.class_predictor.register_forward_hook(self._capture_query_embed)

def _capture_query_embed(self, _m, inputs, _out):
    self._query_embed = inputs[0]   # [B, Q, hidden]

Default head arch is a 2-layer MLP (Linear → LayerNorm → GELU → Linear); layers=1 makes it a pure linear probe. The arch is stored in the checkpoint, so reloading rebuilds identical modules.

It rides on the detector's own matching

The clever part of training is that attributes never run their own matcher. Detection already solves "which query is responsible for which ground-truth object" via the Hungarian matcher. Attributes simply reuse that same query→GT assignment, then read the answer off the matched queries.

1. MatchReuse model.eomt.criterion.matcher on the model's geometry (masks for instance, boxes for detect) to get (query_idx, gt_idx) pairs — the same assignment the detection loss used.
2. Gate (optional)The matcher pairs every GT, even barely-overlapping ones (common early on). gate_indices drops pairs whose predicted↔GT IoU is below a threshold, so attributes only learn from queries that actually localize the object.
3. Gather_gather_matched pulls just the matched rows out of query_embed → a tight [N, hidden] tensor of "real" instances.
4. LossEach head runs on that tensor; cross-entropy against the per-instance attribute labels, summed across heads, scaled by aux_w (default 1.0), and added to the detector loss. Missing labels use ignore_index and contribute nothing.
5. Graph-safe zeroIf no query matched in a batch, it returns embed.sum()*0 — a real zero that keeps autograd happy instead of crashing.

# aux_cls.py — the whole supervision, condensed
indices  = match_queries(model, out, geom_labels, class_labels)  # reuse Hungarian
indices  = gate_indices(out, indices, ..., iou_thr=0.5)          # keep well-localized
matched  = gather_matched(out, indices)                          # [N, hidden]
for name, head in model.aux_heads.items():
    logits = head(matched)
    loss  += w * F.cross_entropy(logits, gt[name], ignore_index=-100)

It never hijacks model selection

Per-head matched-query accuracy is printed live and written to metrics.csv, but it never drives which checkpoint becomes best.pt — that's still segm/mAP (or bbox/mAP). The attribute genuinely tags along without changing the detector's training contract.

Attributes live inside the COCO annotations

Because every COCO annotation is already a per-instance object, alignment is automatic — pycocotools still parses the file untouched. Two additions define the heads; no YAML changes, the heads (count, classes, names) are discovered straight from the JSON, just like nc.

1 · a top-level attributes list (the vocabulary)

"attributes": [
  {"name": "ripeness", "categories": [
     {"id": 0, "name": "unripe"}, {"id": 1, "name": "turning"}, {"id": 2, "name": "ripe"}]},
  {"name": "grade", "categories": [
     {"id": 10, "name": "A"}, {"id": 20, "name": "B"}]}
]

2 · a per-annotation attributes map

{ "id": 1, "image_id": 42, "category_id": 1, "bbox": [...], "segmentation": [...],
  "attributes": {"ripeness": 1, "grade": 20} }

Raw ids are remapped to a contiguous 0..n-1 per head (so grade's 10/20 → 0/1). A missing value defaults to 0; a JSON with no attributes is just detection-only, exactly as before.

At inference

Each kept detection gets an aux = {head: {"ids", "probs"}} entry, and predict(plot=True) renders every attribute next to the class label, using the names stored in the checkpoint — the primary class on the first row, each attribute + its confidence on the row beneath.

Factorize the labels, tap the embedding, reuse the match

Keep the primary taxonomy small and well-populated. For everything orthogonal — colour, state, grade, laterality — attach an independent head that reads the per-query embedding the detector already produces, supervise it on the detector's own Hungarian match, and add a cross-entropy term. You get composable, generalizing per-instance attributes for almost no extra compute, and the primary detection metric stays exactly as it was.