Metadata-Version: 2.4
Name: evalanchor
Version: 0.1.0
Summary: Beyond Hit@1: standardized evaluation protocol + harness for zero-shot multi-label recognition
Author: Lizhuo Zhang, Yan Ma
License: MIT License
        
        Copyright (c) 2026 Lizhuo Zhang and Yan Ma
        
        Permission is hereby granted, free of charge, to any person obtaining a copy
        of this software and associated documentation files (the "Software"), to deal
        in the Software without restriction, including without limitation the rights
        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
        copies of the Software, and to permit persons to whom the Software is
        furnished to do so, subject to the following conditions:
        
        The above copyright notice and this permission notice shall be included in all
        copies or substantial portions of the Software.
        
        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
        SOFTWARE.
        
Project-URL: homepage, https://github.com/zhanglizhuo/EvalAnchor
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: numpy>=1.20
Requires-Dist: scipy>=1.7
Dynamic: license-file

# Beyond Hit@1: Metric Choice, Implementation Channels, and a Standardized Evaluation Protocol for Zero-Shot Multi-Label Recognition

**The problem.** Naive evaluation of zero-shot multi-label recognition defaults to **Hit@1 (top-1 accuracy)**, which is lenient for multi-label tasks and systematically flatters zero-shot models — even making zero-shot appear to *beat* supervised models. This project formalizes the **metric-asymmetry finding** (first surfaced in `scb5-zeroshot`) into a **standardized evaluation protocol + open-source benchmark harness**.

> **Status:** 🚧 **M0–M3 draft complete (2026-08-14).** Evidence chain independently cross-validated (109/109 checks); manuscript drafted (`paper/EvalAnchor.tex`, 8pp, compiles clean). Pre-submission items: author lists, one bib verification, series-status re-grep.

## What this research is about

For a multi-label image, a "correct" prediction means hitting *any* of the true labels — so top-1 accuracy (Hit@1) systematically under-punishes misses and overstates zero-shot performance. The seed finding: on TeacherBehavior (8-class multi-label), zero-shot Hit@1 = **85.56%** (appears to beat the supervised linear probe's 77.10%), yet proper multi-label **Sample-F1 = 59.94%** vs the supervised **88–90%**.

This project delivers three things:

1. **The ranking-flip matrix** — a systematic cross-model × cross-dataset × cross-metric study showing that *metric choice changes which zero-shot model appears best*.
2. **The protocol** — primary metrics (Sample-F1 + Macro-F1 + mAP), Hit@1 demoted to coarse screening with an explicit caveat, and a threshold-selection procedure.
3. **The harness** — open-source evaluation framework (splits + metric suite + one-command reproduction) on existing public datasets.

**What this is NOT:** a new method, theory, or dataset. "Hit@1 is lenient for multi-label" is *known* (Zhang & Zhou 2014, TKDE review; Madjarov et al. 2012). The contribution is the **zero-shot × multi-label** intersection and the systematic protocol — see `AGENTS.md` for the full novelty boundary.

## Series relationship

This is the capstone of a 6-paper series on visual anchoring in the SCB5 classroom-behavior setting. It audits the series' own metric choices and provides the shared evaluation standard.

```
LA (llm-annotation)       IEEE Access ✅    concept: visual anchoring / AnchorProxy
  └─> VA (VisualAnchor)    EAAI (review)     class-level AnchorScore diagnostic
        └─> IA (InstanceAnchor) preprint      instance-level cross-backbone agreement
  └─> SV (SharedVisual)    preprint          mechanism: shared class-neighborhood geometry
scb5-zeroshot             Symmetry ✅         CAPE prompts, 5-backbone registry, metric-asymmetry seed
evalanchor (THIS)         🚧                capstone: evaluation protocol + harness
```

## Planned repository structure

```
analysis/     — statistical analyses (CPU-only; ranking-flip matrix, protocol validation)
experiments/  — feature/metric extraction (GPU; reuses sibling caches)
results/      — committed evidence JSONs (single source of truth)
paper/        — manuscript (LaTeX)
todo.md       — executable task breakdown
AGENTS.md     — absorbed discipline + project-specific rules
```

## Key results

**The ranking flip (M1 v2, `results/ranking_flip_report.txt`; siglip2 on the corrected open_clip channel, see `results/cache/README.md`):** the model that Hit@1 crowns "best" is overturned by every strict multi-label metric on 2 of 4 datasets, with coupled-bootstrap flip probability 1.000 and BH-FDR q ≤ 4e-4:

| dataset | Hit@1 winner | strict-metric winner(s) | verdict |
|---|---|---|---|
| TeacherBehavior | siglip2 | eva02 (all strict metrics) | **stable flip** (q≤4e-4) |
| MultiTHUMOS | eva02 | openai (sample-F1, exact, Hamming) | **stable flip** (q≤4e-4) |
| VOC2007 | siglip2 | siglip2 (dominates all metrics) | stable |
| COCO | siglip2 | siglip2 (dominates all metrics) | stable |

The flip is *conditional*, and the conditions are diagnostic: it occurs on the high-label-cardinality classroom data (card. 3.61, 94.6% multi-label) and the dense-annotation action data (24% empty frames), and not on the moderate-cardinality detection sets. On TeacherBehavior even Hit@1→Hit@3 flips the winner (openai vs siglip2). A loading-channel audit additionally shows the same architecture name ("siglip2") yields hit@1 0.046 vs 0.965 depending on whether weights come from HF transformers or open_clip (logit corr ≈ 0) — implementation choices act as hidden evaluation variables.

*Honesty note:* an earlier run of this matrix (before the channel fix) showed voc2007/coco flips that were artifacts of a collapsed siglip2 cache; they disappeared after re-extraction and a full-chain re-run (`results/README.md`).

Seed finding (from scb5-zeroshot, accepted at Symmetry): TeacherBehavior zero-shot Hit@1 85.56% vs Sample-F1 59.94% vs supervised 88–90%.

## The protocol (M2, `paper/EvalAnchor.tex`, evidence `results/protocol_validation.json`)

- **Primary suite:** Sample-F1 + Macro-F1 + mAP, reported together; split verdicts reported as split.
- **Hit@1 demoted to screening** with a mandatory caveat: its leniency is a per-sample function of cardinality (on TeacherBehavior P(Hit@1) climbs 0.20→0.83 from 1→4 true labels while Sample-F1 plateaus at ~0.78). No predictive flip rule is claimed (n=4 datasets; the dataset-level Hit@1−Sample-F1 gap does NOT separate flipping from stable sets — documented).
- **Threshold procedure:** 5-fold out-of-fold per-class τ + argmax fallback. Validation: within 0.002–0.005 Sample-F1 of the in-sample oracle, vs 0.07–0.52 lost by fixed global τ.
- **Retroactive series audit:** scb5's CAPE prompt-set ranking (Set-C best under Hit@1, unanimous) inverts under Sample-F1 (Set-B on 4/5 backbones); VisualAnchor-style argmax-recall class difficulty is uncorrelated (point estimates) with protocol F1 on TeacherBehavior (ρ∈[−0.36,0.24], underpowered n=8) but preserved on VOC2007 (ρ=0.858) — diagnostics inherit the same conditionality.

## The harness (M3, `evalanchor/`)

```bash
python -m evalanchor.run --dataset voc2007 --output report.json   # full suite + verdict
```

Audited metric suite (30 known-value + sklearn cross-checks), channel-pinned cache loading with override dir, coupled bootstrap + BH-FDR. P3.4 self-check: outputs bit-identical to the committed evidence on teacher/voc/coco.

## Reproduce

```bash
python analysis/audit_metrics.py          # metric suite audit (30 checks)
python analysis/scope_inventory.py        # cache inventory -> results/scope_inventory.json
python analysis/ranking_flip.py           # headline matrix -> results/ranking_flip_report.json
python analysis/protocol_validation.py    # protocol validation + series audits
python -m evalanchor.run --dataset coco    # one-command evaluation
```

## Citation

🚧 TBD (preprint; manuscript under preparation).

## Data

Multi-label datasets (TeacherBehavior, VOC2007, COCO, MultiTHUMOS) are third-party public benchmarks; feature caches already exist in the sibling repos (`InstanceAnchor/results/per_image/`, `scb5-zeroshot/data/feature_cache/`). See `AGENTS.md` §"Existing assets to reuse".
