======================================================================
Carbon Factor Matcher - Evaluation Report
======================================================================

[1/4] Loading factor database...
  Loaded 7742 factors in 20.7s
[2/4] Initializing matcher...

Loading weights:   0%|          | 0/199 [00:00<?, ?it/s]
Loading weights: 100%|| 199/199 [00:00<00:00, 2302.27it/s]
  Mode: hybrid (embedding=available)
[3/4] Loading ground truth...
  17 test cases loaded
[4/4] Running evaluation...
----------------------------------------------------------------------
======================================================================
SUMMARY
======================================================================

  Retrieval Quality:
    Test cases:     17
    Recall@1:       9/17 (52.9%)
    Recall@3:       9/17 (52.9%)
    Recall@5:       11/17 (64.7%)
    MRR:            0.665

  Value Error Analysis (6 cases with CO2 data):
    [WRONG] elec-uk-230v: UK electricity mix (230-240V)
      CO2: matched=0.5604 vs expected=0.6224 kg/unit
      Error: 0.0620 kg/unit (10.0%)
      Impact: 62.0 kg CO2 per 1000 units
    [OK] elec-finland: Finnish electricity grid (230V)
      CO2: matched=0.2889 vs expected=0.2889 kg/unit
      Error: 0.0000 kg/unit (0.0%)
      Impact: 0.0 kg CO2 per 1000 units
    [WRONG] elec-eu-low-voltage: EU low voltage electricity  keyword search matches 'low voltage' in name suffix
      CO2: matched=0.5633 vs expected=0.5281 kg/unit
      Error: 0.0352 kg/unit (6.7%)
      Impact: 35.2 kg CO2 per 1000 units
    [WRONG] elec-sweden: Swedish electricity (low carbon, mainly hydro/nuclear)
      CO2: matched=1.6287 vs expected=0.1075 kg/unit
      Error: 1.5212 kg/unit (1415.1%)
      Impact: 1521.2 kg CO2 per 1000 units
    [WRONG] elec-poland: Polish electricity (high carbon, coal-heavy)
      CO2: matched=1.6287 vs expected=1.0595 kg/unit
      Error: 0.5692 kg/unit (53.7%)
      Impact: 569.2 kg CO2 per 1000 units
    [WRONG] elec-estonia: Estonian electricity (highest carbon in EU, oil shale)
      CO2: matched=0.5604 vs expected=1.3507 kg/unit
      Error: 0.7903 kg/unit (58.5%)
      Impact: 790.3 kg CO2 per 1000 units

    Aggregate Statistics:
      Mean absolute error:   0.4963 kg CO2/unit
      Mean relative error:   257.3%
      Max absolute error:    1.5212 kg CO2/unit
      Max relative error:    1415.1%
      Max emission impact:   1521.2 kg CO2 per 1000 units

    Correct matches avg impact:  0.0 kg CO2 / 1000 units
    Wrong matches avg impact:    595.6 kg CO2 / 1000 units
    Wrong/Correct ratio:         33355.2x

  FAILED CASES (8):
    - elec-uk-230v: UK electricity mix (230-240V) (expected at #4) [=62.0 kg CO2/1000u]
      Got: Electricity Mix, consumption mix, at consumer, AC, < 1kV
    - elec-eu-low-voltage: EU low voltage electricity  keyword search matches 'low voltage' in name suffix (expected at #2) [=35.2 kg CO2/1000u]
      Got: heat and power co-generation, oil | electricity, high voltage | Cutoff, U
    - transport-barge: Barge transport (expected at #1)
      Got: transport, freight, inland waterways, barge | transport, freight, inland waterways, barge | Cutoff, U
    - steam-natural-gas: Steam from natural gas
      Got: heat and power co-generation, natural gas, combined cycle power plant, 400MW electrical | heat, district or industrial, natural gas | Cutoff, U
    - no-match-unit: Unit mismatch - kWh vs MJ, should find no match
      Got: Electricity Mix, consumption mix, at consumer, AC, < 1kV
    - elec-sweden: Swedish electricity (low carbon, mainly hydro/nuclear) (expected at #4) [=1521.2 kg CO2/1000u]
      Got: electricity production, oil | electricity, high voltage | Cutoff, U
    - elec-poland: Polish electricity (high carbon, coal-heavy) (expected at #6) [=569.2 kg CO2/1000u]
      Got: electricity production, oil | electricity, high voltage | Cutoff, U
    - elec-estonia: Estonian electricity (highest carbon in EU, oil shale) (expected at #7) [=790.3 kg CO2/1000u]
      Got: Electricity Mix, consumption mix, at consumer, AC, < 1kV

