load_backend: loaded RPC backend from /var/home/rybens/.local/share/ggufone/runtime/b11026-linux-x64-vulkan/libggml-rpc.so
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = NVIDIA GeForce RTX 3060 Ti (NVIDIA) | uma: 0 | fp16: 1 | bf16: 1 | fp4: 0 | warp size: 32 | shared memory: 49152 | int dot: 1 | matrix cores: NV_coopmat2
load_backend: loaded Vulkan backend from /var/home/rybens/.local/share/ggufone/runtime/b11026-linux-x64-vulkan/libggml-vulkan.so
load_backend: loaded CPU backend from /var/home/rybens/.local/share/ggufone/runtime/b11026-linux-x64-vulkan/libggml-cpu-haswell.so
~llama_context:    Vulkan0 compute buffer size is 483.8672 MiB, matches expectation of 483.8672 MiB
~llama_context: Vulkan_Host compute buffer size is   7.9877 MiB, matches expectation of   7.9877 MiB
~llama_context:    Vulkan0 compute buffer size is 483.8672 MiB, matches expectation of 483.8672 MiB
~llama_context: Vulkan_Host compute buffer size is   7.9877 MiB, matches expectation of   7.9877 MiB
F
=================================== FAILURES ===================================
___________ test_the_bench_sends_the_same_prompt_as_the_serving_path ___________

monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f8d66670190>

    @pytest.mark.model
    def test_the_bench_sends_the_same_prompt_as_the_serving_path(monkeypatch):
        """[card t_6de5fc53] The bench executes the plan the serving path executes — byte for byte.
    
        The instrument must run the product wiring: `cli.ask`/`run` plan the context from the **model
        handle** (so the model's chat template resolves) and the bench used to re-plan it from the
        live session, which resolves nothing and silently fell back to the plain E1b framing. This
        gate captures the tokens each path really prefills, on one real model and one real dev item,
        and asserts they are identical — on the pre-fix tree it fails with the plain prefix (102
        tokens on Spark-X2.5-4B's `c01`) against the serving one (119).
        """
        path = _model()
        item = devset.load()[0]
        payload = devset.request_for(item, model=str(path), threads=4)
    
        prefilled: list[list[int]] = []
        original = session_module.ModelSession.prefill
    
        def spy(self, tokens, **kwargs):
            prefilled.append([int(token) for token in tokens])
            return original(self, tokens, **kwargs)
    
        monkeypatch.setattr(session_module.ModelSession, "prefill", spy)
    
        # the serving path, exactly as `ggufone ask/run` reaches the engine
        serving = cli.decide_payload(payload, home=None, fit_enabled=False)
        serving_tokens = prefilled[-1]
        prefilled.clear()
    
        # the bench path, exactly as `ggufone bench --suite quality` reaches the engine
        report = suites.run_suite(
            harness.BenchConfig(suite="quality", model_path=str(path), items=1, threads=4,
                                backend=harness.CPU_BACKEND),
            factory=suites.live_factory)
        bench_tokens = prefilled[-1]
    
        # the load-bearing claim first: what each path really sent
>       assert bench_tokens == serving_tokens, (
            f"the bench sends {len(bench_tokens)} prefix tokens and the serving path "
            f"{len(serving_tokens)}: the executed plan is not the one the product resolves")
E       AssertionError: the bench sends 102 prefix tokens and the serving path 119: the executed plan is not the one the product resolves
E       assert [3683, 599, 2...8067, 27, ...] == [0, 130972, 1...599, 259, ...]
E         
E         At index 0 diff: 3683 != 0
E         Right contains 17 more items, first extra item: 2945
E         Use -v to get more diff

tests/test_bench_live.py:186: AssertionError
=========================== short test summary info ============================
FAILED tests/test_bench_live.py::test_the_bench_sends_the_same_prompt_as_the_serving_path
1 failed in 23.69s
