Conpack E2E Eval Results

Source: 1771901522s-since-epoch

Summary

Baseline WORST

44%

recall | 27.9s avg | 9/10 ok | 1 timeouts

Ctx+CLI BEST

100%

recall | 19.3s avg | 10/10 ok | 0 timeouts

MCP+Tools

60%

recall | 23.9s avg | 10/10 ok | 0 timeouts

MCPOnly

50%

recall | 17.5s avg | 10/10 ok | 0 timeouts

Comparison

Metric	Baseline	Ctx+CLI	MCP+Tools	MCPOnly
Total Recall	44.4%	100.0%	60.0%	50.0%
Avg Response Time	27.9s	19.3s	23.9s	17.5s
Success Rate	9/10	10/10	10/10	10/10
Timeouts	1	0	0	0

Per-Query Recall

Query	Baseline	Ctx+CLI	MCP+Tools	MCPOnly
q-001What is the Zephyr Query Protocol and how does its three-phase handshake work?	0%	100%	0%	0%
q-002How does ConpackDB implement ring-buffer sharding with BLAKE3 partition keys?	100%	100%	100%	100%
q-003Explain the Meridian consensus protocol used by Nexara and its temporal sharding approach	0%	100%	0%	0%
q-004How do you configure the Vortex ingestion pipeline stages and dead-letter handling?	100%	100%	100%	100%
q-005What traffic shaping policies does the Auralis service mesh support?	100%	100%	100%	100%
q-006How do you train custom embeddings with the Helix Embedder and export to ONNX?	0%	100%	100%	0%
q-007Describe the Prismoid cache coherence protocol and its gossip-based invalidation	0%	100%	100%	100%
q-008What resources does the conpack-cluster Terraform module provision and what are the auto-scaling triggers?	100%	100%	0%	0%
q-009How does the Spectral Query Analyzer optimize execution plans for multi-upstream searches?	0%	100%	0%	0%
q-010Explain the Obsidian Vault Sync protocol modes and delta sync mechanism for knowledge files	TIMEOUT	100%	100%	100%

Query Details

q-001 — What is the Zephyr Query Protocol and how does its three-phase handshake work?

Expected: eval-001 — Zephyr Query Protocol v3.2 Specification

The Zephyr Query Protocol (ZQP) v3.2 introduces scatter-gather query routing with adaptive fan-out. ZQP uses a three-phase handshake: DISCOVER, NEGOTIATE, STREAM. The DISCOVER phase broadcasts topology probes to all registered shards using multicast group 239.17.42.0/24. NEGOTIATE selects optimal shards based on affinity scores computed via the Kessler-Huang ranking algorithm. STREAM delivers results using chunked transfer with backpressure signaling via ZQP-FlowControl headers. Configuration requires setting zqp.scatter_factor (default 3), zqp.negotiate_timeout_ms (default 150), and zqp.stream_window_size (default 64KB). The protocol supports both synchronous request-response and asynchronous push modes controlled by the zqp.mode parameter.

Vertical	Doc	Match	Confidence	Time	Matched Terms
Baseline	eval-001	NO	0.273	13.1s	title-partial:4/5tag:zephyrtag:query-protocolterm:specificationterm:handshaketerm:phaseterm:three-phase
Ctx+CLI	eval-001	YES	0.569	17.3s	title-partial:4/5phrase:Kessler-Huang rankingtag:zephyrtag:query-protocoltag:scatter-gathertag:zqpterm:adaptiveterm:scoresterm:fan-outterm:resultsterm:signalingterm:pushterm:handshaketerm:phaseterm:three-phase
MCP+Tools	eval-001	NO	0.273	17.5s	title-partial:4/5tag:zephyrtag:query-protocolterm:specificationterm:handshaketerm:phaseterm:three-phase
MCPOnly	eval-001	NO	0.250	17.1s	title-partial:3/5tag:zephyrtag:query-protocolterm:handshaketerm:phaseterm:three-phase

Agent Responses

Baseline — recall 0%

No response captured (run with EVAL_DIRTY=1 to capture responses)

Ctx+CLI — recall 100%

No response captured (run with EVAL_DIRTY=1 to capture responses)

MCP+Tools — recall 0%

No response captured (run with EVAL_DIRTY=1 to capture responses)

MCPOnly — recall 0%

No response captured (run with EVAL_DIRTY=1 to capture responses)

q-002 — How does ConpackDB implement ring-buffer sharding with BLAKE3 partition keys?

Expected: eval-002 — ConpackDB Ring-Buffer Sharding Architecture

ConpackDB implements ring-buffer sharding using BLAKE3 partition keys for deterministic data distribution. Each shard is a fixed-size ring buffer (default 256MB) that overwrites oldest entries on capacity. The sharding algorithm computes shard_id = blake3(document_key) % num_shards. Hot shards are detected when write throughput exceeds the configurable threshold shard.hot_threshold_ops_sec (default 10000). Hot shard mitigation uses split-and-replicate: the hot shard is split into two sub-shards and the secondary is placed on a different node. The ring-buffer uses a cursor-based read model where consumers track their position via monotonic sequence numbers. ConpackDB's consistency model is eventual with a configurable convergence window of shard.convergence_ms (default 500).

Vertical	Doc	Match	Confidence	Time	Matched Terms
Baseline	eval-002	YES	0.410	14.1s	title-partial:3/4phrase:BLAKE3 partitiontag:conpackdbtag:ring-buffertag:shardingtag:blake3term:keysterm:blake3
Ctx+CLI	eval-002	YES	0.728	18.9s	title-partial:3/4phrase:BLAKE3 partitionphrase:ConpackDB implementsphrase:Each shardtag:conpackdbtag:ring-buffertag:shardingtag:blake3term:trackterm:keysterm:blake3term:fixed-sizeterm:num_shardsterm:differentterm:consistencyterm:shards
MCP+Tools	eval-002	YES	0.410	16.6s	title-partial:3/4phrase:BLAKE3 partitiontag:conpackdbtag:ring-buffertag:shardingtag:blake3term:keysterm:blake3
MCPOnly	eval-002	YES	0.432	15.5s	title-partial:3/4phrase:BLAKE3 partitiontag:conpackdbtag:ring-buffertag:shardingtag:blake3term:keysterm:blake3term:different

Agent Responses

Baseline — recall 100%

No response captured (run with EVAL_DIRTY=1 to capture responses)

Ctx+CLI — recall 100%

No response captured (run with EVAL_DIRTY=1 to capture responses)

MCP+Tools — recall 100%

No response captured (run with EVAL_DIRTY=1 to capture responses)

MCPOnly — recall 100%

No response captured (run with EVAL_DIRTY=1 to capture responses)

q-003 — Explain the Meridian consensus protocol used by Nexara and its temporal sharding approach

Expected: eval-003 — Nexara Consensus: The Meridian Protocol

Nexara implements the Meridian consensus protocol, a Byzantine fault-tolerant algorithm designed for geo-distributed clusters. Meridian operates in three rounds: PROPOSE, CERTIFY, COMMIT. Unlike traditional BFT protocols, Meridian uses a rotating committee of 2f+1 validators selected via verifiable random functions (VRFs). The protocol achieves 50,000 TPS in WAN configurations with 100ms finality. Key parameters include meridian.committee_size (default 7), meridian.vrf_threshold (0.67), and meridian.proposal_timeout_ms (default 200). Meridian introduces the concept of 'temporal sharding' where consensus rounds are partitioned across time slots to reduce message complexity from O(n^2) to O(n*log(n)). The Nexara implementation uses the Ed25519 signature scheme for vote signing.

Vertical	Doc	Match	Confidence	Time	Matched Terms
Baseline	eval-003	NO	0.250	9.1s	title-partial:3/4tag:nexaratag:meridiantag:consensusterm:temporal
Ctx+CLI	eval-003	YES	0.569	22.8s	title-partial:3/4phrase:Byzantine fault-toleranttag:nexaratag:meridiantag:bfttag:consensusterm:vrfsterm:validatorsterm:parametersterm:temporalterm:ed25519term:voteterm:certifyterm:timeterm:across
MCP+Tools	eval-003	NO	0.250	21.9s	title-partial:3/4tag:nexaratag:meridiantag:consensusterm:temporal
MCPOnly	eval-003	NO	0.250	17.6s	title-partial:3/4tag:nexaratag:meridiantag:consensusterm:temporal

Agent Responses

Baseline — recall 0%

No response captured (run with EVAL_DIRTY=1 to capture responses)

Ctx+CLI — recall 100%

No response captured (run with EVAL_DIRTY=1 to capture responses)

MCP+Tools — recall 0%

No response captured (run with EVAL_DIRTY=1 to capture responses)

MCPOnly — recall 0%

No response captured (run with EVAL_DIRTY=1 to capture responses)

q-004 — How do you configure the Vortex ingestion pipeline stages and dead-letter handling?

Expected: eval-004 — Vortex Ingestion Pipeline Configuration Guide

The Vortex ingestion pipeline processes streaming data through configurable stages: PARSE, ENRICH, VALIDATE, ROUTE. Each stage runs as an isolated actor with bounded mailboxes (vortex.stage.mailbox_size, default 4096). The PARSE stage supports JSON, Avro, Protobuf, and custom formats via pluggable deserializers registered in vortex.parsers[]. ENRICH applies transformation rules defined in vortex.enrichment_rules[], supporting field mapping, regex extraction, and lookup joins against reference tables. VALIDATE enforces schema contracts using Vortex Schema Language (VSL) files referenced by vortex.schema_path. ROUTE distributes validated records to downstream sinks based on content-based routing rules in vortex.routing_table[]. Dead-letter handling sends invalid records to the vortex.dlq_topic (default 'vortex-dlq'). Backpressure propagates upstream when any stage's mailbox exceeds 80% capacity.

Vertical	Doc	Match	Confidence	Time	Matched Terms
Baseline	eval-004	YES	0.364	43.5s	title-partial:5/5phrase:Dead-letter handlingtag:vortextag:ingestiontag:pipelinetag:streaming
Ctx+CLI	eval-004	YES	0.569	21.2s	title-partial:4/5phrase:Dead-letter handlingtag:vortextag:ingestiontag:pipelinetag:streamingterm:mailboxesterm:throughterm:schematerm:contractsterm:schema_pathterm:enforcesterm:4096term:backpressureterm:vortex-dlq
MCP+Tools	eval-004	YES	0.319	23.1s	title-partial:3/5phrase:Dead-letter handlingtag:vortextag:ingestiontag:pipeline
MCPOnly	eval-004	YES	0.319	17.0s	title-partial:3/5phrase:Dead-letter handlingtag:vortextag:ingestiontag:pipeline

Agent Responses

Baseline — recall 100%

No response captured (run with EVAL_DIRTY=1 to capture responses)

Ctx+CLI — recall 100%

No response captured (run with EVAL_DIRTY=1 to capture responses)

MCP+Tools — recall 100%

No response captured (run with EVAL_DIRTY=1 to capture responses)

MCPOnly — recall 100%

No response captured (run with EVAL_DIRTY=1 to capture responses)

q-005 — What traffic shaping policies does the Auralis service mesh support?

Expected: eval-005 — Auralis Service Mesh: Traffic Shaping Policies

Auralis service mesh provides traffic shaping through declarative policies defined in AuralisPolicy CRDs. Traffic splitting uses weighted backends specified as auralis.backends[].weight (0-100, must sum to 100). Circuit breaking triggers when error_rate exceeds auralis.circuit.threshold (default 0.5) over auralis.circuit.window_sec (default 30). The mesh supports canary deployments via header-based routing: requests with X-Auralis-Canary: true are routed to canary backends. Rate limiting is per-service with auralis.ratelimit.rps (default 1000) and auralis.ratelimit.burst (default 50). Auralis uses sidecar proxies based on the Luminex engine, which processes L4/L7 traffic with sub-millisecond overhead. Mutual TLS is enforced by default between all mesh services using SPIFFE identities (spiffe://auralis.local/service/<name>).

Vertical	Doc	Match	Confidence	Time	Matched Terms
Baseline	eval-005	YES	0.341	19.9s	title-partial:5/6phrase:Auralis servicetag:auralistag:service-meshtag:traffic-shapingterm:service
Ctx+CLI	eval-005	YES	0.569	15.8s	title-partial:5/6phrase:Auralis servicetag:auralistag:service-meshtag:traffic-shapingtag:luminexterm:x-auralis-canaryterm:enforcedterm:luminexterm:window_secterm:header-basedterm:serviceterm:routedterm:requeststerm:ratelimit
MCP+Tools	eval-005	YES	0.341	19.1s	title-partial:5/6phrase:Auralis servicetag:auralistag:service-meshtag:traffic-shapingterm:service
MCPOnly	eval-005	YES	0.341	17.7s	title-partial:5/6phrase:Auralis servicetag:auralistag:service-meshtag:traffic-shapingterm:service

Agent Responses

Baseline — recall 100%

No response captured (run with EVAL_DIRTY=1 to capture responses)

Ctx+CLI — recall 100%

No response captured (run with EVAL_DIRTY=1 to capture responses)

MCP+Tools — recall 100%

No response captured (run with EVAL_DIRTY=1 to capture responses)

MCPOnly — recall 100%

No response captured (run with EVAL_DIRTY=1 to capture responses)

q-006 — How do you train custom embeddings with the Helix Embedder and export to ONNX?

Expected: eval-006 — Helix Embedder: Custom Model Training Workflow

The Helix Embedder produces domain-specific text embeddings using a fine-tuned transformer architecture. Training requires a corpus file in Helix Format (.hxf) containing tab-separated (anchor, positive, negative) triplets. The training pipeline is: helix train --corpus data.hxf --model-dim 384 --epochs 20 --lr 1e-5 --loss contrastive-margin --margin 0.3. Evaluation uses helix eval --model checkpoint.hx --benchmark internal-retrieval --metrics ndcg@10,mrr. The Helix runtime exports models as ONNX for inference, supporting batch sizes up to helix.inference.max_batch (default 32). Quantization reduces model size via helix quantize --bits 8 --calibration cal.hxf. The Helix index format uses product quantization (PQ) with helix.index.pq_segments (default 48) and helix.index.pq_bits (default 8) for compressed vector storage.

Vertical	Doc	Match	Confidence	Time	Matched Terms
Baseline	eval-006	NO	0.205	39.4s	tag:helixtag:embeddertag:trainingtag:onnxterm:helix
Ctx+CLI	eval-006	YES	0.910	16.9s	title-partial:4/6phrase:--benchmark internal-retrievalphrase:--calibration cal.hxfphrase:--corpus data.hxfphrase:--loss contrastive-marginphrase:--lr 1e-5tag:helixtag:embeddertag:trainingtag:onnxterm:storageterm:helixterm:evalterm:sizeterm:pq_bitsterm:compressedterm:--lrterm:contrastive-margin
MCP+Tools	eval-006	YES	0.319	53.6s	title-partial:5/6tag:helixtag:embeddertag:trainingtag:onnxterm:workflowterm:helix
MCPOnly	eval-006	NO	0.296	17.0s	title-partial:4/6tag:helixtag:embeddertag:trainingtag:onnxterm:helix

Agent Responses

Baseline — recall 0%

No response captured (run with EVAL_DIRTY=1 to capture responses)

Ctx+CLI — recall 100%

No response captured (run with EVAL_DIRTY=1 to capture responses)

MCP+Tools — recall 100%

No response captured (run with EVAL_DIRTY=1 to capture responses)

MCPOnly — recall 0%

No response captured (run with EVAL_DIRTY=1 to capture responses)

q-007 — Describe the Prismoid cache coherence protocol and its gossip-based invalidation

Expected: eval-007 — Prismoid Cache Coherence Protocol

The Prismoid protocol maintains cache coherence across distributed proxy nodes using a gossip-based invalidation scheme. Each cache entry carries a vector clock (prismoid.vclock) that tracks the last-write timestamp per node. On cache miss, Prismoid performs a quorum read: it queries prismoid.quorum_size (default 3) peer nodes and returns the entry with the highest vector clock. Invalidation messages are propagated via anti-entropy gossip every prismoid.gossip_interval_ms (default 100). Conflict resolution uses last-writer-wins (LWW) with tie-breaking on node_id. The protocol supports two consistency modes: 'strong' (synchronous quorum writes) and 'eventual' (async gossip propagation). Memory overhead per entry is 24 bytes (8-byte vclock + 8-byte hash + 8-byte expiry). Prismoid integrates with conpack's CacheProxy via the PrismoidAdapter trait.

Vertical	Doc	Match	Confidence	Time	Matched Terms
Baseline	eval-007	NO	0.250	11.5s	title-partial:4/4tag:prismoidtag:cache-coherencetag:gossipterm:gossip-based
Ctx+CLI	eval-007	YES	0.614	27.5s	title:Prismoid Cache Coherence Protocolphrase:Conflict resolutiontag:prismoidtag:cache-coherencetag:gossiptag:vector-clockterm:nodeterm:gossip-basedterm:proxyterm:peerterm:hashterm:traitterm:entryterm:acrossterm:quorum_size
MCP+Tools	eval-007	YES	0.318	25.3s	title:Prismoid Cache Coherence Protocoltag:prismoidtag:cache-coherencetag:gossipterm:gossip-basedterm:proxy
MCPOnly	eval-007	YES	0.318	16.7s	title:Prismoid Cache Coherence Protocoltag:prismoidtag:cache-coherencetag:gossipterm:gossip-basedterm:proxy

Agent Responses

Baseline — recall 0%

No response captured (run with EVAL_DIRTY=1 to capture responses)

Ctx+CLI — recall 100%

No response captured (run with EVAL_DIRTY=1 to capture responses)

MCP+Tools — recall 100%

No response captured (run with EVAL_DIRTY=1 to capture responses)

MCPOnly — recall 100%

No response captured (run with EVAL_DIRTY=1 to capture responses)

q-008 — What resources does the conpack-cluster Terraform module provision and what are the auto-scaling triggers?

Expected: eval-008 — Terraform Module: ConpackCluster Deployment

The conpack-cluster Terraform module deploys a production-ready conpack proxy cluster on AWS. Resources provisioned: ALB (conpack_alb), ECS Fargate tasks (conpack_proxy, 3 replicas), ElastiCache Redis (conpack_cache), and CloudWatch alarms. Required variables: cluster_name, vpc_id, subnet_ids[], conpack_image, upstream_urls[]. Optional variables: proxy_cpu (default 1024), proxy_memory (default 2048), cache_node_type (default 'cache.r6g.large'), min_tasks (default 3), max_tasks (default 12). Auto-scaling triggers on CloudWatch metric ConpackProxyCacheHitRate: scales out when hit rate drops below 70% for 5 minutes, scales in when above 90% for 15 minutes. Health checks use /ready endpoint with 30-second interval. The module outputs: alb_dns_name, ecs_cluster_arn, redis_endpoint, cloudwatch_dashboard_url.

Vertical	Doc	Match	Confidence	Time	Matched Terms
Baseline	eval-008	YES	0.432	50.6s	phrase:30-second intervaltag:terraformtag:conpack-clustertag:awstag:ecs-fargateterm:intervalterm:cloudwatchterm:clusterterm:outputsterm:proxyterm:1024term:triggers
Ctx+CLI	eval-008	YES	0.455	17.5s	phrase:30-second intervaltag:terraformtag:conpack-clustertag:awstag:ecs-fargateterm:intervalterm:cloudwatchterm:clusterterm:outputsterm:cache_node_typeterm:proxyterm:1024term:triggers
MCP+Tools	eval-008	NO	0.159	18.3s	tag:terraformtag:conpack-clusterterm:clusterterm:proxyterm:triggers
MCPOnly	eval-008	NO	0.136	16.1s	tag:terraformtag:conpack-clusterterm:clusterterm:proxy

Agent Responses

Baseline — recall 100%

No response captured (run with EVAL_DIRTY=1 to capture responses)

Ctx+CLI — recall 100%

No response captured (run with EVAL_DIRTY=1 to capture responses)

MCP+Tools — recall 0%

No response captured (run with EVAL_DIRTY=1 to capture responses)

MCPOnly — recall 0%

No response captured (run with EVAL_DIRTY=1 to capture responses)

q-009 — How does the Spectral Query Analyzer optimize execution plans for multi-upstream searches?

Expected: eval-009 — Spectral Query Analyzer: Execution Plan Optimization

The Spectral Query Analyzer optimizes query execution plans for multi-upstream searches. It decomposes queries into sub-plans using cost-based optimization with the Spectral cost model. Cost factors include: upstream_latency_p99 (measured), result_cardinality (estimated via histogram sketches), and network_hop_count. The analyzer supports three plan strategies: spectral.strategy = 'parallel' (fan-out all upstreams simultaneously), 'sequential' (try upstreams in cost order, stop on threshold), or 'adaptive' (start parallel, switch to sequential after first result). Plan caching uses spectral.plan_cache_size (default 1000) with LRU eviction. The EXPLAIN command outputs the chosen plan: EXPLAIN QUERY 'search terms' shows estimated costs per upstream, chosen strategy, and predicted latency. Statistics are refreshed every spectral.stats_refresh_interval_sec (default 300).

Vertical	Doc	Match	Confidence	Time	Matched Terms
Baseline	eval-009	NO	0.114	17.3s	tag:spectraltag:query-analyzerterm:analyzer
Ctx+CLI	eval-009	YES	0.886	16.7s	title:Spectral Query Analyzer: Execution Plan Optimizationphrase:Analyzer optimizesphrase:Cost factorsphrase:EXPLAIN QUERYphrase:Plan cachingtag:spectraltag:query-analyzertag:cost-basedtag:execution-planterm:stopterm:analyzerterm:cost-basedterm:factorsterm:predictedterm:executionterm:parallelterm:cachingterm:statistics
MCP+Tools	eval-009	NO	0.114	19.5s	tag:spectraltag:query-analyzerterm:analyzer
MCPOnly	eval-009	NO	0.114	16.9s	tag:spectraltag:query-analyzerterm:analyzer

Agent Responses

Baseline — recall 0%

No response captured (run with EVAL_DIRTY=1 to capture responses)

Ctx+CLI — recall 100%

No response captured (run with EVAL_DIRTY=1 to capture responses)

MCP+Tools — recall 0%

No response captured (run with EVAL_DIRTY=1 to capture responses)

MCPOnly — recall 0%

No response captured (run with EVAL_DIRTY=1 to capture responses)

q-010 — Explain the Obsidian Vault Sync protocol modes and delta sync mechanism for knowledge files

Expected: eval-010 — Obsidian Vault Sync Protocol for Knowledge Files

The Obsidian Vault Sync protocol synchronizes knowledge files between conpack's local knowledge folder and a remote vault. Sync operates in three modes: obsidian.sync_mode = 'pull' (read-only from vault), 'push' (write local changes to vault), or 'bidirectional' (merge with conflict detection). File identity uses content-addressed hashing: obsidian.hash_algorithm (default 'blake3'). Delta sync transfers only changed blocks using rolling checksums with obsidian.block_size (default 4KB). Conflict resolution in bidirectional mode creates .conflict files with timestamps. The vault manifest (vault.json) tracks file metadata: path, hash, last_modified, size_bytes. Sync scheduling uses obsidian.sync_interval_sec (default 60) with jitter of ±10%. Authentication to remote vaults uses Ed25519 key pairs stored in obsidian.key_path (default '~/.conpack/vault_key').

Vertical	Doc	Match	Confidence	Time	Matched Terms
Baseline	eval-010	NO	0.000	60.0s
Ctx+CLI	eval-010	YES	0.637	18.2s	title-partial:6/6phrase:Delta syncphrase:File identitytag:obsidiantag:vault-synctag:knowledgetag:delta-syncterm:ed25519term:jsonterm:manifestterm:conflictterm:bidirectionalterm:content-addressedterm:modeterm:size_bytes
MCP+Tools	eval-010	YES	0.387	23.9s	title-partial:6/6phrase:Delta synctag:obsidiantag:vault-synctag:knowledgetag:delta-syncterm:mode
MCPOnly	eval-010	YES	0.432	23.6s	title-partial:6/6phrase:Delta synctag:obsidiantag:vault-synctag:knowledgetag:delta-syncterm:vaultsterm:conflictterm:mode

Agent Responses

Baseline — TIMEOUT

No response captured (run with EVAL_DIRTY=1 to capture responses)

Ctx+CLI — recall 100%

No response captured (run with EVAL_DIRTY=1 to capture responses)

MCP+Tools — recall 100%

No response captured (run with EVAL_DIRTY=1 to capture responses)

MCPOnly — recall 100%

No response captured (run with EVAL_DIRTY=1 to capture responses)