Conpack E2E Eval Results

Generated: 2026-07-18 14:03:18 UTC

Summary

No Ctx BEST CONFIDENCE
35.9%
confidence  |  pass rate: 100%
19.9s
avg time  |  p95: 30.5s  |  p99: 31.4s  |  best: 7.6s
MCP Tools BEST TIME
10.3%
confidence  |  pass rate: 0%
11.7s
avg time  |  p95: 19.7s  |  p99: 20.3s  |  best: 1.8s

Comparison

MetricNo CtxMCP Tools
Avg Confidence35.9%10.3%
Pass Rate100%0%
Avg Time19.9s11.7s
Time P9530.5s19.7s
Time P9730.9s20.0s
Time P9931.4s20.3s
Best Time7.6s1.8s
Success Rate3/33/3
Timeouts00
Errors00

Coverage Matrix (Strategy × Vertical)

StrategyNo CtxMCP Tools
fts100% (1/1)0% (0/1)
vector100% (1/1)0% (0/1)
cascade100% (1/1)0% (0/1)

Per-Strategy Recall

StrategyNo CtxMCP Tools
fts100% (1/1)0% (0/1)
vector100% (1/1)0% (0/1)
cascade100% (1/1)0% (0/1)

Per-Query Confidence

QueryStrategyNo CtxMCP Tools
q-fts-1Tokio async runtime schedulerfts41%-
q-vector-1similarity search over high-dimensional embeddingsvector31%-
q-cascade-1Meilisearch typo tolerance rankingcascade35%-
q-fts-2Cache-Control ETag header freshnessfts-29%
q-vector-2in-memory data store with TTL evictionvector-2%
q-cascade-2Result enum pattern matching errorscascade-0%

Query Details

q-fts-1 No Ctx — Tokio async runtime scheduler
Expected: doc-001 — Rust async tokio runtime
Tokio is an async runtime for the Rust programming language, providing the building blocks for writing network applications. It offers a multi-threaded work-stealing scheduler and supports both TCP and UDP protocols.
VerticalDocMatchConfidenceTimeMatched Terms
No Ctxdoc-001YES0.41220.4s
title-partial:3/4tag:asynctag:tokiotag:runtimeterm:tokioterm:asyncterm:applicationsterm:runtime
Agent Responses
No Ctx — recall 100%
1. **Dependency Injection**: The dependency injection (DI) pattern is a design pattern in which an object relies on other objects rather than being a subject of their own creation. It allows for code reuse and facilitates the creation of complex, object-oriented systems. 2. **Event Loop**: An event loop is a programming mechanism that waits for events such as timer intervals or user input to occur. It processes these events and performs the appropriate actions. 3. **C++11**: C++11 introduced several features that improved the performance and usability of the C++ runtime environment, including the introduction of the `async` and `async_await` keywords, the `async_return` function, and the `async` function. 4. **Future**: A future represents an asynchronous operation that returns a result. It is an object that represents the result of an asynchronous operation. 5. **Task Parallel Library (TPL)**: The TPL is a library for parallel programming that allows developers to write parallel code that can be executed on multiple threads or processes simultaneously. 6. **Timer Intervals**: Timer intervals are used to schedule asynchronous operations, such as tasks, threads, or coroutines, to run at specific times. The interval is typically defined in milliseconds or microseconds. 7. **Event-Driven Programming**: Event-driven programming is a programming paradigm in which the program is controlled by events, such as timer intervals or user input. It allows for more responsive and dynamic behavior. 8. **Synchronization Primitives**: Synchronization primitives, such as locks and atomic operations, are used to manage shared resources and prevent concurrent access to them. 9. **Native C++ Code**: The native C++ code is the code that is compiled to native code by a C++ compiler (such as GCC). This code is executed directly by the underlying operating system. 10. **Async/Await**: Async/await is a programming pattern that allows asynchronous code to be executed in a synchronous manner. It is used to handle asynchronous operations that can take a significant amount of time to complete. By implementing the above concepts, the Tokio async runtime scheduler can be utilized to create asynchronous applications that are scalable, responsive, and efficient.
q-vector-1 No Ctx — similarity search over high-dimensional embeddings
Expected: doc-004 — Vector databases for embeddings
Vector databases like Qdrant, Pinecone, and Milvus store high-dimensional embeddings and enable similarity search using cosine distance, dot product, or Euclidean distance. They power modern RAG and recommendation systems.
VerticalDocMatchConfidenceTimeMatched Terms
No Ctxdoc-004YES0.31031.6s
title-partial:2/3tag:vectortag:embeddingsterm:high-dimensionalterm:vectorterm:similarityterm:systemsterm:cosine
Agent Responses
No Ctx — recall 100%
The similarity search over high-dimensional embeddings is a technique used in natural language processing and information retrieval systems. Here are the technical details: **Technique:** The similarity search over high-dimensional embeddings is a technique that compares the similarity between a given input document or query and the top-k most similar documents in a high-dimensional vector space. This is done using the cosine similarity formula, which calculates the similarity between two vectors as the cosine of the angle between them. The similarity score is then normalized by the maximum possible value to obtain a similarity score for each document. The similarity score is used to identify documents that are similar to the given query, and the top-k most similar documents are returned as the results. **Methodology:** The similarity search over high-dimensional embeddings is performed using a combination of dimensionality reduction techniques and machine learning algorithms. One common method is to use the K-Nearest Neighbors (KNN) algorithm, which calculates the similarity between each document and all other documents in the training dataset. The most similar documents are then selected to form a K-nearest neighbors set. The similarity score between each document and the K-nearest neighbors is calculated using the cosine similarity formula. The top-k most similar documents are then selected as the results of the similarity search. **Hardware and Software:** The similarity search over high-dimensional embeddings is performed on a wide range of hardware and software platforms, including large-scale distributed computing systems, cloud computing platforms, and specialized hardware devices. For example, Google's Cloud AI Platform provides an SDK for Python to perform similarity search over high-dimensional embeddings, while Amazon's SageMaker platform provides a framework for building and deploying machine learning models. **Implementation:** The implementation of the similarity search over high-dimensional embeddings typically involves the following steps: 1. Pre-processing: The input data is pre-processed to reduce the dimensionality of the data, such as through PCA or t-SNE. 2. Feature extraction: The data is extracted from the pre-processed data to obtain the high-dimensional representation of the data. 3. K-Nearest Neighbors algorithm: The similarity score is calculated using the cosine similarity formula for each document to form a K-nearest neighbors set. 4. Top-k selection: The K-nearest neighbors are selected as the top-k most similar documents. 5. Similarity search: The similarity score between the input document and the top-k most similar documents is calculated using the cosine similarity formula. **Datasets:** The similarity search over high-dimensional embeddings is typically performed on large datasets, such as labeled documents or large-scale text corpora. Examples of datasets include: * Word embeddings: Word embeddings, such as Word2Vec and GloVe, are used to represent words as dense vectors in a high-dimensional space. * Text documents: Text documents, such as articles and books, are used to perform similarity search over high-dimensional embeddings. * Machine learning datasets: Machine learning datasets, such as ImageNet and Stanford Question Answering Dataset, are used to perform similarity search over high-dimensional embeddings. **Conclusion:** The similarity search over high-dimensional embeddings is a powerful technique used in natural language processing and information retrieval systems to identify documents that are similar to a given query. The technique is performed using a combination of dimensionality reduction techniques and machine learning algorithms, such as the K-Nearest Neighbors (KNN) algorithm. The implementation of the similarity search over high-dimensional embeddings typically involves the use of large-scale distributed computing systems, cloud computing platforms, and specialized hardware devices.
q-cascade-1 No Ctx — Meilisearch typo tolerance ranking
Expected: doc-002 — Meilisearch full-text search
Meilisearch is a lightweight, fast, and open-source search engine that delivers instant and relevant search results out of the box. It uses BM25-like ranking and supports typo tolerance, filters, and faceting.
VerticalDocMatchConfidenceTimeMatched Terms
No Ctxdoc-002YES0.3547.6s
title-partial:2/3tag:meilisearchtag:searchterm:toleranceterm:engineterm:rankingterm:typo
Agent Responses
No Ctx — recall 100%
Meilisearch typo tolerance ranking is a feature in Meilisearch that allows users to customize the tolerance values for different types of typos. Here's how it works: 1. The Meilisearch engine can be configured to use different tolerance values for different types of typos, such as "exact", "close enough", "no change", "minimal change", "maximal change", "exact matching", and "no matching". 2. When a typo is detected, Meilisearch checks the type of typo and its corresponding tolerance value. For example, "exact" typos are penalized by a small increase in the matching percentage. 3. Meilisearch then adjusts the matching percentage based on the type of typo and the tolerance value. If a typo is a "exact" match, the matching percentage will be adjusted to match the default value. If a typo is a "close enough" match, the matching percentage will be adjusted to the default value, but the actual matching percentage will be slightly lower due to the tolerance. 4. The user can then view the ranking of the typos based on their tolerance values, with "exact" typos being the top tier and "no matching" typos being the lowest. By adjusting the tolerance values, users can tailor Meilisearch to their specific needs and preferences. For example, a user may want to focus on typos with a low matching percentage, while another user may prefer to focus on typos with a high matching percentage.
q-fts-2 MCP Tools — Cache-Control ETag header freshness
Expected: doc-005 — HTTP caching headers explained
Cache-Control, ETag, Last-Modified, and Vary headers control how HTTP responses are cached by browsers and CDNs. Cache-Control: max-age=N sets freshness in seconds. Must-revalidate forces rechecking stale entries.
VerticalDocMatchConfidenceTimeMatched Terms
MCP Toolsdoc-005NO0.28612.9s
phrase:Cache-Control ETagtag:httptag:cachetag:headersterm:headersterm:vary
Agent Responses
MCP Tools — recall 0%
To find relevant documents using the cache-control ETag header freshness, I would use the following command: ```bash curl -H "Cache-Control: max-age=3600" -H "ETag: 1" -H "Content-Type: application/json" http://my-service.com/api/documents ``` In this command: - `curl` is the client tool for curl. - `-H` is a syntax used in curl commands to set headers. In this case, we're setting the cache-control header to `max-age=3600`. This means the document will expire after 1 hour. - `-H "ETag: 1"` sets the ETag header to `1`. - `-H "Content-Type: application/json"` sets the Content-Type header to `application/json`, allowing us to pass JSON data as the response. - `http://my-service.com/api/documents` specifies the endpoint we're trying to reach. Once we run this command, the cache-control header will be updated to `max-age=3600` and the ETag header set to `1`. This indicates that the document has been aged and is now fresh. Note: The exact format of the JSON response may vary based on the specific endpoint, but the basic syntax should remain the same.
q-vector-2 MCP Tools — in-memory data store with TTL eviction
Expected: doc-003 — Caching strategies in distributed systems
Distributed caches reduce latency and backend load. Strategies include read-through, write-through, write-behind, and cache-aside. TTL and eviction policies determine freshness versus memory pressure.
VerticalDocMatchConfidenceTimeMatched Terms
MCP Toolsdoc-003NO0.02420.4s
term:eviction
Agent Responses
MCP Tools — recall 0%
To find relevant documents in a in-memory data store with TTL eviction, you can use the following conproxy tools: 1. conproxy_search: This command provides a search interface for searching in-memory data stores. It can be used to find relevant documents and retrieve specific information. 2. conproxy_search_by_id: This command can be used to search for documents by their unique identifiers. 3. conproxy_search_by_id_and_authority: This command can be used to search for documents that match both their unique identifiers and a specific authority. Here are some examples of using these commands to find relevant documents in an in-memory data store with TTL eviction: ``` # Search for documents by their unique identifier conproxy_search_by_id 12345 # Search for documents that match their unique identifier and a specific authority conproxy_search_by_id_and_authority 12345 abc ``` To retrieve specific information from a document, you can use the `doc` command. Here's an example: ``` # Retrieve the title of a document doc 12345 ``` To retrieve the author of a document, you can use the `author` command: ``` # Retrieve the author of a document author 12345 ``` To retrieve a list of documents that match a specific search criteria, you can use the `search` command: ``` # Search for documents that match a specific search criteria search -d 12345 -t 12345 -a "abc" ``` To get the TTL of a document, you can use the `ttl` command: ``` # Get the TTL of a document ttl 12345 ``` To verify the existence of a document, you can use the `exists` command: ``` # Verify if a document exists exists 12345 ```
q-cascade-2 MCP Tools — Result enum pattern matching errors
Expected: doc-006 — Rust error handling with Result
Rust uses Result<T, E> for recoverable errors and panic! for unrecoverable ones. The ? operator propagates errors up the call stack. anyhow::Result is a common choice for application code; thiserror for library code.
VerticalDocMatchConfidenceTimeMatched Terms
MCP Toolsdoc-006NO0.0001.8s
Agent Responses
MCP Tools — recall 0%
Sorry for the misunderstanding, but I don't have access to the conproxy MCP tools mentioned in your question. I'm here to help you find relevant documents and answer with specific technical details.