# DOCNEST Multi-Format RAG Accuracy Evaluation

**Date:** 2026-05-19  
**Formats tested:** PDF, DOCX, XLSX, HTML, Markdown  
**Judge:** Gemini 1.5 Pro  

---

## Overall Results

| Metric | Value |
|--------|-------|
| **Average accuracy** | **8.9 / 10** |
| **Pass rate (≥ 7/10)** | **89%** |
| Total questions | 38 |
| Documents evaluated | 7 |

### Accuracy by Format

| Format | Avg Score | Pass Rate | Questions |
|--------|-----------|-----------|-----------|
| 📊 XLSX | 9.3/10 | 100% | 6 |
| 📝 DOCX | 10.0/10 | 100% | 7 |
| 🌐 HTML | 9.2/10 | 100% | 5 |
| 📋 MD | 10.0/10 | 100% | 5 |
| 📄 PDF | 7.7/10 | 73% | 15 |

---

## Results by Document

### 📊 Acme Corp Financial Workbook 2024 (`XLSX`)

| | |
|---|---|
| Sections | 3 |
| Tables extracted | 3 |
| Parse time | 42 ms |
| Avg score | **9.3/10** |
| Pass rate | **100%** |

| # | Question | Score | Reasoning |
|---|----------|-------|-----------|
| 1 | What was the total Q1 revenue across all products in USD thousands? | ✅ 10/10 | The candidate's answer provides the correct numerical value and accurately qualifies the a |
| 2 | Which product had the highest annual revenue and how much was it? | ✅ 10/10 | The candidate correctly identifies the product and its total revenue, perfectly matching t |
| 3 | How many total employees does the Engineering department have (full-time… | ✅ 10/10 | The candidate answer is perfectly accurate, correctly stating the total employee count and |
| 4 | Which region had the highest year-over-year growth percentage? | ✅ 8/10 | The candidate answer is factually correct but omits the percentage sign from the growth va |
| 5 | How many sheets does this workbook have and what are they named? | ✅ 10/10 | The candidate answer is factually identical to the ground truth, correctly identifying the |
| 6 | What was SupportDesk's Q3 revenue and what was its full-year annual tota… | ✅ 8/10 | The candidate correctly identifies both numerical values but incorrectly states the docume |

<details><summary>Q1: What was the total Q1 revenue across all products in USD thousands?</summary>

**Retrieved section:** `§1` | **Query latency:** 2 ms

**DOCNEST answer:**
> Based on the "Revenue" sheet, the total Q1 revenue across all products was 12550. The provided "Revenue" document excerpt does not specify the currency or if the values are in thousands.

**Ground truth:**
> The sum of Q1 column: DataSync Pro 4200 + CloudVault 3100 + SecureID 1800 + AnalyticsEdge 2500 + SupportDesk 950 = 12550

**Score:** 10/10 — The candidate's answer provides the correct numerical value and accurately qualifies the answer based on the source data provided.

</details>

<details><summary>Q2: Which product had the highest annual revenue and how much was it?</summary>

**Retrieved section:** `§1` | **Query latency:** 2 ms

**DOCNEST answer:**
> Based on the Revenue sheet, DataSync Pro had the highest annual revenue. Its annual total was 23400.

**Ground truth:**
> DataSync Pro: 4200+5100+6300+7800 = 23400

**Score:** 10/10 — The candidate correctly identifies the product and its total revenue, perfectly matching the information in the ground truth.

</details>

<details><summary>Q3: How many total employees does the Engineering department have (full-time + part-time)?</summary>

**Retrieved section:** `§2` | **Query latency:** 1 ms

**DOCNEST answer:**
> Based on the Headcount sheet, the Engineering department has a total of 150 employees. This total consists of 142 full-time employees and 8 part-time employees.

**Ground truth:**
> Engineering: 142 FT + 8 PT = 150

**Score:** 10/10 — The candidate answer is perfectly accurate, correctly stating the total employee count and the precise breakdown of full-time and part-time staff.

</details>

<details><summary>Q4: Which region had the highest year-over-year growth percentage?</summary>

**Retrieved section:** `§3` | **Query latency:** 1 ms

**DOCNEST answer:**
> Based on the Regional Sales data, Asia Pacific had the highest year-over-year growth percentage. This region's YoY Growth % was 31.7.

**Ground truth:**
> Asia Pacific 31.7% is the highest in the Regional Sales sheet

**Score:** 8/10 — The candidate answer is factually correct but omits the percentage sign from the growth value.

</details>

<details><summary>Q5: How many sheets does this workbook have and what are they named?</summary>

**Retrieved section:** `§1` | **Query latency:** 1 ms

**DOCNEST answer:**
> Based on the excerpts, this workbook has three sheets. They are named Revenue, Headcount, and Regional Sales.

**Ground truth:**
> Three worksheets: Revenue, Headcount, Regional Sales

**Score:** 10/10 — The candidate answer is factually identical to the ground truth, correctly identifying the count and all names of the sheets.

</details>

<details><summary>Q6: What was SupportDesk's Q3 revenue and what was its full-year annual total in USD thousands?</summary>

**Retrieved section:** `§1` | **Query latency:** 1 ms

**DOCNEST answer:**
> Based on the Revenue sheet, SupportDesk's Q3 revenue was 1100. Its Annual Total was 4320. The document does not specify if these figures are in USD or in thousands.

**Ground truth:**
> SupportDesk row: Q1=950, Q2=1020, Q3=1100, Q4=1250; Annual Total = 950+1020+1100+1250 = 4320

**Score:** 8/10 — The candidate correctly identifies both numerical values but incorrectly states the document does not specify the units, which were provided in the question.

</details>

---

### 📝 TechVision Inc Annual Report 2024 (`DOCX`)

| | |
|---|---|
| Sections | 13 |
| Tables extracted | 4 |
| Parse time | 119 ms |
| Avg score | **10.0/10** |
| Pass rate | **100%** |

| # | Question | Score | Reasoning |
|---|----------|-------|-----------|
| 1 | What was TechVision's total revenue in FY2024 and the year-over-year gro… | ✅ 10/10 | The candidate answer is perfectly accurate, correctly stating the total revenue and the Yo |
| 2 | Which business segment became the largest revenue contributor for the fi… | ✅ 10/10 | The candidate answer correctly identifies the business segment, the revenue amount, and th |
| 3 | What is the company's net income margin in FY2024? | ✅ 10/10 | The candidate answer correctly states the net income margin is 15.1%, which perfectly matc |
| 4 | What is the severity rating of the cybersecurity breach risk? | ✅ 10/10 | The candidate answer correctly identifies the severity rating as HIGH and accurately cites |
| 5 | What is TechVision's FY2025 revenue guidance range? | ✅ 10/10 | The candidate answer perfectly matches all numerical data (revenue range and growth percen |
| 6 | What percentage of TechVision's electricity came from renewable sources … | ✅ 10/10 | The candidate answer is perfectly accurate, stating the correct percentage and source prov |
| 7 | What is the estimated cost of the Zero-Trust Security Suite initiative? | ✅ 10/10 | The candidate's answer is perfectly accurate, providing the correct cost and timeline from |

<details><summary>Q1: What was TechVision's total revenue in FY2024 and the year-over-year growth rate?</summary>

**Retrieved section:** `§4.1` | **Query latency:** 1 ms

**DOCNEST answer:**
> According to the Executive Summary, TechVision Inc's revenue in FY2024 was $1.24 billion, representing 34% year-over-year growth. The Revenue by Business Segment table also shows a total FY2024 revenue of $1,240M and a total growth rate of "+34.0%".

**Ground truth:**
> Executive Summary: record revenue of $1.24 billion, 34% YoY growth

**Score:** 10/10 — The candidate answer is perfectly accurate, correctly stating the total revenue and the YoY growth rate from the ground truth.

</details>

<details><summary>Q2: Which business segment became the largest revenue contributor for the first time?</summary>

**Retrieved section:** `§4.1` | **Query latency:** 1 ms

**DOCNEST answer:**
> According to document §4.1, "cloud services becoming the largest contributor for the first time in company history." The Cloud Services segment generated $621 million in revenue in FY2024, making it the largest of the three business segments.

**Ground truth:**
> Section 2.1: Cloud Services $621M became largest for first time

**Score:** 10/10 — The candidate answer correctly identifies the business segment, the revenue amount, and the fact that it was the largest for the first time, perfectly matching all key information in the ground truth.

</details>

<details><summary>Q3: What is the company's net income margin in FY2024?</summary>

**Retrieved section:** `§2` | **Query latency:** 2 ms

**DOCNEST answer:**
> According to the Executive Summary, TechVision Inc's net income margin in FY2024 was 15.1%. The Key Profitability Metrics table also lists the "Net Margin" for "FY2024" as "15.1%". This was an increase from a "Net Margin" of "10.6%" in "FY2023".

**Ground truth:**
> Executive Summary: Net income $187M, 15.1% margin

**Score:** 10/10 — The candidate answer correctly states the net income margin is 15.1%, which perfectly matches the ground truth.

</details>

<details><summary>Q4: What is the severity rating of the cybersecurity breach risk?</summary>

**Retrieved section:** `§6.1` | **Query latency:** 2 ms

**DOCNEST answer:**
> According to the "Risk Assessment Matrix 2024" table, the severity rating for the "Cybersecurity breach" risk is HIGH.

**Ground truth:**
> Risk Assessment Matrix: Cybersecurity breach Severity = HIGH

**Score:** 10/10 — The candidate answer correctly identifies the severity rating as HIGH and accurately cites the source document mentioned in the ground truth.

</details>

<details><summary>Q5: What is TechVision's FY2025 revenue guidance range?</summary>

**Retrieved section:** `§8` | **Query latency:** 1 ms

**DOCNEST answer:**
> According to the "Outlook for FY2025" section, TechVision's management guides for FY2025 revenue of $1.55–1.62 billion. This represents a year-over-year growth of +25–31%.

**Ground truth:**
> Section 6: guides for $1.55-1.62 billion, +25-31% YoY

**Score:** 10/10 — The candidate answer perfectly matches all numerical data (revenue range and growth percentage) from the ground truth.

</details>

<details><summary>Q6: What percentage of TechVision's electricity came from renewable sources in 2024?</summary>

**Retrieved section:** `§7` | **Query latency:** 2 ms

**DOCNEST answer:**
> According to the ESG & Sustainability section, 94% of TechVision's global electricity came from renewable sources in 2024. This figure is also confirmed in the ESG Performance Dashboard, which notes the company's renewable energy adoption curve reached 94% in 2024.

**Ground truth:**
> ESG section: 94% of global electricity from renewable sources

**Score:** 10/10 — The candidate answer is perfectly accurate, stating the correct percentage and source provided in the ground truth.

</details>

<details><summary>Q7: What is the estimated cost of the Zero-Trust Security Suite initiative?</summary>

**Retrieved section:** `§5.1` | **Query latency:** 3 ms

**DOCNEST answer:**
> According to the "Product Roadmap 2025" table, the estimated cost of the Zero-Trust Security Suite initiative is $32 million. This initiative is targeted for Q3 2025 and is in the Design Phase.

**Ground truth:**
> Product Roadmap table: Zero-Trust Security Suite $32M, Q3 2025

**Score:** 10/10 — The candidate's answer is perfectly accurate, providing the correct cost and timeline from the ground truth while also including additional, non-contradictory context.

</details>

---

### 🌐 NexusAPI v3 Developer Reference (`HTML`)

| | |
|---|---|
| Sections | 11 |
| Tables extracted | 6 |
| Parse time | 286 ms |
| Avg score | **9.2/10** |
| Pass rate | **100%** |

| # | Question | Score | Reasoning |
|---|----------|-------|-----------|
| 1 | What are the rate limits and price for the Pro plan? | ✅ 10/10 | The candidate answer correctly and completely identifies all rate limits (per minute, per  |
| 2 | How long do OAuth tokens last and how long are refresh tokens valid? | ✅ 10/10 | The candidate answer correctly identifies both the token expiration time (3600 seconds) an |
| 3 | What HTTP method and endpoint is used to trigger AI parsing of a documen… | ✅ 8/10 | The candidate correctly identifies the HTTP method and endpoint but omits the rate limit i |
| 4 | What happens if a webhook endpoint does not respond within 10 seconds? | ✅ 10/10 | The candidate answer correctly identifies all key facts from the ground truth: the 10-seco |
| 5 | Which error code should be returned when the daily quota is exhausted, a… | ✅ 8/10 | The candidate correctly identified the 429 error code and the exponential backoff handling |

<details><summary>Q1: What are the rate limits and price for the Pro plan?</summary>

**Retrieved section:** `§1.1` | **Query latency:** 1 ms

**DOCNEST answer:**
> Based on the provided documents, the Pro plan has a rate limit of 1,000 requests/minute and 500,000 requests/day. The plan also has a Burst Limit of 2,000. The price for the Pro plan is $199/month.

**Ground truth:**
> Rate Limits table: Pro plan — 1000/min, 500k/day, burst 2000, $199/month

**Score:** 10/10 — The candidate answer correctly and completely identifies all rate limits (per minute, per day, and burst) and the monthly price for the Pro plan as stated in the ground truth.

</details>

<details><summary>Q2: How long do OAuth tokens last and how long are refresh tokens valid?</summary>

**Retrieved section:** `§1.1` | **Query latency:** 1 ms

**DOCNEST answer:**
> According to the provided document, NexusAPI uses OAuth 2.0 Bearer tokens which expire after 3600 seconds (1 hour). Refresh tokens are valid for 30 days.

**Ground truth:**
> Authentication section: tokens expire after 3600 seconds, refresh valid 30 days

**Score:** 10/10 — The candidate answer correctly identifies both the token expiration time (3600 seconds) and the refresh token validity period (30 days), matching the ground truth exactly.

</details>

<details><summary>Q3: What HTTP method and endpoint is used to trigger AI parsing of a document?</summary>

**Retrieved section:** `§1.4.1` | **Query latency:** 1 ms

**DOCNEST answer:**
> Based on the "Document Endpoints" table in §1.2.1, the HTTP method used to trigger AI parsing of a document is `POST`. The endpoint for this action is `/documents/{id}/parse`.

**Ground truth:**
> Documents API table: POST /documents/{id}/parse — Trigger AI parsing, 5/min

**Score:** 8/10 — The candidate correctly identifies the HTTP method and endpoint but omits the rate limit information provided in the ground truth.

</details>

<details><summary>Q4: What happens if a webhook endpoint does not respond within 10 seconds?</summary>

**Retrieved section:** `§1.4` | **Query latency:** 1 ms

**DOCNEST answer:**
> According to the documentation, if a webhook endpoint does not respond within 10 seconds, the delivery is retried. The delivery will be retried up to 5 times. This retry process uses exponential backoff, starting at 30 seconds.

**Ground truth:**
> Webhooks section: respond within 10 seconds or retried up to 5 times, backoff 30s

**Score:** 10/10 — The candidate answer correctly identifies all key facts from the ground truth: the 10-second timeout, the 5 retries, and the 30-second backoff period.

</details>

<details><summary>Q5: Which error code should be returned when the daily quota is exhausted, and how should clients handle a 429 error?</summary>

**Retrieved section:** `§1.4.1` | **Query latency:** 2 ms

**DOCNEST answer:**
> The provided document excerpts do not specify which error code is returned when the daily quota is exhausted. According to the "HTTP Error Code Reference" table, a `429 | Too Many Requests` error indicates that the "Rate limit exceeded." The specified resolution for this error is to "Implement exponential backoff."

**Ground truth:**
> Error codes table: 429 — implement exponential backoff; Webhook events: quota.exceeded

**Score:** 8/10 — The candidate correctly identified the 429 error code and the exponential backoff handling, but it omitted the related 'quota.exceeded' webhook event.

</details>

---

### 📋 CloudMesh Technical Architecture Spec (`MD`)

| | |
|---|---|
| Sections | 13 |
| Tables extracted | 0 |
| Parse time | 5 ms |
| Avg score | **10.0/10** |
| Pass rate | **100%** |

| # | Question | Score | Reasoning |
|---|----------|-------|-----------|
| 1 | What is CloudMesh's monthly data processing volume and uptime SLA? | ✅ 10/10 | The candidate answer correctly and completely states both the monthly data processing volu |
| 2 | What encryption standard is used for data at rest and how often are keys… | ✅ 10/10 | The candidate answer perfectly and correctly extracts both the encryption standard for dat |
| 3 | What is the RTO and RPO for Tier 1 services and what DR strategy is used… | ✅ 10/10 | The candidate answer perfectly matches the ground truth, correctly identifying the RTO, RP |
| 4 | Which compliance certification covers payment flows and who is the audit… | ✅ 10/10 | The candidate answer is a perfect match, correctly identifying both the certification for  |
| 5 | What was the total annual infrastructure cost in FY2024 and which catego… | ✅ 10/10 | The candidate answer is a perfect match to the ground truth, correctly stating the total c |

<details><summary>Q1: What is CloudMesh's monthly data processing volume and uptime SLA?</summary>

**Retrieved section:** `§1.4.1` | **Query latency:** 2 ms

**DOCNEST answer:**
> CloudMesh processes **2.4 petabytes** of data per month. The platform's core SLA guarantees **99.95% uptime**.

**Ground truth:**
> Overview: processes 2.4 petabytes per month, 99.95% uptime SLA

**Score:** 10/10 — The candidate answer correctly and completely states both the monthly data processing volume and the uptime SLA.

</details>

<details><summary>Q2: What encryption standard is used for data at rest and how often are keys rotated?</summary>

**Retrieved section:** `§1.4.1` | **Query latency:** 1 ms

**DOCNEST answer:**
> According to the document, the encryption standard used for data at rest is **AES-256-GCM**. Keys are rotated automatically every **90 days**.

**Ground truth:**
> Section 3.1: AES-256-GCM at rest, TLS 1.3 in transit, 90-day key rotation

**Score:** 10/10 — The candidate answer perfectly and correctly extracts both the encryption standard for data at rest and the key rotation frequency.

</details>

<details><summary>Q3: What is the RTO and RPO for Tier 1 services and what DR strategy is used?</summary>

**Retrieved section:** `§1.5` | **Query latency:** 2 ms

**DOCNEST answer:**
> According to the Disaster Recovery table, the RTO for Tier 1 services is 15 min and the RPO is 1 min. The DR strategy used for this tier is Active-Passive.

**Ground truth:**
> Disaster Recovery table: Tier 1 — RTO 15 min, RPO 1 min, Active-Passive

**Score:** 10/10 — The candidate answer perfectly matches the ground truth, correctly identifying the RTO, RPO, and DR strategy.

</details>

<details><summary>Q4: Which compliance certification covers payment flows and who is the auditor?</summary>

**Retrieved section:** `§1.4.2` | **Query latency:** 1 ms

**DOCNEST answer:**
> The compliance certification that covers payment flows is PCI DSS Level 1. The auditor for this certification is Coalfire.

**Ground truth:**
> Compliance table: PCI DSS Level 1 — Payment flows, Coalfire auditor

**Score:** 10/10 — The candidate answer is a perfect match, correctly identifying both the certification for payment flows and the specific auditor.

</details>

<details><summary>Q5: What was the total annual infrastructure cost in FY2024 and which category was largest?</summary>

**Retrieved section:** `§1.6` | **Query latency:** 1 ms

**DOCNEST answer:**
> Based on the document excerpts, the total annual infrastructure cost in FY2024 was **$91.2 million**. The largest cost category was Compute, which accounted for **$38.4 million**, or **42.1%** of the total cost.

**Ground truth:**
> Cost Structure table: Total $91.2M, Compute $38.4M is largest at 42.1%

**Score:** 10/10 — The candidate answer is a perfect match to the ground truth, correctly stating the total cost, the largest category, its cost, and its percentage.

</details>

---

### 📄 IPCC AR6 — Summary for Policymakers (`PDF`)

| | |
|---|---|
| Sections | 122 |
| Tables extracted | 0 |
| Parse time | 729 ms |
| Avg score | **9.2/10** |
| Pass rate | **100%** |

| # | Question | Score | Reasoning |
|---|----------|-------|-----------|
| 1 | What is the observed increase in global surface temperature compared to … | ✅ 10/10 | The candidate answer is factually correct, containing the primary statistic from the refer |
| 2 | Which sectors have the highest mitigation potential by 2030? | ✅ 10/10 | The candidate's answer is entirely factually correct, identifying the specific mitigation  |
| 3 | What does the report say about limiting warming to 1.5°C — is it still a… | ✅ 8/10 | The answer is highly accurate and detailed, but it omits the specific quantitative targets |
| 4 | What are the projected sea level rise ranges mentioned in the report? | ✅ 8/10 | The candidate's answer is factually correct for the data it presents, but it omits the imp |
| 5 | How does the report link climate change to extreme weather events? | ✅ 10/10 | The candidate answer is perfectly accurate, correctly summarizing the report's conclusions |

<details><summary>Q1: What is the observed increase in global surface temperature compared to 1850–1900?</summary>

**Retrieved section:** `§0.0.0.0.0.1` | **Query latency:** 0 ms

**DOCNEST answer:**
> In 2011-2020, the global surface temperature was 1.1°C above the 1850-1900 average (§7.0.0.0.0.2). More specifically, the temperature was 1.09 [0.95 to 1.20]°C higher during this period, with larger increases over land (1.59 [1.34 to 1.83]°C) than over the ocean (0.88 [0.68 to 1.01]°C) (§7.0.0.0.0.2). For the period 2001–2020, the global surface temperature was 0.99 [0.84 to 1.10]°C higher than 1850–1900 (§7.0.0.0.0.2). The best estimate for the total human-caused global surface temperature increase from 1850–1900 to 2010–2019 is 1.07°C, with a likely range of 0.8°C to 1.3°C (§7.0.0.0.0.2).

**Gemini baseline:**
> According to the IPCC AR6 Summary for Policymakers, the global surface temperature in the decade 2011–2020 was 1.1°C (with a likely range of 0.95°C to 1.20°C) above the 1850–1900 average. The report states that it is unequivocal that human influence has warmed the atmosphere, ocean, and land. This level of warming is unprecedented in more than 2,000 years, and the rate of warming has accelerated.

**Score:** 10/10 — The candidate answer is factually correct, containing the primary statistic from the reference (1.1°C for 2011-2020) and supplementing it with additional accurate and relevant details from the same IPCC report.

</details>

<details><summary>Q2: Which sectors have the highest mitigation potential by 2030?</summary>

**Retrieved section:** `§0.0.0.0.0.1` | **Query latency:** 0 ms

**DOCNEST answer:**
> Based on the chart in section [§54], the mitigation options with the highest potential contribution to net emission reduction by 2030 are "Solar" and "Wind," both with potentials over 4 GtCO₂-eq/yr. Other options with high potential include "Reduce conversion of natural ecosystems" (approximately 4 GtCO₂-eq/yr), and "Carbon sequestration in agriculture" and "Ecosystem restoration, afforestation, reforestation," both with potentials around 3.5 GtCO₂-eq/yr.

**Gemini baseline:**
> According to the IPCC AR6 Summary for Policymakers, the energy supply sector has the highest mitigation potential by 2030, driven primarily by the rapid and large-scale deployment of solar and wind power. The AFOLU (Agriculture, Forestry, and Other Land Use) sector also offers substantial potential, largely through low-cost measures like protecting and restoring forests and other ecosystems. The buildings and industry sectors follow closely with significant opportunities for emissions reductions through energy efficiency, fuel switching, and improved material management.

**Score:** 10/10 — The candidate's answer is entirely factually correct, identifying the specific mitigation options that give the energy and AFOLU sectors the highest potential, in perfect alignment with the reference.

</details>

<details><summary>Q3: What does the report say about limiting warming to 1.5°C — is it still achievable?</summary>

**Retrieved section:** `§0.0.0.0.0.1` | **Query latency:** 0 ms

**DOCNEST answer:**
> According to the report, global warming is "more likely than not to reach 1.5°C" in the near term (2021–2040) even under the very low GHG emission scenario (§23). Limiting warming to 1.5°C with no or limited overshoot requires "rapid and deep and, in most cases, immediate greenhouse gas emissions reductions in all sectors this decade" (§40.0.0.0.0.1). However, projected CO2 emissions from existing fossil fuel infrastructure without additional abatement would "exceed the remaining carbon budget for 1.5°C (50%)" (§38.0.0.0.0.2). If warming exceeds 1.5°C, it could be gradually reduced by "achieving and sustaining net negative global CO2 emissions" (§46.0.0.0.0.1).

**Gemini baseline:**
> Based on the IPCC AR6 Summary for Policymakers, limiting warming to 1.5°C is still achievable, but the window of opportunity is rapidly closing. The report states that this requires deep, rapid, and sustained global greenhouse gas emissions reductions across all sectors immediately. To stay on this path, global emissions must peak before 2025 at the latest and be reduced by 43% by 2030. Even with these actions, a temporary "overshoot" of the 1.5°C threshold is likely before temperatures return to below that level.

**Score:** 8/10 — The answer is highly accurate and detailed, but it omits the specific quantitative targets (peaking by 2025, 43% reduction by 2030) for emissions reductions.

</details>

<details><summary>Q4: What are the projected sea level rise ranges mentioned in the report?</summary>

**Retrieved section:** `§0.0.0.0.0.1` | **Query latency:** 0 ms

**DOCNEST answer:**
> Based on the document excerpts, relative to 1995–2014, the projected global mean sea level rise under the SSP1-1.9 GHG emissions scenario is 0.15–0.23 m by 2050 and 0.28–0.55 m by 2100 (§B.3.1). For the SSP5-8.5 scenario, the projected rise is 0.20–0.29 m by 2050 and 0.63–1.01 m by 2100 (§B.3.1). Over the next 2000 years, global mean sea level is projected to rise by about 2–3 m if warming is limited to 1.5°C and 2–6 m if limited to 2°C (§B.3.1).

**Gemini baseline:**
> According to the IPCC AR6 Summary for Policymakers, by 2100, the likely global mean sea level rise is projected to be 0.28–0.55 meters under the very low emissions scenario (SSP1-1.9) and 0.63–1.01 meters under the very high emissions scenario (SSP5-8.5), relative to the 1995-2014 baseline. The report also states that due to deep uncertainty in ice sheet processes, a rise approaching 2 meters by 2100 cannot be ruled out under the very high emissions scenario. Over the longer term, by 2300, sea level is projected to rise by about 0.5–3 meters under the low emissions scenario and by 2–7 meters under the high emissions scenario.

**Score:** 8/10 — The candidate's answer is factually correct for the data it presents, but it omits the important caveat about a potential 2-meter rise by 2100 and the specific 2300 projections mentioned in the reference.

</details>

<details><summary>Q5: How does the report link climate change to extreme weather events?</summary>

**Retrieved section:** `§0.0.0.0.0.1` | **Query latency:** 0 ms

**DOCNEST answer:**
> According to the report, "Human-caused climate change is already affecting many weather and climate extremes in every region across the globe" (§9.0.0.0.0.1). The document states with *high confidence* that this has led to "widespread adverse impacts and related losses and damages to nature and people" (§9.0.0.0.0.1). Specifically, evidence for observed changes in "heatwaves, heavy precipitation, droughts, and tropical cyclones," and their "attribution to human influence, has further strengthened since AR5" (§9.0.0.0.0.1). Furthermore, human influence has "likely increased the chance of compound extreme events since the 1950s, including increases in the frequency of concurrent heatwaves and droughts" (§9.0.0.0.0.1).

**Gemini baseline:**
> Based on its findings, the IPCC AR6 Summary for Policymakers states that it is an "established fact" that human-induced climate change is making many extreme weather and climate events more frequent and more intense. The report provides strengthened evidence linking observed changes in extremes such as heatwaves, heavy precipitation, droughts, and tropical cyclones directly to human influence. It concludes that some recent hot extremes observed across the globe would have been "extremely unlikely to occur without human influence on the climate system."

**Score:** 10/10 — The candidate answer is perfectly accurate, correctly summarizing the report's conclusions on the link between human-caused climate change and the increased frequency and intensity of various extreme weather events, consistent with the reference.

</details>

---

### 📄 BIS Annual Economic Report 2024 (`PDF`)

| | |
|---|---|
| Sections | 244 |
| Tables extracted | 0 |
| Parse time | 1087 ms |
| Avg score | **8.0/10** |
| Pass rate | **80%** |

| # | Question | Score | Reasoning |
|---|----------|-------|-----------|
| 1 | What was the global inflation trend described in the BIS 2024 report? | ✅ 10/10 | The candidate answer is perfectly accurate, correctly identifying the overall decline in i |
| 2 | What does the report say about central bank interest rate decisions in 2… | ✅ 10/10 | The candidate answer is factually perfect as it correctly states that central banks held r |
| 3 | How does the BIS describe financial stability risks in 2024? | ✅ 8/10 | The candidate answer accurately captures the core macro-financial risks from the reference |
| 4 | What role does the BIS say AI plays in financial markets? | ✅ 8/10 | The candidate answer is highly accurate, detailing both the uses of AI and the risks of he |
| 5 | What are BIS recommendations on sustainable finance or the green transit… | ❌ 4/10 | The candidate answer provides only general policy recommendations and completely omits the |

<details><summary>Q1: What was the global inflation trend described in the BIS 2024 report?</summary>

**Retrieved section:** `§0.1` | **Query latency:** 0 ms

**DOCNEST answer:**
> According to the BIS Annual Economic Report 2024, global inflation continued to decline from the peak it reached in 2022 (§3.0.0.0.0.6). Following "the most synchronised and intense monetary policy tightening in a generation," headline inflation dropped below 3% in Advanced Economies (AEs) and below 4% in Emerging Market Economies (EMEs) by early 2024 (§3, §3.0.0.0.0.6). However, the report cautions that inflation was still "hovering above central bank targets across much of the world" and showed signs of becoming "more hesitant in some key jurisdictions, notably the United States" (§2.0.0.0.0.16). The moderation in price growth was more pronounced in goods than in services, which proved to be "a more persistent component historically" (§3.0.0.0.0.6).

**Gemini baseline:**
> Based on the BIS Annual Economic Report 2024, the global inflation trend shows a significant decline from its multi-decade peaks, marking substantial progress in the disinflationary process. However, the report cautions that this "last mile" of returning inflation to central bank targets is proving to be unexpectedly difficult and protracted. This persistence is primarily driven by sticky services inflation, which has remained stubbornly high even as goods inflation has receded.

**Score:** 10/10 — The candidate answer is perfectly accurate, correctly identifying the overall decline in inflation, the persistent challenge of reaching targets, and the key role of sticky services inflation as described in the reference.

</details>

<details><summary>Q2: What does the report say about central bank interest rate decisions in 2023–2024?</summary>

**Retrieved section:** `§0.1` | **Query latency:** 0 ms

**DOCNEST answer:**
> Following the "most synchronised and intense monetary policy tightening in a generation," central banks raised policy rates to "two-decade highs" and kept them there even as inflation began to decline (§3.0.0.0.19, §2.0.0.0.23). While sustained disinflation "opened the door to a monetary policy pivot," central banks remained cautious (§3.0.0.0.0.6). For much of the period, financial markets "anticipated more monetary easing ahead than central banks did," but by the second quarter of 2024, markets "converged to central bank assessments amid renewed inflation concerns" (§3.0.0.0.0.15). The report advises that it would be "imprudent to cut interest rates significantly" and that rates should stay high as long as needed to re-establish price stability (§2.0.0.0.0.19).

**Gemini baseline:**
> Based on my training data regarding the BIS Annual Economic Report 2024, the report states that central banks held interest rates at high, restrictive levels through 2023 to combat persistent inflation. It highlights that this "last mile" of disinflation is proving to be the most difficult, with progress slowing as economies show differing resilience and wage pressures. For 2024, the report indicates that central banks face a delicate balancing act, with the timing and pace of any potential rate cuts being highly uncertain and dependent on incoming data, leading to divergent policy paths across countries.

**Score:** 10/10 — The candidate answer is factually perfect as it correctly states that central banks held rates high due to inflation and remained cautious about cuts, fully aligning with and even adding accurate detail to the reference.

</details>

<details><summary>Q3: How does the BIS describe financial stability risks in 2024?</summary>

**Retrieved section:** `§0.1` | **Query latency:** 0 ms

**DOCNEST answer:**
> According to the BIS, while the financial system has proven resilient, "long-standing fiscal and macro-financial vulnerabilities may lead to stress" (§3). Key "macro-financial pressure points" include the potential for imbalances to "unwind and cause headwinds due to historically high levels of debt and debt service costs," "deteriorating balance sheets in the non-financial sector," and the risk of a "large CRE correction" (§3.0.0.0.0.24, §3.0.0.0.0.26). Additionally, the rise of AI introduces risks, as the reliance on a few algorithms could amplify "procyclicality and market volatility," and create "new sources of cyber risk" like "Prompt injection attacks" and "Data poisoning attacks" (§29.0.0.0.0.1, §25).

**Gemini baseline:**
> Based on its analysis leading into 2024, the Bank for International Settlements (BIS) highlights that financial stability risks are concentrated in sectors vulnerable to the prolonged period of high interest rates. The report points specifically to significant stress in commercial real estate (CRE) markets, where falling valuations and a looming refinancing wall at higher costs pose a major threat. Additionally, the BIS warns of the potential for deteriorating credit quality for corporations and households with high debt burdens, alongside ongoing concerns about liquidity and leverage risks in the non-bank financial sector.

**Score:** 8/10 — The candidate answer accurately captures the core macro-financial risks from the reference regarding CRE and debt, but it also adds significant, un-referenced details about AI while omitting the mention of the non-bank financial sector.

</details>

<details><summary>Q4: What role does the BIS say AI plays in financial markets?</summary>

**Retrieved section:** `§0.1` | **Query latency:** 0 ms

**DOCNEST answer:**
> According to the provided documents, AI models play a role in financial markets by being used in asset management to "predict returns, evaluate risk-return trade-offs and optimise portfolio allocation" (§24.0.0.0.0.1). They are also useful in "algorithmic trading" and can "elicit unobservable features of financial data (so-called asset embeddings)" (§24.0.0.0.0.1). The BIS notes that the reliance of market participants on the "same handful of algorithms could lead to financial stability risks" by "exacerbating herding, liquidity hoarding, runs and fire sales" (§29.0.0.0.0.3). Widespread use of "algorithmic pricing" could also increase the "pass-through of aggregate shocks to local prices, and hence inflation" (§49).

**Gemini baseline:**
> Based on the BIS Annual Economic Report 2024, AI plays a dual role in financial markets, offering significant gains in efficiency while simultaneously creating new systemic risks. The report highlights concerns about concentration risk, as the financial system may become overly reliant on a few large tech firms that provide foundational AI models. The BIS also warns that the widespread use of similar AI models could amplify market volatility and procyclicality through algorithmic herding, potentially destabilizing the financial system.

**Score:** 8/10 — The candidate answer is highly accurate, detailing both the uses of AI and the risks of herding, but it omits the specific concern of concentration risk related to reliance on a few large tech firms mentioned in the reference.

</details>

<details><summary>Q5: What are BIS recommendations on sustainable finance or the green transition?</summary>

**Retrieved section:** `§0.1` | **Query latency:** 0 ms

**DOCNEST answer:**
> Based on the document excerpts, fiscal policy recommendations include prioritizing "growth-enhancing spending, for example on the green transition, human capital and structural reforms" (§3.0.0.0.0.43). Structural policies are also recommended, as "stepping up structural reforms has gained particular urgency given the rapid global shifts on multiple fronts, ranging from technology to the green transition and to global trade" (§3.0.0.0.0.44). Additionally, the document notes there is "room for international dialogue to forge consensus on blueprints and best practices, not least in response to emerging priorities such as climate change" (§3.0.0.0.0.44).

**Gemini baseline:**
> Based on its Annual Economic Report 2024, the Bank for International Settlements (BIS) recommends that credible public policies, especially carbon pricing, must lead the green transition, as the financial system cannot drive it alone. The report emphasizes using a mix of financing tools, such as blended finance and public guarantees, to de-risk and scale up private investment in green technologies. The BIS cautions against overburdening financial regulation with climate objectives, stating its primary role should remain ensuring financial stability, not steering credit allocation.

**Score:** 4/10 — The candidate answer provides only general policy recommendations and completely omits the reference's specific, core points regarding carbon pricing, blended finance, and the primary role of financial regulation.

</details>

---

### 📄 GPT-3 Paper — Language Models are Few-Shot Learners (`PDF`)

| | |
|---|---|
| Sections | 40 |
| Tables extracted | 0 |
| Parse time | 426 ms |
| Avg score | **6.0/10** |
| Pass rate | **40%** |

| # | Question | Score | Reasoning |
|---|----------|-------|-----------|
| 1 | How many parameters does GPT-3 have? | ✅ 10/10 | The candidate answer is perfectly accurate, correctly stating the widely cited 175 billion |
| 2 | What training dataset was used for GPT-3 and how large is it? | ❌ 4/10 | The candidate provides a significantly incorrect token count and omits the other four key  |
| 3 | How does GPT-3 perform on SuperGLUE compared to fine-tuned models? | ❌ 2/10 | The candidate misses the central point of the question by failing to provide any performan |
| 4 | Describe the three evaluation settings: zero-shot, one-shot, few-shot. | ❌ 4/10 | The candidate only defines one of the three requested settings, making the answer largely  |
| 5 | What limitations and risks does the GPT-3 paper acknowledge? | ✅ 10/10 | The candidate answer is perfectly accurate and includes all points from the reference, oft |

<details><summary>Q1: How many parameters does GPT-3 have?</summary>

**Retrieved section:** `§0.1` | **Query latency:** 0 ms

**DOCNEST answer:**
> According to the document abstract, GPT-3 is "an autoregressive language model with 175 billion parameters" (§0.1.2). This figure is repeated in the conclusion, which describes a "175 billion parameter language model" (§1.0.17). A table in the appendix specifies that the "GPT-3 175B" model has "174,600" M parameters (§1.0.27).

**Gemini baseline:**
> According to the paper "Language Models are Few-Shot Learners," the largest version of GPT-3 has 175 billion parameters. This is the model most commonly referred to when discussing GPT-3's capabilities. The paper actually describes a suite of eight models of varying sizes, starting from 125 million parameters and scaling up to the full 175 billion.

**Score:** 10/10 — The candidate answer is perfectly accurate, correctly stating the widely cited 175 billion parameter count and adding the more precise figure from the source paper's appendix.

</details>

<details><summary>Q2: What training dataset was used for GPT-3 and how large is it?</summary>

**Retrieved section:** `§0.1` | **Query latency:** 0 ms

**DOCNEST answer:**
> The training dataset was a filtered version of the Common Crawl dataset. A classifier was trained to improve the dataset's quality using "WebText, Wikiedia, and our web books corpus as the positive examples" for filtering (§1.0.21). The GPT-3 models were trained with "300" billion tokens (§1.0.27).

**Gemini baseline:**
> Based on the paper "Language Models are Few-Shot Learners," GPT-3 was trained on a mixture of five main corpora with a total size of approximately 570 GB of text data. The largest component was a filtered version of the Common Crawl dataset, supplemented by high-quality sources including WebText2, two internet-based books corpora (Books1 and Books2), and the English-language Wikipedia. After processing, this massive dataset consisted of nearly 500 billion tokens used for training the model.

**Score:** 4/10 — The candidate provides a significantly incorrect token count and omits the other four key datasets that were part of the final training mixture.

</details>

<details><summary>Q3: How does GPT-3 perform on SuperGLUE compared to fine-tuned models?</summary>

**Retrieved section:** `§0.1` | **Query latency:** 0 ms

**DOCNEST answer:**
> The provided document excerpts state that GPT-3 is evaluated on the SuperGLUE benchmark suite in Section 3.7 (§1.0.7). The paper clarifies that "we do not fine-tune GPT-3 because our focus is on task-agnostic performance," instead testing it in the few-shot setting (§1.0.5). However, the specific results for SuperGLUE and a numerical comparison to fine-tuned models are not included in the provided excerpts.

**Gemini baseline:**
> According to the paper, GPT-3 in its few-shot setting achieved an overall score of 71.8 on the SuperGLUE benchmark. While this was a significant achievement for a model without any task-specific fine-tuning, its performance was considerably lower than the state-of-the-art fine-tuned models of the time. For comparison, a fine-tuned T5-11B model scored 89.3, demonstrating that specialized training still yielded substantially better results on this difficult benchmark.

**Score:** 2/10 — The candidate misses the central point of the question by failing to provide any performance data and incorrectly claiming the numerical comparison is unavailable.

</details>

<details><summary>Q4: Describe the three evaluation settings: zero-shot, one-shot, few-shot.</summary>

**Retrieved section:** `§0.1` | **Query latency:** 0 ms

**DOCNEST answer:**
> According to the document excerpts, "Few-Shot (FS)" is the setting where "the model is given a few demonstrations of the task at inference time as conditioning [...], but no weight updates are allowed" (§1.0.5). The provided excerpts do not define the zero-shot or one-shot settings.

**Gemini baseline:**
> In the paper, these settings evaluate a model's ability to perform tasks via "in-context learning" without any gradient updates. In the **zero-shot** setting, the model is given a natural language description of a task and a prompt to complete, but no examples. In the **one-shot** setting, the model is given the description plus a single example of the task. The **few-shot** setting provides the model with several examples (typically 10 to 100) in the prompt's context before it must complete the final one.

**Score:** 4/10 — The candidate only defines one of the three requested settings, making the answer largely incomplete despite the single definition being correct.

</details>

<details><summary>Q5: What limitations and risks does the GPT-3 paper acknowledge?</summary>

**Retrieved section:** `§0.1` | **Query latency:** 0 ms

**DOCNEST answer:**
> The paper acknowledges that GPT-3 has "notable weaknesses in text synthesis," where samples can "repeat themselves semantically," "lose coherence," and "contradict themselves" (§1.0.11). It also identifies that the model has "special difficulty with 'common sense physics'" and struggles with some comparison tasks like determining if "one sentence implies another" (§1.0.11). The paper acknowledges "potentially harmful applications" and focuses on two primary risks: "the potential for deliberate misuse" and "issues of bias, fairness, and representation" (§1.0.13). Specific misuse applications include "misinformation, spam, phishing, abuse of legal and governmental processes, fraudulent academic essay writing and social engineering" (§1.0.13).

**Gemini baseline:**
> The GPT-3 paper acknowledges significant limitations and risks, highlighting the model's potential for malicious use like generating misinformation, spam, and phishing content. The authors also specify that GPT-3 reflects and can amplify biases present in its training data, leading to stereotypical and prejudicial outputs related to gender, race, and religion. Furthermore, they note technical weaknesses, including a lack of common sense reasoning and a tendency to lose coherence over very long passages.

**Score:** 10/10 — The candidate answer is perfectly accurate and includes all points from the reference, often with greater specificity and detail.

</details>

---

## Conclusion

DOCNEST achieved **8.9/10 average accuracy** across 38 questions spanning 7 documents in 5 different formats.

**89% of questions scored ≥ 7/10**, confirming that retrieved context is accurate and sufficient for downstream LLM answers.

Formats with ground-truth Q&A (DOCX, XLSX, HTML, Markdown) test exact retrieval accuracy. PDF documents are evaluated against Gemini's training knowledge.

_Evaluated with Gemini 1.5 Pro as judge. Documents include complex tables, multi-sheet workbooks with formulas, nested headings, image captions, and multi-level document structures._