PatentBench-Mini v0.1.0

1. Executive Summary

PatentBench-Mini v0.1.0 is the first reproducible, open benchmark designed to evaluate AI systems on real-world patent prosecution tasks. Drawing from 82 genuine USPTO Office Actions across 8 Technology Centers, the benchmark tests four core competencies that patent practitioners perform daily: classifying Office Action types, computing response timelines, calculating USPTO fees, and reasoning through substantive rejections.

The benchmark achieves an overall accuracy of 92.3% across 298 individual test cases, scoring 941 out of 1,020 total points. Three of four task categories achieved perfect 100% accuracy. The sole source of errors — Deadline Calculation at 81.1% — traces to JavaScript date arithmetic edge cases rather than substantive legal reasoning failures.

2. Methodology

2.1 Data Collection

The benchmark dataset was constructed from 321 real USPTO patent applications, of which 268 received at least one Office Action. From this pool, 82 Office Actions were selected to ensure coverage across all eight Technology Centers, diverse rejection types (35 U.S.C. §101, §102, §103, §112), and varying procedural postures (non-final, final, advisory).

2.2 Task Tiers

Tests are organized into three tiers of increasing complexity:

*Tier 3 reasoning tasks are evaluated qualitatively and are presented as samples rather than scored numerically in the overall accuracy figure.

2.3 Scoring

Each test awards points for correct sub-components. For example, an Action Classification test awards 4 points: 1 for action type, 1 for finality, 1 for rejection basis, and 1 for claim count identification. Points are summed and divided by the maximum to yield accuracy percentages. All tests are deterministic and reproducible.

2.4 Technology Center Coverage

Office Actions were sampled from all eight major Technology Centers to ensure the benchmark is not biased toward any single patent domain:

3. Results

3.1 Results by Task Type

3.2 Accuracy by Task Type

3.3 Results by Technology Center

All Technology Centers achieved 100% accuracy on non-deadline tasks (Action Classification, Timeline Analysis, and Fee Computation), confirming that the benchmark generalizes across patent domains.

3.4 Point Distribution

The following chart shows the proportion of total points contributed by each task type, illustrating that Deadline Calculation dominates the scoring weight — and is the sole source of errors.

4. Tier 3 Reasoning Samples

Tier 3 evaluates the system's ability to generate substantive legal arguments in response to real USPTO rejections. The benchmark includes 25 reasoning tasks spanning the major rejection types:

Below are two representative samples demonstrating the quality of generated arguments.

Sample 1: §103 Obviousness Traversal

Sample 2: §101 Alice/Mayo Eligibility Argument

5. Limitations & Next Steps

5.1 Current Limitations

• Deadline calculation date arithmetic: The 81.1% accuracy on deadline tasks reflects JavaScript date handling edge cases (month-end rollover, leap years, federal holiday table completeness), not legal reasoning errors. This is the primary area for improvement.
• Tier 3 evaluation is qualitative: Legal reasoning samples are currently evaluated by human review rather than automated scoring. Developing reliable automated metrics for argument quality remains an open research problem.
• Dataset size: While 82 Office Actions across 8 Technology Centers provides meaningful coverage, the benchmark would benefit from expansion. Some TCs (notably TC 2100 with 3 tests) have limited representation.
• Design patent and PCT coverage: The current benchmark focuses exclusively on utility patent prosecution. Design patents, provisional applications, and PCT international phase actions are not yet covered.
• Single-action scope: Tests evaluate responses to individual Office Actions. Multi-round prosecution strategy (e.g., RCE decisions, appeal timing) is not yet benchmarked.

5.2 Planned Improvements

5.3 Reproducibility

All test cases, expected outputs, and scoring logic are included in the PatentBench-Mini repository. Tests can be executed with a single command and produce deterministic results. The dataset of 321 USPTO applications is derived entirely from public USPTO records via the Patent Examination Data System (PEDS) and PAIR APIs.

6. Appendix

A. Expanded Dataset Statistics

B. Rejection Type Distribution (Tier 3)

C. Scoring Methodology Detail

Action Classification (4 points per test)

• 1 pt — Correct action type (Non-Final / Final / Advisory / Allowance / Restriction)
• 1 pt — Correct finality determination
• 1 pt — Correct rejection basis identification (§101, §102, §103, §112)
• 1 pt — Correct affected claim count

Timeline Analysis (3 points per test)

• 1 pt — Correct mailing date extraction
• 1 pt — Correct shortened statutory period identification
• 1 pt — Correct maximum extension period

Fee Computation (3 points per test)

• 1 pt — Correct base fee amount
• 1 pt — Correct entity size classification (micro / small / large)
• 1 pt — Correct extension fee calculation

Deadline Calculation (variable points per test)

• 1 pt — Correct base deadline date
• 1 pt — Correct weekend/holiday adjustment
• 1 pt — Correct final deadline with extensions (where applicable)

D. References

• 35 U.S.C. §§101, 102, 103, 112 — Patentability Conditions
• 37 C.F.R. §1.136 — Extensions of Time
• MPEP §710 — Time for Reply to Office Actions
• MPEP §2106 — Patent Subject Matter Eligibility (Alice/Mayo Framework)
• MPEP §2141 — Examination Guidelines for Obviousness
• Alice Corp. v. CLS Bank Int'l, 573 U.S. 208 (2014)
• KSR Int'l Co. v. Teleflex Inc., 550 U.S. 398 (2007)
• Enfish, LLC v. Microsoft Corp., 822 F.3d 1327 (Fed. Cir. 2016)
• DDR Holdings, LLC v. Hotels.com, 773 F.3d 1245 (Fed. Cir. 2014)
• USPTO Patent Examination Data System (PEDS) API
• USPTO Patent Application Information Retrieval (PAIR)