Geoprocessing for AI agents, with provenance you can check.

basins.tif.provenance.json

What came with the picture

Not a log. A record complete enough that someone who was not there can re-run the analysis and disagree with it — which is a different and much higher bar than being able to read what happened.

Operations

{{TOOL_COUNT}} tools, not a catalogue

Tool-selection accuracy degrades past a few dozen exposed tools, so capability lives behind a searchable catalogue rather than in the tool list. What is exposed is what an agent has to choose between.

Finding the right operation among hundreds

{{TOOL_COUNT}} tools are exposed; the catalogue behind them holds {{CATALOG_COUNT}} operations and is built for thousands. That makes reaching scale a retrieval problem, so it is treated as one — and measured like one, against 800 real GIS operations with queries phrased the way somebody with a problem phrases them.

Ranking is the wrong verb.
When the two labellers who wrote the ground truth disagree with each other three times in ten, there is no single right answer to rank toward — so the search hands over the set and says the order is a hint.

Those are measured on 118 requests written by two other model families from job scenarios — a hydrologist with a flood report, a surveyor arguing with a field measurement — and neither generator was shown this catalogue, because a model handed the entry writes a paraphrase of the entry. The requests, both labels and the harness that turns them into these percentages are all in the repository — benchmarks/discovery_report.py recomputes every figure here with no network and no model, and a test fails if this page and the harness disagree. All three of the numbers above are over the same 118 requests; agreement measured over all 155 in the file is 68%, and the difference is the pairs where both labellers agreed a request was unanswerable.

It narrows first, deterministically, on facts the caller already has. What data is in hand, and what should come back. Declaring those two takes the candidates from 74 to 31 — and below thirty survivors the search stops choosing for you: it returns status: "choose" with every candidate, ordered as a hint that says so, each carrying the sentence that separates it from its neighbours. On 115 of those 118 requests the right operation is in what comes back; the other three leave a set still too large to hand over whole and fall back to a ranked shortlist. Not as an accuracy figure — as a property: the narrowing never drops the answer, and the 3% is the gap between «cannot lose it» and «can show you all of it».

The family orders instead of filtering, and that is a correction. It used to be a hard filter like the others, and it is not like the others: input kind is a fact about your data, family is a guess about our taxonomy that you cannot see. It removed six candidates out of seventeen — and when the guess was wrong it removed the right operation, silently, leaving a confident answer built from neighbours. Every request here has 4.4 plausible families. That is the failure Argleton measures in other people's systems, found in our own discovery layer.

The scaling wall arrived, and it took an afternoon. This page used to say the guarantee held at fifty-one operations and not at eight hundred. On 29 August the catalogue went to sixty-one, the commonest surviving set went from 26 candidates to 34 — past the point where the whole set can be handed over — and delivered fell from 100% to 45% while found@3 fell from 48% to 36%. Adding capability had made discovery worse, and nothing raised: a catalogue with too many candidates is a perfectly valid catalogue.

Raising the threshold would have bought about ten operations. What fixed it is the fourth facet in the table above: how many datasets you are holding. One layer or two. It is a fact about your situation rather than a guess about our vocabulary, it is derivable from each operation's own signature so a test checks the declaration against the code, and it takes the median surviving set from 34 to 9. The catalogue grew by a fifth and discovery got better — but only because a facet arrived with it, and that is the trade rather than a free lunch. It is now a required field of the published entry specification, with the measurement attached.

Then it happened again, and that is what watching a curve is for. On 30 August the catalogue went to seventy-one. Every ranking figure fell — 25% bare where it had been 28%, 27% on the input kind where it had been 34% — and delivery at the second facet fell from 48% to 27%, because more requests now leave a set too large to hand over whole. The bottom row did not move: 97%, the same as at 61. Ten more operations absorbed by the facets already there, for the first time. It held again at seventy-four, and those two were operations a caller is unusually likely to want — the remedies MapSmith's own error messages had been recommending — so they landed in the busiest corner of the facet space.

So the figure to watch is not found@3. A ranker gets worse as a catalogue grows and it is a hint either way. It is the average surviving set at the fullest declaration — 9 at fifty-one operations, 14 at sixty-one, 16 at seventy-two, 17 at seventy-four. (This page said median until 29 August and published the mean; the median at seventy-four is 14. The distribution is skewed, so the two say different things, and the mean is the pessimistic one.) When that crosses thirty, delivery stops being a property and becomes a ranking again, and the answer will be another fact the caller already knows rather than a bigger threshold.

And no model is needed to extract those facets, because the caller is one. An MCP client is an LLM holding the context we lack: it knows what file it has open and what it is trying to produce. So the tool asks for them in its schema and its description leads with why. Same shape as the auto-retrieval patterns in the RAG toolkits, minus the model they have to host — ours is already on the other end of the protocol.

Then it ranks, with two engines that both always run, and every result says which one answered: BM25 over the same corpus — forty lines, no model, no network — and static embeddings pinned to an exact model revision, with the vectors themselves pinned by a test, so a change in the model or the tokenizer fails a test instead of an analysis.

And when the two agree on nothing, it says so instead of answering. Asked “send an email to my accountant”, the embedding engine returned idw_interpolation with the confidence of a real answer — a silent error in the layer whose job is to prevent them, found by pointing our own method at our own product. No confidence threshold fixes it: “convert this mp4 to a gif” scores above sixteen of twenty genuine queries. What separates them is the two rankers landing on nothing in common, 0.90 of 3 in domain against 0.18 out. Below that the answer is status: "unsure", carrying both guesses and the question that narrows the catalogue. It fires on 9 of 11 unanswerable queries and costs one suppressed correct answer in 20. Below the choose threshold it stops refusing and warns instead: refusing made sense while the search was deciding, and handing over every candidate is not deciding.

A discovery layer that always answers is one that sometimes lies.
This one can ask instead — the same rule the operations follow when a number would be wrong.

One caveat we will not bury: that 68% ceiling was measured between two language models. Whether human GIS analysts agree with each other more, less or about the same is unknown, and two analysts with thirty years each routinely do the same job with different tools. Until people have tried it, these are reported as agreement with model-written labels — never as accuracy.

How an entry has to be written is a published specification, not a house convention: catalog-entry-spec.md, with a normative JSON Schema every entry validates against in CI. Each field is there because a measurement said so — including the two that measured to nothing and are documented as such, and the MUST NOT that stops an implementation filtering on the family. It is the second specification this project publishes, after the manifest format, and both are written to be implemented by somebody else.

How we know, including the parts that caught us

Three test files hold this up, and each exists because a measurement contradicted something we had already written down.

ablation	Each facet is worth measuring alone, over the 118 independent requests: nothing declared → 74 candidates, 16% found@3 with embeddings and 25% with BM25; what data I have → 49, 20% and 27%; + what I want back → 31, 32% and 40%; + how many datasets I have → 17, 53% ranked and 97% delivered. Both engines, because this row carried one unnamed column until a CI run met a rate limit from the model host, fell back to BM25 and recomputed the first figure as 28% — the published number had been a measurement of what the machine could download. The jump in the last column is not the ranker getting better — it is the set becoming small enough to hand over whole, and the 3% that is not 100% is the handful of requests where it still is not.
the default	BM25 was the default until the golden queries were re-read: they had been written by whoever wrote the catalogue, so they shared its vocabulary and tested word overlap dressed as retrieval. On those BM25 scores 100% found@1 and embeddings 60%, and on queries phrased like a problem rather than like a manual the finding reversed — at fifty-one entries. It has since reversed back. Recomputed on 30 August at seventy-four, BM25 leads from thirty entries upward, and it leads at every row of the facet table above. The embedding engine is still the default, and the measurement that made it one no longer says so: that is a decision to take rather than a number to quietly restate.
and reverses again	Measured against 800 near neighbours rather than our own spread-out entries, the embedding engine degrades faster: 35% against 20% found@3. Embeddings blur near neighbours; an exact term either matches or does not. This used to be the measurement that disagreed with the one above it, and both were kept because they answered different questions. They agree now: the near-neighbour effect this predicted has arrived in our own catalogue at seventy-four entries. At 800 the better engine alone is wrong two times in three, which is exactly why ranking is the tie-break and not the mechanism.
per operation	A catalogue-wide average hides the entry nobody can reach: 90% found@3 over fifty means five are invisible and the average will not say which. So every entry is probed with the worked example it advertises. It used to have to rank in the top three; that gate is gone, and removing it is the point — the only way to repair such a failure is to reword the entry until the ranker likes it, which is how a 70% on this page became 51% overnight. What stays under contract is what is ours and deterministic: the facets an entry declares must never drop it, and it must reach the caller.
and it bit us	centroid_layer advertised “label points for a polygon layer” and ranked below point_on_surface. The ranking was right: a centroid can fall outside its own polygon — Argleton trap 014 — so our own catalogue was recommending the defect our own suite measures. We changed the example, not the score.

Determinism is the reason for building it this way rather than reaching for a hosted embedding API: that would make tool discovery a network call whose answer can change under you, and an agent that finds a different tool tomorrow for the same question is not reproducible, whatever its manifest says. The one network access left is the model download on first use, at the pinned revision; after that it is local, and a machine that never gets it keeps BM25 and is told so.

How it gets better, and what it refuses to do

Every figure above rests on 155 requests written by two language models. That is the best set obtainable without users, and it is not what users ask. So MapSmith can record its own: MAPSMITH_DISCOVERY_LOG writes one line per search — the query, the facets declared, every candidate delivered, and the operation actually run afterwards, with its rank in the list it was picked from. A harness in the repository turns those lines into rows shaped like the benchmark and flags the two that matter: a choice the ranking did not put first, and a search nothing followed. A dashboard writes the same material, and the rest of what the project knows about itself, as one self-contained page — no CDN, works offline — where the cases are answered by clicking and the percentages recompute against those answers as you go. It carries a panel the log cannot produce: every request where the two model labellers disagreed with each other, which is what the 70% ceiling is made of and the part that needs somebody who has done the job. Beside it: whether every operation can actually be found, the degradation curve as the catalogue grows, and what each engine does with each family of trap.

It does not train anything, and that is the design. A ranker that learns from what callers pick learns from an ordering it produced: what it showed first gets picked more, gets learned as correct, gets shown first harder — a confident answer nothing contradicts, which is the failure this whole project exists to measure. The model revision stays pinned, held by a golden-vector test, so the same question gets the same answer next year. What changes is the catalogue text, as a diff somebody can read and revert. That loop is not the timid option: it is what took found@3 from 18% to 57% and delivery to 97%.

It found something on its first recorded session, which is the argument for it in one line. A request for the area of each parcel, declared as wanting an answer rather than a layer, left zero candidates — and zero candidates came back as “0 operations survive, which is few enough to read”, an empty list an agent reads as MapSmith cannot do this. It can: the operation writes the number into a column, so it declares that it produces a dataset. That case now answers with which declaration excluded everything, computed rather than ranked.

Off unless the variable is set. It holds queries and operation names, never dataset paths or arguments, is confined by the workspace like every other path MapSmith writes, and never leaves the machine.

buffer_layer · clip_layer · overlay_layers · dissolve_layer	Vector geometry, with the metric-on-geographic trap handled explicitly rather than silently
measure_area	Ground or planar area, with the plane checked against the ellipsoid — the first check here that asks whether the number is right rather than whether the operation ran
merge_layers · simplify_layer · centroid_layer · convert_format	Layer plumbing with the cost recorded: null-filled columns named, simplification drift measured before and after, and the two lossy conversions — shapefile output, GeoJSON for a non-WGS84 layer — refused with the reason instead of performed
reproject_layer	Coordinate systems, with the decision and its reason recorded
zonal_statistics	Raster statistics per polygon, via exactextract — partial pixels weighted, not counted or dropped
hillshade · slope · aspect · flow_accumulation · watershed	Terrain and hydrology on Whitebox Workflows
run_sql	Spatial SQL on DuckDB, sandboxed to the workspace with external access off
validate_plan · execute_plan	A typed DAG checked before anything executes: stable error codes, simulated CRS flow
preview_map	An in-chat map panel that shows the verification status of every layer it draws

One question, end to end

Six parcels, a river and an elevation grid, and five operations picked out of {{CATALOG_COUNT}}. Everything below was recorded from a run that happened while this page was built — the search at each step, the arguments, the coordinate-system decision and the verification. Nothing here is drawn.

{{WORKED_EXAMPLE}}

The rejected plan is the honest half: steps in the wrong order are the dominant failure class in the agent benchmark, so the example includes one and shows what the validator says before any file is touched. It earned that place while this was being written — the first version passed distance_m where the operation declares distance_meters, and the validator named the argument and listed the three it accepts.

Evidence

The headline of our own benchmark is a null result

Invariants

What it will not do