Geoprocessing for AI agents, with provenance you can check.

basins.tif.provenance.json

What came with the picture

Not a log. A record complete enough that someone who was not there can re-run the analysis and disagree with it — which is a different and much higher bar than being able to read what happened.

Operations

{{TOOL_COUNT}} tools, not a catalogue

Tool-selection accuracy degrades past a few dozen exposed tools, so capability lives behind a searchable catalogue rather than in the tool list. What is exposed is what an agent has to choose between.

Finding the right operation among hundreds

{{TOOL_COUNT}} tools are exposed; the catalogue behind them holds {{CATALOG_COUNT}} operations and is built for thousands. That makes reaching scale a retrieval problem, so it is treated as one — and measured like one, against 800 real GIS operations with queries phrased the way somebody with a problem phrases them.

Ranking is the wrong verb.
When the two labellers who wrote the ground truth disagree with each other three times in ten, there is no single right answer to rank toward — so the search hands over the set and says the order is a hint.

Those are measured on 118 requests written by two other model families from job scenarios — a hydrologist with a flood report, a surveyor arguing with a field measurement — and neither generator was shown this catalogue, because a model handed the entry writes a paraphrase of the entry. The requests, both labels and the harness that turns them into these percentages are all in the repository — benchmarks/discovery_report.py recomputes every figure here with no network and no model, and a test fails if this page and the harness disagree. All three of the numbers above are over the same 118 requests; agreement measured over all 155 in the file is 68%, and the difference is the pairs where both labellers agreed a request was unanswerable.

It narrows first, deterministically, on facts the caller already has. What data is in hand, and what should come back. Declaring those two takes the candidates from 51 to 21 — and below thirty survivors the search stops choosing for you: it returns status: "choose" with every candidate, ordered as a hint that says so, each carrying the sentence that separates it from its neighbours. On all 118 requests the right operation is in what comes back. Not as an accuracy figure — as a property, because nothing was dropped.

The family orders instead of filtering, and that is a correction. It used to be a hard filter like the others, and it is not like the others: input kind is a fact about your data, family is a guess about our taxonomy that you cannot see. It removed six candidates out of twenty-one — and when the guess was wrong it removed the right operation, silently, leaving a confident answer built from neighbours. Every request here has 4.4 plausible families. That is the failure Argleton measures in other people's systems, found in our own discovery layer.

What we have not solved is scale, and this page claimed we had. It said the facets leave sixteen candidates at 800 operations as they do at 200. The sixteen is real and the family produces almost all of it — those 800 are all raster in, raster out, so the honest facets cut nothing and only the taxonomy does, a guess among 43. What is needed is more facts a caller can state without knowing our vocabulary: how many inputs an operation takes, whether it changes geometry or only attributes, whether the output has as many features as the input. Structural, checkable against the code. Not built yet. The guarantee above holds at fifty-one operations and not at eight hundred, and we would rather say so than round it.

And no model is needed to extract those facets, because the caller is one. An MCP client is an LLM holding the context we lack: it knows what file it has open and what it is trying to produce. So the tool asks for them in its schema and its description leads with why. Same shape as the auto-retrieval patterns in the RAG toolkits, minus the model they have to host — ours is already on the other end of the protocol.

Then it ranks, with two engines that both always run, and every result says which one answered: BM25 over the same corpus — forty lines, no model, no network — and static embeddings pinned to an exact model revision, with the vectors themselves pinned by a test, so a change in the model or the tokenizer fails a test instead of an analysis.

And when the two agree on nothing, it says so instead of answering. Asked “send an email to my accountant”, the embedding engine returned idw_interpolation with the confidence of a real answer — a silent error in the layer whose job is to prevent them, found by pointing our own method at our own product. No confidence threshold fixes it: “convert this mp4 to a gif” scores above sixteen of twenty genuine queries. What separates them is the two rankers landing on nothing in common, 0.90 of 3 in domain against 0.18 out. Below that the answer is status: "unsure", carrying both guesses and the question that narrows the catalogue. It fires on 9 of 11 unanswerable queries and costs one suppressed correct answer in 20. Below the choose threshold it stops refusing and warns instead: refusing made sense while the search was deciding, and handing over every candidate is not deciding.

A discovery layer that always answers is one that sometimes lies.
This one can ask instead — the same rule the operations follow when a number would be wrong.

One caveat we will not bury: that 68% ceiling was measured between two language models. Whether human GIS analysts agree with each other more, less or about the same is unknown, and two analysts with thirty years each routinely do the same job with different tools. Until people have tried it, these are reported as agreement with model-written labels — never as accuracy.

How an entry has to be written is a published specification, not a house convention: catalog-entry-spec.md, with a normative JSON Schema every entry validates against in CI. Each field is there because a measurement said so — including the two that measured to nothing and are documented as such, and the MUST NOT that stops an implementation filtering on the family. It is the second specification this project publishes, after the manifest format, and both are written to be implemented by somebody else.

How we know, including the parts that caught us

Three test files hold this up, and each exists because a measurement contradicted something we had already written down.

ablation	Each facet is worth measuring alone, over the 118 independent requests: nothing declared → 51 candidates, 25% found@3; what data I have → 33, 29%; + what I want back → 21, 48% ranked and 100% delivered. The last row is what ships, and the jump in the last column is not the ranker getting better — it is the set becoming small enough to hand over whole.
the default	BM25 was the default until the golden queries were re-read: they had been written by whoever wrote the catalogue, so they shared its vocabulary and tested word overlap dressed as retrieval. On those BM25 scores 100% found@1 and embeddings 60% — and the finding reverses on queries phrased like a problem rather than like a manual.
and reverses again	Measured against 800 near neighbours rather than our own spread-out entries, the embedding engine degrades faster: 35% against 20% found@3. Embeddings blur near neighbours; an exact term either matches or does not. Both measurements are kept, because they answer different questions — and at 800 the better engine alone is wrong two times in three, which is exactly why ranking is the tie-break and not the mechanism.
per operation	A catalogue-wide average hides the entry nobody can reach: 90% found@3 over fifty means five are invisible and the average will not say which. So every entry is probed with the worked example it advertises. It used to have to rank in the top three; that gate is gone, and removing it is the point — the only way to repair such a failure is to reword the entry until the ranker likes it, which is how a 70% on this page became 51% overnight. What stays under contract is what is ours and deterministic: the facets an entry declares must never drop it, and it must reach the caller.
and it bit us	centroid_layer advertised “label points for a polygon layer” and ranked below point_on_surface. The ranking was right: a centroid can fall outside its own polygon — Argleton trap 014 — so our own catalogue was recommending the defect our own suite measures. We changed the example, not the score.

Determinism is the reason for building it this way rather than reaching for a hosted embedding API: that would make tool discovery a network call whose answer can change under you, and an agent that finds a different tool tomorrow for the same question is not reproducible, whatever its manifest says. The one network access left is the model download on first use, at the pinned revision; after that it is local, and a machine that never gets it keeps BM25 and is told so.

buffer_layer · clip_layer · overlay_layers · dissolve_layer	Vector geometry, with the metric-on-geographic trap handled explicitly rather than silently
measure_area	Ground or planar area, with the plane checked against the ellipsoid — the first check here that asks whether the number is right rather than whether the operation ran
merge_layers · simplify_layer · centroid_layer · convert_format	Layer plumbing with the cost recorded: null-filled columns named, simplification drift measured before and after, and the two lossy conversions — shapefile output, GeoJSON for a non-WGS84 layer — refused with the reason instead of performed
reproject_layer	Coordinate systems, with the decision and its reason recorded
zonal_statistics	Raster statistics per polygon, via exactextract — partial pixels weighted, not counted or dropped
hillshade · slope · aspect · flow_accumulation · watershed	Terrain and hydrology on Whitebox Workflows
run_sql	Spatial SQL on DuckDB, sandboxed to the workspace with external access off
validate_plan · execute_plan	A typed DAG checked before anything executes: stable error codes, simulated CRS flow
preview_map	An in-chat map panel that shows the verification status of every layer it draws

One question, end to end

Six parcels, a river and an elevation grid, and five operations picked out of {{CATALOG_COUNT}}. Everything below was recorded from a run that happened while this page was built — the search at each step, the arguments, the coordinate-system decision and the verification. Nothing here is drawn.

{{WORKED_EXAMPLE}}

The rejected plan is the honest half: steps in the wrong order are the dominant failure class in the agent benchmark, so the example includes one and shows what the validator says before any file is touched. It earned that place while this was being written — the first version passed distance_m where the operation declares distance_meters, and the validator named the argument and listed the three it accepts.

Evidence

The headline of our own benchmark is a null result

Invariants

What it will not do