Metadata-Version: 2.5
Name: portalkit
Version: 0.1.0
Summary: One Python API for Socrata, CKAN, ArcGIS and OpenDataSoft open data portals, into pandas or geopandas.
Project-URL: Homepage, https://github.com/aviseth/portalkit
Project-URL: Repository, https://github.com/aviseth/portalkit
Project-URL: Changelog, https://github.com/aviseth/portalkit/blob/main/CHANGELOG.md
Project-URL: Issues, https://github.com/aviseth/portalkit/issues
Author-email: Avi Seth <avi@crispa.ai>
License-Expression: MIT
License-File: LICENSE
Keywords: arcgis,civic-tech,ckan,geopandas,gis,open-data,opendatasoft,pandas,socrata,urban-planning
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Scientific/Engineering :: GIS
Classifier: Topic :: Sociology
Classifier: Typing :: Typed
Requires-Python: >=3.10
Requires-Dist: httpx>=0.27
Requires-Dist: pandas>=2.0
Requires-Dist: pyarrow>=14
Requires-Dist: tomli>=2.0; python_version < '3.11'
Provides-Extra: geo
Requires-Dist: geopandas>=1.0; extra == 'geo'
Requires-Dist: shapely>=2.0; extra == 'geo'
Description-Content-Type: text/markdown

# portalkit

One Python API for the four pieces of software that nearly every government open data portal
runs on: Socrata, CKAN, ArcGIS and OpenDataSoft. Point it at a URL, get a DataFrame back.

```shell
pip install portalkit          # add [geo] if you want geometry
```

```python
import portalkit as pk

df = pk.load("chicago/311", limit=5000)  # a known dataset, by name

portal = pk.open("https://data.sfgov.org")  # or any portal URL
portal.search("evictions")
gdf = portal.dataset("6z8x-wfk4").to_geopandas(where="file_date > '2026-01-01'")
```

`pk.open` works out which software the portal runs by asking it. You do not have to know, and
you do not have to import a different library per city.

## Why this exists

The Python client most people reach for is `sodapy`, which has been unmaintained since 2022 and
only speaks Socrata. `ckanapi` is a faithful wrapper around CKAN's Action API and has no opinion
about DataFrames. ArcGIS either means Esri's SDK or writing the pagination yourself. So you end
up with three half-written clients per project, and each of them has the same three bugs.

**Your download is silently truncated.** Point `geopandas.read_file()` at an ArcGIS
FeatureServer layer and you get 1000 rows with no warning, because that is the server's
`maxRecordCount`. Socrata's default is 1000 too. People publish analyses on top of that. Paging
past it is not just a loop: without an explicit sort, `resultOffset` and `$offset` are free to
return the same row twice and never show you another one, so both need an explicit stable order
(`:id` on Socrata, the object id field on ArcGIS). OpenDataSoft caps `offset` at 10000 and
errors past it, so anything bigger has to go through its export endpoint instead. portalkit
does all of that and returns the number of rows the portal itself says the dataset has.

**Your ZIP codes turn into floats.** Socrata returns every value as a string, dates included.
Let pandas infer types afterwards and `02134` becomes `2134.0`. The portal already published a
schema saying which column is a number and which is an identifier, so portalkit uses it. When
the schema turns out to be wrong, which happens, the column comes back untouched rather than as
a column of nulls, so you can see the mess and decide what to do about it.

**Your geometry arrives as JSON.** Socrata sends GeoJSON, ArcGIS sends `rings` and `paths`,
OpenDataSoft wraps a GeoJSON Feature in one column and its centroid in another. All of it comes
out as shapely, with the CRS set. `geopandas` stays an optional install for people who only want
the table.

## The rest of it

**Caching.** Every tabular fetch is stored as Parquet, keyed by portal, dataset and the exact
query, so re-running a notebook does not re-run the download. `PORTALKIT_CACHE` sets the
directory, `cache=False` skips it, `portalkit clear-cache` empties it. `to_geopandas()` does not
cache: shapely geometry does not survive a Parquet round trip, so it would be a cache that
silently never hits.

**Filtering happens on the server.** `where`, `select` and `order` are passed through in the
portal's own query language, which is SoQL on Socrata, SQL-ish on ArcGIS, ODSQL on
OpenDataSoft, and CKAN's SQL endpoint when a `where` is given. This is usually the difference
between two seconds and five minutes.

**Manners.** One connection at a time, a real user agent, retries with jitter on the failures
worth retrying, and `Retry-After` respected. Most of these portals are one box run by a team of
two. Pass `token=` for a Socrata app token if you are going to be hammering one.

**Adaptive page sizes on ArcGIS.** Layers advertise a `maxRecordCount` that describes what they
will return for cheap rows. Ask for the same number of full-resolution county polygons and the
server returns a 500 wrapped inside a 200. There is no way to know the real ceiling except to
find it, so portalkit halves the page size and retries until the server copes, then keeps the
smaller size. It also asks for six decimal places of coordinate precision, which is about ten
centimetres and roughly halves the size of a polygon response.

## Command line

```text
$ portalkit catalog chicago
reference                 dataset                  portal
------------------------  -----------------------  ------------------------------
chicago/311               311 Service Requests     https://data.cityofchicago.org
chicago/building-permits  Building permits         https://data.cityofchicago.org
chicago/crimes            Crimes, 2001 to present  https://data.cityofchicago.org

$ portalkit info chicago/311
311 Service Requests
id       v6vf-nfxy
portal   https://data.cityofchicago.org (socrata)
rows     14,517,544

column              type           label
------------------  -------------  ----------------
sr_number           text           SR_NUMBER
created_date        calendar_date  CREATED_DATE
...

$ portalkit get chicago/311 --where "sr_type = 'Pothole in Street Complaint'" -o potholes.parquet
wrote 422,462 rows to potholes.parquet
```

`-o` takes `.parquet`, `.csv`, `.json`, `.geojson` or `.gpkg`. The last two fetch geometry.
Without `-o` you get the first twenty rows on stdout.

## The catalog

`pk.load("nyc/311")` works because there is a small list of well-known datasets shipped with the
package. It is not meant to be exhaustive, it is meant to save you the five minute detour of
finding out that Chicago's 311 dataset is `v6vf-nfxy`. `portalkit catalog` lists what is there.

Adding your city is a pull request against `src/portalkit/catalog.toml` with a portal URL, a
dataset id and a title. For entries you do not want to publish, point `PORTALKIT_CATALOG` at
your own TOML file and it gets merged in.

## Notes

CKAN is a catalogue first. Only resources that have been pushed into its DataStore can be
queried at all; everything else is a file on disk. `portalkit` resolves a package to its first
DataStore-backed resource, or tells you plainly that there is not one and where the files are.
Pass `package:resource-id` to pick a specific one.

ArcGIS addressing follows the REST URLs. Give it a service and name a layer
(`portal.dataset("0")`), or give it the layer URL directly.

Big geometry downloads from municipal servers are genuinely slow: all 3235 US counties at full
resolution is a few hundred megabytes and takes a while. The CLI shows a running row count, and
`to_pandas(progress=...)` takes a callback if you want one in a script. Filter server-side
first if you can.

Rate limits are the portal's, not portalkit's. If you are pulling millions of rows from a city
of 40,000 people, consider whether their server would rather you did that once and cached it.

Python 3.10+. MIT.
