Metadata-Version: 2.4
Name: baltimore-camden
Version: 0.8.1
Summary: Python toolkit for the Baltimore City Mayor's Office of Performance and Innovation (OPI).
Keywords: baltimore,civic-tech,microsoft-graph,sharepoint,open-data
Author: Baltimore City Office of Performance and Innovation
License-Expression: MIT
License-File: LICENSE
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Scientific/Engineering :: GIS
Classifier: Typing :: Typed
Requires-Dist: httpx>=0.28
Requires-Dist: msal>=1.37
Requires-Dist: pandas>=3.0
Requires-Dist: pyarrow>=25.0.0
Requires-Dist: pydantic>=2.13
Requires-Dist: pydantic-settings>=2.14
Requires-Dist: baltimore-camden[db,excel,geo,postgres] ; extra == 'all'
Requires-Dist: pyodbc>=5.3 ; extra == 'db'
Requires-Dist: sqlalchemy>=2.0 ; extra == 'db'
Requires-Dist: openpyxl>=3.1 ; extra == 'excel'
Requires-Dist: geopandas>=1.1 ; extra == 'geo'
Requires-Dist: psycopg[binary]>=3.3.4 ; extra == 'postgres'
Requires-Dist: sqlalchemy>=2.0 ; extra == 'postgres'
Requires-Python: >=3.13
Project-URL: Homepage, https://github.com/city-of-baltimore/camden
Project-URL: Issues, https://github.com/city-of-baltimore/camden/issues
Provides-Extra: all
Provides-Extra: db
Provides-Extra: excel
Provides-Extra: geo
Provides-Extra: postgres
Description-Content-Type: text/markdown

# Camden

**OPI's Python toolkit for civic data, automation, and reporting.**

[![CI](https://github.com/city-of-baltimore/camden/actions/workflows/ci.yml/badge.svg)](https://github.com/city-of-baltimore/camden/actions/workflows/ci.yml)
[![Python 3.13+](https://img.shields.io/badge/python-3.13%2B-blue.svg)](https://www.python.org/downloads/)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)
[![Ruff](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ruff/main/assets/badge/v2.json)](https://github.com/astral-sh/ruff)
[![uv](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/uv/main/assets/badge/v0.json)](https://github.com/astral-sh/uv)
[![Checked with mypy](https://img.shields.io/badge/mypy-checked-blue)](https://mypy-lang.org/)

Camden gives analysts and data staff in the Baltimore City Mayor's Office of Performance and Innovation (OPI) one consistent, safe, repeatable way to do the things we do every week: pull SharePoint lists into pandas, query city databases, fetch Open Baltimore layers, geocode addresses, send report emails, and post to Teams.

## Part of the Baltimore civic foundations

Camden is one of three independently versioned foundations OPI maintains for Baltimore civic software:

- **[camden](https://github.com/city-of-baltimore/camden)** (this repo) — Python **data** SDK (`baltimore-camden`): Graph/SharePoint, Open Baltimore, city databases, geocoding, provenance, delivery.
- **[patapsco](https://github.com/city-of-baltimore/patapsco)** — **platform / integration point** (`baltimore-patapsco` + `@city-of-baltimore/patapsco`): request IDs, safe error envelopes, observability, health/readiness, typed frontend client, and app scaffold. Authentication remains app-local today.
- **[bromo / Baltimore Design System](https://github.com/city-of-baltimore/bromo)** — TypeScript **presentation** layer (`@city-of-baltimore/bromo`): tokens, theme, shell, accessible components.

How the pieces fit together: **[patapsco/docs/foundations.md](https://github.com/city-of-baltimore/patapsco/blob/main/docs/foundations.md)** is the canonical architecture overview. The shared naming rule (`baltimore-<name>` on PyPI, `@city-of-baltimore/<name>` on npm) is [ADR-0005](docs/adr/0005-civic-packages-share-the-baltimore-namespace.md).

## Contents

- [Part of the Baltimore civic foundations](#part-of-the-baltimore-civic-foundations)
- [Philosophy](#philosophy)
- [What Camden supports](#what-camden-supports)
- [Installation](#installation)
- [Configuration](#configuration)
- [Quickstart](#quickstart)
- [Coming from r4OPI](#coming-from-r4opi)
- [Development](#development)
- [Roadmap](#roadmap)
- [Contributing](#contributing)
- [License](#license)

## Philosophy

**Camden exists to lower the ramp**: reusable helpers, copy-pasteable recipes, and shared standards, so each analyst doesn't hand-roll their own auth, connection strings, and API calls.

- **Python is the preferred default** for new shared, repeatable, production-facing, integrated, or automated workflows.
- **R remains welcome** for statistical analysis, exploratory work, and existing R/Shiny assets.

Design principles: plain-English function names, predictable return types (pandas in, pandas out), small composable functions, no magic, no hard-coded credentials, and everything configurable without editing code.

## What Camden supports

| Area | Module | What you get |
| --- | --- | --- |
| Microsoft Graph auth | `camden.graph` | `GraphAuth` (app-only or signed-in) and a hardened `GraphClient` — retries, paging, and `$batch` built in |
| SharePoint & OneDrive | `camden.sharepoint` | Lists ↔ DataFrames with `where=` filters (non-indexed columns fall back gracefully), display-name labels, choice validation, person lookups, `describe()` diagnostics; schema admin (`create_list`/`delete_list`, typed column builders, `add_column`/`delete_column`); document libraries by path (`ls`, `mkdir`, `versions`, recursive download, upload, delete); file transfer by pasted link; extension-aware `read_sharepoint()` / `write_sharepoint()` |
| Email | `camden.mail` | `send_mail()` through Graph (attachments, HTML, cc/bcc) — no ad hoc SMTP |
| Teams | `camden.teams` | Post to channels by team/channel *name*, list team members/rosters, or fire a webhook from unattended jobs |
| Planner | `camden.planner` | Plans/buckets/tasks as DataFrames; create and update tasks (buckets by name, assignees by email) with Planner's If-Match etag dance handled for you |
| Notifications | `camden.notify` | Provider-neutral `send()` with Graph-mail, SendGrid, and Teams-webhook transports; sent/skipped/failed statuses instead of exceptions; `"all"` or `"failover"` strategies |
| City databases | `camden.db` | Named registry (SQL Server + Postgres/PostGIS via `kind`), `read_sql()`/`read_table()` → DataFrames, `write_sql()` bulk writes, `read_postgis()` → GeoDataFrames, SQLAlchemy URLs, `sql_in()` |
| Open Baltimore | `camden.opendata` | `search_datasets()` finds datasets; `get_layer()`/`get_table()`/`download_layer()`/`iter_layer_rows()` fetch them (parameterized, paginated, retried, logged); provenance built in: `describe_layer()`, `assert_fields()` schema-drift guards, `where_since()` incremental clauses, and a `manifest_for()` sidecar recording exactly what query produced every cached file |
| Census/ACS | `camden.census` | `acs()` Baltimore City estimates at county/tract/block-group level, friendly variable names, numeric columns, join-ready `geoid` |
| Geocoding | `camden.geocoding` | City ESRI locator with Census fallback, address scrubbing; batch `geocode_frame()` dedupes, rate-limits, and logs progress |
| Spatial recipes | `camden.geo` | `spatial_join()` (which neighborhood is each point in — accepts plain lon/lat DataFrames), `buffer_feet()` (no accidental degree buffers), `points_from_frame()`, `reproject()` with named CRS constants |
| Excel reports | `camden.excel` | `write_excel()`: branded headers, frozen rows, autofilter, currency/percent/date formats — one call from DataFrame to presentable workbook |
| Report templates | `camden.reporting` | `html_report()` (branded email-ready HTML from DataFrames + text) and `excel_report()` (About sheet with title/timestamp/run id + one sheet per section) |
| Validation & diffs | `camden.validate` | `check_frame()` — required columns/dtypes, nulls, uniqueness, allowed values, all problems in one teaching error; `diff_frames()` — keyed add/update/delete plans that power list sync |
| Job runtime | `camden.runtime` | `configure_logging()` (UTC, run-id correlation, quiet deps), `log_duration()` timing on success and failure, `safe_summary()` config logging with secrets redacted |
| City utilities | `camden.dates`, `camden.formats`, `camden.branding` | Fiscal years, dollar formatting, CCF→gallons, OPI brand palette |

**What Camden is not:** a stats library, a dashboarding framework, or a place for one-off analysis code. If a helper is only useful to one project, it belongs in that project.

## Installation

Camden supports Python 3.13 and 3.14. The repository also pins the family-wide
Node.js 24 tooling baseline in `.nvmrc`; Camden itself ships no JavaScript
runtime or npm package.

Releases are published to PyPI as `baltimore-camden` from a matching `vX.Y.Z`
GitHub release through OIDC trusted publishing. Consumer applications pin a
released version rather than a moving Git branch.

The PyPI project and trusted-publisher binding are active; releases publish
through the workflow without a stored registry token.

```bash
# with uv (preferred)
uv add "baltimore-camden[all]==0.8.1"

# with pip
pip install "baltimore-camden[all]==0.8.1"
```

Extras keep the core light — install only what you need:

| Extra | Adds | Needed for |
| --- | --- | --- |
| *(none)* | httpx, msal, pandas, pyarrow, pydantic | Graph, SharePoint, mail, Teams, Open Baltimore tables/Parquet, geocoding, utilities |
| `[db]` | pyodbc, SQLAlchemy | `camden.db` SQL Server profiles |
| `[postgres]` | psycopg, SQLAlchemy | `camden.db` Postgres/PostGIS profiles |
| `[excel]` | openpyxl | `camden.excel` styled workbooks, `.xlsx` writes in `write_sharepoint` |
| `[geo]` | geopandas | `camden.opendata` layers, spatial file reading/writing |
| `[all]` | everything above | — |

## Configuration

Camden reads `CAMDEN_*` environment variables (or a local `.env` file — see [.env.example](.env.example)). Every value has an OPI default, and the legacy variables `opi_graph_id` / `opi_graph_secret` still work, so most existing machines need zero setup.

| Variable | Default | Purpose |
| --- | --- | --- |
| `CAMDEN_CLIENT_ID` | legacy `opi_graph_id` | Entra app registration ID |
| `CAMDEN_CLIENT_SECRET` | legacy `opi_graph_secret` | Client secret (app-only auth) |
| `CAMDEN_TENANT` | `bmore.onmicrosoft.com` | Entra tenant |
| `CAMDEN_DEFAULT_SITE_URL` | OPI SharePoint site | Site used when none is given |
| `CAMDEN_MAIL_SENDER` | — | From-mailbox for app-only email |
| `CAMDEN_SENDGRID_API_KEY` | — | Enables the SendGrid notification transport |
| `CAMDEN_CENSUS_API_KEY` | — | Census API key for `camden.census` (free signup) |
| `CAMDEN_SQL_DRIVER` | auto-detected | ODBC driver name |
| `CAMDEN_DATABASES_FILE` | — | TOML file extending the database registry |
| `CAMDEN_CACHE_DIR` | `~/.cache/camden` | Where cached open-data pulls live |
| `CAMDEN_TIMEOUT` | `30` | HTTP timeout (seconds) for outbound requests |
| `CAMDEN_GRAPH_MAX_RETRIES` | `4` | Retry budget for transient Graph failures |
| `CAMDEN_GRAPH_BASE_URL` | Graph v1.0 | Microsoft Graph endpoint (rarely changed) |
| `CAMDEN_GRAPH_SCOPES` | `.default` | OAuth scopes requested for Graph tokens |

Credentials never live in code, and `Settings` never prints secrets. Any function also accepts an explicit `camden.Settings` object when you need per-call overrides; inspect the resolved configuration with `camden.get_settings()`.

## Quickstart

```python
from baltimore import camden

# SharePoint lists <-> pandas
site = camden.get_site()                        # the OPI site by default
updates = site.get_list("Book of Business - Weekly Updates")
on_track = updates.to_df(where="fields/Status eq 'On track'", labels="display")
updates.choices("Status")                       # valid options before you write
updates.append(new_rows_df)                     # person columns: use site.user_lookup_id(email)

# Sync a tracker safely: dry-run the plan, then apply (deletes are opt-in;
# columns you don't manage — like user-written notes — are never touched)
print(updates.diff(weekly_df, key="Title").summary())   # add 3, update 2, delete 0, unchanged 41
updates.sync(weekly_df, key="Title")

# Paste a link, get data — and write it back the same way
df = camden.read_sharepoint("https://bmore.sharepoint.com/:x:/r/sites/opi/...xlsx")
camden.write_sharepoint(df, "weekly.xlsx", folder_link)   # DataFrame -> xlsx -> SharePoint

# Document libraries, by path
docs = site.drive()                             # the default "Documents" library
docs.ls("General", recursive=True)              # name/path/kind/size/modified
docs.download_dir("General/Reports", "reports/")
docs.upload("weekly.xlsx", "General/Reports")

# City databases -> DataFrames (parameterize with ?, never f-strings)
permits = camden.read_sql(
    "SELECT * FROM dbo.Permits WHERE issued_date >= ?", "citistat", params=["2026-01-01"]
)
neighborhoods = camden.read_table("dbo.Neighborhoods", "citistat")
camden.write_sql(
    camden.check_frame(scores_df, columns={"score": "number"}, no_nulls=["agency"]),
    "dbo.Scores", "opi-sandbox", if_exists="append",
)
conn = camden.connect("citistat")               # raw connection when you need one

# Open Baltimore: search the catalog, then pull any dataset by name, page URL, or id
camden.search_datasets("parking fines")         # title, dataset_id, snippet, modified
hoods = camden.get_shape("neighborhoods")       # GeoDataFrame, EPSG:4326
vacants = camden.get_layer(                     # paste a dataset page straight from the catalog
    "https://data.baltimorecity.gov/datasets/8ed6565f93a04455ae6bd5bcb7d272db_19/explore"
)
camden.describe_layer("real_property")          # provenance: rows, last edit, owner
camden.assert_fields("vbns", {"NoticeDate": "Date"})   # fail fast on schema drift
camden.layer_fields("real_property")            # inspect the schema before filtering
camden.layer_count("real_property")             # size up a big pull first (~238k)
fines = camden.get_table(                       # JSON rows only — no geometry, much faster
    "d2a2330d6a374ad39a24a0d7f7b58f19_0",       # parking fines: ~11M rows
    where="ViolFine > 500",
    fields=["Citation", "Description", "ViolFine", "ViolDate"],
)
camden.download_layer(                          # ginormous pulls: stream to Parquet, flat memory
    "d2a2330d6a374ad39a24a0d7f7b58f19_0", "fines.parquet", where="ViolDate >= DATE '2026-01-01'"
)  # then pd.read_parquet(...) — or pl.scan_parquet(...) if you prefer polars
fines = camden.cached_table(                    # or cache it: downloads at most once per max_age
    "d2a2330d6a374ad39a24a0d7f7b58f19_0",
    where=camden.where_since("ViolDate", "2026-01-01"),  # incremental pulls
    max_age="6h",
)  # cache_layer(...) returns the Parquet path instead; every download writes a
   # manifest sidecar — camden.manifest_for(path) says exactly what query produced
   # the file — and iter_layer_rows() streams row-by-row

# Geocoding + spatial tagging
located = camden.geocode_frame(df["address"], fallback=True, rate_limit_per_second=5)
tagged = camden.spatial_join(located, hoods, keep=["Name"])   # neighborhood per address

# Styled Excel in one call (branded header, filters, number formats)
camden.write_excel(summary_df, "weekly.xlsx", currency=["TotalFines"], percent=["ShareResolved"])

# Email a report; post to Teams
camden.send_mail(to="team@baltimorecity.gov", subject="Weekly VBN report",
                 body="Attached.", attachments=["vbns.xlsx"])
camden.send_channel_message("OPI", "Book of Business", "Reporting period is <b>open</b>.")

# City utilities
camden.fiscal_year_label("2026-07-01")          # "FY27"
camden.week_bounds("2026-07-08")                # Fri-Thu reporting week: (Jul 3, Jul 9)
camden.dollars(121324520.45)                    # "$121,324,520.45"
```

> **Teams note:** Graph only allows channel messages from *signed-in users* (delegated auth). For unattended/scheduled jobs, use `camden.teams.send_webhook()` with a channel's Workflows URL — no credentials required.

## Coming from r4OPI

| r4OPI | Camden |
| --- | --- |
| `sharepointLogin()` | `GraphAuth()` (automatic inside every client) |
| `sharepointConnect()` | `camden.get_site()` |
| `sharedWithMe(url)` | `camden.sharepoint.download_file(url)` — direct API, no browser |
| `sqlConnection("citistat")` | `camden.connect("citistat")` |
| `ordTbl()` | `camden.db.ordered_select(conn, table)` / `ordered_columns()` |
| `getGeoShapeFromOB("neighborhoods")` | `camden.get_shape("neighborhoods")` |
| `getOBdatasets("vbns")` | `camden.get_dataset("vbns")` |
| `cityGeocoder(addr)` / `censusGeocoder(addr)` | `camden.geocode(addr)` / `camden.geocode(addr, source="census")`; batches: `camden.geocode_frame(addresses)` |
| `scrubAddress(addr)` | `camden.scrub_address(addr)` |
| `whatFY(d)` / `whatFY(d, abbrv=TRUE)` | `camden.fiscal_year(d)` / `camden.fiscal_year_label(d)` |
| `dollarWithCents(x)` | `camden.dollars(x)` |
| `ccfToGallons(x)` | `camden.ccf_to_gallons(x)` |
| `vectorToSql(v)` | `camden.sql_in(v)` |
| `opiColors()` | `camden.opi_colors()` / `camden.opi_palette()` |

New in Camden (no r4OPI equivalent): `send_mail`, `send_channel_message`, `send_webhook`, `upload_file`, `read_sharepoint`, list `update`/`delete`, SharePoint URL parsing, `shared_with_me()` listing.

## Development

One command contract for humans, CI, and agents — [Taskfile.yml](Taskfile.yml) ([install go-task](https://taskfile.dev/installation/)):

```bash
git clone https://github.com/city-of-baltimore/camden && cd camden
task setup      # dev environment from the committed uv.lock
task check      # everything a PR must pass: lint, types, tests + coverage gate
task fmt        # auto-format and fix lint findings
task ci         # exactly what CI runs
task docs-serve # live-preview the documentation site (mkdocs-material + mkdocstrings)
uvx pre-commit install   # lint + types on every commit
```

Unit tests run offline with no credentials. Live-tenant checks are opt-in: put credentials in `.env` and run `task integration` (each check skips individually when its target isn't configured — see [.env.example](.env.example)).

See [AGENTS.md](AGENTS.md) for architecture, conventions, and testing patterns, and [CONTRIBUTING.md](CONTRIBUTING.md) for the pull-request workflow.

## Roadmap

Direction lives in [ROADMAP.md](ROADMAP.md); execution is tracked with GitHub [milestones](https://github.com/city-of-baltimore/camden/milestones) and issues. Highlights: Excel read/write helpers, geospatial recipes (joins, buffers, CRS), Postgres/PostGIS support, and standard report outputs.

## Contributing

Contributions are welcome — new helpers, recipes, docs, and bug reports. Branch, open a PR to `main`, and CI must pass. If you have an idea but aren't sure where it fits, open an issue and ask.

## License

Distributed under the MIT License. See [LICENSE](LICENSE) for more information.

## Maintainers

Mayor's Office of Performance and Innovation (OPI).
