Metadata-Version: 2.4
Name: geosint
Version: 1.0.0
Summary: Geospatial OSINT collection and monitoring for points of interest and organisations - OpenStreetMap by default, optionally Google Places.
Author: Jenil Mistry
License-Expression: MIT
Project-URL: Homepage, https://github.com/jenilmistryhq/geosint
Project-URL: Repository, https://github.com/jenilmistryhq/geosint
Project-URL: Issues, https://github.com/jenilmistryhq/geosint/issues
Keywords: osint,geospatial,openstreetmap,overpass,geojson,reconnaissance,monitoring,investigation
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Information Technology
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: GIS
Classifier: Topic :: Security
Classifier: Topic :: Internet :: WWW/HTTP
Classifier: Environment :: Console
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: requests>=2.31
Provides-Extra: xlsx
Requires-Dist: openpyxl>=3.1; extra == "xlsx"
Provides-Extra: dotenv
Requires-Dist: python-dotenv>=1.0; extra == "dotenv"
Provides-Extra: bulk
Requires-Dist: osmium>=3.7; extra == "bulk"
Provides-Extra: dev
Requires-Dist: pytest>=7.4; extra == "dev"
Requires-Dist: osmium>=3.7; extra == "dev"
Dynamic: license-file

# geosint

Geospatial OSINT collection and monitoring for points of interest and
organisations. Collects from public map data, records **where every fact came
from**, reports **what changed between collections**, and correlates entities
by shared infrastructure.

[![Python](https://img.shields.io/pypi/pyversions/geosint.svg)](https://pypi.org/project/geosint/)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)

```
$ geosint collect --what "bureau de change" --area "Austin, TX" --case meridian

3 entities collected (0 new to this case)

ENTITY             CATEGORY          ADDRESS    PHONE            DOMAIN             CONF  TRIAGE
-----------------  ----------------  ---------  ---------------  -----------------  ----  ----------
Alpha Exchange     bureau_de_change  1 Main St  +1 555 077 2200  newholdco.example  83    Unreviewed
Beta Money Ser...  bureau_de_change  1 Main St  +1 555 010 0100  holdco.example     83    Unreviewed
Cedar Currency     bureau_de_change  Cedar Ave  +1 555 044 7788                     67    Unreviewed

  3/3 have usable coordinates.

3 field change(s) since the last collection:

WHEN              ENTITY          FIELD    WAS                     NOW                    SOURCE
----------------  --------------  -------  ----------------------  ---------------------  ------
2026-09-09 19:31  Alpha Exchange  phone    +1 555 010 0100         +1 555 077 2200        osm
2026-09-09 19:31  Alpha Exchange  website  https://holdco.example  https://newholdco.e... osm
2026-09-09 19:31  Alpha Exchange  domain   holdco.example          newholdco.example      osm

1 previously-seen entity did not reappear:
    - Northside Exchange
```

## Scope

This collects **organisations, businesses, and infrastructure** from public map
data — entities that publish an address and a phone number precisely so they
can be found. It queries each source's own official public API, at documented
rate limits, identifying itself with a contact address.

**It is not a people-search tool.** There is no person lookup, no
cross-referencing of individuals, and none should be added. Everything it
returns describes a *place* or an *organisation*.

## Start here

```bash
pip install geosint
export GEOSINT_CONTACT="you@example.com"   # required, see below
geosint start
```

`start` asks five plain questions, remembers the answers, and runs your first
collection. Everything below is the same thing with flags.

---

## Why this rather than a scraper

Three things, and they are the whole design.

**1. Provenance on every field.** Each record carries its source, source
reference, collection time, case, and the query that found it. `geosint show`
prints all of it. An answer you cannot attribute is not intelligence, it is
trivia.

**2. Change is the product.** Re-run a collection and the tool diffs it against
what is stored, writing an append-only observation for every field that moved.
A phone number changing, a website going dark, an entity vanishing from the
map — these are findings, and a tool that just overwrites its database destroys
them.

```bash
geosint collect --what pharmacy --area "Austin, TX" --case alpha   # week 1
geosint collect --what pharmacy --area "Austin, TX" --case alpha   # week 2
geosint changes --since 7d --case alpha
```

**3. Correlation between entities.** Everything else describes entities one at
a time. `geosint links` looks for what connects them:

| Link | Meaning |
|---|---|
| shared phone | Two "different" organisations answering one number |
| shared domain | Two entities publishing the same web host |
| operator / brand | Affiliation the source states outright — the strongest signal available |
| co-location | Entities within metres of each other |

```
$ geosint links --case meridian

Stated operator / brand affiliations

  Affiliation: Meridian Holdings   (2 entities)
      - Alpha Exchange
      - Beta Money Services

Co-located entities (within 25m)

  Cluster: 2 entities within 25m   (2 entities)
      - Alpha Exchange
      - Beta Money Services
```

A shared value is **a lead, not a conclusion**. Shared numbers and domains have
mundane explanations — a shared reception, a franchise call centre, a serviced
office — at least as often as interesting ones. The tool reports the evidence
and leaves the judgement to you.

---

## Setup

```bash
pip install geosint                     # core
pip install "geosint[xlsx,dotenv]"      # + Excel export, .env support
pip install "geosint[bulk]"             # + offline .osm.pbf extraction
```

### The contact address is required

OpenStreetMap's Nominatim and Overpass are run by volunteers, and their usage
policy requires requests to carry a real contact so an operator can email you
rather than block you.

```bash
export GEOSINT_CONTACT="you@example.com"          # macOS / Linux
$env:GEOSINT_CONTACT="you@example.com"            # Windows PowerShell
```

Or pass `--contact` per run. It is sent to OSM and to any site you explicitly
choose to check with `--check-websites`, and nowhere else.

### Optional: the Google collector

`--provider google` has better contact coverage but no `operator`, `brand`,
`ref`, or `wikidata` fields — the metadata that makes OSM more useful for
investigation. It also costs money past a free monthly allowance, and requires
billing enabled with a card on file even to stay inside it. Google prices a
whole response at the tier of its most expensive requested field, so
`--include-ratings` drops the free allowance from 5,000 to 1,000 calls/month.

1. https://console.cloud.google.com → new project
2. Enable **Places API (New)**, enable billing
3. Create an API key, restrict it to "Places API (New)"
4. `export GOOGLE_PLACES_API_KEY="your-key"`

`--near` / `--radius` become a Places `locationRestriction` rectangle around the
circle (the only shape Text Search accepts there), and results are then narrowed
to the true radius. The radius is capped at 50 km, since Text Search returns at
most 60 results - a wider radius is refused rather than quietly clamped. Use
`--provider osm` for a bigger sweep. `--tile` is osm-only.

---

## Commands

### Collect

```bash
# A named area (geocoded to its bounding box)
geosint collect --what pharmacy --area "Austin, TX" --case alpha

# A point and radius - more precise, and usually what you want.
# A box around a city centre is mostly not the city centre.
geosint collect --what pharmacy --near 30.2672,-97.7431 --radius 2

# Any OSM tag, for anything not in the built-in category list
geosint collect --osm-tag "office=diplomatic" --area "Vienna, Austria"
geosint collect --osm-tag "man_made=mast" --near 51.5,-0.12 --radius 10

# Dense area under-returning? Split it into a 3x3 grid (9 queries, deduped)
geosint collect --what pharmacy --area "Chicago, IL" --tile 3

# Many collections from a CSV (header: what,area[,osm_tag]).
# Each row is its own tracked collection, so re-running the batch diffs
# every row and reports what stopped appearing, per row.
geosint batch --input targets.csv --case alpha

# A whole country or region, offline, no rate limit
geosint pbf --pbf texas-latest.osm.pbf --what pharmacy --case alpha
```

For `pbf`, download the smallest extract covering your region from
<https://download.geofabrik.de/>.

### Analyse

```bash
geosint entities --case alpha                    # browse
geosint entities --indicator shared_phone        # filter by indicator
geosint entities --missing                       # what stopped appearing
geosint show "Alpha Exchange"                    # one entity + full provenance
geosint links --case alpha --tag                 # correlate, and tag the results
geosint near 30.2672,-97.7431 2                  # what's within 2km
```

### Record judgement

```bash
geosint triage "Alpha Exchange" Flagged --note "same address as Beta"
geosint note "Alpha Exchange" "registration MSB-11021 predates the website"
geosint recheck "Alpha Exchange" +7d             # also +2w, +3m, +1y, 2026-10-01
geosint recheck                                  # what's due
```

Triage states: `Unreviewed`, `Reviewing`, `Corroborated`, `Dismissed`,
`Flagged`. Anything you write — triage, notes, recheck dates, analyst — is
**never overwritten by a re-collection**.

### Monitor and report

```bash
geosint changes --since 7d --case alpha          # what moved
geosint runs                                     # collection history
geosint report --case alpha                      # indicator + link summary
geosint cases                                    # all cases
geosint export --case alpha                      # GeoJSON by default
```

`export` writes **GeoJSON** by default because the data is inherently
geospatial — drop the file into [geojson.io](https://geojson.io), QGIS,
kepler.gl, or Google Earth with no conversion. `--format csv|xlsx|json` also
available.

---

## Indicators

Findings are named indicators, not a single score. A score collapses independent
facts into one number and hides which fact drove it.

| Indicator | Meaning |
|---|---|
| `no_web_presence` | No website published |
| `website_dead` | Website published but unreachable (needs `--check-websites`) |
| `no_contact_details` | No phone, website, or email at all |
| `no_address` / `no_coordinates` | Incomplete location data |
| `operator_named` | The source names who runs it |
| `chain_affiliated` | Carries a brand tag |
| `registration_ref` | Carries a licence/registration reference |
| `wikidata_linked` | Hard link to a structured record |
| `stale_survey` | Map data not verified recently |
| `not_operational` | Source reports it closed |
| `shared_phone` / `shared_domain` | Written by `geosint links --tag` |

`confidence` is a separate number: the percentage of core fields actually
populated. It measures **record completeness**, not how interesting the entity
is — those are different questions and should not share a scale.

---

## What the sources can and cannot tell you

- **OSM has no rating, review, or operational-status data.** Those come back as
  unknown, and every check skips rather than inventing a value. A blank never
  reads as a zero.
- **Ways carry tags but not coordinates** in offline `pbf` mode (full geometry
  assembly needs a location index and far more memory). They are flagged
  `no_coordinates` so the gap is visible in the data rather than silently absent.
  Nodes — the common case for POIs — carry their coordinates.
- **`--check-websites` is not an audit.** One GET to each entity's own published
  homepage, once, recording only whether it answered. Nothing is read out of the
  page, and it is never a crawl. Sites whose host resolves into private or
  loopback address space are declined rather than fetched — the URL comes from a
  world-writable OSM tag.
- **Coverage varies enormously by region.** OSM is crowdsourced; absence of an
  entity is weak evidence of absence in reality.
- **This tool does not scrape.** `pbf` reaches country scale through OSM's own
  bulk distribution channel — a bigger free, ToS-clean source, not evasion of
  anyone's rate limits. It does not harvest contact details off web pages
  either: `email` comes only from the source's own `email`/`contact:email` tag,
  published as a contact detail by a mapper.
- **Spreadsheet exports are formula-escaped.** An OSM `name` tag is
  world-writable, and Excel evaluates a cell beginning `=`, `+`, `-` or `@`.
  Values that would execute are prefixed so they stay text. GeoJSON and JSON are
  untouched — nothing that reads them executes them.

---

## Options (`collect` / `batch` / `pbf`)

| Flag | Default | What it does |
|---|---|---|
| `--what` | — | Category, e.g. `"pharmacy"` |
| `--osm-tag` | — | Any OSM tag, e.g. `"office=ngo"` (overrides `--what`) |
| `--area` | — | Place name, geocoded to a bounding box |
| `--near` / `--radius` | — | Centre point (`lat,lon`) and radius in km. google caps the radius at 50km |
| `--case` | `default` | Scopes entities, changes, and correlation |
| `--contact` | `$GEOSINT_CONTACT` | Your email/URL, sent as User-Agent |
| `--provider` | `osm` | `osm` (free, richer metadata) or `google` |
| `--tile` | 1 | osm only — N x N grid split for fuller coverage. Raise `--max-results` with it |
| `--check-websites` | off | Record whether each published site answers. 8 at a time |
| `--no-store` | off | One-off collection; disables change detection |
| `--db-path` | `.geosint.db` | Database location |
| `--max-results` | 60 | osm ceiling 3000; google hard cap 60 |
| `--min-confidence` | 0 | Drop records less than N% complete |
| `--format` | geojson | `geojson`, `csv`, `xlsx`, `json` |
| `--output` | — | Write results to this file |
| `--cache-ttl` | 24 | Hours before a cached query refetches (0 = never reuse) |
| `--no-cache` | off | Skip the cache entirely |
| `--dry-run` | off | Collect and analyse, write nothing |
| `--verbose` / `--quiet` | normal | Debug logging / warnings only |

A stored collection always bypasses the cache on read: it exists to answer
"what changed?", and only the network can answer that. The cache key covers
every flag that changes the result set, `--max-results` included, so raising a
cap refetches rather than replaying the smaller answer.

`--max-results` is applied **after** tiling merges and dedupes, so `--tile 3`
with the default 60 still keeps 60 — of the whole area, ranked by record
completeness, not the first tiles queried. When results are dropped to fit the
cap the run says so and names the flag to raise. A radius collection keeps the
nearest N instead, because there distance is a meaningful order.

---

## Development

```bash
git clone https://github.com/jenilmistryhq/geosint.git
cd geosint
pip install -e ".[dev,xlsx,dotenv]"
pytest -q
```

388 tests, all against mocked HTTP and temporary SQLite, or a real synthetic
`.osm.pbf` fixture — no API key, no network, no cost. (One skips unless the
`bulk` extra is installed.) Each test runs in its own working directory with a
neutralised `$HOME`, so nothing leaks between tests or into the repo.

Two rules this suite learned the hard way:

- **Build fixtures from a real normalizer, not by hand.** `dedupe` once keyed on
  field names no normalizer produced. Every test passed; every branch of every
  chain silently merged into one record. `tests/test_dedupe.py` now builds its
  input by calling the collectors' own `_normalize`.
- **Mock the transport, assert on the contract.** Tests that mock a client
  wholesale prove only that the mock works — `--provider google` had no
  end-to-end test and silently discarded `--near` for as long as the flag
  existed.

`tests/test_regressions.py` holds one named test per bug that produced
confident, wrong output.

### Layout

```
geosint/
├── cli.py             argument parsing + command orchestration
├── wizard.py          `start` - guided setup
├── display.py         tables, bar charts, entity cards
├── config.py          env/.env/CLI resolution + validation (one path, all commands)
├── useragent.py       required contact -> User-Agent
├── geo.py             coordinates, radius maths, GeoJSON
├── entities.py        Entity record, indicators, confidence
├── analysis.py        correlation: shared phone/domain/operator/location
├── db.py              SQLite: entities, observations, collections, aliases, actions
├── osm_client.py      Overpass + Nominatim collector
├── api_client.py      Google Places API (New) collector
├── bulk_extract.py    offline .osm.pbf extraction
├── dedupe.py          identity keys, in-memory and at the DB boundary
├── website_health.py  does the published site resolve
├── batch.py           CSV multi-collection runner
├── cache.py           per-query cache with a TTL
└── exporters.py       GeoJSON / CSV / XLSX / JSON
```

### Data model

```
collections   one row per run: query, source, area, counts, timestamps
     |
entities      current best-known state, one row per entity
     |
observations  APPEND-ONLY. Every field change, with source and timestamp.
aliases       maps a second source's id onto the canonical entity
actions       analyst triage changes and notes
```

Adding a category? Add it to `CATEGORY_TAG_MAP` in `osm_client.py` plus a line
in `test_osm_client.py`.

## Legal and ethical note

This queries each data source's own official public API (OpenStreetMap's
Nominatim and Overpass, Google's Places API) and, with `--check-websites`, makes
a plain GET to an entity's own published homepage. It does not scrape search
results, bypass authentication or CAPTCHAs, evade rate limits, or attempt to
disguise itself — the contact address is mandatory precisely so it does not.

You are responsible for the lawfulness and proportionality of your collection.
Public availability is not the same as a right to compile, and the answer
differs by jurisdiction and by purpose. Where collected records touch personal
data, data-protection law generally applies to you as the one compiling them.

## License

[MIT](LICENSE)
