Metadata-Version: 2.5
Name: ats-scrapers
Version: 0.3.0
Summary: Open job datasets and Python scrapers for ATS platforms and direct job sources.
Project-URL: Homepage, https://github.com/kalil0321/ats-scrapers
Project-URL: Documentation, https://github.com/kalil0321/ats-scrapers#readme
Project-URL: Repository, https://github.com/kalil0321/ats-scrapers
Project-URL: Issues, https://github.com/kalil0321/ats-scrapers/issues
Project-URL: Changelog, https://github.com/kalil0321/ats-scrapers/blob/main/CHANGELOG.md
Project-URL: Dataset, https://storage.stapply.ai/jobhive/v1/manifest.json
Author-email: Kalil Bouzigues <kalil.bouzigues@gmail.com>
License: MIT
License-File: LICENSE
Keywords: ashby,ats,data,greenhouse,hiring,job-scraper,job-search,jobs,lever,recruiting,smartrecruiters,workable,workday
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Internet :: WWW/HTTP :: Indexing/Search
Classifier: Topic :: Office/Business
Classifier: Topic :: Scientific/Engineering :: Information Analysis
Classifier: Typing :: Typed
Requires-Python: >=3.11
Requires-Dist: httpx>=0.27
Requires-Dist: pandas>=2.0
Requires-Dist: pydantic>=2.6
Provides-Extra: all
Requires-Dist: aiohttp>=3.9; extra == 'all'
Requires-Dist: beautifulsoup4>=4.12; extra == 'all'
Requires-Dist: cloakbrowser>=0.3; extra == 'all'
Requires-Dist: html2text>=2024.0; extra == 'all'
Requires-Dist: httpcloak>=1.6; extra == 'all'
Requires-Dist: pyarrow>=15.0; extra == 'all'
Requires-Dist: pycryptodome>=3.20; extra == 'all'
Provides-Extra: dev
Requires-Dist: aiohttp>=3.9; extra == 'dev'
Requires-Dist: beautifulsoup4>=4.12; extra == 'dev'
Requires-Dist: cloakbrowser>=0.3; extra == 'dev'
Requires-Dist: html2text>=2024.0; extra == 'dev'
Requires-Dist: httpcloak>=1.6; extra == 'dev'
Requires-Dist: mypy>=1.10; extra == 'dev'
Requires-Dist: pandas-stubs; extra == 'dev'
Requires-Dist: pyarrow>=15.0; extra == 'dev'
Requires-Dist: pycryptodome>=3.20; extra == 'dev'
Requires-Dist: pytest-asyncio>=0.23; extra == 'dev'
Requires-Dist: pytest-httpx>=0.30; extra == 'dev'
Requires-Dist: pytest>=8.0; extra == 'dev'
Requires-Dist: ruff>=0.5; extra == 'dev'
Requires-Dist: types-boto3; extra == 'dev'
Provides-Extra: parquet
Requires-Dist: pyarrow>=15.0; extra == 'parquet'
Provides-Extra: scrapers
Requires-Dist: aiohttp>=3.9; extra == 'scrapers'
Requires-Dist: beautifulsoup4>=4.12; extra == 'scrapers'
Requires-Dist: cloakbrowser>=0.3; extra == 'scrapers'
Requires-Dist: html2text>=2024.0; extra == 'scrapers'
Requires-Dist: httpcloak>=1.6; extra == 'scrapers'
Requires-Dist: pycryptodome>=3.20; extra == 'scrapers'
Provides-Extra: test
Requires-Dist: aiohttp>=3.9; extra == 'test'
Requires-Dist: beautifulsoup4>=4.12; extra == 'test'
Requires-Dist: cloakbrowser>=0.3; extra == 'test'
Requires-Dist: html2text>=2024.0; extra == 'test'
Requires-Dist: httpcloak>=1.6; extra == 'test'
Requires-Dist: pyarrow>=15.0; extra == 'test'
Requires-Dist: pycryptodome>=3.20; extra == 'test'
Requires-Dist: pytest-asyncio>=0.23; extra == 'test'
Requires-Dist: pytest-httpx>=0.30; extra == 'test'
Requires-Dist: pytest>=8.0; extra == 'test'
Description-Content-Type: text/markdown

<p align="center">
  <img src="https://raw.githubusercontent.com/kalil0321/ats-scrapers/main/assets/banner.jpeg" alt="ats-scrapers" />
</p>


# ats-scrapers

An open dataset and Python toolkit for job data from ATS platforms and public
sources.

[![PyPI version](https://img.shields.io/pypi/v/ats-scrapers)](https://pypi.org/project/ats-scrapers/)
[![Python versions](https://img.shields.io/pypi/pyversions/ats-scrapers)](https://pypi.org/project/ats-scrapers/)
[![License: MIT](https://img.shields.io/badge/license-MIT-green)](https://github.com/kalil0321/ats-scrapers/blob/main/LICENSE)
[![Live jobs](https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fstorage.stapply.ai%2Fjobhive%2Fv1%2Fmanifest.json&query=%24.stats.total_jobs&label=live%20jobs&cacheSeconds=300)](https://storage.stapply.ai/jobhive/v1/manifest.json)
[![Companies](https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fstorage.stapply.ai%2Fjobhive%2Fv1%2Fmanifest.json&query=%24.stats.total_companies&label=companies&cacheSeconds=300)](https://storage.stapply.ai/jobhive/v1/manifest.json)
[![Sources](https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fstorage.stapply.ai%2Fjobhive%2Fv1%2Fmanifest.json&query=%24.stats.ats_count&label=sources&cacheSeconds=300)](https://storage.stapply.ai/jobhive/v1/manifest.json)
[![Dataset snapshot](https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fstorage.stapply.ai%2Fjobhive%2Fv1%2Fmanifest.json&query=%24.generated_at&label=snapshot&cacheSeconds=300)](https://storage.stapply.ai/jobhive/v1/manifest.json)

`ats-scrapers` provides two layers:

- A free, hosted dataset of live jobs from company career sites, ATS
  platforms, and public job feeds. The badges above read current counts and
  snapshot time directly from the live manifest.
- Reusable scraper adapters for Workday, Greenhouse, Lever,
  Ashby, SmartRecruiters, and SuccessFactors.

Jobs are collected from ATS endpoints, company career sites, and public job
feeds, then normalized into one typed schema. Querying the hosted dataset
requires no API key or account.

## Install

```bash
pip install ats-scrapers
```

The package is imported as `ats_scrapers`. Optional extras add only what you
need:

```bash
pip install "ats-scrapers[parquet]"   # query the full Parquet snapshot
pip install "ats-scrapers[scrapers]"  # run the scraper library
pip install "ats-scrapers[all]"       # install every runtime extra
```

## Query the public dataset

```python
from ats_scrapers import search

# Per-source searches work with the base install.
jobs = search(
    query="machine learning engineer",
    location="Paris",
    ats="greenhouse",
    limit=100,
)

# The result is a pandas DataFrame.
print(jobs[["company", "title", "location", "apply_url"]])
```

For practical full-dataset queries, install the `parquet` extra. The base
install is intended for smaller per-source CSV slices.

```python
from ats_scrapers import search

jobs = search(query="data engineer", remote=True, salary_min=80_000)
```

The [live manifest](https://storage.stapply.ai/jobhive/v1/manifest.json)
contains current row counts and artifact URLs. See the
[job schema](https://github.com/kalil0321/ats-scrapers/blob/main/docs/JOB_SCHEMA.md)
for field definitions and normalization rules.

## Scrape a company

You don't need to know which ATS a company uses. Paste its careers URL:

```python
from ats_scrapers import get_scraper_for_url

scraper = get_scraper_for_url("https://jobs.ashbyhq.com/openai")
jobs = scraper.fetch()
```

Or look it up by name in the hosted companies directory:

```python
from ats_scrapers import find_company
from ats_scrapers.scrapers import get_scraper

find_company("openai")          # → ats="ashby", slug="openai", url=...
scraper = get_scraper("ashby", "openai")
jobs = scraper.fetch()
```

Scraper classes are also available directly:

```python
from ats_scrapers.scrapers import GreenhouseScraper

jobs = GreenhouseScraper("anthropic").fetch()
```

Scrapers are async-first — in async code (or for concurrency) use
`await scraper.afetch()` instead. The sync `fetch()` also works from
inside a running event loop (Jupyter, FastAPI): it transparently runs
on a worker thread.

Scraper adapters include:

- Major ATS platforms: ADP Workforce Now, Greenhouse, Lever, Ashby, Workday, SmartRecruiters,
  SuccessFactors, Oracle, iCIMS, HERP Hire, HRMOS, Keka, Paycom, Softgarden, Workable, Personio,
  and more.
- First-party company APIs: Amazon, Apple, Google, TikTok, and Uber.
- Public and regional sources: EURES, Bundesagentur, Arbetsformedlingen,
  Welcome to the Jungle, and others.

Run `python -c "from ats_scrapers import list_ats; print(*list_ats())"` for the
sources currently present in the hosted dataset. The dataset can list a source
before this package ships a scraper for it — `search()` still returns those
rows; only building your own scraper needs one. To skip sources without a
scraper:

```python
from ats_scrapers import list_ats
from ats_scrapers.scrapers import ScraperRegistry, get_scraper

scrapeable = [a for a in list_ats() if ScraperRegistry.has_scraper(a)]
```

## Contributing

Contributions can add a source, improve an existing scraper, or add companies
to the CSV inventories in
[`ats-companies/`](https://github.com/kalil0321/ats-scrapers/tree/main/ats-companies).
The scraper API is intentionally tiny: subclass `BaseScraper`, set `ats`, and
implement `async def afetch()` using `self.make_fetcher()` for HTTP — retries,
backoff, and error mapping come for free. See
`src/ats_scrapers/scrapers/greenhouse.py` for a compact reference and the `Job`
model in `src/ats_scrapers/models.py` for the schema you populate.

```bash
git clone https://github.com/kalil0321/ats-scrapers
cd ats-scrapers
uv sync --extra dev
uv pip install -r pipeline/requirements.txt  # repo-only ops/publisher tests
uv run pytest
uv run ruff check .
```

## License

[MIT](https://github.com/kalil0321/ats-scrapers/blob/main/LICENSE)
