Metadata-Version: 2.4
Name: atrium-job-scraper
Version: 2.0.2
Summary: A powerful job scraper with a FastAPI Web UI
Author: devkunal2812
Description-Content-Type: text/markdown
Requires-Dist: requests>=2.31.0
Requires-Dist: beautifulsoup4>=4.12.3
Requires-Dist: lxml>=5.2.0
Requires-Dist: selenium>=4.20.0
Requires-Dist: python-dotenv>=1.0.1
Requires-Dist: pytest>=8.2.0
Requires-Dist: gspread
Requires-Dist: google-auth
Requires-Dist: fastapi>=0.111.0
Requires-Dist: uvicorn[standard]>=0.29.0
Requires-Dist: jinja2>=3.1.4
Requires-Dist: sse-starlette>=2.1.0
Requires-Dist: python-multipart>=0.0.9
Requires-Dist: prompt_toolkit>=3.0.0
Dynamic: author
Dynamic: description
Dynamic: description-content-type
Dynamic: requires-dist
Dynamic: summary

# Atrium Job Scraper

A modular, polite, production-oriented scraper that collects publicly
available internship/job listings for Atrium's student community.

## ⚠️ Read this first: a note on live-site verification

This project was built in a sandboxed environment with **no network
access to linkedin.com, indeed.com, naukri.com, or internshala.com** —
only package registries (PyPI/npm/GitHub) were reachable. That means:

- The architecture, pipeline, tests, error handling, and CLI are fully
  built and **verified end-to-end using saved HTML fixtures**
  (`tests/fixtures/*.html`) that mirror each site's real DOM structure.
- **Internshala** is the fully working reference implementation — its
  selectors are based on Internshala's known server-rendered markup and
  pass parsing tests against a realistic fixture.
- **Indeed, Naukri, and LinkedIn** are implemented with the same
  architecture and best-effort selectors, but **you must run them
  against the live sites yourself** (from a machine with normal internet
  access) and adjust selectors in `scraper/sources/<name>.py` if a site
  has changed its markup since this was written. Each source file
  isolates its selectors in one or two small methods specifically so
  this kind of update is quick and low-risk.
- Run `python main.py --source internshala` first, then check
  `output/*.json` and `logs/scraper.log`. If Internshala's HTML has
  drifted, the log will show 0 jobs found on page 1 — that's your signal
  to inspect the live page and update the selectors in
  `scraper/sources/internshala.py`.

This is standard practice for any scraper anyway: sites change their
markup periodically, so **selector maintenance is an ongoing task**, not
a one-time build step, regardless of who wrote the original code.

## Architecture

```
scraper/
  sources/          # one file per site, isolated selectors
    base.py          # BaseSource ABC — pagination loop, error isolation
    internshala.py    # requests + BeautifulSoup (fully working reference)
    naukri.py          # requests, JSON-payload parsing + DOM fallback
    indeed.py            # Selenium (site requires JS / blocks plain requests)
    linkedin.py            # requests, guest endpoint only (see limitations)
  core/
    scraper.py       # orchestrator: runs sources, isolates failures
    normalizer.py     # shared text/date/skill cleanup helpers
    filters.py         # KEYWORDS / LOCATION / REMOTE_ONLY / etc.
    deduplicator.py     # stable UID-based dedup
    output.py            # JSON + CSV writers, Postgres stub
  models/
    job.py           # the Job dataclass every source normalizes into
  utils/
    config.py        # .env -> typed Config object
    logger.py          # console + rotating file logging
    http.py             # retries, backoff, rate limiting, robots.txt
    browser.py           # reusable headless Selenium manager
main.py              # CLI entrypoint
tests/               # pytest suite + HTML fixtures (no live network needed)
```

**Data flow:** `main.py` → `core.scraper.run_sources()` → for each
source: `BaseSource.scrape()` paginates, calls the source's
`parse_listing_page()` + `normalize()` → all `Job` objects are merged →
`Deduplicator` removes duplicates → `filters.apply_filters()` applies
your `.env` filters → `core.output` writes JSON + CSV.

## requests vs. Selenium — which sources use which, and why

| Source | Method | Why |
|---|---|---|
| **Internshala** | `requests` + BeautifulSoup | Listing pages are server-rendered; the job cards are present in the raw HTML, no JS execution needed. |
| **Naukri** | `requests` + BeautifulSoup | The listing data ships as a JSON blob embedded in a `<script>` tag (Next.js hydration payload) inside the server-rendered shell — no browser needed to read it. A DOM-based fallback parser is included in case a page variant doesn't include the payload. |
| **Indeed** | **Selenium** | Indeed serves a JS/CAPTCHA challenge page to most non-browser HTTP clients, even with correct headers — a plain `requests.get` reliably returns a challenge page, not listings. A real browser is a genuine technical requirement here, not a workaround. We do **not** attempt to solve or bypass the challenge — if Selenium itself lands on a challenge page, we log and skip it. |
| **LinkedIn** | `requests` (guest endpoint only) | LinkedIn's public guest job-search page returns cards without login. Full job **descriptions** redirect to a login wall, which we deliberately do not cross — `description`/`skills`/`qualifications` are left empty for LinkedIn jobs. See the docstring in `scraper/sources/linkedin.py` for details. |

## Setup & Installation

```bash
# 1. Install via pip
pip install atrium-job-scraper

# 2. Or, if installing from source:
git clone https://github.com/devkunal2812/Atrium-Scraper.git
cd Atrium-Scraper
python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt

# 3. Configuration
cp .env.example .env
# Edit .env to set your KEYWORDS, LOCATION, MAX_PAGES, etc.
```

Selenium 4.6+ uses **Selenium Manager**, which auto-downloads the correct ChromeDriver the first time it runs — no manual driver setup needed. You do need Google Chrome (or Chromium) installed on the machine running the Indeed source.

## Configuration (`.env`)

```bash
KEYWORDS=react,node.js,python,java   # matched against title/description/skills
SKILLS=                              # optional, narrower than KEYWORDS
LOCATION=india                       # substring match
REMOTE_ONLY=false
INTERNSHIP_ONLY=false
MAX_PAGES=5
```
*(All config settings can be overridden per-run via CLI flags).*

## Usage Guide

The package provides multiple ways to run the scraper depending on your workflow:

### 1. Interactive Terminal (TUI) + Web UI 🌟
Run the following command to start both the modern Interactive Terminal Interface and the Web UI in the background:
```bash
atrium-scraper
```
* **Interactive Terminal**: An `opencode`-style interface will appear in your terminal. You can run commands like `/scrape naukri`, `/scrape all`, or even standard shell commands (e.g., `!ls`).
* **Web UI**: Automatically launches at `http://localhost:8000`. You can monitor scrape progress and results via the web dashboard simultaneously!

### 2. Standalone Web UI
If you only want to start the Web UI (and access the API/dashboard without the interactive terminal):
```bash
uvicorn app:app --host 0.0.0.0 --port 8000
```

### 3. Traditional CLI
If you prefer standard CLI arguments (great for cron jobs or CI/CD pipelines), use `atrium-cli`:

```bash
# Single source
atrium-cli --source internshala
atrium-cli --source linkedin

# All sources at once
atrium-cli --source all

# Override filters for one run
atrium-cli --source all --keyword "frontend developer" --location "india"
atrium-cli --source internshala --max-pages 5 --remote-only
```

Output goes to `output/jobs_<timestamp>.json` and `.csv`. Logs go to `logs/scraper.log` and the console.

## Running the tests

```bash
python -m pytest -q
```

44 tests cover: normalization helpers (dates, remote-status detection,
skill parsing), the `Job` UID/dedup logic, filter combinations, and
HTML parsing for Internshala and Naukri against saved fixtures — so the
suite runs fully offline and doesn't depend on live sites being up or
unchanged.

## Deduplication

Each `Job` computes a stable UID via `Job.compute_uid()`:
1. `source + source_job_id` if the site exposes a real job ID (best).
2. `source + application_url` if no ID but a canonical URL exists.
3. `sha256(source + title + company + location)` as a last resort.

The `Deduplicator` runs across the **combined** results of all sources
in one invocation, so `--source all` won't double-count a job that
happens to be posted identically on two sites under different IDs (it
will still count it once per source if the title/company/location
differ even slightly — true cross-source identity matching would need
fuzzy text matching, which isn't implemented).

## Error handling & rate limiting

- Every page fetch and every per-job normalization is wrapped so one bad
  URL or malformed card **never kills the whole run** — it's logged and
  skipped.
- `HttpClient` retries transient failures (timeouts, 5xx, 429) with
  exponential backoff, respects `robots.txt` by default, and treats
  anything that looks like a CAPTCHA/bot-block page as a failure to skip,
  never something to "solve."
- A configurable delay is enforced between every request
  (`REQUEST_DELAY_SECONDS`).

## Connecting to Postgres later

`scraper/core/output.py` has a `save_to_postgres()` stub with the
intended implementation shape documented in its docstring. Because every
source already normalizes into the same `Job` dataclass and `job_uid` is
a natural primary key, wiring up a real DB (SQLAlchemy/psycopg2, an
`INSERT ... ON CONFLICT (job_uid) DO NOTHING`) is a small, isolated
change — no scraping code needs to change.

## Known limitations (be upfront about these with stakeholders)

- **Selector drift**: sites redesign their frontends periodically.
  Expect to revisit selectors every so often, especially for
  Naukri/Indeed/LinkedIn. This is normal maintenance for any scraper.
- **LinkedIn descriptions are unavailable** without logging in, which
  this scraper deliberately does not do.
- **Indeed depends on a headless browser succeeding past its bot-check**;
  on a machine/IP with a poor reputation, more pages may come back
  blocked. This is expected and by design (we don't try to force through).
- **Location filtering is a plain substring match**, not geocoding — see
  the comment in `scraper/core/filters.py`.
- Job descriptions/skills from Internshala's **listing** page are
  limited to what the card shows; getting the full description would
  require an additional fetch per job to its detail page (not done by
  default to keep request volume low — can be added as an optional
  `--deep` flag if Atrium wants it).
