Metadata-Version: 2.4
Name: threaded-fetcher
Version: 0.2.1
Summary: Concurrent threaded URL fetcher for data engineering workflows
Author-email: "S.Sanjay" <sanjays13022003@gmail.com>
License: MIT
Keywords: threading,fetch,http,scrape,data-engineering,cli
Classifier: Programming Language :: Python :: 3
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Requires-Python: >=3.8
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: requests>=2.28
Requires-Dist: pandas>=2.0
Requires-Dist: colorama>=0.4
Requires-Dist: openpyxl>=3.0
Requires-Dist: pyarrow>=8.0
Dynamic: license-file

# threaded-fetcher

A small Python package that fetches many URLs concurrently using threads, records per-task timing, supports retries with exponential backoff, and optional rate limiting.

## Features
- ThreadPoolExecutor-based concurrency
- Per-task start/end timestamps and elapsed time
- Retries with exponential backoff
- Token-bucket rate limiter
- Load URLs from `.txt`, `.csv`, `.xlsx`, `.json`
- Save results to `.csv`, `.json`, `.xlsx` (multi-sheet summary), `.parquet`, `.feather`, `.sqlite`
- Auto-detect URL column names (e.g. `url`, `link`, `Career Page Link`)
- Colored console output and summary metrics

## Installation
Create and activate a virtualenv, then install:

```bash
python -m venv .venv
# Windows (PowerShell)
.\.venv\Scripts\Activate.ps1
# macOS / Linux
# source .venv/bin/activate

pip install -e .
pip install openpyxl pyarrow colorama tqdm
