Metadata-Version: 2.5
Name: proxmint-haystack
Version: 0.1.0
Summary: Haystack component that fetches web pages through Proxmint residential, datacenter and mobile proxies, with country, city and ASN targeting and sticky sessions.
Project-URL: Homepage, https://proxmint.com
Project-URL: Documentation, https://proxmint.com/docs/haystack
Project-URL: Repository, https://github.com/proxmint/proxmint-haystack
Project-URL: Issues, https://github.com/proxmint/proxmint-haystack/issues
Author-email: Proxmint <support@proxmint.com>
License-Expression: MIT
License-File: LICENSE
Keywords: agents,fetcher,haystack,proxmint,proxy,residential-proxy,web-scraping
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Internet :: Proxy Servers
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.10
Requires-Dist: haystack-ai>=2.10
Requires-Dist: httpx<1,>=0.27
Description-Content-Type: text/markdown

# proxmint-haystack

[Haystack](https://haystack.deepset.ai) integration for [Proxmint](https://proxmint.com) proxies: fetch web pages
from residential, datacenter or 4G/5G mobile IPs and get them back as `Document`s. Pick the exit country, state,
city, ZIP or ASN, and keep one IP for up to two hours when a site needs it.

- **`ProxmintFetcher`**: a component that takes a list of URLs and returns `documents` (readable page text, or the
  raw HTML) and `failed_urls`. Use it as the first step of an indexing pipeline, or wrap it with `ComponentTool` to
  give an agent a "read this page from Germany" tool.

Requests go straight from your machine to the Proxmint gateway (`gw.proxmint.com`). Nothing else sits in
between, and bandwidth comes out of your own Proxmint account.

## Install

```bash
pip install proxmint-haystack
```

## Credentials

Create an API key under **Settings** in the [Proxmint dashboard](https://proxmint.com/dashboard) and export it:

```bash
export PROXMINT_API_KEY="pmk_..."
```

The key reads the proxy login for the product you choose (`residential` by default). You can use the proxy login
instead, with `PROXMINT_USERNAME` and `PROXMINT_PASSWORD`.

## Fetch pages

```python
from haystack_integrations.components.fetchers.proxmint import ProxmintFetcher

fetcher = ProxmintFetcher(country="gb")
result = fetcher.run(urls=["https://example.com", "https://example.org"])

print(result["documents"][0].content)
print(result["documents"][0].meta)  # source, requested_url, status, title, content_type, country
print(result["failed_urls"])
```

## Use it in a pipeline

```python
from haystack import Pipeline
from haystack.components.preprocessors import DocumentSplitter
from haystack.components.writers import DocumentWriter
from haystack.document_stores.in_memory import InMemoryDocumentStore
from haystack_integrations.components.fetchers.proxmint import ProxmintFetcher

store = InMemoryDocumentStore()
pipe = Pipeline()
pipe.add_component("fetcher", ProxmintFetcher(country="us", city="New York", session_minutes=30))
pipe.add_component("splitter", DocumentSplitter(split_by="sentence", split_length=5))
pipe.add_component("writer", DocumentWriter(store))
pipe.connect("fetcher.documents", "splitter.documents")
pipe.connect("splitter.documents", "writer.documents")

pipe.run({"fetcher": {"urls": ["https://example.com/pricing", "https://example.org/pricing"]}})
```

## Use it as an agent tool

```python
from haystack.tools import ComponentTool
from haystack_integrations.components.fetchers.proxmint import ProxmintFetcher

tool = ComponentTool(
    component=ProxmintFetcher(),
    name="fetch_pages",
    description="Fetch web pages through a proxy in a chosen country and return their text.",
)
```

Pass the tool to an `Agent` or a chat generator as usual. The model can set `urls` and an optional `country` per call.

## Options

| Option | Default | What it does |
| --- | --- | --- |
| `api_key` | `PROXMINT_API_KEY` | A Haystack `Secret` |
| `product` | `"residential"` | `residential`, `premium-residential`, `datacenter` or `mobile` |
| `country` | any | Exit country (two-letter code). `run(urls, country=...)` overrides it for one run |
| `state`, `city`, `zip_code`, `asn` | none | Finer targeting; needs `country` |
| `session_minutes` | none | Keep one exit IP across runs for 1 to 120 minutes. Default: a new IP for every URL |
| `output` | `"text"` | `"text"` for readable text, `"html"` for the raw page |
| `timeout` | `60` | Seconds per request |
| `raise_on_failure` | `False` | Raise on a failed URL instead of listing it in `failed_urls` |

A gateway refusal (no bandwidth left, no IP matching the targeting, a blocked site) names what to change.

## Good to know

- Web traffic works on destination ports 80, 443, 8080, 8443, 8888 and 8090 ([port access](https://proxmint.com/docs/ports)).
- A short list of sites, mostly banking and government, is refused by the network ([blocked websites](https://proxmint.com/docs/blocked-websites)).
- City, state, ZIP and ASN targeting meter at 2x on residential, datacenter and mobile ([targeting](https://proxmint.com/docs/targeting)).
- Each product allows up to 2,000 concurrent connections.

Full guide: [proxmint.com/docs/haystack](https://proxmint.com/docs/haystack). Pricing: [proxmint.com/pricing](https://proxmint.com/pricing).
Bandwidth is billed per GB and never expires.

## Development

```bash
uv run --group test pytest
```

The tests run offline against a stand-in for the gateway.

## License

MIT
