Metadata-Version: 2.4
Name: subterranean
Version: 1.5.0
Summary: Give your AI Agent eyes to see the entire internet. Search + Read 10+ platforms.
Author: Chetbrathna Sam (jame) - Subterranean, Neo Reid
License: MIT
Project-URL: Homepage, https://pypi.org/project/subterranean/
Project-URL: Repository, https://pypi.org/project/subterranean/
Keywords: ai-agent,llm-tools,agent-infrastructure,mcp,web-reader,web-scraper,search,twitter-scraper,reddit-scraper,youtube-transcript,bilibili,xiaohongshu,ai-search,cli,automation,claude-code,cursor,openai,free-api,no-api-key
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Internet :: WWW/HTTP
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: requests>=2.28
Requires-Dist: feedparser>=6.0
Requires-Dist: python-dotenv>=1.0
Requires-Dist: loguru>=0.7
Requires-Dist: pyyaml>=6.0
Requires-Dist: rich>=13.0
Requires-Dist: yt-dlp[default]>=2026.07.04
Provides-Extra: browser
Requires-Dist: playwright>=1.40; extra == "browser"
Provides-Extra: cookies
Requires-Dist: browser-cookie3>=0.19; extra == "cookies"
Provides-Extra: all
Requires-Dist: playwright>=1.40; extra == "all"
Requires-Dist: mcp[cli]>=1.0; extra == "all"
Requires-Dist: browser-cookie3>=0.19; extra == "all"
Provides-Extra: dev
Requires-Dist: pytest>=8.0; extra == "dev"
Requires-Dist: ruff>=0.8; extra == "dev"
Requires-Dist: mypy>=1.12; extra == "dev"
Requires-Dist: types-requests>=2.32; extra == "dev"
Requires-Dist: types-PyYAML>=6.0; extra == "dev"
Dynamic: license-file
Dynamic: requires-python

<h1 align="center">🃏 Subterranean</h1>

<p align="center">
  <strong>Give your AI agent internet access — in one command.</strong>
</p>

<p align="center">
  The most reliable way in, right now: chosen, installed, and health-checked for you. Routers change over time — you don't have to care.
</p>

<p align="center">
  <a href="https://pypi.org/project/subterranean/"><img src="https://img.shields.io/badge/License-MIT-blue.svg?style=for-the-badge" alt="MIT License"></a>
  <a href="https://www.python.org/"><img src="https://img.shields.io/badge/Python-3.10+-green.svg?style=for-the-badge&logo=python&logoColor=white" alt="Python 3.10+"></a>
  <a href="https://pypi.org/project/subterranean/"><img src="https://img.shields.io/pypi/v/subterranean?style=for-the-badge" alt="PyPI version"></a>
</p>

<p align="center">
  <a href="#quick-start">Quick Start</a> · <a href="#supported-platforms">Supported Platforms</a> · <a href="#design-philosophy">Design</a> · <a href="#security">Security</a>
</p>

---

## Why Subterranean?

AI agents can write code, edit docs, and manage projects — but ask one to look something up online and it falls apart:

- 📺 "Watch this YouTube tutorial and tell me what it says" → **can't**, no transcript access
- 🐦 "Search Twitter for what people say about this product" → **can't**, the Twitter API costs money
- 📖 "Check Reddit for anyone with the same bug" → **403**, server IP blocked
- 📕 "Look up reviews of this product on Xiaohongshu" → **won't open**, login required
- 📺 "Summarize this Bilibili tech video" → **can't**, generic downloaders are blocked by Bilibili's risk controls
- 🔍 "Find a comparison of the latest LLM frameworks" → **no good search**, either paid or low quality
- 🌐 "What does this webpage say?" → **raw HTML soup**, unreadable
- 📦 "What's this GitHub repo about? What did the issues say?" → possible, but auth setup is painful
- 📡 "Subscribe to these RSS feeds and tell me when they update" → requires writing your own code

**None of this is hard — it's just tedious to set up.**

Every platform has its own gate: paid APIs, blocks to dodge, accounts to log into, data to clean. You fight them one by one. Subterranean turns it into one sentence:

```
Install Subterranean for me: pip install subterranean && subterranean install
```

Paste that to your agent. In a few minutes it can read tweets, search Reddit, watch YouTube, and browse Xiaohongshu.

**Already installed? Updating is one sentence too:**

```
Update Subterranean for me: pip install --upgrade subterranean
```

### ✅ Before you use it, you should know

| | |
|---|---|
| 💰 **Completely free** | All tools are open source, all APIs free. The only possible cost is a server proxy (~$1/month) — local machines need none |
| 🔒 **Private by design** | Cookies stay on your machine only — never uploaded, never shared. Fully open source, auditable any time |
| 🔄 **Always current** | Every platform uses a "primary + fallback" backend route. When one path dies we swap it silently (example: 2026-06, yt-dlp got blocked by Bilibili → switched to bili-cli, zero user action) |
| 🤖 **Works with any agent** | Claude Code, OpenClaw, Cursor, Windsurf — anything that can run shell commands |
| 🩺 **Self-diagnosis** | `subterranean doctor` tells you what works, what doesn't, and how to fix it |

---

## Supported Platforms

| Platform | Works out of the box | Unlocks after setup | How to set up |
|----------|---------------------|--------------------|--------------------|
| 🌐 **Web** | Read any webpage | — | No config |
| 📺 **YouTube** | Transcript extraction + video search | — | No config |
| 📡 **RSS** | Read any RSS/Atom feed | — | No config |
| 🔍 **Web search** | — | Semantic web search | Automatic (MCP, free, no key) |
| 📦 **GitHub** | Read public repos + search | Private repos, Issues/PRs, Fork | Tell your agent "log me into GitHub" |
| 🐦 **Twitter/X** | Read single tweets | Search, timelines, long-form | Tell your agent "set up Twitter" |
| 📺 **Bilibili** | Search + video details (bili-cli, no login) | Subtitles (OpenCLI) | Tell your agent "set up Bilibili" |
| 📖 **Reddit** | — (no zero-config path: anonymous API blocked) | Search + read posts/comments | OpenCLI on desktop with browser session; or rdt-cli + cookies |
| 📘 **Facebook** | — | Search, pages, feeds, group lists | OpenCLI on desktop (reuses Chrome session) |
| 📷 **Instagram** | — | User search, profiles, recent posts, Explore | OpenCLI on desktop (reuses Chrome session) |
| 📕 **Xiaohongshu** | — | Search, read, comment | OpenCLI uses your existing Chrome session; MCP/legacy tools use Cookie-Editor export |
| 💼 **LinkedIn** | Jina Reader for public pages | Profile details, company pages, job search | Tell your agent "set up LinkedIn" |
| 🎯 **Boss Zhipin** | CDP link check | Job search + full JDs (dedicated Chrome) | Tell your agent "set up Boss Zhipin"; agent opens a dedicated Chrome, you log in manually |
| 💻 **V2EX** | Hot posts, node posts, post details + replies, user info | — | No config |
| 📈 **Xueqiu** | Stock quotes, stock search, hot posts, rankings | — | Tell your agent "set up Xueqiu" |
| 🎙️ **Xiaoyuzhou Podcast** | — | Podcast audio → text (Whisper transcription, free key) | Tell your agent "set up Xiaoyuzhou" |

> **Don't know how to configure it? Skip the docs.** Just tell your agent "set up XXX" — it knows what's needed and will walk you through it.
>
> 🍪 Twitter accepts only cookies you export manually via Cookie-Editor. Subterranean does not perform Xiaohongshu logins on your behalf or read Xiaohongshu browser cookies; OpenCLI only uses Chrome sessions you already created and control. `subterranean configure xhs-cookies` does not inject cookies into OpenCLI / Chrome — without an existing session, export via Cookie-Editor and configure xiaohongshu-mcp / legacy tools instead.
>
> Twitter cookies saved via Subterranean are only used by `subterranean doctor` to verify your setup; before running the upstream `twitter` command directly, set `TWITTER_AUTH_TOKEN` and `TWITTER_CT0` explicitly in your process environment.
>
> 🔒 Cookies stay local — never uploaded, never shared.
> 💻 No proxy needed on a local machine. Proxies are only for server deployments (~$1/month).

---

## Quick Start

> ⚠️ **OpenClaw users: enable exec permissions first**
>
> Subterranean needs your agent to run shell commands (`pip install`, `mcporter`, `twitter`, etc.). If your OpenClaw uses the default `messaging` tool profile, the agent cannot execute commands. **Enable exec permissions before installing:**
>
> ```bash
> openclaw config set tools.profile "coding"
> ```
> Or set `"tools": { "profile": "coding" }` in `~/.openclaw/openclaw.json`. Restart the gateway (`openclaw gateway restart`) and start a new conversation. Other platforms (Claude Code, Cursor, Windsurf, etc.) are not affected.

Paste this to your AI agent (Claude Code, OpenClaw, Cursor, etc.):

```
Install Subterranean for me: pip install subterranean && subterranean install
```

That's it — your agent handles the rest.

> 🔄 **Already installed?** Updating is one sentence:
> ```
> Update Subterranean for me: pip install --upgrade subterranean
> ```

> 🛡️ **Safe by default:** `subterranean install` only inspects your environment — it never installs system packages or writes config on its own:
> ```
> Check my environment and install Subterranean safely: pip install subterranean && subterranean install
> ```
> Use `subterranean install --system` only when you explicitly allow system changes.

<details>
<summary>What does it do? (click to expand)</summary>

1. **Installs the CLI** — installs the `subterranean` command from PyPI (bundles yt-dlp, feedparser)
2. **Checks system prerequisites** — Node.js, gh CLI, mcporter, with install commands for anything missing
3. **Installs and configures only with permission** — dependencies and the Exa MCP route are only set up when you explicitly pass `--system`
4. **Detects your environment** — local machine vs. server, with matching config advice
5. **Registers SKILL.md only with permission** — your agent's skill files are only written with explicit `--system`; the default check touches nothing
6. **Asks what else you want** — 6 zero-config channels activate by default; login-gated platforms (Xiaohongshu, Twitter, Reddit, Facebook, Instagram) appear in a menu and install only if you name them

After install, `subterranean doctor` shows every channel's status and which backend it's currently using.
</details>

---

## Works Right After Install

No configuration needed — just ask your agent:

- "Read this link" → `curl https://r.jina.ai/URL` reads any webpage
- "What's this GitHub repo about?" → `gh repo view owner/repo`
- "What does this YouTube video say?" → `yt-dlp` extracts transcripts
- "Search Bilibili for AI tutorials" → `bili search` (no login)
- "Compare the latest LLM frameworks" → Exa semantic search
- "Subscribe to this RSS feed" → `feedparser` parses it

**No commands to memorize.** Your agent reads SKILL.md and knows what to call. For login-gated platforms (Xiaohongshu, Twitter, Reddit, Facebook, Instagram), just say "set up XXX" to unlock them.

---

## Design Philosophy

**Subterranean is a capability layer — not another tool.**

It sits one level above any specific implementation, handling **selection, installation, health checks, and routing** — not the reading itself. Actual reading is done by your agent calling upstream tools directly. No wrapper layer.

When you set up an agent from scratch, you always burn time hunting tools and fighting config — what reads Twitter? How does Reddit login work? That CLI for Xiaohongshu got abandoned, what now? You re-tread the same path every time. Subterranean does one thing: **the most reliable route in, right now — chosen, installed, and health-checked for you. Routes change over time (in March 2026 a batch of single-platform CLIs was abandoned and we swapped the routing); you don't have to care.**

### 🔌 Every platform = an ordered list of primary + fallback backends

Swapping access methods = reordering the list, not rewriting code. `subterranean doctor` tells you **which backend each platform is using right now**.

```
channels/
├── web.py          → Jina Reader
├── twitter.py      → twitter-cli ▸ OpenCLI ▸ bird
├── youtube.py      → yt-dlp
├── github.py       → gh CLI
├── bilibili.py     → bili-cli ▸ OpenCLI ▸ search API (yt-dlp blocked by Bilibili risk controls, retired)
├── reddit.py       → OpenCLI ▸ rdt-cli (no zero-config path, login required)
├── facebook.py     → OpenCLI (desktop browser session)
├── instagram.py    → OpenCLI (desktop browser session)
├── xiaohongshu.py  → OpenCLI ▸ xiaohongshu-mcp ▸ xhs-cli
├── linkedin.py     → mcp-server-linkedin ▸ Jina Reader
├── rss.py          → feedparser
├── exa_search.py   → Exa via mcporter
└── __init__.py     → channel registry (for doctor checks)
```

Each channel file **actually probes** its candidate backends in order (not just checking whether a command exists) — the first fully working one wins, and a broken one prints a fix recipe. The agent then calls the upstream tools directly.

### Current Selections

| Use case | Primary | Fallback | Why |
|----------|---------|----------|-----|
| Read web pages | [Jina Reader](https://github.com/jina-ai/reader) | — | Free, no API key |
| Read Twitter | [twitter-cli](https://github.com/public-clis/twitter-cli) | [OpenCLI](https://github.com/jackwener/opencli) | Search proven stable; OpenCLI covers it via browser session |
| Reddit | [OpenCLI](https://github.com/jackwener/opencli) (desktop) | [rdt-cli](https://github.com/public-clis/rdt-cli) | Anonymous API blocked, official API approval-only — only login-session paths remain |
| Facebook | [OpenCLI](https://github.com/jackwener/opencli) (desktop) | — | Graph/Groups API permissions tightened; browser session is the practical path |
| Instagram | [OpenCLI](https://github.com/jackwener/opencli) (desktop) | Official Graph API (Business/Creator + approval) | instaloader-style paths unstable; OpenCLI reuses a real browser session |
| YouTube transcripts + search | [yt-dlp](https://github.com/yt-dlp/yt-dlp) | — | 154K stars, still best for YouTube (no longer used for Bilibili) |
| Bilibili | [bili-cli](https://github.com/public-clis/bilibili-cli) | OpenCLI ▸ search API | yt-dlp got hard-blocked (HTTP 412) by Bilibili risk controls (tested 2026-06); bili-cli searches and reads without login |
| Web search | [Exa](https://exa.ai) via [mcporter](https://github.com/nicobailon/mcporter) | — | AI semantic search, MCP-based, key-free |
| GitHub | [gh CLI](https://cli.github.com) | — | Official tool, full API once authenticated |
| Read RSS | [feedparser](https://github.com/kurtmckee/feedparser) | — | Standard in the Python ecosystem |
| Xiaohongshu | [OpenCLI](https://github.com/jackwener/opencli) (desktop) | [xiaohongshu-mcp](https://github.com/xpzouying/xiaohongshu-mcp) (server) ▸ xhs-cli | OpenCLI only uses your existing session; other backends use manual Cookie-Editor exports |
| LinkedIn | [mcp-server-linkedin](https://github.com/stickerdaniel/linkedin-mcp-server) | Jina Reader | MCP service, browser automation |

> 📌 These are **current selections**, re-verified periodically on real machines. When a route dies, we move to the next — `subterranean doctor` always tells you which one you're on.

---

## Security

| Measure | Detail |
|---------|--------|
| 🔒 **Credentials stay local** | Cookies and tokens live only in `~/.subterranean/config.yaml`, mode 600 (owner-only), never uploaded, never shared |
| 🛡️ **Safe by default** | `subterranean install` doesn't modify your system; only explicit `--system` installs external tools or writes config |
| 👀 **Fully open source** | Transparent code, auditable anytime. All dependency tools are open source too |
| 🔍 **Dry run** | `subterranean install --dry-run` previews every action without changing anything |
| 🧩 **Pluggable** | Don't trust a component? Replace its channel file — nothing else is affected |

### 🍪 Cookie Safety

> ⚠️ **Ban risk:** platforms where you log in with cookies (Twitter, Xiaohongshu, etc.) may detect scripted/API access and **restrict or ban the account**. Always use a **dedicated secondary account**, never your main.

Two reasons for a burner account on cookie/login-gated platforms (Twitter, Xiaohongshu, Reddit, Facebook, Instagram, etc.):
1. **Ban risk** — platforms can flag API calls that don't look like a normal browser
2. **Security risk** — a cookie equals full login rights; a secondary account caps the blast radius if credentials leak

### 📦 Install Options

| Mode | Command | Use when |
|------|---------|----------|
| Default safe check | `subterranean install --env=auto` | Any environment; read-only check listing what's missing |
| Explicit system install | `subterranean install --env=auto --system` | You explicitly allow machine changes |
| Compatible safe flag | `subterranean install --env=auto --safe` | Same as default |
| Preview only | `subterranean install --env=auto --dry-run` | See what it would do |

### 🗑️ Uninstall

```bash
subterranean uninstall
```

Removes: `~/.subterranean/` (all tokens/cookies), agent skill files, MCP config in mcporter.

```bash
# Preview only, delete nothing
subterranean uninstall --dry-run

# Remove skill files only, keep token config (for reinstall)
subterranean uninstall --keep-config
```

Uninstall the Python package itself: `pip uninstall subterranean`

---

## ⭐ Why Star This

I use this project every day, so I keep maintaining it:

- New channel requests get added over time
- Every channel stays **working, pleasant, and free** where possible
- When platforms change their anti-bot rules or APIs, I fix the routing

Star it so you can find it next time you need it. ⭐

---

## Acknowledgments

[OpenCLI](https://github.com/jackwener/opencli) · [twitter-cli](https://github.com/public-clis/twitter-cli) · [rdt-cli](https://github.com/public-clis/rdt-cli) · [xiaohongshu-mcp](https://github.com/xpzouying/xiaohongshu-mcp) · [xhs-cli](https://github.com/jackwener/xiaohongshu-cli) · [bili-cli](https://github.com/public-clis/bilibili-cli) · [yt-dlp](https://github.com/yt-dlp/yt-dlp) · [Jina Reader](https://github.com/jina-ai/reader) · [Exa](https://exa.ai) · [mcporter](https://github.com/nicobailon/mcporter) · [feedparser](https://github.com/kurtmckee/feedparser) · [mcp-server-linkedin](https://github.com/stickerdaniel/linkedin-mcp-server)

## Contact

- 📧 **Email:** jetbrathna@gmail.com
- 🐛 Bugs and feature requests: open an issue on the PyPI project page or email

## License

[MIT](LICENSE)
