Metadata-Version: 2.4
Name: langchain-trawl
Version: 0.1.0
Summary: LangChain integration for the Trawl unified content API — transcripts from 10+ sources
License: MIT
License-File: LICENSE
Keywords: langchain,trawl,transcripts,youtube,podcasts,rag
Author: Trawl
Author-email: support@gettrawl.com
Requires-Python: >=3.10,<4.0
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Requires-Dist: langchain-core (>=0.3.0,<1.0.0)
Requires-Dist: trawl-sdk (>=0.1.3,<1.0.0)
Project-URL: Homepage, https://gettrawl.com
Project-URL: Repository, https://github.com/gettrawl/langchain-trawl
Description-Content-Type: text/markdown

# langchain-trawl

LangChain integration for the [Trawl](https://gettrawl.com) unified content API. Extract transcripts and text from 10+ sources as LangChain `Document` objects, ready for RAG pipelines, summarization chains, and more.

## Installation

```bash
pip install langchain-trawl
```

This installs `langchain-core` and `trawl-sdk` as dependencies.

## Quick Start

```python
from langchain_trawl import TrawlLoader

loader = TrawlLoader(
    urls=["https://youtube.com/watch?v=dQw4w9WgXcQ"],
    api_key="trawl_...",  # or set TRAWL_API_KEY env var
)
docs = loader.load()

print(docs[0].page_content[:200])
print(docs[0].metadata)
# {'video_id': 'dQw4w9WgXcQ', 'title': '...', 'channel': '...', ...}
```

## Supported Sources

| Source | Example URL |
|---|---|
| YouTube | `https://youtube.com/watch?v=...` |
| Podcasts | `https://podcasts.apple.com/...` or RSS feed URL |
| TikTok | `https://tiktok.com/@user/video/...` |
| Instagram Reels | `https://instagram.com/reel/...` |
| Twitter/X Spaces | `https://twitter.com/i/spaces/...` |
| Reddit | `https://reddit.com/r/sub/comments/...` |
| News Articles | Any news article URL |
| Earnings Calls | Ticker symbol via Trawl API |
| SEC Filings | EDGAR filing URLs |
| Academic Papers | ArXiv or DOI URLs |

## Parameters

| Parameter | Type | Default | Description |
|---|---|---|---|
| `urls` | `list[str]` | required | URLs to extract content from |
| `api_key` | `str \| None` | `None` | Trawl API key (falls back to `TRAWL_API_KEY` env var) |
| `language` | `str \| None` | `None` | Language code filter (e.g. `"en"`, `"es"`) |
| `base_url` | `str` | `"https://api.gettrawl.com"` | API base URL |

## Document Metadata

Each loaded document includes these metadata fields:

- `video_id` -- Source content identifier
- `title` -- Content title
- `channel` -- Channel or author name
- `language` -- Detected language code
- `is_auto_generated` -- Whether the transcript was auto-generated
- `segment_count` -- Number of transcript segments
- `source` -- Original URL

If extraction fails, the document will have empty `page_content` and an `error` key in metadata.

## Use in RAG Pipelines

```python
from langchain_trawl import TrawlLoader
from langchain_openai import OpenAIEmbeddings
from langchain_community.vectorstores import FAISS
from langchain.text_splitter import RecursiveCharacterTextSplitter

# Load transcripts
loader = TrawlLoader(
    urls=[
        "https://youtube.com/watch?v=video1",
        "https://youtube.com/watch?v=video2",
    ],
)
docs = loader.load()

# Split and embed
splitter = RecursiveCharacterTextSplitter(chunk_size=1000, chunk_overlap=100)
chunks = splitter.split_documents(docs)
vectorstore = FAISS.from_documents(chunks, OpenAIEmbeddings())

# Query
results = vectorstore.similarity_search("your question here")
```

## Development

```bash
# Install dependencies
poetry install

# Run tests
make test

# Lint and format
make lint
make format
```

## License

MIT

