Metadata-Version: 2.5
Name: langchain-socialscrape
Version: 0.1.0
Summary: LangChain tools and document loader for SocialScrape: profiles, posts, comments, search and video transcripts from social platforms as JSON.
Project-URL: Homepage, https://socialscrape.dev/?utm_source=pypi&utm_medium=package&utm_campaign=langchain-socialscrape
Project-URL: Documentation, https://socialscrape.dev/docs/langchain?utm_source=pypi&utm_medium=package&utm_campaign=langchain-socialscrape
Project-URL: Repository, https://github.com/SocialScraper/langchain-socialscrape
Author-email: SocialScrape <hello@socialscrape.dev>
License-Expression: MIT
License-File: LICENSE
Keywords: agents,document-loader,instagram,langchain,rag,social media,tiktok,toolkit,tools,transcript,youtube
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Typing :: Typed
Requires-Python: >=3.11
Requires-Dist: langchain-core<2,>=0.3.0
Requires-Dist: socialscrape-sdk<1,>=0.1.2
Provides-Extra: test
Requires-Dist: langchain-tests>=0.3; extra == 'test'
Requires-Dist: pytest>=8; extra == 'test'
Description-Content-Type: text/markdown

# langchain-socialscrape

LangChain tools and a document loader for [SocialScrape](https://socialscrape.dev/?utm_source=github&utm_medium=integration&utm_campaign=langchain-socialscrape), a social media data API: profiles, posts, comments, search results and video transcripts from YouTube, TikTok, Instagram, X, Facebook, LinkedIn, Reddit and more, as JSON in one response shape, with one API key.

```bash
pip install -U langchain-socialscrape
export SOCIALSCRAPE_API_KEY="ss_live_..."   # get a key: https://socialscrape.dev/sign-up
```

Python 3.11 or newer. The package builds on the official [`socialscrape-sdk`](https://pypi.org/project/socialscrape-sdk/) client, so retries, timeouts and long transcription jobs are handled for you.

## Give an agent the tools

```python
from langchain.agents import create_agent
from langchain_socialscrape import SocialScrapeToolkit

tools = SocialScrapeToolkit().get_tools()   # reads SOCIALSCRAPE_API_KEY
agent = create_agent("openai:gpt-5", tools=tools)

result = agent.invoke({"messages": [
    ("user", "What did NASA post on YouTube this week, and what are people saying in the comments?")
]})
print(result["messages"][-1].content)
```

`get_tools()` returns eight tools that share one client:

| Tool | What it does |
|---|---|
| `socialscrape_get_profile` | A profile, channel or artist: name, bio, links, verification, follower counts. 17 platforms. |
| `socialscrape_list_posts` | The latest posts or videos of a profile, channel or page, one page per call. 14 platforms. |
| `socialscrape_get_post` | One post, video, tweet, pin or clip from its URL. 18 platforms. |
| `socialscrape_list_comments` | The comments on a post or video, one page per call. 7 platforms. |
| `socialscrape_get_transcript` | The transcript of a video from its URL: captions when the video has them, AI transcription of the audio when it has none. |
| `socialscrape_search` | Keyword search on one platform, one page per call. 14 platforms. |
| `socialscrape_list_endpoints` | What else exists: every endpoint id with its parameters and price. Makes no request, so it is free. |
| `socialscrape_call_endpoint` | Runs any of the 193 endpoints by id: ad libraries, followers, hashtags, playlists, music, shops and the rest. |

The platform is an argument (`{"platform": "tiktok", "handle": "nasa"}`), and for a post, its comments or a transcript it is read from the link. Tools that list things return a `cursor` to pass back for the next page.

## Use one tool directly

```python
from langchain_socialscrape import SocialScrapeProfileTool, SocialScrapeTranscriptTool

SocialScrapeProfileTool().invoke({"platform": "tiktok", "handle": "nasa"})
# '{"data": {"object": "profile", "platform": "tiktok", "handle": "nasa", "name": "NASA", ...,
#   "stats": {"followers": ..., "following": ..., "posts": ..., "likes": ...}},
#   "credits_charged": 1, "credits_remaining": 4999}'

SocialScrapeTranscriptTool().invoke({"url": "https://www.youtube.com/watch?v=dQw4w9WgXcQ"})
# '{"data": {"object": "transcript", "language": "en", "source": "captions", "text": "..."}, ...}'
```

Every tool answers with a JSON string: `data` (the object, or a list page with `items`, `cursor` and `has_more`), `credits_charged` and `credits_remaining`.

Any endpoint, by id:

```python
from langchain_socialscrape import SocialScrapeCallEndpointTool, SocialScrapeEndpointsTool

SocialScrapeEndpointsTool().invoke({"search": "ad library"})        # which endpoints, what they take
SocialScrapeCallEndpointTool().invoke({
    "endpoint": "facebook.ad_library.search",
    "params": {"query": "running shoes"},
})
```

Parameters are checked against the endpoint before the request is made, so a misspelt name is answered at once and costs nothing.

### Errors

A request the API refuses (no such account, a private post, a video with no transcript) is free. By default the tool returns the error as its answer, for example `not_found: No TikTok account with that handle. | HTTP 404 | request req_...`, so an agent can read it and try something else. Pass `handle_tool_error=False` to have the tool raise `ToolException` instead. A missing API key always raises.

### Options

Each tool, and the toolkit, takes:

- `api_key`: the SocialScrape key. Left out, it is read from `SOCIALSCRAPE_API_KEY`.
- `max_age_days`: accept a cached result up to this many days old (0 to 30). A cached result costs 0 credits.
- `trim` (default `True`): ask for the smaller response, without null fields and the `extra` object, which saves context.
- `client` / `async_client`: a `socialscrape.SocialScrape` or `AsyncSocialScrape` you configured yourself (timeouts, retries, your own `httpx` client).
- `base_url`: API base URL override.

The tools work with `invoke` and `ainvoke`.

## Load documents for RAG

`SocialScrapeLoader` turns the result of any endpoint into LangChain `Document`s.

```python
from langchain_socialscrape import SocialScrapeLoader

# One Document per comment, three pages of them
docs = SocialScrapeLoader(
    "youtube.video.comments",
    {"url": "https://www.youtube.com/watch?v=dQw4w9WgXcQ"},
    max_pages=3,
).load()

docs[0].page_content   # the comment text
docs[0].metadata       # {'source': 'https://www.youtube.com/watch?v=...&lc=...', 'author': '...',
                       #  'published_at': '...', 'likes': 12, 'platform': 'youtube', ...}

# A transcript as one Document
transcript = SocialScrapeLoader("transcript", {"url": "https://www.tiktok.com/@nasa/video/7300000000000000000"}).load()

# A subreddit search, ready for a vector store
posts = SocialScrapeLoader("reddit.search", {"query": "mechanical keyboards", "sort": "top"}, max_items=50, max_pages=3).load()
```

- A list endpoint gives one Document per item and is followed across pages up to `max_pages` (default 1; each page is one request). `max_items` stops early.
- Any other endpoint (a profile, one post, a transcript) gives one Document.
- The page content is the item's text: a post's title and text, a comment's text, a transcript's text, a profile's name and bio. An object with no text of its own is kept whole as JSON.
- The item's other scalar fields, its counts (`views`, `likes`, `comments`, ...) and its author go to `metadata`, flat, with the item's URL as `source`.
- `lazy_load()` yields Documents as each page arrives.

The endpoint ids and their parameters are in the [API reference](https://socialscrape.dev/docs/api?utm_source=github&utm_medium=integration&utm_campaign=langchain-socialscrape), and in code:

```python
from langchain_socialscrape import endpoints

endpoints()["tiktok.profile.videos"].signature()   # 'handle | id (one needed), sort, include_pinned, cursor, region'
```

## Billing

Most requests cost 1 credit: a profile, a post, a page of a list or of search results, a transcript from captions. AI transcription of a video with no captions costs 1 credit up to 2 minutes of audio; longer audio is transcribed only when you pass `allow_long_transcription`, at 5 credits per started minute of the whole video. A failed request and a cached repeat cost 0. A new account starts with 100 free credits, no card needed. See [pricing](https://socialscrape.dev/pricing?utm_source=github&utm_medium=integration&utm_campaign=langchain-socialscrape).

## Tests

```bash
pip install -e ".[test]"
pytest
```

The tests run offline against sample responses from the API reference, and include LangChain's standard unit tests for tools.

## License

MIT
