Metadata-Version: 2.5
Name: comment-scraper-haystack
Version: 0.1.0
Summary: Haystack converter for Reddit and Hacker News threads exported with Comment Scraper: one Document per comment, [cN] numbers kept. Reads local JSON files, makes no network requests.
Project-URL: Homepage, https://comment-scraper.com/
Project-URL: Documentation, https://comment-scraper.com/haystack
Project-URL: Repository, https://github.com/supertj/comment-scraper-haystack
Project-URL: Issues, https://github.com/supertj/comment-scraper-haystack/issues
Author-email: Super Lede <hello@comment-scraper.com>
License-Expression: MIT
License-File: LICENSE
Keywords: comment-scraper,converter,hacker-news,haystack,rag,reddit
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Text Processing
Requires-Python: >=3.10
Requires-Dist: haystack-ai<4.0.0,>=2.9.0
Description-Content-Type: text/markdown

# comment-scraper-haystack

A Haystack converter for Reddit and Hacker News threads you exported with [Comment Scraper](https://comment-scraper.com/). `CommentScraperConverter` turns each comment into one Haystack `Document`. Comments keep the numbers the extension's Markdown export gives them, so `[c12]` in a model's answer is comment 12 in your file.

The converter reads JSON files on your disk. It makes no network requests, Reddit and Hacker News included.

## Install

```sh
pip install comment-scraper-haystack
```

You need Python 3.10 or newer and Haystack 2.9 or newer (`haystack-ai`). Haystack 3 works too.

## Export a thread

1. Install the Comment Scraper extension for Chrome or Edge from [comment-scraper.com](https://comment-scraper.com/). Exports are free and don't need an account.
2. Open a Reddit or Hacker News thread and open the extension's side panel.
3. Pick **JSON, one row per comment** or **JSON, replies nested**, then export and download the file. CSV and Markdown exports can't be converted.

## Convert it

```python
from haystack_integrations.components.converters.comment_scraper import CommentScraperConverter

converter = CommentScraperConverter()
result = converter.run(sources=["which-crm-do-small-teams-actually-stick-with.json"])

for document in result["documents"][1:4]:
    meta = document.meta
    print(meta["comment_number"], meta["reply_to_number"], document.content)
```

```text
1 None We left our CRM after the renewal doubled.
2 1 Ask for the year-two price in writing.
3 2 The export button was the first thing they hid.
```

That output comes from `tests/fixtures/which-crm.json`, a made-up thread used in the tests. The first Document is the thread's post, so the comments start at index 1. Comment 2 replies to comment 1, which the Markdown export shows as `[c2] reply to [c1]`.

`sources` takes file paths or `ByteStream` objects. A batch export holds several threads in one file, and you get Documents for all of them, thread by thread.

`run()` also takes `meta`, like Haystack's own converters: one dict for every Document, or a list with one dict per source. The keys are added to each Document's meta. If one of your keys has the same name as a comment field below, the comment's value wins.

Three options when you create the converter:

- `include_post=True` (the default) gives the thread's post first, as its own Document. Its content is the title, a blank line, then the post body.
- `include_deleted=False` (the default) skips deleted and removed comments, because they have no text. They keep their numbers, so the numbering still matches the Markdown export. Set it to `True` to get them with `(deleted)` or `(removed)` as their content.
- `store_full_path=False` (the default) puts only the file name in `meta["file_path"]`. Set it to `True` for the full path.

## Write the comments to a document store

```python
from haystack import Pipeline
from haystack.components.retrievers.in_memory import InMemoryBM25Retriever
from haystack.components.writers import DocumentWriter
from haystack.document_stores.in_memory import InMemoryDocumentStore
from haystack.document_stores.types import DuplicatePolicy
from haystack_integrations.components.converters.comment_scraper import CommentScraperConverter

store = InMemoryDocumentStore()

indexing = Pipeline()
indexing.add_component("converter", CommentScraperConverter())
indexing.add_component(
    "writer", DocumentWriter(document_store=store, policy=DuplicatePolicy.OVERWRITE)
)
indexing.connect("converter.documents", "writer.documents")
indexing.run({"converter": {"sources": ["which-crm-do-small-teams-actually-stick-with.json"]}})
print(store.count_documents())

retriever = InMemoryBM25Retriever(document_store=store, top_k=3)
only_comments = {"field": "meta.record_type", "operator": "==", "value": "comment"}
for document in retriever.run(query="renewal price", filters=only_comments)["documents"]:
    print(f"[c{document.meta['comment_number']}] {document.content}")
```

```text
7
[c2] Ask for the year-two price in writing.
[c1] We left our CRM after the renewal doubled.
[c8] Same here, the renewal email came with no warning.
```

Seven Documents: the post and the six comments that have text. To use another document store, pass it to `DocumentWriter` and check what meta values it accepts (see the note on `None` below). For embedding retrieval, put an embedder between the converter and the writer.

Each Document's `id` is `site:thread_id:comment_id`, for example `reddit:crm1:a3`, and `site:thread_id` for the post. Export the thread again later, write it with `DuplicatePolicy.OVERWRITE`, and its comments are replaced, not added twice. With the default options, a comment deleted since your last import, or one the new export doesn't include, keeps its old Document. To refresh a thread completely, convert the new export first. If it gives Documents for the thread, delete the thread's old ones, filtering on `meta.site` and `meta.thread_id`, and write the new ones. Check the fetch counts at the top of the new file before you do: comments a partial export lacks are gone afterwards.

## What each Document holds

`content` is the comment text as the export has it: Reddit's own Markdown when the extension read Reddit's JSON, and plain text for Hacker News or when it had to read the Reddit page instead.

| Meta key | What it holds |
|---|---|
| `file_path` | Name of the export file (the full path with `store_full_path=True`); missing for a `ByteStream` without one |
| `site` | `reddit` or `hn` |
| `thread_id`, `thread_title` | The thread's ID and title |
| `thread_url` | Link to the thread |
| `subreddit` | Subreddit name; `None` for Hacker News |
| `fetched_at` | When the extension fetched the thread, ISO 8601 in UTC |
| `record_type` | `post` or `comment` |
| `comment_id` | The site's comment ID, as text; `None` for the post |
| `comment_number` | N in `[cN]`, the number the Markdown export shows; `None` for the post. Numbering starts again in each thread of a batch export |
| `parent_id` | ID of the comment it replies to; the thread ID for top-level comments |
| `reply_to_number` | Number of the comment it replies to; `None` for top-level comments and when the parent wasn't fetched |
| `parent_missing` | `True` when the parent comment wasn't fetched |
| `depth` | 0 for top-level comments, 1 for their replies, and so on |
| `site_score` | Score as shown on the site; `None` when hidden. Hacker News doesn't show comment scores |
| `score_hidden` | `True` when Reddit hides the score |
| `created_utc`, `edited_utc` | When it was posted and last edited, ISO 8601 in UTC; `edited_utc` is `None` if never edited |
| `is_op` | `True` when the person who started the thread wrote it |
| `comment_state` | `normal`, `deleted` or `removed` |
| `permalink` | Link to the comment |

Exports don't contain usernames, so there's no author field. Names someone typed inside a comment are kept as written.

The score is `site_score`, not `score`, because `score` is the `Document` field retrievers fill with relevance. Haystack 2.x and 3.0 put meta keys and `Document` fields in one flat dict when a document store saves or returns a Document, so a meta key called `score` would be lost.

All values are strings, numbers, booleans or `None`, and nothing is nested, so you can filter on any of them, as `meta.record_type` above. If your document store rejects `None` values, drop those keys before you write the Documents.

## When a source can't be converted

Like Haystack's own converters, the converter logs a warning and skips the source, then carries on with the rest. The warning names the source and says why:

- the file can't be read
- it isn't a Comment Scraper JSON export (a CSV or Markdown export, say)
- a thread in it is damaged, for example cut off part way, or with a field removed or changed to the wrong type by hand; the warning says which thread and which field
- a nested export's replies go deeper than Python's JSON parser allows; export the thread as "JSON, one row per comment" instead

An export can be partial: Reddit may have rate-limited the extension, or you stopped it early. The file header records how many comments were fetched and how many the site reports. The converter passes on what's in the file and doesn't fill gaps.

## Privacy

The converter reads the files and `ByteStream`s you pass to it and nothing else. It doesn't write files and doesn't connect to the internet; the tests run with network access disabled to check that.

Haystack itself sends anonymous usage telemetry when a pipeline runs. Set `HAYSTACK_TELEMETRY_ENABLED=False` before importing Haystack to turn it off. If you send the comments to a hosted model or a hosted document store, they go to that provider under its terms.

## Development

```sh
uv sync
make test   # tests, with network access disabled
make lint   # ruff and mypy
```

The tests include indexing and retrieval pipelines, and they run the Python examples in this README and check their output.

The fixtures in `tests/fixtures` are made-up threads written by the extension's own export code, together with the `[cN]` line the Markdown export prints for each comment. `make fixtures` rebuilds them; it needs a checkout of [comment-scraper-mcp](https://github.com/supertj/comment-scraper-mcp) next to this repository, with its dependencies installed.

To check real exports, which aren't committed, run `CS_REAL_EXPORTS_DIR=/path/to/exports uv run pytest tests/test_real_exports.py`. If a thread's `.md` export sits next to its `.json`, the comment numbers are compared line by line.

## Support

Open an issue on GitHub or email hello@comment-scraper.com.

## License

MIT. Comment Scraper isn't affiliated with Reddit, Inc. or Y Combinator.
