Source: Salesforce WikiText-2 raw, test split.
File: test.parquet
Downloaded from:
https://huggingface.co/datasets/Salesforce/wikitext/resolve/main/wikitext-2-raw-v1/test-00000-of-00001.parquet
Dataset card: https://huggingface.co/datasets/Salesforce/wikitext

License: CC BY-SA 3.0 and GFDL, as published on that dataset card.

What the file is: English text from Wikipedia articles that Wikipedia editors
marked Good or Featured. The test split is 62 of those articles, stored as
whitespace-separated tokens. None of the titles are in the validation split.
The ingest pipeline restores hyphen, punctuation, and contraction markers,
then saves every paragraph as its own VectorPrism chunk. Document ids use the
prefix wikitext2test- so a validation replace, which deletes wikitext2-%,
does not remove this split.

What it is not: a complete encyclopedia, a neutral sample of all subjects, or
a bias-free source. Good and Featured is an editorial filter. The articles are
English only. Nothing in this folder was rewritten to sound neutral. The extra
articles give the creative walk more stored paragraphs when retrieval names
one of them. They do not train a decoder.
