Skip to content

Dataset Management

QuranKit uses a dataset registry system for managing Qur'anic text, morphology, and lexical datasets with full provenance tracking.

Dataset Registry

from qurankit import Quran

quran = Quran.load()
registry = quran.registry

# List all registered datasets
for dataset_id, info in registry.list_datasets().items():
    print(f"{dataset_id}: {info.name} v{info.version}")
    print(f"  Status: {info.status}")
    print(f"  Source: {info.source}")
    print(f"  License: {info.license}")

Installing Datasets

Tanzil Text Editions (CC-BY-3.0)

# Simple edition (clean text)
qurankit dataset install tanzil --edition simple --version 1.0.0

# Uthmani edition (with Quranic orthography)
qurankit dataset install tanzil --edition uthmani --version 1.0.0

# Specific version
qurankit dataset install tanzil --edition simple --version 1.0.1

QAC Morphology (GPL + Non-commercial FAQ)

qurankit dataset install qac --version 2.0

Custom Dataset

# Install from local path
qurankit dataset install /path/to/dataset --name custom-text

# Install from URL
qurankit dataset install https://example.com/dataset.zip --name custom-text

Dataset Manifest

Every dataset has a manifest with provenance metadata:

manifest = registry.get_manifest("tanzil", "simple", "1.0.0")

print(f"Dataset ID: {manifest.dataset_id}")
print(f"Name: {manifest.name}")
print(f"Version: {manifest.version}")
print(f"Source: {manifest.source}")
print(f"Source URL: {manifest.source_url}")
print(f"License: {manifest.license}")
print(f"License URL: {manifest.license_url}")
print(f"Redistribution: {manifest.redistribution_status}")
print(f"Checksum: {manifest.checksum}")
print(f"Retrieved: {manifest.retrieved_date}")
print(f"Imported: {manifest.imported_date}")
print(f"Surah count: {manifest.surah_count}")
print(f"Ayah count: {manifest.ayah_count}")
print(f"Word count: {manifest.word_count}")

Verifying Dataset Integrity

# Verify checksums
results = registry.verify_checksums("tanzil", "simple", "1.0.0")
for file_path, verified in results.items():
    status = "✓" if verified else "✗"
    print(f"{status} {file_path}")

# Full validation
validation = registry.validate_dataset("tanzil", "simple", "1.0.0")
print(f"Valid: {validation.is_valid}")
for error in validation.errors:
    print(f"ERROR: {error}")
for warning in validation.warnings:
    print(f"WARNING: {warning}")

Managing Multiple Versions

# List versions for a dataset
versions = registry.list_versions("tanzil", "simple")
for v in versions:
    print(f"v{v.version}: {v.status} ({v.imported_date})")

# Set default version
registry.set_default_version("tanzil", "simple", "1.0.1")

# Remove old version
registry.remove_version("tanzil", "simple", "1.0.0")

Dataset Status

Status Meaning
available Registered but not downloaded
downloading Download in progress
installed Downloaded and verified
imported Imported into database
active Currently loaded edition
error Installation/import failed
removed Uninstalled

Data Directory Structure

~/.local/share/qurankit/
├── config.toml
├── registry.json
├── datasets/
│   ├── tanzil/
│   │   ├── simple/
│   │   │   ├── 1.0.0/
│   │   │   │   ├── manifest.json
│   │   │   │   ├── quran-simple.txt
│   │   │   │   └── metadata.json
│   │   │   └── latest -> 1.0.0
│   │   └── uthmani/
│   │       └── ...
│   └── qac/
│       └── 2.0/
│           ├── manifest.json
│           ├── morphology.xml
│           └── latest -> 2.0
└── cache/
    ├── search_index.db
    └── vocabulary.db

Offline Dataset Preparation

For air-gapped environments:

# On machine with internet
qurankit dataset download tanzil --edition simple --version 1.0.0 \
    --output /tmp/qurankit-datasets/

# Transfer /tmp/qurankit-datasets/ to offline machine

# On offline machine
qurankit dataset install /tmp/qurankit-datasets/tanzil-simple-1.0.0

Custom Dataset Import

from qurankit.datasets import DatasetManifest, DatasetValidator
from qurankit.datasets.import_pipeline import ImportPipeline

# Create manifest for custom dataset
manifest = DatasetManifest(
    dataset_id="custom-text",
    name="Custom Qur'an Text",
    version="1.0.0",
    source="My Organization",
    source_url="https://example.com/dataset",
    license="CC-BY-4.0",
    license_url="https://creativecommons.org/licenses/by/4.0/",
    redistribution_status="allowed",
    surah_count=114,
    ayah_count=6236,
    word_count=77430,
    checksum="sha256:...",
    description="Custom text edition for research"
)

# Validate
validator = DatasetValidator()
validation = validator.validate(manifest, dataset_path)
if validation.is_valid:
    # Import
    pipeline = ImportPipeline(quran.config)
    pipeline.import_text_dataset(manifest, dataset_path)
else:
    for error in validation.errors:
        print(f"Validation error: {error}")

Licensing Compliance

QuranKit tracks licensing for all datasets:

# Check license compatibility
from qurankit.datasets.manifest import RedistributionStatus

for dataset_id, info in registry.list_datasets().items():
    if info.redistribution_status == RedistributionStatus.ALLOWED:
        print(f"{dataset_id}: Can be redistributed")
    elif info.redistribution_status == RedistributionStatus.RESTRICTED:
        print(f"{dataset_id}: Restricted redistribution")
    elif info.redistribution_status == RedistributionStatus.UNKNOWN:
        print(f"{dataset_id}: License unclear - manual review needed")

Dataset Licensing

  • Tanzil: CC-BY-3.0 — Requires attribution, verbatim text only
  • QAC: GPL + FAQ non-commercial — Optional adapter, no redistribution
  • OpenITI: CC-BY-NC-SA — Non-commercial only
  • Always verify current license terms before redistribution

CLI Dataset Commands

# List installed datasets
qurankit dataset list

# List available datasets (from registry)
qurankit dataset list --available

# Install dataset
qurankit dataset install tanzil --edition uthmani

# Show dataset info
qurankit dataset info tanzil simple 1.0.0

# Verify checksums
qurankit dataset verify tanzil simple 1.0.0

# Validate dataset
qurankit dataset validate tanzil simple 1.0.0

# Remove dataset
qurankit dataset remove tanzil simple 1.0.0

# Set default version
qurankit dataset set-default tanzil simple 1.0.1

# Export manifest
qurankit dataset export-manifest tanzil simple 1.0.0 > manifest.json

Next Steps