QuranKit Research Workflows¶
This notebook demonstrates computational research workflows using QuranKit v0.3.0+.
Prerequisites¶
- QuranKit installed (
pip install qurankit-core) - Tanzil dataset installed (
qurankit dataset install tanzil) - QAC morphology dataset installed (
qurankit dataset install qac) for vocabulary features
Claim Categories¶
Every result in QuranKit is tagged with a claim category per AGENTS.md:
| Category | Description | Examples |
|---|---|---|
source_text |
Original Qur'anic text | Ayah text, surah names |
source_annotation |
Imported scholarly data | Tafsir, morphology, dictionary senses |
deterministic_result |
Exact computation | Exact phrase match, n-grams |
computational_suggestion |
Statistical/algorithmic | PMI collocations, near-duplicates |
research_hypothesis |
Exploratory findings | Wujuh wa Nazair candidates |
Always check claim categories — Don't present computational suggestions as facts.
# Setup
from qurankit import Quran
from qurankit.analysis import (
MutashabihatEngine,
SemanticRelationExtractor,
CollocationAnalyzer,
CooccurrenceMeasure,
CrossReferenceAnalyzer,
WujuhWaNazairAnalyzer,
ReportGenerator,
MatchMethod,
SemanticRelationType,
)
from qurankit.models.research import ClaimCategory
import json
# Load Quran with default edition (tanzil-uthmani-v1)
quran = Quran.load()
print(f"Loaded edition: {quran.current_edition.id}")
print(f"Available editions: {[e.id for e in quran.available_editions]}")
# Check if vocabulary is available (requires QAC)
vocab = quran.vocabulary
print(f"Vocabulary available: {vocab is not None}")
if vocab:
print(f"Vocabulary stats: {vocab.get_vocabulary_stats()}")
1. Mutashabihat Analysis (Repeated Phrase Detection)¶
Find exact and near-duplicate phrases across the Qur'an.
engine = MutashabihatEngine()
# Exact phrase matches for Ayat al-Kursi (2:255)
result = engine.find_exact_phrase("2:255", min_length=2)
print(f"Exact phrase matches for 2:255: {len(result.matches)} matches")
for match in result.matches[:5]:
print(f" {match.reference}: {match.matched_text}")
# Normalized matches (diacritics-insensitive)
result = engine.find_normalized_matches(
"2:255",
normalization_profile="diacritics-insensitive-v1",
min_words=2
)
print(f"\nNormalized matches for 2:255: {len(result.matches)} matches")
for match in result.matches[:5]:
print(f" {match.reference}: {match.matched_text} [{match.match_type.value}]")
# Near-duplicate detection (computational suggestion)
result = engine.find_near_duplicates(
"2:255",
threshold=0.7,
normalization_profile="search-broad-v1"
)
print(f"\nNear duplicates for 2:255: {len(result.matches)} matches")
for match in result.matches[:5]:
print(f" {match.reference}: {match.matched_text} (sim: {match.similarity_score:.3f}) [{match.match_type.value}]")
# Recurring n-grams (3-grams appearing 2+ times)
ngrams = engine.find_exact_ngrams(n=3, min_occurrences=2)
print(f"\nRecurring 3-grams (min 2 occurrences): {len(ngrams)} found")
for ng in ngrams[:10]:
print(f" {ng.text} — {ng.count} occurrences")
for ref in ng.references[:3]:
print(f" {ref}")
if len(ng.references) > 3:
print(f" ... and {len(ng.references) - 3} more")
2. Vocabulary & Frequency Analysis¶
Requires QAC morphology dataset for root/lemma information.
if vocab:
# Root frequency list
roots = vocab.get_root_frequency_list(min_occurrences=10, limit=20)
print("Top 20 roots by frequency:")
for r in roots:
print(f" {r['root_form']}: {r['frequency']} (surahs: {r['surah_count']})")
# Lemma frequency list
lemmas = vocab.get_lemma_frequency_list(min_occurrences=5, limit=20)
print("\nTop 20 lemmas by frequency:")
for l in lemmas:
print(f" {l['lemma_form']}: {l['frequency']} (surahs: {l['surah_count']})")
# Vocabulary statistics for a specific root
stats = vocab.get_vocabulary_stats(root_form="ر ح م")
print(f"\nRoot ر ح م stats:")
print(f" Total occurrences: {stats.total_occurrences}")
print(f" Unique surahs: {stats.unique_surahs}")
print(f" Unique lemmas: {stats.unique_lemmas}")
print(f" Juz distribution: {stats.juz_distribution}")
# Concordance
concordance = vocab.get_concordance(root_form="ر ح م", context_words=3, limit=10)
print("\nConcordance for ر ح م:")
for line in concordance:
print(f" {line['reference']}: ...{line['target_form']}...")
# Get occurrences by root/lemma/surface
occurrences = vocab.get_occurrences_by_root("ر ح م")
print(f"\nOccurrences of root ر ح م: {len(occurrences)}")
for occ in occurrences[:5]:
print(f" {occ['reference']}: {occ['surface_form']} (lemma: {occ['lemma_form']})")
else:
print("Vocabulary not available - install QAC dataset for morphology features")
3. Collocation Analysis¶
Find statistically significant word co-occurrences using PMI, Dice coefficient, log-likelihood, or raw frequency.
if vocab:
analyzer = CollocationAnalyzer(vocabulary=vocab, edition_id="tanzil-uthmani-v1")
# Corpus-wide collocations using PMI
results = analyzer.compute_collocations(
window_size=5,
min_cooccurrence=3,
measure=CooccurrenceMeasure.PMI,
top_n=20,
filter_stopwords=True,
)
print("Top 20 corpus-wide collocations (PMI):")
for r in results:
print(f" {r.word1} — {r.word2}: PMI={r.score:.4f} (co-occur: {r.cooccurrence_count})")
# Collocations for a specific word using Dice coefficient
word_colloc = analyzer.get_collocations_for_word(
"الْحَمْدُ",
measure=CooccurrenceMeasure.DICE,
top_n=15
)
print("\nCollocations for الْحَمْدُ (Dice):")
for r in word_colloc:
print(f" {r.word1} — {r.word2}: Dice={r.score:.4f} (co-occur: {r.cooccurrence_count})")
# Compare measures
for measure in [CooccurrenceMeasure.PMI, CooccurrenceMeasure.DICE, CooccurrenceMeasure.LOG_LIKELIHOOD]:
results = analyzer.compute_collocations(
window_size=5,
min_cooccurrence=3,
measure=measure,
top_n=10,
filter_stopwords=True,
)
print(f"\nTop 10 by {measure.value}:")
for r in results[:5]:
print(f" {r.word1} — {r.word2}: {r.score:.4f}")
else:
print("Vocabulary not available - install QAC dataset for collocation analysis")
4. Cross-Reference Analysis¶
Find explicit (dataset-annotated) or implicit (computed) links between ayahs.
analyzer = CrossReferenceAnalyzer(edition_id="tanzil-uthmani-v1")
# Implicit links from shared vocabulary (QAC morphology)
links = analyzer.find_implicit_links(
"2:255", # Ayat al-Kursi
min_shared_roots=2,
min_shared_lemmas=1,
limit=15
)
print(f"Implicit cross-references for 2:255: {len(links)} found")
for link in links:
print(f" {link.source_reference} → {link.target_reference}")
print(f" Evidence: {link.evidence}")
print(f" Confidence: {link.confidence:.2f}")
print(f" Claim: {link.claim_category.value}")
# Explicit cross-references (requires annotated dataset)
explicit = analyzer.get_explicit_cross_references("2:255")
print(f"\nExplicit cross-references for 2:255: {len(explicit)} found")
5. Wujuh wa Nazair Detection¶
Find candidate verse pairs with similar wording but potentially different meanings (computational approximation for scholarly review).
Important: These are research_hypothesis — they need expert scholarly review.
analyzer = WujuhWaNazairAnalyzer(edition_id="tanzil-uthmani-v1")
# For a specific ayah
candidates = analyzer.find_for_reference(
"2:255",
min_shared_words=2,
max_candidates=15
)
print(f"Wujuh wa Nazair candidates for 2:255: {len(candidates)} found")
for pair in candidates:
print(f" {pair.ayah1_reference} ~ {pair.ayah2_reference}")
print(f" Similarity: {pair.similarity_score:.3f}")
print(f" Shared words: {', '.join(pair.shared_words[:8])}")
print(f" Notes: {pair.difference_notes}")
print(f" Claim: {pair.claim_category.value}")
# Corpus-wide candidates
all_candidates = analyzer.find_candidates(
min_shared_words=3,
max_candidates=50,
similarity_threshold=0.4
)
print(f"\nCorpus-wide Wujuh wa Nazair candidates: {len(all_candidates)} found")
for pair in all_candidates[:5]:
print(f" {pair.ayah1_reference} ~ {pair.ayah2_reference} (sim: {pair.similarity_score:.3f})")
6. Semantic Relation Lookup¶
Find synonyms, antonyms, and other semantic relations from dictionary sources (Lane's Lexicon, Al-Mufradat, etc.).
Note: Requires dictionary adapter with semantic relation data loaded.
extractor = SemanticRelationExtractor()
# Find synonyms for a root
synonyms = extractor.find_synonyms(root_form="ر ح م")
print(f"Synonyms for root ر ح م: {len(synonyms)} found")
for rel in synonyms:
print(f" {rel.subject_headword} → {rel.object_headword} ({rel.relation_type.value})")
print(f" Source: {rel.source_resource}")
print(f" Claim: {rel.claim_category.value}")
# Find antonyms for a lemma
antonyms = extractor.find_antonyms(lemma_form="رحمة")
print(f"\nAntonyms for lemma رحمة: {len(antonyms)} found")
for rel in antonyms:
print(f" {rel.subject_headword} → {rel.object_headword} ({rel.relation_type.value})")
# Find all relations for a root
relations = extractor.find_relations_for_root(
"ر ح م",
relation_types=[
SemanticRelationType.SYNONYM,
SemanticRelationType.ANTONYM,
SemanticRelationType.RELATED
]
)
print(f"\nAll relations for ر ح م: {len(relations)} found")
for rel in relations:
print(f" {rel.subject_headword} → {rel.object_headword} ({rel.relation_type.value})")
7. Comparison Reports¶
Generate structured reports for mutashabihat and other analyses.
reporter = ReportGenerator()
# Single ayah report
result = engine.find_normalized_matches(
"2:255",
normalization_profile="diacritics-insensitive-v1",
min_words=2
)
report = reporter.generate_report(result, "tanzil-uthmani-v1", "2:255")
print(reporter.format_report_text(report))
# Multi-ayah reports
reports = reporter.generate_multi_report(["2:255", "3:2", "1:1"])
print(f"\nMulti-report generated for {len(reports)} ayahs")
# Corpus statistics
stats = reporter.generate_corpus_statistics()
print(f"\nCorpus statistics:")
print(f" Total surahs: {stats['total_surahs']}")
print(f" Total ayahs: {stats['total_ayahs']}")
print(f" Total words: {stats['total_words']}")
print(f" Unique words: {stats['unique_words']}")
print(f" Average ayah length: {stats['avg_ayah_length']:.1f} words")
8. Reproducible Research Runs¶
Document your analysis with full provenance for reproducibility.
from qurankit.models.research import ResearchRun, ResearchResult, ClaimCategory
from datetime import datetime, UTC
import uuid
# Create a research run record
run = ResearchRun(
run_id=f"collocation_analysis_{uuid.uuid4().hex[:8]}",
operation="collocation_analysis",
algorithm_name="CollocationAnalyzer.compute_collocations",
algorithm_version="0.3.0",
parameters_json={
"window_size": 5,
"measure": "pmi",
"min_cooccurrence": 3,
"filter_stopwords": True
},
normalization_profile_id="diacritics-insensitive-v1",
software_version="qurankit-core-0.3.0",
started_at=datetime.now(UTC),
status="completed"
)
print(f"Research run: {run.run_id}")
print(f"Operation: {run.operation}")
print(f"Algorithm: {run.algorithm_name} v{run.algorithm_version}")
print(f"Parameters: {json.dumps(run.parameters_json, indent=2)}")
# Create a research result
result = ResearchResult(
result_id=f"result_{uuid.uuid4().hex[:8]}",
run_id=run.run_id,
result_type="collocation_list",
claim_category=ClaimCategory.COMPUTATIONAL_SUGGESTION,
result_payload_json={"top_10": [{"word1": "test", "word2": "test", "score": 0.5}]},
evidence_json={"method": "PMI", "corpus": "tanzil-uthmani-v1"},
claim_category=ClaimCategory.COMPUTATIONAL_SUGGESTION,
)
print(f"\nResearch result: {result.result_id}")
print(f"Claim category: {result.claim_category.value}")
9. Extension Adapters (Tafsir, Hadith, Qira'at, Tajwid, Waqf)¶
QuranKit v0.3.0+ includes an extension framework for scholarly resources.
# Access extension adapters (lazy-loaded)
print("Available adapters:")
print(f" Tafsir: {quran.tafsir is not None}")
print(f" Hadith: {quran.hadith is not None}")
print(f" Qira'at: {quran.qiraat is not None}")
print(f" Tajwid: {quran.tajwid is not None}")
print(f" Waqf: {quran.waqf is not None}")
# Example: Tafsir lookup (requires sample data)
try:
tafsir_entries = quran.tafsir.get_tafsir(1, 1)
print(f"\nTafsir for 1:1: {len(tafsir_entries)} entries")
for entry in tafsir_entries[:2]:
print(f" [{entry.source_id}] {entry.text[:100]}...")
except Exception as e:
print(f"\nTafsir not available: {e}")
# Example: Qira'at comparison
try:
comparison = quran.qiraat.compare_readings(2, 10, ["hafs", "warsh"])
print(f"\nQira'at comparison for 2:10:")
print(f" Base: {comparison.base_text}")
for reading in comparison.readings:
print(f" {reading.qiraat_id}: {reading.text}")
except Exception as e:
print(f"\nQira'at not available: {e}")
# Example: Tajwid rules
try:
rules = quran.tajwid.get_rules()
print(f"\nTajwid rules: {len(rules)} loaded")
for rule in rules[:3]:
print(f" {rule.rule_type.value}: {rule.description[:80]}...")
except Exception as e:
print(f"\nTajwid not available: {e}")
# Example: Waqf/Ibtida rules
try:
waqf_rules = quran.waqf.get_waqf_rules()
print(f"\nWaqf rules: {len(waqf_rules)} loaded")
except Exception as e:
print(f"\nWaqf not available: {e}")
10. Search with Normalization Profiles¶
QuranKit provides 4 built-in normalization profiles for Arabic text search.
# Search examples with different profiles
query = "بسم الله الرحمن الرحيم"
for profile_id in ["exact-source-v1", "canonical-unicode-v1", "diacritics-insensitive-v1", "search-broad-v1"]:
results = quran.search(query, mode=profile_id)
print(f"\nProfile: {profile_id}")
print(f" Results: {len(results)}")
for r in results[:3]:
print(f" {r.reference}: {r.matched_text[:60]} [{r.match_type.value}]")
# Using the normalization engine directly
from qurankit.search import get_engine
engine = get_engine()
for profile_id in engine.list_profiles():
result = engine.normalize(query, profile_id, return_mappings=True)
print(f"\n{profile_id}:")
print(f" Original: {query}")
print(f" Normalized: {result.normalized}")
if result.mappings:
print(f" Mappings: {len(result.mappings)} character mappings")
Summary¶
This notebook demonstrated the key research workflows in QuranKit:
- Mutashabihat Analysis - Exact and near-duplicate phrase detection
- Vocabulary & Frequency - Root/lemma statistics, concordance (requires QAC)
- Collocation Analysis - PMI, Dice, log-likelihood co-occurrence measures
- Cross-Reference Analysis - Implicit (computed) and explicit (annotated) links
- Wujuh wa Nazair - Similar wording, potentially different meanings
- Semantic Relations - Synonyms/antonyms from dictionary sources
- Reports - Structured output for analysis
- Reproducible Research - Full provenance tracking
- Extension Adapters - Tafsir, Hadith, Qira'at, Tajwid, Waqf
- Search - 4 normalization profiles for Arabic text
Key Principles:
- Always check claim categories
- Preserve source text integrity
- Document data sources and normalization
- Computational suggestions ≠ scholarly conclusions
- Require expert review for research hypotheses