Metadata-Version: 2.5
Name: data-refiner-pyspark
Version: 1.0.0
Summary: A PySpark-based big data processing framework with pipeline support.
Project-URL: Homepage, https://github.com/Sonder0731/data-refiner
Project-URL: Repository, https://github.com/Sonder0731/data-refiner
Project-URL: Issues, https://github.com/Sonder0731/data-refiner/issues
Author-email: Sonder0731 <sonderbanana@gmail.com>
License-File: LICENSE
Requires-Python: <3.12,>=3.11
Requires-Dist: botocore>=1.43.16
Requires-Dist: curl-cffi>=0.13.0
Requires-Dist: datefinder==0.7.3
Requires-Dist: docstring-parser>=0.17.0
Requires-Dist: editdistance>=0.8.1
Requires-Dist: fasttext-wheel>=0.9.2
Requires-Dist: fastwarc==0.15.2
Requires-Dist: furl>=2.1.4
Requires-Dist: graphframes>=0.6
Requires-Dist: jieba>=0.42.1
Requires-Dist: json-repair>=0.44.1
Requires-Dist: librosa>=0.11.0
Requires-Dist: litellm>=1.83.0
Requires-Dist: loguru>=0.7.3
Requires-Dist: nltk==3.9.4
Requires-Dist: numpy<2.0.0
Requires-Dist: opencc>=1.1.9
Requires-Dist: orjson>=3.11.3
Requires-Dist: pandas>=2.3.1
Requires-Dist: parsel>=1.10.0
Requires-Dist: pillow>=12.0.0
Requires-Dist: pipreqs>=0.5.0
Requires-Dist: presidio-analyzer>=2.2.360
Requires-Dist: presidio-anonymizer>=2.2.360
Requires-Dist: psutil>=7.0.0
Requires-Dist: pyahocorasick>=2.2.0
Requires-Dist: pyarrow>=22.0.0
Requires-Dist: pybloom-live>=4.0.0
Requires-Dist: pydantic-settings>=2.12.0
Requires-Dist: pydantic>=2.11.7
Requires-Dist: pydub>=0.25.1
Requires-Dist: pyspark==3.5.2
Requires-Dist: python-dotenv>=1.2.1
Requires-Dist: pyyaml>=6.0.2
Requires-Dist: scikit-learn>=1.7.2
Requires-Dist: scipy>=1.13.1
Requires-Dist: tenacity>=9.1.2
Requires-Dist: textdescriptives==2.8.4
Requires-Dist: tomli>=2.3.0
Requires-Dist: tqdm>=4.67.3
Requires-Dist: trafilatura>=2.0.0
Requires-Dist: wget>=3.2
Requires-Dist: xxhash>=3.5.0
Description-Content-Type: text/markdown

<p align="center">
  <img src="logo.png" width="500">
</p>

---

Data-refiner is an extensible data processing framework built on PySpark that enables users to build scalable data pipelines by composing reusable operators. With its built-in Operator Marketplace, users can discover, extend, and combine modular data processing components—like assembling LEGO bricks—to solve complex data tasks without writing complex Spark code.

**Key Features**

- **Composable Pipeline Architecture**

    Workflows are built by assembling modular operators. Pipelines can be reused, extended, swapped, or reconfigured to meet different business needs with minimal effort.

- **Reliable Pipeline Execution**

    Data-refiner provides built-in pipeline validation, execution management, and error handling mechanisms to improve reliability and simplify large-scale data processing.

- **Extensible Operator Ecosystem**

    Data-refiner provides an extensible operator system where users can develop custom operators in Python and share reusable components through the Operator Marketplace.

## Quick Start
Before running, make sure your environment is properly configured to run Apache Spark.

**Install**
```shell
uv pip install data-refiner-pyspark
uv run dr_init && uv run dr_pack_resources
```
**Run**
```python
import data_refiner
from pathlib import Path
from data_refiner import Runner
from pyspark.sql import SparkSession
spark = (
    SparkSession.builder.master("local[1]")
    .appName("data_refiner")
    .enableHiveSupport()
    .getOrCreate()
)
data_path = Path(data_refiner.__file__).parent / "tests/ops/test_data/mapper/html_content_extract_mapper"
pipeline = {
  "read_whole_text_file": {
    "op_name": "whole_text_file_reader",
    "input_path": f"{str(data_path)}",
    "output_df": "input.df",
  },
  "html_content_extract": {
    "op_name": "html_content_extract_mapper",
    "input_df": "input.df",
    "output_df": "html_extract.df",
    "field": "text",
    "output_field": "content",
  },
  "language_identifier": {
    "op_name": "language_identification_mapper",
    "input_df": "html_extract.df",
    "output_df": "language_identification.df",
    "field": "content",
    "output_field": "language",
    "show": {"truncate":True},
  }
}
Runner.from_dict(pipeline).run(spark=spark)
```

## Operator Market
[View the operator market](docs/operator/ops_market.md)

## How to run
- [Run locally](docs/howtouse/run_on_local.md)
- [Run on Cluster](docs/howtouse/run_on_cluster.md)

## Community

---

👉 Join [Discord](https://discord.gg/rmhW2b7UC2) server to:
- Ask questions
- Discuss development
- Share ideas and feedback
- New operator requirements

 OR email to: `sonderbanana@gmail.com`