Metadata-Version: 2.4
Name: mock-pyspark
Version: 0.0.1.dev20261023
Summary: Drop-in mock for Apache PySpark 4.x — Arrow-backed, no JVM required
Author-email: Peter Dowdy <peter.dowdy@gmail.com>
License: MIT
Project-URL: Homepage, https://github.com/PeterDowdy/mock-pyspark
Project-URL: Repository, https://github.com/PeterDowdy/mock-pyspark
Project-URL: Issues, https://github.com/PeterDowdy/mock-pyspark/issues
Keywords: pyspark,mock,testing,arrow,duckdb
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Topic :: Software Development :: Testing
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Requires-Python: >=3.11
Description-Content-Type: text/markdown
Requires-Dist: pyarrow>=14
Requires-Dist: duckdb<3,>=2
Requires-Dist: pandas>=2.0
Requires-Dist: numpy>=1.24
Provides-Extra: avro
Requires-Dist: fastavro>=1.9; extra == "avro"
Provides-Extra: crypto
Requires-Dist: cryptography>=41; extra == "crypto"
Provides-Extra: delta
Requires-Dist: deltalake>=1.0; extra == "delta"
Provides-Extra: plot
Requires-Dist: plotly>=5.24; extra == "plot"
Provides-Extra: s3
Requires-Dist: boto3; extra == "s3"
Provides-Extra: all
Requires-Dist: mock-pyspark[avro,crypto,delta,plot,s3]; extra == "all"

# mockpyspark

A drop-in mock for Apache PySpark 4.x for unit and integration testing. Runs entirely in Python — no JVM, no Hadoop, no cluster — using PyArrow as the compute engine and DuckDB for SQL.

```bash
pip install mock-pyspark
```

`mockpyspark` (the PyPI distribution is `mock-pyspark`; you still `import mockpyspark`) installs alongside real `pyspark`; there is no namespace collision. Tests opt in via a one-liner so that `from pyspark.sql import ...` in your production code transparently resolves to the mock.

---

## Quick start

```python
# conftest.py
import mockpyspark
mockpyspark.activate()
```

```python
# tests/test_my_pipeline.py
from pyspark.sql import SparkSession, functions as F

def test_normalize(spark=None):
    spark = SparkSession.builder.getOrCreate()
    df = spark.createDataFrame([
        {"id": 1, "name": "  alice ", "score": 10},
        {"id": 2, "name": "BOB",      "score": 20},
    ])
    out = (
        df.withColumn("name", F.initcap(F.trim(F.col("name"))))
          .filter(F.col("score") > 15)
          .select("id", "name", "score")
          .collect()
    )
    assert out == [(2, "Bob", 20)]
```

No changes to your Spark logic. The alias just makes `pyspark.*` imports resolve to `mockpyspark.*` for the test process.

---

## Version targeting

`mockpyspark` supports both PySpark 4.0.x and 4.1.x. APIs that only exist in 4.1 (e.g. `current_time`, `TimeType`) raise `AttributeError` when running in 4.0 mode.

It picks a target version automatically, in order:

1. `PYSPARK_VERSION` env var
2. The `pyspark` pin in `pyproject.toml`, `requirements*.txt`, `poetry.lock`, `uv.lock`, `Pipfile.lock`, or `setup.cfg` (walking up from the working directory)
3. `SparkSession.builder.config("spark.mock.targetVersion", "4.0").getOrCreate()` — per-session override
4. If nothing is found, defaults to the latest supported version and emits a `UserWarning`. Set `MOCKPYSPARK_STRICT_VERSION=1` to raise instead.

So if your project pins `pyspark==4.0.1` in `pyproject.toml`, the mock will refuse 4.1-only APIs without any configuration on your part.

---

## Usage modes

| Mode                         | What to do                                                       |
| ---------------------------- | ---------------------------------------------------------------- |
| **Mock for tests (default)** | `mockpyspark.activate()` in conftest.                            |
| **Pytest plugin**            | `pytest_plugins = ["mockpyspark.pytest_plugin"]` in conftest.    |
| **Real PySpark for a run**   | Pass `--no-mock-pyspark` to pytest, or don't call `activate()`.  |

The pytest plugin also accepts `--no-mock-pyspark` to bypass activation for a single invocation, useful for running the same suite against real PySpark for integration coverage.

See **[docs/usage.md](docs/usage.md)** for the full pattern, including switching between mock and real PySpark in the same suite, and known limitations.

---

## Detecting the mock at runtime

```python
import pyspark
if getattr(pyspark, "__mock__", False):
    print("running against mockpyspark")
```

---

## Contributing

**Run unit tests** (no Java, no Docker):

```bash
git clone https://github.com/PeterDowdy/mock-pyspark.git && cd mock-pyspark
pip install pyarrow "duckdb>=2,<3" pandas numpy pytest
PYTHONPATH=. pytest -q -m unit
```

**Run integration tests** against real PySpark (requires Java 17):

```bash
# Via Docker (easiest)
./start.sh 4.1 --integration-test

# Or locally with a PySpark venv
python -m venv /opt/spark-venv && /opt/spark-venv/bin/pip install pyspark==4.1.0
PYTHONPATH=. PYSPARK_VERSION=4.1 PYSPARK_PYTHON=/opt/spark-venv/bin/python pytest -q -m integration
```

See **[docs/contributing.md](docs/contributing.md)** for the full setup guide.

---

## API coverage

Supports PySpark 4.0.x and 4.1.x — the complete PySpark 4.1 `pyspark.sql.functions` surface, full DataFrame/Column/Window/Catalog API, CSV/JSON/Parquet IO, Delta Lake (the `delta.tables` API plus Delta SQL: MERGE/UPDATE/OPTIMIZE/VACUUM/RESTORE/CLONE/time travel, on a built-in in-memory engine — no extra dependency — whose version follows the target PySpark: delta-spark 4.0.1 or 4.1.0) and Unity-Catalog-style three-level namespaces (catalogs, schemas, volumes, grants, `information_schema`).

**Delta tables live in memory.** The default Delta backend keeps each table's transaction log and data in the process: it is
fast and needs no files, but a table's directory holds no `_delta_log` and no Parquet. Code that must read the files of a
Delta table (with `deltalake`, pandas, or a real Spark) can select delta-rs instead, with `pip install "mock-pyspark[delta]"`
and either `MOCKPYSPARK_DELTA_BACKEND=deltalake` or `SparkSession.builder.config("spark.mock.delta.backend", "deltalake")`.
A Delta table that real Delta wrote to disk is read and written through delta-rs on either backend.

**S3.** `s3://`, `s3a://` and `s3n://` paths work for the file formats (CSV, JSON, Parquet, text) and for Delta tables, through boto3
(`pip install "mock-pyspark[s3]"`), configured as in Spark: `spark.hadoop.fs.s3a.endpoint`, `.access.key`, `.secret.key`,
`.path.style.access`, `.connection.ssl.enabled`, `.endpoint.region` (also per bucket, `fs.s3a.bucket.<name>.*`, and through
`sc._jsc.hadoopConfiguration()`). That reaches a local [moto](https://github.com/getmoto/moto) server or MinIO; with no endpoint
set, boto3's own credential chain applies, which is what an in-process `moto.mock_aws()` provides.

**Nothing is written to S3 (or moto) unless you ask.** By default what is written to an `s3://` path stays in the process - a file is
read back from there, with the modes and partition layout of a write to a bucket, and a Delta table lives in memory - so tests pay
nothing for S3. Set `spark.mock.s3.writes.enabled` to `true` (`SparkSession.builder.config(...)`, `spark.conf.set(...)`, or the
environment variable `MOCKPYSPARK_S3_WRITES=true`) to have writes reach the bucket: files are uploaded, and a Delta table there is a
real one (`_delta_log/*.json` and Parquet files that real Spark can read) whose commits by other writers are followed. Reading what a
bucket already holds needs no flag. With the flag on and a bucket that cannot be reached at all (no credentials, no server) a Delta
table stays in memory; a session that names an endpoint which does not answer gets the error.

- [STATUS-4.1.md](STATUS-4.1.md) — PySpark 4.1 coverage
- [STATUS-4.0.md](STATUS-4.0.md) — PySpark 4.0 coverage
