Metadata-Version: 2.4
Name: google-cloud-spark-connect
Version: 2.0.0
Summary: Managed Spark client library for Spark Connect
Home-page: https://github.com/GoogleCloudDataproc/managed-spark-connect-python
Author: Google LLC
License: Apache 2.0
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: google-api-core>=2.19
Requires-Dist: google-cloud-dataproc>=5.18
Requires-Dist: packaging>=20.0
Requires-Dist: tqdm>=4.67
Requires-Dist: websockets>=14.0
Provides-Extra: client
Requires-Dist: pyspark-client~=4.0.0; extra == "client"
Provides-Extra: spark
Requires-Dist: pyspark[connect]~=4.0.0; extra == "spark"
Provides-Extra: interactive
Requires-Dist: google-colabsqlviz>=0.4.0; extra == "interactive"
Requires-Dist: ipython>=8.0; extra == "interactive"
Requires-Dist: sqlparse>=0.5.0; extra == "interactive"
Requires-Dist: traitlets>=5.1; extra == "interactive"
Dynamic: author
Dynamic: description
Dynamic: description-content-type
Dynamic: home-page
Dynamic: license
Dynamic: license-file
Dynamic: provides-extra
Dynamic: requires-dist
Dynamic: summary

# Managed Spark Connect Client

A wrapper of the Apache [Spark Connect](https://spark.apache.org/spark-connect/)
client with additional functionalities that allow applications to communicate
with a remote Managed Spark Session using the Spark Connect protocol without
requiring additional steps.

## Install

For notebooks and interactive development, install with the `client` and
`interactive` extras to get the lightweight `pyspark-client` package along with
the interactive extras (`explore_dataframe()`, `%dpip`, `%%sparksql`):

```sh
pip install 'google-cloud-spark-connect[client,interactive]'
```

Other installation options:

* `pip install 'google-cloud-spark-connect[client]'`: Installs `pyspark-client`
  without the interactive extras (for standalone scripts or applications).
* `pip install 'google-cloud-spark-connect[spark]'` (or
  `'[spark,interactive]'`): Installs the full `pyspark[connect]` distribution
  instead of `pyspark-client`, for environments that also need a local Spark JVM
  or already have full `pyspark` installed.
* `pip install google-cloud-spark-connect` (or `'[interactive]'`): Does not
  install a Spark package, relying on whichever `pyspark` or `pyspark-client` is
  already present in your environment (such as pre-installed Dataproc or Colab
  runtime images). Use this in Dataproc batch and Serverless runtimes, which
  must keep their bundled `pyspark`.

> **Note:** Avoid installing both `pyspark` and `pyspark-client` in the same
> Python environment. Both packages write to the `pyspark` module namespace, so
> mixing them can cause version skew or break the remaining package if one is
> later uninstalled. In particular, never combine the `client` and `spark`
> extras (`'[client,spark]'`).

## Uninstall

```sh
pip uninstall google-cloud-spark-connect pyspark-client
```

`pip uninstall` never removes dependencies, so this also names the
`pyspark-client` that the `client` extra installed. If you installed the `spark`
extra instead, uninstall `pyspark` too if nothing else needs it.

## Setup

This client requires permissions to
manage [Managed Spark Sessions and Session Templates](https://cloud.google.com/dataproc-serverless/docs/concepts/iam).

If you are running the client outside of Google Cloud, you need to provide
authentication credentials. Set the `GOOGLE_APPLICATION_CREDENTIALS` environment
variable to point to
your [Application Credentials](https://cloud.google.com/docs/authentication/provide-credentials-adc)
file.

You can specify the project and region either via environment variables or directly
in your code using the builder API:

* Environment variables: `GOOGLE_CLOUD_PROJECT` and `GOOGLE_CLOUD_REGION`
* Builder API: `.projectId()` and `.location()` methods (recommended)

## Usage

1. Install the latest version of Managed Spark Connect:

   ```sh
   pip install -U 'google-cloud-spark-connect[client,interactive]'
   ```

2. Add the required imports into your PySpark application or notebook and start
   a Spark session using the fluent API:

   ```python
   from google.cloud.managed_spark_connect import ManagedSparkSession
   spark = ManagedSparkSession.builder.getOrCreate()
   ```

3. You can configure Spark properties using the `.config()` method:

   ```python
   from google.cloud.managed_spark_connect import ManagedSparkSession
   spark = ManagedSparkSession.builder.config('spark.executor.memory', '4g').config('spark.executor.cores', '2').getOrCreate()
   ```

4. For advanced configuration, you can use the `Session` class to customize
   settings like subnetwork or other environment configurations:

   ```python
   from google.cloud.managed_spark_connect import ManagedSparkSession
   from google.cloud.dataproc_v1 import Session
   session_config = Session()
   session_config.environment_config.execution_config.subnetwork_uri = '<subnet>'
   session_config.runtime_config.version = '3.0'
   spark = ManagedSparkSession.builder.projectId('my-project').location('us-central1').sessionConfig(session_config).getOrCreate()
   ```

### Builder Configuration

The `ManagedSparkSession.builder` provides a fluent API to configure the session. Below is a list of available methods:

| Method | Description |
|--------|-------------|
| `config(key, value)` | Sets a Spark configuration property. |
| `sessionConfig(session_config)` | Sets the [Dataproc Session](https://docs.cloud.google.com/python/docs/reference/dataproc/latest/google.cloud.dataproc_v1.types.Session) configuration object. |
| `sessionId(session_id)` | Sets a custom session ID for creating or reusing sessions. |
| `idleTtl(duration)` | Sets the idle time-to-live (idle TTL) for the session using a `datetime.timedelta` object. |
| `label(key, value)` | Adds a single label to the session. |
| `labels(labels)` | Adds multiple labels to the session. |
| `location(location)` | Sets the Google Cloud region. |
| `projectId(project_id)` | Sets the Google Cloud project ID. |
| `runtimeVersion(version)` | Sets the Managed Spark runtime version (e.g., "3.0"). |
| `serviceAccount(account)` | Sets the service account for the session. |
| `sessionTemplate(profile)` | Sets the Session Template to use. |
| `subnetwork(subnet)` | Sets the subnetwork URI for the session. |
| `ttl(duration)` | Sets the time-to-live (TTL) for the session using a `datetime.timedelta` object. |

### Reusing Named Sessions Across Notebooks

Named sessions allow you to share a single Spark session across multiple notebooks, improving efficiency by avoiding repeated session startup times and reducing costs.

To create or connect to a named session:

1. Create a session with a custom ID in your first notebook:

   ```python
   from google.cloud.managed_spark_connect import ManagedSparkSession
   session_id = 'my-ml-pipeline-session'
   spark = ManagedSparkSession.builder.sessionId(session_id).getOrCreate()
   df = spark.createDataFrame([(1, 'data')], ['id', 'value'])
   df.show()
   ```

2. Reuse the same session in another notebook by specifying the same session ID:

   ```python
   from google.cloud.managed_spark_connect import ManagedSparkSession
   session_id = 'my-ml-pipeline-session'
   spark = ManagedSparkSession.builder.sessionId(session_id).getOrCreate()
   df = spark.createDataFrame([(2, 'more-data')], ['id', 'value'])
   df.show()
   ```

3. Session IDs must be 4-63 characters long, start with a lowercase letter, contain only lowercase letters, numbers, and hyphens, and not end with a hyphen.

4. Named sessions persist until explicitly terminated or reach their configured TTL.

5. A session with a given ID that is in a TERMINATED state cannot be reused. It must be deleted before a new session with the same ID can be created.

### Interactive Extras

For notebooks and other interactive use, include the `interactive` extra (along
with `client` or `spark`, unless PySpark is already in your environment):

```sh
pip install 'google-cloud-spark-connect[client,interactive]'
```

When you import the package inside an IPython kernel, it then automatically sets
up a few interactive conveniences:

- `explore_dataframe()` from
  [google-colabsqlviz](https://pypi.org/project/google-colabsqlviz/) is injected
  into your notebook globals.
- The [`%dpip` and `%%sparksql`](#bundled-magics) magics are loaded.

```python
import google.cloud.managed_spark_connect  # extras load here
```

Each extra is only set up if its dependency is importable, so without the
`interactive` extra you get whichever of them your environment already happens
to provide. On import, a single line lists the extras that were loaded.

The extras won't override anything you've already set up, for example if `explore_dataframe` is already present, or `%%sparksql` was loaded from somewhere else, it is left alone.

#### Opting out

Set the environment variable before starting the kernel:

```sh
export MANAGED_SPARK_CONNECT_ENABLE_EXTRAS=false
```

Alternatively, configure it through IPython, either persistently in
`~/.ipython/profile_default/ipython_config.py`:

```python
c.ManagedSparkConnect.enable_extras = False
```

Or at runtime:

```python
%config ManagedSparkConnect.enable_extras = False
```

An explicit IPython setting takes precedence over the environment variable.

### Bundled Magics

These are loaded with the interactive extras, or explicitly with
`%load_ext google.cloud.managed_spark_magics`.

#### `%dpip`

Installs PyPI packages on the session's driver and executors, so your UDFs can
import them:

```python
%dpip install spacy==3.8.4 torch
```

See [docs/dpip.md](https://github.com/GoogleCloudDataproc/managed-spark-connect-python/blob/main/docs/dpip.md)
for details.

#### `%%sparksql`

Runs Spark SQL against the active session and displays the result as a table,
with `:name` binding notebook variables as SQL values:

```python
%%sparksql
SELECT * FROM IDENTIFIER(:table) WHERE id > :min_id
```

See [docs/sparksql.md](https://github.com/GoogleCloudDataproc/managed-spark-connect-python/blob/main/docs/sparksql.md)
for options, passing Python values (including `IDENTIFIER()` for table names)
and more.

## Migrating from dataproc-spark-connect

The `dataproc-spark-connect` package has been renamed to `google-cloud-spark-connect`. This is a breaking change with no compatibility shims — you need to update your code in the following places when you switch to the new package.

### 1. Update the package you install

`dataproc-spark-connect` previously pulled in full `pyspark` by default. When
upgrading an existing environment in-place, use the `spark` extra so `pip`
reuses your existing `pyspark` installation without installing `pyspark-client`
alongside it (along with `interactive` for the interactive extras):

```sh
# Before
pip install dataproc-spark-connect

# After (upgrading an existing environment with pyspark already installed)
pip install 'google-cloud-spark-connect[spark,interactive]'
```

In a fresh environment (or after running `pip uninstall dataproc-spark-connect pyspark`),
prefer the lightweight `client` extra instead:

```sh
pip install 'google-cloud-spark-connect[client,interactive]'
```

### 2. Update your imports and session class

`google.cloud.dataproc_spark_connect` is now `google.cloud.managed_spark_connect`, and `DataprocSparkSession` is now `ManagedSparkSession`:

```python
# Before
from google.cloud.dataproc_spark_connect import DataprocSparkSession
spark = DataprocSparkSession.builder.getOrCreate()

# After
from google.cloud.managed_spark_connect import ManagedSparkSession
spark = ManagedSparkSession.builder.getOrCreate()
```

If you use the Jupyter magic commands, `google.cloud.dataproc_magics` is now `google.cloud.managed_spark_magics` and `DataprocMagics` is now `ManagedSparkMagics` (the `%dpip` magic itself is unchanged).

### 3. Rename `dataprocSessionConfig(...)` and `dataprocSessionId(...)` calls

These builder methods drop the `dataproc` prefix — they take the same arguments and behave identically, only the names change:

```python
# Before
spark = (
    DataprocSparkSession.builder
        .dataprocSessionId('my-session')
        .dataprocSessionConfig(session_config)
        .getOrCreate()
)

# After
spark = (
    ManagedSparkSession.builder
        .sessionId('my-session')
        .sessionConfig(session_config)
        .getOrCreate()
)
```

### 4. Rename any `DATAPROC_SPARK_CONNECT_*` environment variables

If you set any of the library's own environment variables (as opposed to standard GCP ones like `GOOGLE_CLOUD_PROJECT`), rename the `DATAPROC_SPARK_CONNECT_` prefix to `MANAGED_SPARK_CONNECT_`:

| Before | After |
|--------|-------|
| `DATAPROC_SPARK_CONNECT_SERVICE_ACCOUNT` | `MANAGED_SPARK_CONNECT_SERVICE_ACCOUNT` |
| `DATAPROC_SPARK_CONNECT_SUBNET` | `MANAGED_SPARK_CONNECT_SUBNET` |
| `DATAPROC_SPARK_CONNECT_AUTH_TYPE` | `MANAGED_SPARK_CONNECT_AUTH_TYPE` |
| `DATAPROC_SPARK_CONNECT_TTL_SECONDS` | `MANAGED_SPARK_CONNECT_TTL_SECONDS` |
| `DATAPROC_SPARK_CONNECT_IDLE_TTL_SECONDS` | `MANAGED_SPARK_CONNECT_IDLE_TTL_SECONDS` |
| `DATAPROC_SPARK_CONNECT_SESSION_TERMINATE_AT_EXIT` | `MANAGED_SPARK_CONNECT_SESSION_TERMINATE_AT_EXIT` |
| `DATAPROC_SPARK_CONNECT_DEFAULT_DATASOURCE` | `MANAGED_SPARK_CONNECT_DEFAULT_DATASOURCE` |
| `DATAPROC_SPARK_CONNECT_ACTIVE_SESSION_FILE_PATH` | `MANAGED_SPARK_CONNECT_ACTIVE_SESSION_FILE_PATH` |

Note that `GOOGLE_CLOUD_DATAPROC_API_ENDPOINT` and other variables naming the actual Dataproc API (not this library's own config) are unchanged.

## Developing

For development instructions see [guide](DEVELOPING.md).

## Contributing

We'd love to accept your patches and contributions to this project. There are
just a few small guidelines you need to follow.

### Contributor License Agreement

Contributions to this project must be accompanied by a Contributor License
Agreement. You (or your employer) retain the copyright to your contribution;
this simply gives us permission to use and redistribute your contributions as
part of the project. Head over to <https://cla.developers.google.com> to see
your current agreements on file or to sign a new one.

You generally only need to submit a CLA once, so if you've already submitted one
(even if it was for a different project), you probably don't need to do it
again.

### Code reviews

All submissions, including submissions by project members, require review. We
use GitHub pull requests for this purpose. Consult
[GitHub Help](https://help.github.com/articles/about-pull-requests/) for more
information on using pull requests.
