Metadata-Version: 2.5
Name: jupyter-storage-explorer-pyspark-support
Version: 0.21.0
Summary: PySpark support for Storage Explorer
Project-URL: Homepage, https://github.com/sugibuchi/jupyter-storage-explorer/tree/main/plugins/pyspark-support
Project-URL: Source, https://github.com/sugibuchi/jupyter-storage-explorer
Project-URL: Issues, https://github.com/sugibuchi/jupyter-storage-explorer/issues
Author: Tsuyoshi SUGIBUCHI
License-Expression: MIT
License-File: LICENSE
Keywords: databricks,delta lake,jupyter,jupyter-storage-explorer,pyspark,spark
Classifier: Development Status :: 4 - Beta
Classifier: Framework :: Jupyter
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Topic :: Scientific/Engineering
Requires-Python: >=3.11
Requires-Dist: jupyter-storage-explorer==0.21.0
Provides-Extra: full
Requires-Dist: delta-spark>=3.0; extra == 'full'
Requires-Dist: pyspark>=3.5; extra == 'full'
Description-Content-Type: text/markdown

# jupyter-storage-explorer-pyspark-support

PySpark support for Storage Explorer,
the [Jupyter Storage Explorer](https://github.com/sugibuchi/jupyter-storage-explorer) plugin for
PySpark as an engine: the data browser on Spark DataFrames of Delta Lake tables, Parquet, CSV and JSON Lines, and code
loading them.

## Installation

```bash
pip install "jupyter-storage-explorer[pyspark]"          # the plugin and its libraries
pip install "jupyter-storage-explorer[pyspark-support]"  # the plugin only, with the libraries you have
```

The libraries are not the plugin's dependencies, so that installing it never changes those of a shared environment.
The first command installs them, as does `jupyter-storage-explorer-pyspark-support[full]`:

| Library       | Needed for                                                       |
|---------------|------------------------------------------------------------------|
| `pyspark`     | Mandatory: the engine                                            |
| `delta-spark` | Delta Lake tables                                                |

Without `delta-spark`, the other formats still work. For Databricks Connect, use
[databricks-support](https://github.com/sugibuchi/jupyter-storage-explorer/blob/main/plugins/databricks-support/README.md)
instead: with Databricks Connect installed, this plugin's engine is not offered.

### Prerequisites

- The plugin expects Spark to reach your storage already, e.g. through
  [`spark-defaults.conf`](https://spark.apache.org/docs/latest/configuration.html#overriding-configuration-directory)
  or a Databricks cluster's notebook.
- `session_options` in [Engine options](#engine-options) can add Spark properties. With a session already running,
  only those Spark can change at run time apply.

## Plugin options

`plugin_options` gives the plugin's options:

```python
StorageExplorer(
    "abfs://lake@myaccount.dfs.core.windows.net/raw",
    plugin_options={"pyspark-support": {"priority": 10}},
)
```

In the launcher's tabs, `plugin_options()` in `launcher.py` gives them:

```python
def plugin_options():
    return {"pyspark-support": {"enabled": False}}
```

| Option     | Default | Meaning                                                                                                  |
|------------|---------|----------------------------------------------------------------------------------------------------------|
| `enabled`  | `True`  | Whether explorers use the plugin. `False`: not even imported.                                            |
| `priority` | lowest  | Plugins of a higher priority come first: their supports are tried before the others'. Equal ones by key. |

## Engine options

The supports' options, in `engine_options["pyspark"]` (by format, or `"*"` for all), configure the Spark session and
the reader, for the data browser and the code:

```python
StorageExplorer(
    "abfs://lake@myaccount.dfs.core.windows.net/raw",
    engine_options={
        "pyspark": {
            "*": {
                "session_options": {
                    "spark.sql.extensions": "io.delta.sql.DeltaSparkSessionExtension",
                    "spark.sql.catalog.spark_catalog": "org.apache.spark.sql.delta.catalog.DeltaCatalog",
                },
            },
            "csv": {"null_value": "NA", "mode": "DROPMALFORMED"},
            "delta": {"version_as_of": 3},
        },
    },
)
```

Every format takes:

| Option            | Default | Meaning                                                                                                                                                                                                                                                        |
|-------------------|---------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `session_options` | None    | Spark configuration, `SparkSession.builder.config(key, value)` for each. With a session already running in the kernel, only what Spark can change at run time applies.                                                                                         |
| `reader_options`  | None    | `spark.read.option(key, value)` for each: options not listed below, e.g. Spark's [generic file source options](https://spark.apache.org/docs/latest/sql-data-sources-generic-options.html) (`{"recursiveFileLookup": True}`). They override the options below. |

The other options are each format's reader options, from Spark's and Delta Lake's documentation, in snake case. Those
left unset (None) are left to Spark, whose defaults are shown. Options that look like credentials are left out of the
generated code.

### CSV (`csv`)

From Spark's [CSV Files](https://spark.apache.org/docs/latest/sql-data-sources-csv.html#data-source-option).

| Option                              | Default                                                                                                      | Meaning                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
|-------------------------------------|--------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `sep`                               | detected                                                                                                     | Sets a separator for each field and value. This separator can be one or more characters. By default, the one detected from the file's beginning.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |
| `encoding`                          | `UTF-8`                                                                                                      | Decodes the CSV files by the given encoding type.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |
| `quote`                             | `"`                                                                                                          | Sets a single character used for escaping quoted values where the separator can be part of the value. To turn off quotations, set not None but an empty string.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |
| `escape`                            | `\`                                                                                                          | Sets a single character used for escaping quotes inside an already quoted value.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |
| `comment`                           | (none)                                                                                                       | Sets a single character used for skipping lines beginning with this character. By default, it is disabled.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |
| `header`                            | `True`                                                                                                       | Uses the first line as names of columns.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
| `infer_schema`                      | `True`                                                                                                       | Infers the input schema automatically from data. It requires one extra pass over the data.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |
| `prefer_date`                       | `True`                                                                                                       | During schema inference (`infer_schema`), attempts to infer string columns that contain dates as `Date` if the values satisfy the `date_format` option or default date format. For columns that contain a mixture of dates and timestamps, try inferring them as `TimestampType` if timestamp format not specified, otherwise infer them as `StringType`.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
| `enforce_schema`                    | `True`                                                                                                       | If `True`, the specified or inferred schema will be forcibly applied to datasource files, and headers in CSV files will be ignored. If `False`, the schema will be validated against all headers in CSV files when `header` is `True`. Field names in the schema and column names in CSV headers are checked by their positions taking into account `spark.sql.caseSensitive`.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| `ignore_leading_white_space`        | `False`                                                                                                      | Whether or not leading whitespaces from values being read should be skipped.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
| `ignore_trailing_white_space`       | `False`                                                                                                      | Whether or not trailing whitespaces from values being read should be skipped.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
| `null_value`                        | (none)                                                                                                       | Sets the string representation of a null value. It applies to all supported types including the string type.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
| `nan_value`                         | `NaN`                                                                                                        | Sets the string representation of a non-number value.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |
| `positive_inf`                      | `Inf`                                                                                                        | Sets the string representation of a positive infinity value.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
| `negative_inf`                      | `-Inf`                                                                                                       | Sets the string representation of a negative infinity value.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
| `date_format`                       | `yyyy-MM-dd`                                                                                                 | Sets the string that indicates a date format. Custom date formats follow the formats at [Datetime Patterns](https://spark.apache.org/docs/latest/sql-ref-datetime-pattern.html). This applies to date type.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
| `timestamp_format`                  | `yyyy-MM-dd'T'HH:mm:ss[.SSS][XXX]`                                                                           | Sets the string that indicates a timestamp format. Custom date formats follow the formats at [Datetime Patterns](https://spark.apache.org/docs/latest/sql-ref-datetime-pattern.html). This applies to timestamp type.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |
| `timestamp_ntz_format`              | `yyyy-MM-dd'T'HH:mm:ss[.SSS]`                                                                                | Sets the string that indicates a timestamp without timezone format. Custom date formats follow the formats at [Datetime Patterns](https://spark.apache.org/docs/latest/sql-ref-datetime-pattern.html). This applies to timestamp without timezone type; zone-offset and time-zone components are not supported when reading this data type.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
| `enable_date_time_parsing_fallback` | Enabled if the time parser policy has legacy settings or if no custom date or timestamp pattern was provided | Allows falling back to the backward compatible (Spark 1.x and 2.0) behavior of parsing dates and timestamps if values do not match the set patterns.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |
| `max_columns`                       | `20480`                                                                                                      | Defines a hard limit of how many columns a record can have.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
| `max_chars_per_column`              | `-1`                                                                                                         | Defines the maximum number of characters allowed for any given value being read. By default, it is -1 meaning unlimited length.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |
| `mode`                              | `PERMISSIVE`                                                                                                 | Allows a mode for dealing with corrupt records during parsing (case-insensitive). Spark tries to parse only required columns in CSV under column pruning, so corrupt records can be different based on required set of fields (`spark.sql.csv.parser.columnPruning.enabled`, enabled by default).<br>• `PERMISSIVE`: when it meets a corrupted record, puts the malformed string into a field configured by `column_name_of_corrupt_record`, and sets malformed fields to null. To keep corrupt records, set a string type field named `column_name_of_corrupt_record` in a user-defined schema; if a schema does not have the field, it drops corrupt records during parsing. A record with less or more tokens than the schema is not a corrupted record: fewer tokens set nulls to the extra fields, more tokens are dropped.<br>• `DROPMALFORMED`: ignores the whole corrupted records.<br>• `FAILFAST`: throws an exception when it meets corrupted records.                                                                                                                                                               |
| `column_name_of_corrupt_record`     | the value of `spark.sql.columnNameOfCorruptRecord`                                                           | Allows renaming the new field having malformed string created by `PERMISSIVE` mode. This overrides `spark.sql.columnNameOfCorruptRecord`.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
| `multi_line`                        | `False`                                                                                                      | Allows a row to span multiple lines, by parsing line breaks within quoted values as part of the value itself.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
| `char_to_escape_quote_escaping`     | `escape` or `\0`                                                                                             | Sets a single character used for escaping the escape for the quote character. The default value is the escape character when escape and quote characters are different, `\0` otherwise.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| `sampling_ratio`                    | `1.0`                                                                                                        | Defines fraction of rows used for schema inferring.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             |
| `empty_value`                       | (none)                                                                                                       | Sets the string representation of an empty value.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |
| `locale`                            | `en-US`                                                                                                      | Sets a locale as language tag in IETF BCP 47 format. For instance, this is used while parsing dates and timestamps.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             |
| `line_sep`                          | `\r`, `\r\n` and `\n`                                                                                        | Defines the line separator that should be used for parsing. Maximum length is 1 character.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |
| `unescaped_quote_handling`          | `STOP_AT_DELIMITER`                                                                                          | Defines how the CSV parser will handle values with unescaped quotes.<br>• `STOP_AT_CLOSING_QUOTE`: if unescaped quotes are found in the input, accumulate the quote character and proceed parsing the value as a quoted value, until a closing quote is found.<br>• `BACK_TO_DELIMITER`: if unescaped quotes are found in the input, consider the value as an unquoted value. The parser accumulates all characters of the current parsed value until the delimiter is found. If no delimiter is found in the value, it continues accumulating characters from the input until a delimiter or line ending is found.<br>• `STOP_AT_DELIMITER`: if unescaped quotes are found in the input, consider the value as an unquoted value. The parser accumulates all characters until the delimiter or a line ending is found in the input.<br>• `SKIP_VALUE`: if unescaped quotes are found in the input, the content parsed for the given value will be skipped and the value set in `null_value` will be produced instead.<br>• `RAISE_ERROR`: if unescaped quotes are found in the input, a `TextParsingException` will be thrown. |
| `time_zone`                         | the value of `spark.sql.session.timeZone`                                                                    | Sets the string that indicates a time zone ID to be used to format timestamps or partition values.<br>• Region-based zone ID: it should have the form 'area/city', such as 'America/Los_Angeles'.<br>• Zone offset: it should be in the format '(+\|-)HH:mm', for example '-08:00' or '+01:00'. Also 'UTC' and 'Z' are supported as aliases of '+00:00'.<br>Other short names like 'CST' are not recommended to use because they can be ambiguous.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              |

### JSON Lines (`json-lines`)

From Spark's [JSON Files](https://spark.apache.org/docs/latest/sql-data-sources-json.html#data-source-option).

| Option                                   | Default                                                                                                      | Meaning                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |
|------------------------------------------|--------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `time_zone`                              | the value of `spark.sql.session.timeZone`                                                                    | Sets the string that indicates a time zone ID to be used to format timestamps or partition values.<br>• Region-based zone ID: it should have the form 'area/city', such as 'America/Los_Angeles'.<br>• Zone offset: it should be in the format '(+\|-)HH:mm', for example '-08:00' or '+01:00'. Also 'UTC' and 'Z' are supported as aliases of '+00:00'.<br>Other short names like 'CST' are not recommended to use because they can be ambiguous.                                                                                                                                                                                                                                    |
| `primitives_as_string`                   | `False`                                                                                                      | Infers all primitive values as a string type.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| `prefers_decimal`                        | `False`                                                                                                      | Infers all floating-point values as a decimal type. If the values do not fit in decimal, then it infers them as doubles.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              |
| `allow_comments`                         | `False`                                                                                                      | Ignores Java/C++ style comment in JSON records.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
| `allow_unquoted_field_names`             | `False`                                                                                                      | Allows unquoted JSON field names.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
| `allow_single_quotes`                    | `True`                                                                                                       | Allows single quotes in addition to double quotes.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
| `allow_numeric_leading_zeros`            | `False`                                                                                                      | Allows leading zeros in numbers (e.g. 00012).                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| `allow_backslash_escaping_any_character` | `False`                                                                                                      | Allows accepting quoting of all character using backslash quoting mechanism.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
| `mode`                                   | `PERMISSIVE`                                                                                                 | Allows a mode for dealing with corrupt records during parsing.<br>• `PERMISSIVE`: when it meets a corrupted record, puts the malformed string into a field configured by `column_name_of_corrupt_record`, and sets malformed fields to null. To keep corrupt records, set a string type field named `column_name_of_corrupt_record` in a user-defined schema; if a schema does not have the field, it drops corrupt records during parsing. When inferring a schema, it implicitly adds a `column_name_of_corrupt_record` field in an output schema.<br>• `DROPMALFORMED`: ignores the whole corrupted records.<br>• `FAILFAST`: throws an exception when it meets corrupted records. |
| `column_name_of_corrupt_record`          | the value of `spark.sql.columnNameOfCorruptRecord`                                                           | Allows renaming the new field having malformed string created by `PERMISSIVE` mode. This overrides `spark.sql.columnNameOfCorruptRecord`.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             |
| `date_format`                            | `yyyy-MM-dd`                                                                                                 | Sets the string that indicates a date format. Custom date formats follow the formats at [Datetime Patterns](https://spark.apache.org/docs/latest/sql-ref-datetime-pattern.html). This applies to date type.                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |
| `timestamp_format`                       | `yyyy-MM-dd'T'HH:mm:ss[.SSS][XXX]`                                                                           | Sets the string that indicates a timestamp format. Custom date formats follow the formats at [Datetime Patterns](https://spark.apache.org/docs/latest/sql-ref-datetime-pattern.html). This applies to timestamp type.                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |
| `timestamp_ntz_format`                   | `yyyy-MM-dd'T'HH:mm:ss[.SSS]`                                                                                | Sets the string that indicates a timestamp without timezone format. Custom date formats follow the formats at [Datetime Patterns](https://spark.apache.org/docs/latest/sql-ref-datetime-pattern.html). This applies to timestamp without timezone type; zone-offset and time-zone components are not supported when reading this data type.                                                                                                                                                                                                                                                                                                                                           |
| `enable_date_time_parsing_fallback`      | Enabled if the time parser policy has legacy settings or if no custom date or timestamp pattern was provided | Allows falling back to the backward compatible (Spark 1.x and 2.0) behavior of parsing dates and timestamps if values do not match the set patterns.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| `multi_line`                             | `False`                                                                                                      | Parse one record, which may span multiple lines, per file.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |
| `allow_unquoted_control_chars`           | `False`                                                                                                      | Allows JSON Strings to contain unquoted control characters (ASCII characters with value less than 32, including tab and line feed characters) or not.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |
| `encoding`                               | detected automatically when `multi_line` is `True`                                                           | Allows to forcibly set one of standard basic or extended encoding for the JSON files. For example UTF-16BE, UTF-32LE.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |
| `line_sep`                               | `\r`, `\r\n` and `\n`                                                                                        | Defines the line separator that should be used for parsing.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |
| `sampling_ratio`                         | `1.0`                                                                                                        | Defines fraction of input JSON objects used for schema inferring.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
| `drop_field_if_all_null`                 | `False`                                                                                                      | Whether to ignore column of all null values or empty array during schema inference.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
| `locale`                                 | `en-US`                                                                                                      | Sets a locale as language tag in IETF BCP 47 format. For instance, it is used while parsing dates and timestamps.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
| `allow_non_numeric_numbers`              | `True`                                                                                                       | Allows JSON parser to recognize set of "Not-a-Number" (NaN) tokens as legal floating number values.<br>• `+INF`: for positive infinity, as well as alias of `+Infinity` and `Infinity`.<br>• `-INF`: for negative infinity, alias `-Infinity`.<br>• `NaN`: for other not-a-numbers, like result of division by zero.                                                                                                                                                                                                                                                                                                                                                                  |
| `use_unsafe_row`                         | the value of `spark.sql.json.useUnsafeRow`                                                                   | Whether to use UnsafeRow to represent struct result in the JSON parser.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |

### Parquet (`parquet`)

From Spark's [Parquet Files](https://spark.apache.org/docs/latest/sql-data-sources-parquet.html#data-source-option).

| Option                 | Default                                                   | Meaning                                                                                                                                                                                                                                                                                                                                                                                                                                             |
|------------------------|-----------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `datetime_rebase_mode` | the value of `spark.sql.parquet.datetimeRebaseModeInRead` | Specifies the rebasing mode for the values of the `DATE`, `TIMESTAMP_MILLIS`, `TIMESTAMP_MICROS` logical types from the Julian to Proleptic Gregorian calendar.<br>• `EXCEPTION`: fails in reads of ancient dates/timestamps that are ambiguous between the two calendars.<br>• `CORRECTED`: loads dates/timestamps without rebasing.<br>• `LEGACY`: performs rebasing of ancient dates/timestamps from the Julian to Proleptic Gregorian calendar. |
| `int96_rebase_mode`    | the value of `spark.sql.parquet.int96RebaseModeInRead`    | Specifies the rebasing mode for INT96 timestamps from the Julian to Proleptic Gregorian calendar.<br>• `EXCEPTION`: fails in reads of ancient INT96 timestamps that are ambiguous between the two calendars.<br>• `CORRECTED`: loads INT96 timestamps without rebasing.<br>• `LEGACY`: performs rebasing of ancient timestamps from the Julian to Proleptic Gregorian calendar.                                                                     |
| `merge_schema`         | the value of `spark.sql.parquet.mergeSchema`              | Sets whether we should merge schemas collected from all Parquet part-files. This overrides `spark.sql.parquet.mergeSchema`.                                                                                                                                                                                                                                                                                                                         |

### Delta Lake (`delta`)

From Delta Lake's [DataFrameReader options](https://docs.delta.io/delta-batch/#dataframereader-options): they load
the table as of a version, a snapshot that later changes to the table do not affect.

| Option            | Default | Meaning                                                                                                                                         |
|-------------------|---------|-------------------------------------------------------------------------------------------------------------------------------------------------|
| `timestamp_as_of` | (none)  | Loads the table as it was at a point in time. Only date or timestamp strings are accepted, e.g. `"2019-01-01"` or `"2019-01-01T00:00:00.000Z"`. |
| `version_as_of`   | (none)  | Loads the table at a version, a number; `DESCRIBE HISTORY` on the table lists them.                                                             |
