Metadata-Version: 2.4
Name: lucidlink
Version: 1.0.1
Summary: Python SDK for LucidLink — programmatic access to LucidLink filespaces
Author-email: LucidLink Corp <support@lucidlink.com>
Maintainer-email: LucidLink Corp <support@lucidlink.com>
License-Expression: LicenseRef-Proprietary
Project-URL: Homepage, https://www.lucidlink.com
Project-URL: Documentation, https://developer.lucidlink.com/docs/sdk/overview
Project-URL: Issues, https://support.lucidlink.com/
Keywords: lucidlink,filesystem,cloud,storage,filespace,cloud-storage,file-management
Classifier: Development Status :: 5 - Production/Stable
Classifier: Intended Audience :: Developers
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Programming Language :: C++
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Topic :: System :: Filesystems
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
License-File: NOTICE
Requires-Dist: typing_extensions>=4.6.0; python_version < "3.12"
Provides-Extra: fsspec
Requires-Dist: fsspec>=2023.10.0; extra == "fsspec"
Dynamic: license-file

# lucidlink

The LucidLink Python SDK lets you programmatically access [LucidLink](https://www.lucidlink.com) filespaces: reading and
writing files, streaming data, integrating with data libraries such as pandas and PyArrow, and managing external
cloud storage via LucidLink Connect (Enterprise add-on).

> Using a version before 1.0.0 (beta)? See the
> [1.0 migration guide and the beta documentation](https://lucidlink.github.io/lucidlink-python-sdk-examples/).

## Key Concepts

- **Workspace** — your organization's LucidLink account.
- **Filespace** — a named storage volume within a workspace. Most SDK operations are scoped to a filespace.
- **Client** — the SDK's main entry point. Authenticates with a service account and manages the SDK's lifecycle.
- **Service Account** — a non-human identity used to authenticate SDK calls. Tokens look like `sa_live:your_key_id:your_secret`.
- **LucidLink Connect** — a feature (Enterprise add-on) that surfaces files from external S3 storage inside a filespace
  without copying them.

## Authentication

All SDK operations require a service account token. Create one in the [LucidLink web client](https://app.lucidlink.com)
(requires admin permissions). Service accounts are available on Business and Enterprise plans, as well as during Trial.

```python
credentials = lucidlink.ServiceAccountCredentials(
    token="sa_live:your_key_id:your_secret"
)
```

## Features

- **Service Account Authentication** — authenticate using service account tokens
- **File Operations** — read, write, create, delete files and directories
- **Streaming I/O** — full `io.RawIOBase` interface for standard Python I/O compatibility
- **fsspec Integration** — read and write `lucidlink://` URLs from libraries that work with fsspec, such as pandas
- **LucidLink Connect** — manage external S3 data stores and link files into filespaces (Enterprise)

## Installation

Requires Python 3.11 or later.

```bash
pip install lucidlink
```

For [fsspec](https://filesystem-spec.readthedocs.io/) integration:

```bash
pip install "lucidlink[fsspec]"
```

## Quick Start

```python
import lucidlink

# Create a Client (ephemeral temp storage by default)
with lucidlink.Client() as client:
    # Authenticate with service account
    credentials = lucidlink.ServiceAccountCredentials(
        token="sa_live:your_key_id:your_secret"
    )
    client.login(credentials)

    # Service-account tokens are workspace-scoped
    workspace = client.workspace

    # Link to a filespace by id (see list_filespaces())
    filespace = workspace.link_filespace("fs-uuid-12345")

    # List directory
    for entry in filespace.file_system.read_dir("/"):
        print(f"{entry.name}: {entry.size} bytes")

    # Write a file
    lucidlink.FsUtils(filespace.file_system).write_bytes("/example.txt", b"Hello from LucidLink!")

    # Read a file
    data = lucidlink.FsUtils(filespace.file_system).read_bytes("/example.txt")
    print(data)

    # Save the changes: block until they have reached LucidLink services
    filespace.sync()

# client.close() runs on __exit__ — unlinks filespaces (with a best-effort
# final sync) and stops the runtime; a failed final sync is raised afterwards.
```

### Streaming File Access

Full `io.RawIOBase` compatibility for streaming:

```python
# Binary streaming
with filespace.file_system.open("/large_file.dat", "rb", buffering=8192) as f:
    for chunk in iter(lambda: f.read(4096), b""):
        process(chunk)

# Text streaming with encoding
with filespace.file_system.open("/document.txt", "rt", encoding="utf-8") as f:
    for line in f:
        print(line.strip())

# Byte range reads
with filespace.file_system.open("/data.bin", "rb") as f:
    f.seek(1000)
    data = f.read(100)  # Read 100 bytes from offset 1000
```

### fsspec Integration

Access LucidLink files from libraries that work with fsspec, such as pandas (requires
`pip install "lucidlink[fsspec]"`):

```python
import pandas as pd

# Read CSV directly from LucidLink
df = pd.read_csv(
    "lucidlink://workspace/filespace/data.csv",
    storage_options={"token": "sa_live:..."}
)

# Write Parquet to LucidLink
df.to_parquet(
    "lucidlink://workspace/filespace/output.parquet",
    storage_options={"token": "sa_live:..."}
)
```

PyArrow does not open `lucidlink://` URLs: give it a `LucidLinkFileSystem` as `filesystem`:

```python
import pyarrow.parquet as pq
from lucidlink.fsspec import LucidLinkFileSystem

with LucidLinkFileSystem(token="sa_live:...") as fs:
    table = pq.read_table("lucidlink://workspace/filespace/data.parquet", filesystem=fs)
```

Direct fsspec usage:

```python
from lucidlink.fsspec import LucidLinkFileSystem

# Closed, with a final sync, at the end of the with block
with LucidLinkFileSystem(token="sa_live:...") as fs:
    # List, download, upload
    entries = fs.ls("lucidlink://workspace/filespace/", detail=True)
    fs.get("lucidlink://workspace/filespace/file.txt", "local_file.txt")
    fs.put("local_file.txt", "lucidlink://workspace/filespace/uploaded.txt")

    # Move/rename (native operation, faster than copy+delete)
    fs.mv("lucidlink://workspace/filespace/old.txt",
          "lucidlink://workspace/filespace/new.txt")
```

Each `LucidLinkFileSystem` instance is a separate client, and fsspec creates one per thread, per process and per set
of storage options; see [Limitations](#limitations).

## Storage Modes

### Ephemeral Mode (Default)

Uses a temporary directory that is automatically cleaned up:

```python
client = lucidlink.Client()  # ephemeral by default
```

### Persistent Mode

Uses a `.lucid` folder under the given directory, kept after the client closes. Not available with
`LucidLinkFileSystem`, which always uses ephemeral storage:

```python
from pathlib import Path

from lucidlink import Client, StorageConfig, StorageMode

client = Client(storage=StorageConfig(
    mode=StorageMode.PERSISTENT,
    root_path=Path.home() / "lucid_data",
))
```

Multiple clients must not share persistent storage: give each client its own `root_path`. Use ephemeral storage
whenever more than one client is involved.

## Limitations

- **A client is not thread safe.** Do not use one `Client`, or the objects obtained from it, from several threads at
  the same time. The same holds for one `LucidLinkFileSystem` instance, which owns one client.
- **Multiple clients must not share persistent storage.** With `StorageMode.PERSISTENT`, each client needs its own
  `root_path`. Use ephemeral storage (the default) whenever more than one client is involved.
- **Multiple clients are only eventually consistent.** A client sees changes made through another client — in the
  same process, in another process or on another machine — only after a delay. For example, after one client closes
  a file, another client can find it missing, empty or truncated for a few seconds; two clients creating the same
  new directory conflict, and one of the directories ends up as `<name>_conflicted`. `filespace.sync()` (or
  `sync_all()` on a `LucidLinkFileSystem`) returns once the changes have reached LucidLink services; it does not by
  itself make them visible to another client at once.
- **A process forked while a client runs cannot use LucidLink.** Every LucidLink call there raises `ClientError`.
  `fork` is the default `multiprocessing` start method on Linux before Python 3.14, e.g. for PyTorch `DataLoader`
  workers: use `spawn` or `forkserver` instead, and give each worker its own client.

### fsspec creates clients implicitly

Each `LucidLinkFileSystem` instance owns its own client. fsspec reuses a cached instance only for the same storage
options in the same thread and process, so `fsspec.filesystem("lucidlink", ...)`, `fsspec.core.url_to_fs(...)`,
`fsspec.open("lucidlink://...")` and every library call that takes a `lucidlink://` URL create another instance,
and so another client:

- in another thread, including the threads a library starts for its own I/O;
- in another process, including worker processes that receive an instance by pickling;
- for other storage options, even in the same thread: files written through an instance created with
  `auto_mkdir=True` and read with `pd.read_csv("lucidlink://...")` without it go through two clients, and the reader
  can miss files or get them truncated;
- with `skip_instance_cache=True`.

These clients are eventually consistent with each other. `LucidLinkFileSystem` always uses ephemeral storage, since
its per-thread and per-process instances would otherwise all get the same `root_path`.

The first and third limitations meet here: sharing one instance between threads uses one client from several
threads, while an instance per thread means several clients.

### Libraries that do not cope with eventual consistency

Libraries that reach `lucidlink://` data through several clients, from worker processes or from their own I/O
threads, can miss files that were just written or read them incomplete, often without an error. In our tests this
happened with:

- dask with every scheduler but `synchronous`
- Ray Data
- zarr 3, and xarray and anndata zarr stores, opened by `lucidlink://` URL
- kerchunk (`reference://` with `remote_protocol="lucidlink"`)
- Hugging Face `datasets` with `lucidlink://` data files
- `fsspec.generic.rsync` and `GenericFileSystem`

The same applies to any code that runs work in other processes, such as PyTorch `DataLoader` workers,
`multiprocessing` or `concurrent.futures.ProcessPoolExecutor`.

## Telemetry

The SDK sends usage telemetry to LucidLink. Each event records the user or service account, the workspace, the
filespace and the client node it concerns, together with the operating system and its version, the SDK version and the
time of the event. LucidLink collects this data under the Developer Platform License Agreement (see "License and
usage") and handles personal data under its privacy policy: https://www.lucidlink.com/privacy.

## License and usage

This Software Development Kit (the “SDK”) is proprietary to LucidLink Corp. and is part of the LucidLink Developer
Platform.

Use of this SDK is governed by the LucidLink Developer Platform License Agreement, available at
https://developer.lucidlink.com/LucidLink%20Developer%20Platform%20License%20Agreement.pdf (the “Agreement”), or, if you
or your organization have entered into a separate written agreement with LucidLink that governs your use of the
Developer Platform, by that agreement. See the `LICENSE` file for details.

Under the Agreement you may modify the sample code and examples in the documentation for use in your own applications,
and you may distribute the SDK, unmodified, as a component of your applications solely for use by their end users.

This SDK is not licensed under an open‑source license. All rights are reserved by LucidLink except as expressly granted
in the Agreement.

This SDK includes third-party open-source components, each licensed under its own terms. See the `NOTICE` file for the
list of components and their licenses.

## Support

- **Support**: support@lucidlink.com
