Metadata-Version: 2.4
Name: ytclone
Version: 0.1.24
Summary: Resumable copying of YTsaurus tables, files and directories to and from S3
Author: Alexander Vasin
Author-email: Alexander Vasin <hi@alvass.in>
License-Expression: MIT
License-File: LICENSE
Classifier: Operating System :: POSIX :: Linux
Classifier: Operating System :: MacOS :: MacOS X
Requires-Dist: aiochannel>=1.3,<2
Requires-Dist: aiohttp>=3.14.3,<4
Requires-Dist: aiohttp-s3-client>=1.0.14,<2
Requires-Dist: aiomisc>=17.10.3,<19
Requires-Dist: argclass>=1.11.1,<2
Requires-Dist: ytsaurus-client>=0.13.54,<0.14
Requires-Dist: ytsaurus-yson>=0.4.10,<0.5
Requires-Dist: zstandard>=0.25.0,<1
Requires-Python: >=3.10
Description-Content-Type: text/markdown

# ytclone

Copy static tables, files, YSON documents, and directories between YTsaurus and S3,
preserving schemas and portable attributes.

Restoring from S3 requires a `yt-s3-v2` archive created by `ytclone`. Earlier archive
formats are rejected; re-export the source to a new S3 prefix.

## Installation

```bash
pip install ytclone
```

Requires **CPython 3.10+** on **Linux x86_64 with glibc** or **macOS ARM64**
(Apple Silicon).

Also available as [standalone downloads](https://github.com/alvassin/ytclone/blob/main/USAGE.md#standalone-downloads)
and [Docker images](https://github.com/alvassin/ytclone/blob/main/USAGE.md#docker).

## Quick start

Configure access to YTsaurus and S3:

```bash
export YT_PROXY="https://yt.example.com"
export YT_TOKEN="..."
export AWS_ACCESS_KEY_ID="..."
export AWS_SECRET_ACCESS_KEY="..."
export AWS_DEFAULT_REGION="us-east-1"
```

The S3 bucket must already exist. For an S3-compatible service, also set
`S3_ENDPOINT` to its URL. Use `AWS_SESSION_TOKEN` for temporary credentials.

**Copy to S3:**

```bash
ytclone //home/analytics/events s3://my-bucket/export --log-file export.log
```

The full YT path is appended to the S3 prefix: this command stores `events` at
`s3://my-bucket/export/home/analytics/events/`.

**Restore to YTsaurus:**

```bash
ytclone s3://my-bucket/export/home/analytics/events \
  //home/analytics/events_copy --log-file restore.log
```

The restore destination is used exactly as given: the children of `events` go
directly under `//home/analytics/events_copy`. With `--log-file`, progress appears
on the console and detailed logs are appended to the file.

## Repeated runs

Rerun the command after an interruption or source update. Completed, unchanged
nodes are skipped; other nodes are copied again. An unfinished table, file, or
document restarts from the beginning.

Source data is never deleted. Destination nodes absent from the source are kept.
Use a separate S3 prefix for each source cluster.

## Parallel export

Use [`--max-streams`](https://github.com/alvassin/ytclone/blob/main/USAGE.md#tuning)
to export large tables through several streams, or
[`ytclone-parallel`](https://github.com/alvassin/ytclone/blob/main/USAGE.md#parallel-export)
for multiple copy processes. Both apply to YT → S3; restore with `ytclone`.

## Before you copy

- Dynamic tables are skipped and reported; links are unsupported.
- **One current archive version.** Failed uploads keep the previous copy available.
  Updates publish first, then try to delete the old copy. No history is kept.
- **Permissions are not copied.** ACLs, ownership, and account settings are
  excluded. Replacing an existing YT table, file, or document recreates the node,
  so its previous settings are not preserved either.
- **Consistency is per node.** A directory tree is not a single point-in-time
  snapshot.
- **Concurrent exports can fail on conflicts.** Updating an archive during restore
  can also fail the restore; rerun it. Concurrent restores to the same YT nodes
  are unsupported.
- S3 must support [atomic conditional writes](https://github.com/alvassin/ytclone/blob/main/USAGE.md#connection-settings).
  See [update and recovery details](https://github.com/alvassin/ytclone/blob/main/USAGE.md#errors-and-recovery)
  for rollback and cleanup behavior.

## Documentation

Run `ytclone --help` or `ytclone-parallel --help` for all command-line options.

- [User guide](https://github.com/alvassin/ytclone/blob/main/USAGE.md): paths,
  configuration, progress, retries, and parallel export.
- [Architecture](https://github.com/alvassin/ytclone/blob/main/ARCHITECTURE.md):
  archive format, change detection, and execution diagrams.
- [Contributing](https://github.com/alvassin/ytclone/blob/main/CONTRIBUTING.md):
  installation from source, development, and tests.

License: [MIT](https://github.com/alvassin/ytclone/blob/main/LICENSE).
