Metadata-Version: 2.5
Name: atlasopenmagic
Version: 1.10.1
Summary: A utility package for retrieving ATLAS open data URLs and metadata.
Project-URL: Homepage, https://opendata.atlas.cern/
Project-URL: Repository, https://github.com/atlas-outreach-data-tools/atlasopenmagic
Project-URL: Documentation, https://opendata.atlas.cern/docs/atlasopenmagic
Author-email: ATLAS Collaboration <atlas-outreach-opendata-support@cern.ch>
License-Expression: Apache-2.0
License-File: LICENSE
Keywords: ATLAS,Open Data,Utilities
Classifier: Development Status :: 5 - Production/Stable
Classifier: Intended Audience :: Education
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Natural Language :: English
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Documentation
Classifier: Topic :: Education
Classifier: Topic :: Scientific/Engineering
Classifier: Topic :: Scientific/Engineering :: Physics
Classifier: Topic :: Utilities
Classifier: Typing :: Typed
Requires-Python: >=3.10
Requires-Dist: pyyaml
Requires-Dist: requests
Requires-Dist: tqdm
Provides-Extra: dev
Requires-Dist: pre-commit; extra == 'dev'
Requires-Dist: pylint; extra == 'dev'
Requires-Dist: pytest; extra == 'dev'
Requires-Dist: pytest-cov; extra == 'dev'
Description-Content-Type: text/markdown

# ATLAS Open Magic 🪄📊
[![Tests](https://github.com/atlas-outreach-data-tools/atlasopenmagic/actions/workflows/test.yml/badge.svg)](https://github.com/atlas-outreach-data-tools/atlasopenmagic/actions/workflows/test.yml)
![Dynamic TOML Badge](https://img.shields.io/badge/dynamic/toml?url=https%3A%2F%2Fraw.githubusercontent.com%2Fatlas-outreach-data-tools%2Fatlasopenmagic%2Frefs%2Fheads%2Fmain%2Fpyproject.toml&query=%24.project.version&label=pypi)
[![codecov](https://codecov.io/gh/atlas-outreach-data-tools/atlasopenmagic/graph/badge.svg?token=CNTZ8AEHIG)](https://codecov.io/gh/atlas-outreach-data-tools/atlasopenmagic)


**`atlasopenmagic`** is a Python package made to simplify working with ATLAS Open Data by providing utilities to manage metadata and URLs for streaming the data.

### Key Features:
- Simple functions to set the active data release (e.g., `2024r-pp`).
- Efficient local caching of metadata to minimize API calls.
- Helper functions to retrieve specific dataset information, including file URLs for different "skims" (filtered versions of datasets).
- Support for multiple URL protocols (root, https, eos).
- Configuration via environment variables for easy integration into different workflows.

## **Installation**
You can install this package using `pip`.

```bash
pip install atlasopenmagic
```
Alternatively, clone the repository and install locally:
```bash
git clone https://github.com/atlas-outreach-data-tools/atlasopenmagic.git
cd atlasopenmagic
pip install .
```

## Documentation
You can find the full documentation for ATLAS Open Magic in the [ATLAS Open Data website](https://opendata.atlas.cern/docs/atlasopenmagic).

## Quick start
First, import the package:
```python
import atlasopenmagic as atom
```
See the available releases and set to one of the options given by `available_releases()`
```python
atom.available_releases()
set_release('2024r-pp')
```
Check in the [Monte Carlo Metadata](https://opendata.atlas.cern/docs/data/for_research/metadata) which datasets do you want to retrieve and use the 'Dataset ID'. For example, to get the metadata from *Pythia8EvtGen_A14MSTW2008LO_Zprime_NoInt_ee_SSM3000*:
```python
all_metadata = atom.get_metadata('301204')
```
If we only want a specific variable:
```python
xsec = atom.get_metadata('301204', 'cross_section_pb')
```
To get the URLs to stream the files for that MC dataset:
```python
all_mc = atom.get_urls('301204')
```
To get some data instead, check the available options:
```python
atom.available_data()
```
And get the URLs for the one that's to be used:
```python
all_mc = atom.get_urls('data')
```

## Command-line interface
Installing the package also installs a CLI, available as both `atlasopenmagic` and the shorter `atom`. The two are the same program, so if `atom` clashes with something else on your machine, `atlasopenmagic` always works and can be aliased to whatever you prefer. It follows a `atom <group> <command> [arguments] [options]` layout and prints JSON to stdout, so it composes well with tools like `jq` or with shell scripts.

Pick a release once, then query without having to repeat yourself:
```bash
atom release set 2024r-pp
atom dataset show 301204 --field cross_section_pb
atom dataset urls 301204 --protocol https
atom dataset search process "pp>Zprime>ee"
atom weights names 301204
```

The command groups are:

| Group | Commands |
|---|---|
| `release` | `list`, `show`, `set <name>`, `unset` |
| `dataset` | `list`, `show <key>`, `urls <key>`, `search <field> <value>`, `build <defs.json>` |
| `metadata` | `fields`, `keywords`, `skims`, `dump`, `export <file>`, `import <file>` |
| `weights` | `show <key>`, `names <key>`, `list` |
| `cache` | `info`, `clear`, `localize <path>` |
| `env` | `install [packages...]` |

### Searching
`dataset search` reads its value as JSON where it can, so searches keep the types the Python API expects:
```bash
atom dataset search nEvents 20000                     # number, not the string "20000"
atom dataset search keywords '["2electron","BSM"]'    # requires both keywords
atom dataset search Filters null                      # datasets where the field is empty
atom dataset search process "pp>Zprime>ee"            # not valid JSON, so plain text
atom dataset search keywords 2024 --raw               # force text for a numeric-looking value
```

Quote lists and objects, or the shell will take them apart before `atom` ever sees them: `zsh` reads the brackets as a filename pattern and refuses to run the command, while `bash` strips the inner quotes and hands over `[2electron,BSM]`, which is no longer valid JSON. The CLI warns when it receives a value that opens like a list but doesn't parse, rather than silently searching for it as text.

### Reading data from disk
If you already have the files locally, point the CLI at them and the URLs come back as paths:
```bash
atom release set 2024r-pp --local-path /data/atlas    # remembered for this release
atom --local-path eos dataset urls 301204             # native POSIX /eos/... paths
```
`--local-path` on its own applies to a single command; on `release set` it is saved alongside the release. Use `atom cache localize <path>` instead when you want only the files that actually exist locally rewritten, leaving the rest as remote URLs.

Your local copy must mirror the dataset's path on EOS, minus the constant `/eos/opendata/atlas/` prefix. For example, `--local-path /data/atlas` expects:
```
/data/atlas/rucio/mc20_13TeV/DAOD_PHYSLITE.37620644._000012.pool.root.1
```

Run `atom --help` or `atom <group> <command> --help` for the full set of options. The deprecated library functions (`get_urls_data`, `build_mc_dataset`, `build_data_dataset`) are intentionally not exposed; use `dataset urls` and `dataset build` instead.

### Output
Commands that return data (`dataset urls`, `dataset show`, `metadata dump`, `weights ...`) print JSON, so they can be piped straight into `jq`. Commands that report state (`release show`, `release list`, `cache info`) print a short human-readable summary instead. Pass `--json` to force JSON everywhere:
```bash
atom release show            # Release: 2024r-pp / Source: config / Cache: fresh, just now
atom --json release show     # {"cache": "fresh, just now", "release": "2024r-pp", ...}
```

### Release selection
Each invocation is a separate process, so the release is resolved from, in order of precedence: the `--release` flag, the `ATLAS_RELEASE` environment variable, the release saved by `atom release set` (in `~/.config/atlasopenmagic/config.json`), and finally the library default. `atom release show` reports which one is in effect and where it came from.

### Caching
Because nothing persists in memory between invocations, the CLI caches each release's metadata under `~/.cache/atlasopenmagic/` so repeated commands don't refetch the whole release. `atom release set` downloads and caches the release up front, so the wait happens where you asked for it rather than on whichever query you happen to run first; pass `--no-fetch` to just save the setting. After that, queries are served from disk:
```bash
atom release set 2024r-pp    # fetches once, ~1.3s
atom dataset list            # served from cache, ~0.1s, no network
```
Entries expire after 7 days. Use `--refresh` to bypass the cache for one command, `atom cache info` to see what is cached, and `atom cache clear` to delete it. The cache is disposable: deleting it only costs one refetch.

`atom metadata import` loads a previously exported file into that cache under a name of your choice, which can then be selected like any other release:
```bash
atom metadata export snapshot.json
atom metadata import snapshot.json --as-release mysnapshot
atom --release mysnapshot dataset list
```

### Update notification
The CLI checks PyPI at most once a day for a newer release and prints a short notice to stderr if one is available. The check belongs to the CLI alone: using the package from Python (`import atlasopenmagic`) never triggers it, so imports in notebooks and scripts stay offline and just as fast as before. Disable it with `--no-update-check` or by setting `ATLASOPENMAGIC_NO_UPDATE_CHECK=1`.


## Contributing
Contributions are welcome! To contribute:

1. Fork the repository.
2. Create a new branch (`git checkout -b feature-name`).
3. Commit your changes (`git commit -am 'Add some feature'`).
4. Push to the branch (`git push origin feature-name`).
5. Create a Pull Request.

Please ensure all tests pass before submitting a pull request (just run `pytest` from the main directory of the package).

Developers can also `pip install` including additional tools required for testing:
```bash
pip install atlasopenmagic[dev]
```
or with a local copy of the repository:
```bash
pip install '.[dev]'
```

### Pre-commit Hooks

We use pre-commit hooks, find below how to use them.

#### Installation

1. Install the `[dev]` dependencies if you haven't already, as shown above.

2. Install the git hook scripts:

```sh
pre-commit install
```

3. (Optional) Run against all files:

```sh
pre-commit run --all-files
```

#### What the hooks do

- **black**: Formats Python code consistently
- **isort**: Sorts imports alphabetically and separates them into sections
- **ruff**: Fast Python linter that catches common errors and style issues
- **codespell**: Checks for common misspellings in code and comments
- **trailing-whitespace**: Removes trailing whitespace
- **end-of-file-fixer**: Ensures files end with a newline
- **pydocstyle**: Checks docstring style (Google convention)

The hooks will run automatically on `git commit`.
If any hook fails, the commit will be blocked until the issues are fixed.

## License
This project is licensed under the [Apache 2.0 License](https://github.com/atlas-outreach-data-tools/atlasopenmagic/blob/main/LICENSE)
