Metadata-Version: 2.5
Name: atlasopenmagic
Version: 1.10.0
Summary: A utility package for retrieving ATLAS open data URLs and metadata.
Project-URL: Homepage, https://opendata.atlas.cern/
Project-URL: Repository, https://github.com/atlas-outreach-data-tools/atlasopenmagic
Project-URL: Documentation, https://opendata.atlas.cern/docs/atlasopenmagic
Author-email: ATLAS Collaboration <atlas-outreach-opendata-support@cern.ch>
License-Expression: Apache-2.0
License-File: LICENSE
Keywords: ATLAS,Open Data,Utilities
Classifier: Development Status :: 5 - Production/Stable
Classifier: Intended Audience :: Education
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Natural Language :: English
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Documentation
Classifier: Topic :: Education
Classifier: Topic :: Scientific/Engineering
Classifier: Topic :: Scientific/Engineering :: Physics
Classifier: Topic :: Utilities
Classifier: Typing :: Typed
Requires-Python: >=3.10
Requires-Dist: pyyaml
Requires-Dist: requests
Requires-Dist: tqdm
Provides-Extra: dev
Requires-Dist: pre-commit; extra == 'dev'
Requires-Dist: pylint; extra == 'dev'
Requires-Dist: pytest; extra == 'dev'
Requires-Dist: pytest-cov; extra == 'dev'
Description-Content-Type: text/markdown

# ATLAS Open Magic 🪄📊
[![Tests](https://github.com/atlas-outreach-data-tools/atlasopenmagic/actions/workflows/test.yml/badge.svg)](https://github.com/atlas-outreach-data-tools/atlasopenmagic/actions/workflows/test.yml)
![Dynamic TOML Badge](https://img.shields.io/badge/dynamic/toml?url=https%3A%2F%2Fraw.githubusercontent.com%2Fatlas-outreach-data-tools%2Fatlasopenmagic%2Frefs%2Fheads%2Fmain%2Fpyproject.toml&query=%24.project.version&label=pypi)
[![codecov](https://codecov.io/gh/atlas-outreach-data-tools/atlasopenmagic/graph/badge.svg?token=CNTZ8AEHIG)](https://codecov.io/gh/atlas-outreach-data-tools/atlasopenmagic)


**`atlasopenmagic`** is a Python package made to simplify working with ATLAS Open Data by providing utilities to manage metadata and URLs for streaming the data.

### Key Features:
- Simple functions to set the active data release (e.g., `2024r-pp`).
- Efficient local caching of metadata to minimize API calls.
- Helper functions to retrieve specific dataset information, including file URLs for different "skims" (filtered versions of datasets).
- Support for multiple URL protocols (root, https, eos).
- Configuration via environment variables for easy integration into different workflows.

## **Installation**
You can install this package using `pip`.

```bash
pip install atlasopenmagic
```
Alternatively, clone the repository and install locally:
```bash
git clone https://github.com/atlas-outreach-data-tools/atlasopenmagic.git
cd atlasopenmagic
pip install .
```

## Documentation
You can find the full documentation for ATLAS Open Magic in the [ATLAS Open Data website](https://opendata.atlas.cern/docs/atlasopenmagic).

## Quick start
First, import the package:
```python
import atlasopenmagic as atom
```
See the available releases and set to one of the options given by `available_releases()`
```python
atom.available_releases()
set_release('2024r-pp')
```
Check in the [Monte Carlo Metadata](https://opendata.atlas.cern/docs/data/for_research/metadata) which datasets do you want to retrieve and use the 'Dataset ID'. For example, to get the metadata from *Pythia8EvtGen_A14MSTW2008LO_Zprime_NoInt_ee_SSM3000*:
```python
all_metadata = atom.get_metadata('301204')
```
If we only want a specific variable:
```python
xsec = atom.get_metadata('301204', 'cross_section_pb')
```
To get the URLs to stream the files for that MC dataset:
```python
all_mc = atom.get_urls('301204')
```
To get some data instead, check the available options:
```python
atom.available_data()
```
And get the URLs for the one that's to be used:
```python
all_mc = atom.get_urls('data')
```

## Command-line interface
Installing the package also installs a CLI, available as both `atlasopenmagic` and the shorter `atom`. The two are the same program, so if `atom` clashes with something else on your machine, `atlasopenmagic` always works and can be aliased to whatever you prefer. It follows a `atom <group> <command> [arguments] [options]` layout and prints JSON to stdout, so it composes well with tools like `jq` or with shell scripts.

Pick a release once, then query without having to repeat yourself:
```bash
atom release set 2024r-pp
atom dataset show 301204 --field cross_section_pb
atom dataset urls 301204 --protocol https
atom dataset search process "pp>Zprime>ee"
atom weights names 301204
```

The command groups are:

| Group | Commands |
|---|---|
| `release` | `list`, `show`, `set <name>`, `unset` |
| `dataset` | `list`, `show <key>`, `urls <key>`, `search <field> <value>`, `build <defs.json>` |
| `metadata` | `fields`, `keywords`, `skims`, `dump`, `export <file>`, `import <file>` |
| `weights` | `show <key>`, `names <key>`, `list` |
| `cache` | `info`, `clear`, `localize <path>` |
| `env` | `install [packages...]` |

### Searching
`dataset search` reads its value as JSON where it can, so searches keep the types the Python API expects:
```bash
atom dataset search nEvents 20000                     # number, not the string "20000"
atom dataset search keywords '["2electron","BSM"]'    # requires both keywords
atom dataset search Filters null                      # datasets where the field is empty
atom dataset search process "pp>Zprime>ee"            # not valid JSON, so plain text
atom dataset search keywords 2024 --raw               # force text for a numeric-looking value
```

Quote lists and objects, or the shell will take them apart before `atom` ever sees them: `zsh` reads the brackets as a filename pattern and refuses to run the command, while `bash` strips the inner quotes and hands over `[2electron,BSM]`, which is no longer valid JSON. The CLI warns when it receives a value that opens like a list but doesn't parse, rather than silently searching for it as text.

### Reading data from disk
If you already have the files locally, point the CLI at them and the URLs come back as paths:
```bash
atom release set 2024r-pp --local-path /data/atlas    # remembered for this release
atom --local-path eos dataset urls 301204             # native POSIX /eos/... paths
```
`--local-path` on its own applies to a single command; on `release set` it is saved alongside the release. Use `atom cache localize <path>` instead when you want only the files that actually exist locally rewritten, leaving the rest as remote URLs.

Run `atom --help` or `atom <group> <command> --help` for the full set of options. The deprecated library functions (`get_urls_data`, `build_mc_dataset`, `build_data_dataset`) are intentionally not exposed; use `dataset urls` and `dataset build` instead.

### Output
Commands that return data (`dataset urls`, `dataset show`, `metadata dump`, `weights ...`) print JSON, so they can be piped straight into `jq`. Commands that report state (`release show`, `release list`, `cache info`) print a short human-readable summary instead. Pass `--json` to force JSON everywhere:
```bash
atom release show            # Release: 2024r-pp / Source: config / Cache: fresh, just now
atom --json release show     # {"cache": "fresh, just now", "release": "2024r-pp", ...}
```

### Release selection
Each invocation is a separate process, so the release is resolved from, in order of precedence: the `--release` flag, the `ATLAS_RELEASE` environment variable, the release saved by `atom release set` (in `~/.config/atlasopenmagic/config.json`), and finally the library default. `atom release show` reports which one is in effect and where it came from.

### Caching
Because nothing persists in memory between invocations, the CLI caches each release's metadata under `~/.cache/atlasopenmagic/` so repeated commands don't refetch the whole release. `atom release set` downloads and caches the release up front, so the wait happens where you asked for it rather than on whichever query you happen to run first; pass `--no-fetch` to just save the setting. After that, queries are served from disk:
```bash
atom release set 2024r-pp    # fetches once, ~1.3s
atom dataset list            # served from cache, ~0.1s, no network
```
Entries expire after 7 days. Use `--refresh` to bypass the cache for one command, `atom cache info` to see what is cached, and `atom cache clear` to delete it. The cache is disposable: deleting it only costs one refetch.

`atom metadata import` loads a previously exported file into that cache under a name of your choice, which can then be selected like any other release:
```bash
atom metadata export snapshot.json
atom metadata import snapshot.json --as-release mysnapshot
atom --release mysnapshot dataset list
```

### Update notification
The CLI checks PyPI at most once a day for a newer release and prints a short notice to stderr if one is available. The check belongs to the CLI alone: using the package from Python (`import atlasopenmagic`) never triggers it, so imports in notebooks and scripts stay offline and just as fast as before. Disable it with `--no-update-check` or by setting `ATLASOPENMAGIC_NO_UPDATE_CHECK=1`.


## Contributing
Contributions are welcome! To contribute:

1. Fork the repository.
2. Create a new branch (`git checkout -b feature-name`).
3. Commit your changes (`git commit -am 'Add some feature'`).
4. Push to the branch (`git push origin feature-name`).
5. Create a Pull Request.

Please ensure all tests pass before submitting a pull request (just run `pytest` from the main directory of the package).

Developers can also `pip install` including additional tools required for testing:
```bash
pip install atlasopenmagic[dev]
```
or with a local copy of the repository:
```bash
pip install '.[dev]'
```

### Pre-commit Hooks

We use pre-commit hooks, find below how to use them.

#### Installation

1. Install the `[dev]` dependencies if you haven't already, as shown above.

2. Install the git hook scripts:

```sh
pre-commit install
```

3. (Optional) Run against all files:

```sh
pre-commit run --all-files
```

#### What the hooks do

- **black**: Formats Python code consistently
- **isort**: Sorts imports alphabetically and separates them into sections
- **ruff**: Fast Python linter that catches common errors and style issues
- **codespell**: Checks for common misspellings in code and comments
- **trailing-whitespace**: Removes trailing whitespace
- **end-of-file-fixer**: Ensures files end with a newline
- **pydocstyle**: Checks docstring style (Google convention)

The hooks will run automatically on `git commit`.
If any hook fails, the commit will be blocked until the issues are fixed.

## License
This project is licensed under the [Apache 2.0 License](https://github.com/atlas-outreach-data-tools/atlasopenmagic/blob/main/LICENSE)
