Metadata-Version: 2.4
Name: cernml-coi
Version: 0.9.6
Summary: Common optimization interfaces for RL/num. optimization
Author-email: Penny Madysa <penny.madysa@cern.ch>
License: Except as otherwise noted (below and/or in individual files), this work is
        licensed under either of GNU Public License, Version 3.0 or later
        <LICENSES/GPL-3.0-or-later.txt>, or European Union Public License, Version 1.2
        or later <LICENSES/EUPL-1.2.txt>, at your option.
        
        Unless You explicitly state otherwise, any contribution intentionally submitted
        by You for inclusion in this Work (the Covered Work) shall be dual-licensed as
        above, without any additional terms or conditions.
        
        For full authorship information, see the version control history.
        
        * The registration package, located in <src/cernml/coi/registration/>, is based
          on parts of the Gymnasium package, available under at
          <https://github.com/Farama-Foundation/Gymnasium/>. It is
          Copyright (c) 2016 OpenAI
          Copyright (c) 2022 Farama Foundation
          and licensed under the MIT License <LICENSES/MIT.txt>.
        
Project-URL: Homepage, https://geoff.docs.cern.ch/
Project-URL: Download, https://gitlab.cern.ch/geoff/cernml-coi/-/releases
Project-URL: Documentation, https://geoff.docs.cern.ch/cernml-coi/
Project-URL: Internal Documentation Mirror, https://acc-py.web.cern.ch/gitlab/geoff/cernml-coi/
Project-URL: Source Code, https://gitlab.cern.ch/geoff/cernml-coi
Project-URL: Bug Tracker, https://gitlab.cern.ch/geoff/cernml-coi/-/work_items
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Science/Research
Classifier: Natural Language :: English
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Scientific/Engineering :: Physics
Classifier: Typing :: Typed
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: COPYING
Requires-Dist: gymnasium<1
Requires-Dist: numpy>=1.22
Requires-Dist: scipy>=1.1
Requires-Dist: importlib-metadata>=4.8.0; python_version < "3.10"
Requires-Dist: typing-extensions>=4.3.0
Provides-Extra: matplotlib
Requires-Dist: matplotlib~=3.8; extra == "matplotlib"
Requires-Dist: pyparsing<3.3; python_version < "3.10" and extra == "matplotlib"
Provides-Extra: optimizers
Requires-Dist: cernml-coi-optimizers~=3.0; extra == "optimizers"
Provides-Extra: robotics
Requires-Dist: gymnasium-robotics~=1.0; extra == "robotics"
Provides-Extra: all
Requires-Dist: cernml-coi[matplotlib,optimizers]; extra == "all"
Provides-Extra: doc
Requires-Dist: sphinx>=4.0; extra == "doc"
Requires-Dist: sphinx-inline-tabs; extra == "doc"
Requires-Dist: python_docs_theme; extra == "doc"
Provides-Extra: examples
Requires-Dist: cernml-coi[all]; extra == "examples"
Requires-Dist: PyQt5~=5.0; extra == "examples"
Requires-Dist: stable_baselines3~=2.0; extra == "examples"
Provides-Extra: test
Requires-Dist: cernml-coi[all]; extra == "test"
Requires-Dist: pytest>=6.0; extra == "test"
Requires-Dist: pytest-cov; extra == "test"
Requires-Dist: types-docutils; extra == "test"
Requires-Dist: typing-extensions; extra == "test"
Dynamic: license-file

<!--
SPDX-FileCopyrightText: 2020-2026 CERN
SPDX-FileCopyrightText: 2023-2026 GSI Helmholtzzentrum für Schwerionenforschung
SPDX-FileNotice: All rights not expressly granted are reserved.

SPDX-License-Identifier: GPL-3.0-or-later OR EUPL-1.2+
-->

Common Optimization Interfaces
==============================

CernML is the project of bringing numerical optimization, machine learning and
reinforcement learning to the operation of the CERN accelerator complex.

CernML-COI defines common interfaces that facilitate using numerical
optimization and reinforcement learning (RL) on the same optimization problems.
This makes it possible to unify both approaches into a generic optimization
application in the CERN Control Center.

This repository can be found online on CERN's [Gitlab][].

[Gitlab]: https://gitlab.cern.ch/geoff/cernml-coi/

Table of Contents
-----------------

[[_TOC_]]

Motivation
----------

Several problems in accelerator control can be solved both using reinforcement
learning (RL) and numerical optimization. However, both approaches usually
slightly differ in their expected input and output:

- Optimizers pick certain *points* in the phase space of modifiable parameters
  and evaluate the loss of these parameters. They minimize this loss through
  multiple evaluations and ultimately yield the optimal parameters.
- RL agents assume that the problem has a certain state, which usually contains
  the values of all modifiable parameters. They receive an observation (which
  is usually higher-dimensional than the loss) and calculate a *correction* of
  the parameters. This correction yields a certain reward to them. Their goal
  is to optimize the parameters incrementally by optimzing their corrections
  for maximal *cumulative* reward.

More informally, optimizers start from scratch each time they are applied and
they yield a point in phase space. RL agents learn once, can be applied many
times, and they yield a sequence of deltas in the phase space.

Even more informally, on a given machine, an optimizer performs the state
transition `machine.parameters = new_parameters`, whereas an RL agent performs
the state transition `machine.parameters += corrections` iteratively.

This package provides interfaces to implement for problems that should be
compatible both with numerical optimizers and RL agents. It is based on the
[Gym][] environment API and enhances it with the `SingleOptimizable` interface.

In addition, the output and metadata of the environments is *restricted* to
make the behavior of environments more uniform and compatible to make them more
easily visualizable and integrable into a generic machine-optimization
application.

[Gym]: https://github.com/openai/gym/

Quickstart
----------

Start a Python project. In your `setup.cfg` or `setup.py`, add dependencies on
Gymnasium and the COI. Make sure to pick a COI version that is supported by the
application that will optimize your problem.

```conf
# setup.cfg
[options]
install_requires =
    gymnasium >= 0.29
    cernml-coi >= 0.9.0
```

Then, write a class that implements one or multiple of the optimization
interfaces. Finally *register* it so that an application that imports your
package may find it. (See the [Parabola example](/examples/parabola.py) for a
more fully featured version of the code below.)

```python
# my_project/__init__.py
import gymnasium as gym
import numpy as np
from cernml import coi

class Parabola(coi.SingleOptimizable, gym.Env):
    observation_space = gym.spaces.Box(-2.0, 2.0, shape=(2,))
    action_space = gym.spaces.Box(-1.0, 1.0, shape=(2,))
    optimization_space = gym.spaces.Box(-2.0, 2.0, shape=(2,))
    metadata = {
        "render_modes": [],
        "cern.machine": coi.Machine.NO_MACHINE,
    }

    def __init__(self, render_mode=None):
        self.render_mode = render_mode
        self.pos = np.zeros(2)
        self._train = True

    def reset(self, *, seed=None, options=None):
        super.reset(seed=seed, options=options)
        self.pos = self.action_space.sample()
        return self.pos.copy()

    def step(self, action):
        next_pos = self.pos + action
        ob_space = self.observation_space
        self.pos = np.clip(next_pos, ob_space.low, ob_space.high)
        reward = -sum(self.pos ** 2)
        terminated = (reward > -0.05) or next_pos not in ob_space
        truncated = False
        return self.pos.copy(), reward, terminated, truncated, {}

    def get_initial_params(self, *, seed=None, options=None):
        return self.reset(seed=seed, options=options)

    def compute_single_objective(self, params):
        ob_space = self.observation_space
        self.pos = np.clip(params, ob_space.low, ob_space.high)
        loss = sum(self.pos ** 2)
        return loss

coi.register("Parabola-v0", entry_point=Parabola, max_episode_steps=10)
```

Any [*host application*][GeOFF] may then import your package and instantiate
your optimization problem.

```python
import my_project
from cernml import coi

problem = coi.make("Parabola-v0")
optimize_in_some_way(problem)
```

[GeOFF]: https://gitlab.cern.ch/geoff/geoff-app

Documentation
-------------

Inside the CERN network, you can read the package documentation on the [Acc-Py
documentation server][acc-py-docs]. [The same documentation is available
outside of CERN][pages-docs] via CERN's [Gitlab Pages][pages-howto] service,
though some cross-links to CERN-internal projects may not work. Finally, API
documentation is provided through extensive Python docstrings.

[acc-py-docs]: https://acc-py.web.cern.ch/gitlab/geoff/cernml-coi/
[pages-docs]: https://cernml-coi.docs.cern.ch/
[pages-howto]: https://how-to.docs.cern.ch/

Changelog
---------

[See here](https://cernml-coi.docs.cern.ch/changelog.html).

Stability
---------

This package uses a variant of [Semantic Versioning](https://semver.org/) that
makes additional promises during the initial development (major version 0):
whenever breaking changes to the public API are published, the first non-zero
version number will increase. This means that code that uses COI version 0.9.0
will continue to work with version 0.9.1, but may break with version 0.10.0.

License
-------

Except as otherwise noted, this work is licensed under either of [GNU Public
License, Version 3.0 or later](LICENSES/GPL-3.0-or-later.txt), or [European
Union Public License, Version 1.2 or later](LICENSES/EUPL-1.2.txt), at your
option. See [COPYING](COPYING) for details.

Unless You explicitly state otherwise, any contribution intentionally submitted
by You for inclusion in this Work (the Covered Work) shall be dual-licensed as
above, without any additional terms or conditions.

For full authorship information, see the version control history.
