Metadata-Version: 2.4
Name: gradient-free-optimizers
Version: 1.13.0
Summary: Lightweight optimization with local, global, population-based and sequential techniques across mixed search spaces 
Author-email: Simon Blanke <simon.blanke@yahoo.com>
Maintainer-email: Simon Blanke <simon.blanke@yahoo.com>
License-Expression: MIT
Project-URL: Homepage, https://github.com/SimonBlanke/Gradient-Free-Optimizers
Project-URL: Documentation, https://gradient-free-optimizers.readthedocs.io/
Project-URL: Bug Reports, https://github.com/SimonBlanke/Gradient-Free-Optimizers/issues
Project-URL: Source, https://github.com/SimonBlanke/Gradient-Free-Optimizers/
Keywords: optimization,hyperparameter-optimization,hyperparameter-tuning,machine-learning,meta-heuristic,bayesian-optimization,genetic-algorithm,particle-swarm,evolution-strategy,hill-climbing,simulated-annealing,gradient-free,black-box-optimization,numerical-optimization,automl,search-algorithms,optimization-algorithms,data-science,visualization
Classifier: Development Status :: 5 - Production/Stable
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Operating System :: OS Independent
Classifier: Natural Language :: English
Classifier: Topic :: Scientific/Engineering
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Scientific/Engineering :: Information Analysis
Classifier: Topic :: Scientific/Engineering :: Mathematics
Classifier: Topic :: Software Development :: Libraries
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Education
Classifier: Intended Audience :: Information Technology
Classifier: Intended Audience :: Science/Research
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: numpy<3.0.0,>=1.18.1
Requires-Dist: pandas<3.0.0
Provides-Extra: progress
Requires-Dist: tqdm<5.0.0,>=4.48.0; extra == "progress"
Provides-Extra: scipy
Requires-Dist: scipy<2.0.0; extra == "scipy"
Provides-Extra: sklearn
Requires-Dist: scikit-learn!=0.23.*,>=0.21; extra == "sklearn"
Provides-Extra: full
Requires-Dist: scipy<2.0.0; extra == "full"
Requires-Dist: scikit-learn!=0.23.*,>=0.21; extra == "full"
Requires-Dist: tqdm<5.0.0,>=4.48.0; extra == "full"
Provides-Extra: build
Requires-Dist: setuptools; extra == "build"
Requires-Dist: build; extra == "build"
Requires-Dist: wheel; extra == "build"
Provides-Extra: test
Requires-Dist: pytest==9.0.3; extra == "test"
Requires-Dist: flake8; extra == "test"
Requires-Dist: pytest-cov; extra == "test"
Requires-Dist: surfaces; extra == "test"
Requires-Dist: scikit-learn!=0.23.*,>=0.21; extra == "test"
Requires-Dist: scipy<2.0.0; extra == "test"
Dynamic: license-file

<p align="center">
  <a href="https://github.com/SimonBlanke/Gradient-Free-Optimizers">
    <picture>
      <source media="(prefers-color-scheme: dark)" srcset="./docs/images/gradient_logo_dark_c.svg">
      <source media="(prefers-color-scheme: light)" srcset="./docs/images/gradient_logo_ink_paths.svg">
      <img src="./docs/images/gradient_logo_ink_paths.svg" width="400" alt="Gradient-Free-Optimizers Logo">
    </picture>
  </a>
</p>

---

<h3 align="center">
Lightweight optimization with local, global, population-based and sequential techniques across mixed search spaces
</h3>

<p align="center">
  <a href="https://github.com/SimonBlanke/Gradient-Free-Optimizers/actions"><img src="https://img.shields.io/github/actions/workflow/status/SimonBlanke/Gradient-Free-Optimizers/ci.yml?style=for-the-badge&logo=githubactions&logoColor=white&label=tests" alt="Tests"></a>
  <a href="https://app.codecov.io/gh/SimonBlanke/Gradient-Free-Optimizers"><img src="https://img.shields.io/codecov/c/github/SimonBlanke/Gradient-Free-Optimizers?style=for-the-badge&logo=codecov&logoColor=white" alt="Coverage"></a>
</p>

<br>

<table align="center">
  <tr>
    <td align="right"><b>Documentation</b></td>
    <td align="center">▸</td>
    <td>
      <a href="https://gradient-free-optimizers.readthedocs.io/en/latest/">Homepage</a> ·
      <a href="https://gradient-free-optimizers.readthedocs.io/en/latest/user_guide/optimizers/index.html">Optimizers</a> ·
      <a href="https://gradient-free-optimizers.readthedocs.io/en/latest/api_reference.html">API Reference</a> ·
      <a href="https://gradient-free-optimizers.readthedocs.io/en/latest/examples.html">Examples</a>
    </td>
  </tr>
  <tr>
    <td align="right"><b>On this page</b></td>
    <td align="center">▸</td>
    <td>
      <a href="#key-features">Features</a> ·
      <a href="#examples">Examples</a> ·
      <a href="#core-concepts">Concepts</a> ·
      <a href="#citation">Citation</a>
    </td>
  </tr>
</table>

<br>

---

<a href="https://github.com/SimonBlanke/Gradient-Free-Optimizers">
  <img src="./docs/gifs/3d_optimizer_animation.gif" width="240" align="right" alt="Bayesian Optimization on Ackley Function">
</a>

**Gradient-Free-Optimizers** is a Python library for gradient-free optimization of black-box functions. It provides a unified interface to 23 optimization algorithms, from simple hill climbing to Bayesian optimization, all operating on mixed search spaces that combine continuous ranges, discrete grids, categorical choices, and SciPy distribution-backed dimensions.

Designed for hyperparameter tuning, simulation optimization, feature selection, engineering design, and any scenario where gradients are unavailable or impractical. The library prioritizes simplicity: define your objective function, specify the search space, and run. All algorithms share one consistent API, so switching from hill climbing to Bayesian optimization is a one-line change. SciPy is optional; GFO works with only pandas as a required dependency, making it suitable as an optimization backend or for minimal environments, containers, and embedded systems.

<p>
  <a href="https://www.linkedin.com/in/simonblanke/"><img src="https://img.shields.io/badge/LinkedIn-Follow-0A66C2?style=flat-square&logo=linkedin" alt="LinkedIn"></a>
</p>

<br>

## Installation

```bash
pip install gradient-free-optimizers
```

<p>
  <a href="https://pypi.org/project/gradient-free-optimizers/"><img src="https://img.shields.io/pypi/v/gradient-free-optimizers?style=flat-square&color=blue" alt="PyPI"></a>
  <a href="https://pypi.org/project/gradient-free-optimizers/"><img src="https://img.shields.io/pypi/pyversions/gradient-free-optimizers?style=flat-square" alt="Python"></a>
  <a href="https://pepy.tech/project/gradient-free-optimizers"><img src="https://img.shields.io/pepy/dt/gradient-free-optimizers?style=flat-square&color=green" alt="Total Downloads"></a>
</p>

<details>
<summary>Optional dependencies</summary>

```bash
pip install gradient-free-optimizers[progress]  # Progress bar with tqdm
pip install gradient-free-optimizers[sklearn]   # scikit-learn for surrogate models
pip install gradient-free-optimizers[full]      # All optional dependencies
```

</details>

<br>

## Key Features

<table>
  <tr>
    <td width="33%">
      <a href="https://gradient-free-optimizers.readthedocs.io/en/latest/user_guide/optimizers/index.html"><b>23 Optimization Algorithms</b></a><br>
      <sub>Local, global, population-based, and sequential model-based optimizers. Switch algorithms with one line of code.</sub>
    </td>
    <td width="33%">
      <a href="https://gradient-free-optimizers.readthedocs.io/en/latest/api_reference.html"><b>Zero Configuration</b></a><br>
      <sub>Sensible defaults for all parameters. Start optimizing immediately without tuning the optimizer itself.</sub>
    </td>
    <td width="33%">
      <a href="https://gradient-free-optimizers.readthedocs.io/en/latest/user_guide/memory.html"><b>Memory System</b></a><br>
      <sub>Built-in caching prevents redundant evaluations. Critical for expensive objective functions like ML models.</sub>
    </td>
  </tr>
  <tr>
    <td width="33%">
      <a href="https://gradient-free-optimizers.readthedocs.io/en/latest/user_guide/search_spaces.html"><b>Mixed Search Spaces</b></a><br>
      <sub>Combine continuous ranges, discrete grids, categorical choices, and SciPy distributions in a single search space.</sub>
    </td>
    <td width="33%">
      <a href="https://gradient-free-optimizers.readthedocs.io/en/latest/user_guide/constraints.html"><b>Constraints Support</b></a><br>
      <sub>Define constraint functions to restrict the search space. Invalid regions are automatically avoided.</sub>
    </td>
    <td width="33%">
      <a href="https://github.com/SimonBlanke/Gradient-Free-Optimizers"><b>Minimal Dependencies</b></a><br>
      <sub>Only pandas required. Optional integrations for progress bars (tqdm) and surrogate models (scikit-learn).</sub>
    </td>
  </tr>
</table>

<br>

## Quick Start

```python
import numpy as np
from gradient_free_optimizers import HillClimbingOptimizer

def objective(params):
    x, y = params["x"], params["y"]
    return -(x**2 + y**2)

search_space = {
    "x": (-5.0, 5.0),            # continuous range
    "y": np.arange(-5, 5, 0.1),  # discrete grid
}

opt = HillClimbingOptimizer(search_space)
opt.search(objective, n_iter=1000)

print(f"Best score: {opt.best_score}")
print(f"Best params: {opt.best_para}")
```

<br>

## Core Concepts

```mermaid
flowchart LR
    O["Optimizer
    ━━━━━━━━━━
    23 algorithms"]

    S["Search Space
    ━━━━━━━━━━━━
    mixed dimensions"]

    F["Objective
    ━━━━━━━━━━
    f(params) → score"]

    D[("Search Data
    ━━━━━━━━━━━
    history")]

    O -->|propose| S
    S -->|params| F
    F -->|score| O

    O -.-> D
    D -.->|warm start| O
```

**Optimizer**: Implements the search strategy. Choose from 23 algorithms across four categories: local search, global search, population-based, and sequential model-based.

**Search Space**: Defines valid parameter ranges and choices. Each key is a parameter name, each value is a tuple `(min, max)` for continuous, a NumPy array for discrete, a list for categorical, or a SciPy distribution.

**Objective Function**: Your function to maximize. Takes a dictionary of parameters, returns a score. Use negation to minimize.

**Search Data**: Complete history of all evaluations accessible via `opt.search_data` for analysis and warm-starting future searches.

<br>

## Examples

<details open>
<summary><b>Hyperparameter Optimization</b></summary>

```python
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.model_selection import cross_val_score
from sklearn.datasets import load_wine
import numpy as np

from gradient_free_optimizers import BayesianOptimizer

X, y = load_wine(return_X_y=True)

def objective(params):
    model = GradientBoostingClassifier(
        n_estimators=params["n_estimators"],
        max_depth=params["max_depth"],
        learning_rate=params["learning_rate"],
    )
    return cross_val_score(model, X, y, cv=5).mean()

search_space = {
    "n_estimators": np.arange(50, 300, 10),
    "max_depth": np.arange(2, 10),
    "learning_rate": np.logspace(-3, 0, 20),
}

opt = BayesianOptimizer(search_space)
opt.search(objective, n_iter=50)
```

</details>



<details>
<summary><b>Bayesian Optimization</b></summary>

```python
import numpy as np
from gradient_free_optimizers import BayesianOptimizer

def ackley(params):
    x, y = params["x"], params["y"]
    return -(
        -20 * np.exp(-0.2 * np.sqrt(0.5 * (x**2 + y**2)))
        - np.exp(0.5 * (np.cos(2 * np.pi * x) + np.cos(2 * np.pi * y)))
        + np.e + 20
    )

search_space = {
    "x": np.arange(-5, 5, 0.01),
    "y": np.arange(-5, 5, 0.01),
}

opt = BayesianOptimizer(search_space)
opt.search(ackley, n_iter=100)
```

</details>



<details>
<summary><b>Particle Swarm Optimization</b></summary>

```python
import numpy as np
from gradient_free_optimizers import ParticleSwarmOptimizer

def rastrigin(params):
    A = 10
    values = [params[f"x{i}"] for i in range(5)]
    return -sum(v**2 - A * np.cos(2 * np.pi * v) + A for v in values)

search_space = {f"x{i}": np.arange(-5.12, 5.12, 0.1) for i in range(5)}

opt = ParticleSwarmOptimizer(search_space, population=20)
opt.search(rastrigin, n_iter=500)
```

</details>



<details>
<summary><b>Simulated Annealing</b></summary>

```python
import numpy as np
from gradient_free_optimizers import SimulatedAnnealingOptimizer

def sphere(params):
    return -(params["x"]**2 + params["y"]**2)

search_space = {
    "x": np.arange(-10, 10, 0.1),
    "y": np.arange(-10, 10, 0.1),
}

opt = SimulatedAnnealingOptimizer(
    search_space,
    start_temp=1.2,
    annealing_rate=0.99,
)
opt.search(sphere, n_iter=1000)
```

</details>



<details>
<summary><b>Constrained Optimization</b></summary>

```python
import numpy as np
from gradient_free_optimizers import RandomSearchOptimizer

def objective(params):
    return params["x"] + params["y"]

def constraint(params):
    # Only positions where x + y < 5 are valid
    return params["x"] + params["y"] < 5

search_space = {
    "x": np.arange(0, 10, 0.1),
    "y": np.arange(0, 10, 0.1),
}

opt = RandomSearchOptimizer(search_space, constraints=[constraint])
opt.search(objective, n_iter=1000)
```

</details>



<details>
<summary><b>Mixed Search Space</b></summary>

```python
import numpy as np
from scipy import stats
from gradient_free_optimizers import BayesianOptimizer

def objective(params):
    x = params["x"]
    n_layers = params["n_layers"]
    lr = params["learning_rate"]
    activation_scores = {"relu": 0.0, "tanh": 0.1, "gelu": 0.3}
    return -(x**2) - 0.1 * n_layers + activation_scores[params["activation"]] - abs(lr - 0.001)

search_space = {
    "x": (-5.0, 5.0),                          # continuous
    "n_layers": np.arange(1, 6),                # discrete
    "activation": ["relu", "tanh", "gelu"],     # categorical
    "learning_rate": stats.loguniform(1e-5, 1),  # distribution
}

opt = BayesianOptimizer(search_space)
opt.search(objective, n_iter=100)
```

</details>

<br>

<details>
<summary><b>Memory and Warm Starting</b></summary>

```python
import numpy as np
from gradient_free_optimizers import HillClimbingOptimizer

def expensive_function(params):
    # Simulating an expensive computation
    return -(params["x"]**2 + params["y"]**2)

search_space = {
    "x": np.arange(-10, 10, 0.1),
    "y": np.arange(-10, 10, 0.1),
}

# First search
opt1 = HillClimbingOptimizer(search_space)
opt1.search(expensive_function, n_iter=100, memory=True)

# Continue with warm start using previous search data
opt2 = HillClimbingOptimizer(search_space)
opt2.search(expensive_function, n_iter=100, memory_warm_start=opt1.search_data)
```

</details>



<details>
<summary><b>Ask/Tell Interface</b></summary>

```python
import numpy as np
from gradient_free_optimizers import BayesianOptimizer

def objective(params):
    return -(params["x"]**2 + params["y"]**2)

search_space = {
    "x": np.arange(-10, 10, 0.1),
    "y": np.arange(-10, 10, 0.1),
}

# Manual control over the optimization loop
opt = BayesianOptimizer(search_space)
opt.setup_search(objective, n_iter=100)

for _ in range(100):
    params = opt.ask()           # Get next parameters to evaluate
    score = objective(params)
    opt.tell(params, score)      # Report result back
```

</details>



<details>
<summary><b>Early Stopping</b></summary>

```python
import numpy as np
from gradient_free_optimizers import BayesianOptimizer

def objective(params):
    return -(params["x"]**2 + params["y"]**2)

search_space = {
    "x": np.arange(-10, 10, 0.1),
    "y": np.arange(-10, 10, 0.1),
}

opt = BayesianOptimizer(search_space)
opt.search(
    objective,
    n_iter=1000,
    max_time=60,           # Stop after 60 seconds
    max_score=-0.01,       # Stop when score reaches -0.01
    early_stopping={       # Stop if no improvement for 50 iterations
        "n_iter_no_change": 50,
    },
)
```

</details>

<br>

## Ecosystem

GFO is used as the optimization engine in other packages and integrates with the broader Python optimization ecosystem. For updates, [follow on GitHub](https://github.com/SimonBlanke).

| Package | Description |
|---------|-------------|
| [Hyperactive](https://github.com/SimonBlanke/Hyperactive) | High-level hyperparameter optimization framework, uses GFO as its optimization backend |
| [Surfaces](https://github.com/SimonBlanke/Surfaces) | Test functions and benchmark surfaces for optimization algorithm evaluation |

<br>

## Documentation

| Resource | Description |
|----------|-------------|
| [User Guide](https://gradient-free-optimizers.readthedocs.io/en/latest/user_guide.html) | Comprehensive tutorials and explanations |
| [API Reference](https://gradient-free-optimizers.readthedocs.io/en/latest/api_reference.html) | Complete API documentation |
| [Optimizers](https://gradient-free-optimizers.readthedocs.io/en/latest/user_guide/optimizers/index.html) | Detailed description of all 23 algorithms |
| [Examples](https://gradient-free-optimizers.readthedocs.io/en/latest/examples.html) | Code examples for various use cases |

<br>

## Contributing

Contributions welcome! See [CONTRIBUTING.md](./CONTRIBUTING.md) for guidelines.

- **Bug reports**: [GitHub Issues](https://github.com/SimonBlanke/Gradient-Free-Optimizers/issues)
- **Feature requests**: [GitHub Discussions](https://github.com/SimonBlanke/Gradient-Free-Optimizers/discussions)
- **Questions**: [GitHub Issues](https://github.com/SimonBlanke/Gradient-Free-Optimizers/issues)

<br>

## Citation

If you use this software in your research, please cite:

```bibtex
@software{gradient_free_optimizers,
  author = {Simon Blanke},
  title = {Gradient-Free-Optimizers: Simple and reliable optimization with local, global, population-based and sequential techniques in mixed search spaces},
  year = {2020},
  url = {https://github.com/SimonBlanke/Gradient-Free-Optimizers},
}
```

<br>

## License

[MIT License](./LICENSE) - Free for commercial and academic use.
