Metadata-Version: 2.4
Name: grimoirelab-core
Version: 0.1.0
Summary: Core of GrimoireLab
License: GPL-3.0+
License-File: AUTHORS
License-File: LICENSE
Keywords: development,grimoirelab
Author: GrimoireLab Developers
Requires-Python: >=3.11,<4.0
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: GNU General Public License v3 or later (GPLv3+)
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Software Development
Requires-Dist: django (>=5.2,<6.0)
Requires-Dist: django-cors-headers (>=4.6.0,<5.0.0)
Requires-Dist: django-rq (>=3.1,<4.0)
Requires-Dist: django-storages[google] (>=1.14.6,<2.0.0)
Requires-Dist: djangorestframework (>=3.15.2,<4.0.0)
Requires-Dist: djangorestframework-simplejwt (>=5.4.0,<6.0.0)
Requires-Dist: drf-spectacular (>=0.28.0,<0.29.0)
Requires-Dist: grimoirelab-chronicler (>=0.0.1rc2)
Requires-Dist: grimoirelab-toolkit (>=1.2.0)
Requires-Dist: mysqlclient (>=2.0.3,<3.0.0)
Requires-Dist: opensearch-py (>=3.0.0,<4.0.0)
Requires-Dist: perceval (>=1.3.4)
Requires-Dist: sortinghat (>=1.11.0)
Requires-Dist: uWSGI (>=2.0,<3.0)
Project-URL: Bug Tracker, https://github.com/chaoss/grimoirelab-core/issues
Project-URL: Homepage, https://chaoss.github.io/grimoirelab/
Project-URL: Repository, https://github.com/chaoss/grimoirelab-core
Description-Content-Type: text/markdown

# GrimoireLab Core

GrimoireLab scheduler to fetch data from software repositories.

The scheduler is a distributed job queue that schedules and executes Perceval.
The platform has one or more workers that will run each Perceval job.

The repositories whose data will be fetched are added to the platform using a
REST API. Then, the server transforms these repositories into Perceval jobs and
schedules them between its job queues.

Workers are waiting for new jobs checking these queues. Workers only execute
a job at a time. When a new job arrives, an idle worker will take and run it.
Once a job is finished, if the result is successful, the server will re-schedule
it to retrieve new data.

By default, items fetched by each job will be published using a Redis queue.

## Requirements

- Python >= 3.11
- Redis database
- MySQL database

You will also need some other libraries for running the tool, you can find the
whole list of dependencies in [pyproject.toml](pyproject.toml) file.


## Installation

There are several ways to install GrimoireLab core on your system: packages or
source code using Poetry or pip.

### mysql_config

Before you install GrimoireLab core you might need to install `mysql_config`
command. If you are using a Debian based distribution, this command can be
found either in `libmysqlclient-dev` or `libmariadbclient-dev` packages
(depending on if you are using MySQL or MariaDB database server). You can
install these packages in your system with the next commands:

* **MySQL**

```
$ apt install libmysqlclient-dev
```

* **MariaDB**

```
$ apt install libmariadbclient-dev-compat
```

### PyPI

GrimoireLab core can be installed using pip, a tool for installing Python
packages. To do it, run the next command:
```
$ pip install grimoirelab-core
```

### Source code

To install from the source code you will need to clone the repository first:
```
$ git clone https://github.com/grimoirelab/grimoirelab-core
$ cd grimoirelab-core
```

Then use pip or Poetry to install the package along with its dependencies.

#### Pip
To install the package from local directory run the following command:
```
$ pip install .
```
In case you are a developer, you should install grimoirelab-core in editable mode:
```
$ pip install -e .
```

#### Poetry
We use [poetry](https://python-poetry.org/) for dependency management and 
packaging. You can install it following its [documentation](https://python-poetry.org/docs/#installation).
Once you have installed it, you can install grimoirelab-core and the dependencies
in a project isolated environment using:
```
$ poetry install
```
To spaw a new shell within the virtual environment use:
```
$ poetry shell
```

To build the static files automatically you can install the pre-hook plugin
included in the project. Each time you run `poetry install` or `poetry build`
the static files will be built automatically:
```
$ poetry self add "$PWD/build/poetry-prehook"
```

If you prefer to build the static files manually, you can run:
```
$ poetry run python build/build-ui.py
```

## Usage

```
$ grimoirelab
Usage: grimoirelab [OPTIONS] COMMAND [ARGS]...

  Toolset for software development analytics.

  GrimoireLab is a set of tools and a platform to retrieve, analyze, and
  provide insights about data coming from software development repositories.
  With this command, you'll be able to configure and run its different
  services.

  It requires to pass a configuration file module using the Python path syntax
  (e.g. grimoirelab.core.config.settings). Take into account the configuration
  should be accessible by your PYTHON_PATH. You can also use the environment
  variable GRIMOIRELAB_CONFIG to define the config location.

Options:
  --config TEXT  Configuration module in Python path syntax  [default:
                 grimoirelab.core.config.settings]
  --help         Show this message and exit.

Commands:
  admin  GrimoireLab administration tool.
  run    Run a GrimoireLab service.
```

## Configuration

The first step is to run a Redis server, a MySQL database and an OpenSearch 
container that will be used for communicating components and storing results.
Please refer to their documentation to know how to install and run them.

### Configuration variables

By default, GrimoireLab runs using a configuration file defined at
`grimoirelab.core.config.settings`. You can update that file or use
environment variables to override the default values.

#### Configure the database

This command will create the database tables and the initial admin user.

```
grimoirelab admin setup
```

#### Run eventizer workers

Run the eventizer workers that will fetch data from the repositories
and insert it into a Redis Stream.

```
grimoirelab run eventizers
```

#### Run archivist workers

Run the archivist workers that will read data from the Redis Stream and
store it into OpenSearch.

```
grimoirelab run archivists
```

### Run identities storage workers

Run the identities storage workers that will read data from the Redis Stream
and store it into SortingHat database.

```
grimoirelab run ushers
```

#### Run the backend API

Run the backend API server that will provide a REST API to manage the
repositories, jobs and identities.

It also provides a web interface to manage the platform.

```
grimoirelab run server --dev
```

## Running tests

GrimoireLab core comes with a comprehensive list of unit and integration tests.

- Unitary tests requires a SQL database running (MySQL or MariaDB).

- Integration tests uses testcontainers to spawn temporary containers (MariaDB,
Valkey, and OpenSearch).

```
(.venv)$ pytest
```

