Metadata-Version: 2.4
Name: datacube-explorer
Version: 3.1.10
Summary: Web-based exploration of Open Data Cube collections
Author-email: Geoscience Australia <earth.observation@ga.gov.au>
License-Expression: Apache-2.0
Project-URL: Homepage, https://github.com/opendatacube/datacube-explorer
Project-URL: Bug Reports, https://github.com/opendatacube/datacube-explorer/issues
Project-URL: Source, https://github.com/opendatacube/datacube-explorer
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Requires-Python: >=3.12
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: cachetools
Requires-Dist: click>=8.3.2
Requires-Dist: datacube[postgres]<1.10.0,>=1.9.21
Requires-Dist: eodatasets3>=1.9
Requires-Dist: fiona>=1.10.0
Requires-Dist: geoalchemy2>=0.8
Requires-Dist: odc-geo
Requires-Dist: pandas
Requires-Dist: psycopg[c]
Requires-Dist: pygeofilter[backend-sqlalchemy]>=0.2.2
Requires-Dist: pyproj
Requires-Dist: python-rapidjson
Requires-Dist: shapely
Requires-Dist: sqlalchemy>=1.4
Requires-Dist: structlog>=20.2.0
Requires-Dist: werkzeug
Provides-Extra: deployment
Requires-Dist: datacube[performance]; extra == "deployment"
Requires-Dist: datacube-explorer[webserver]; extra == "deployment"
Requires-Dist: gunicorn[gevent,setproctitle]>=22.0.0; extra == "deployment"
Requires-Dist: sentry-sdk[flask]; extra == "deployment"
Requires-Dist: blinker; extra == "deployment"
Requires-Dist: prometheus-flask-exporter; extra == "deployment"
Provides-Extra: test
Requires-Dist: datacube-explorer[deployment]; extra == "test"
Requires-Dist: deepdiff; extra == "test"
Requires-Dist: docker; extra == "test"
Requires-Dist: docutils; extra == "test"
Requires-Dist: jsonschema>=4.18; extra == "test"
Requires-Dist: lxml_html_clean; extra == "test"
Requires-Dist: mypy; extra == "test"
Requires-Dist: pre-commit; extra == "test"
Requires-Dist: pytest; extra == "test"
Requires-Dist: pytest-benchmark[histogram]; extra == "test"
Requires-Dist: pytest-cov; extra == "test"
Requires-Dist: ruff; extra == "test"
Requires-Dist: selectolax; extra == "test"
Requires-Dist: types-cachetools; extra == "test"
Requires-Dist: types-docker; extra == "test"
Requires-Dist: types-pyyaml; extra == "test"
Provides-Extra: webserver
Requires-Dist: flask; extra == "webserver"
Requires-Dist: flask-caching; extra == "webserver"
Requires-Dist: flask-cors; extra == "webserver"
Requires-Dist: flask-themer>=1.4.3; extra == "webserver"
Requires-Dist: jinja2; extra == "webserver"
Requires-Dist: markupsafe; extra == "webserver"
Requires-Dist: pystac; extra == "webserver"
Dynamic: license-file

# Data Cube Explorer
[![Linting](https://github.com/opendatacube/datacube-explorer/workflows/Code%20Linting/badge.svg)](https://github.com/opendatacube/datacube-explorer/actions?query=workflow%3ALinting)
[![Tests](https://github.com/opendatacube/datacube-explorer/workflows/Tests/badge.svg)](https://github.com/opendatacube/datacube-explorer/actions?query=workflow%3ATests)
[![Docker](https://github.com/opendatacube/datacube-explorer/workflows/Docker/badge.svg)](https://github.com/opendatacube/datacube-explorer/actions?query=workflow%3ADocker)
[![Scan](https://github.com/opendatacube/datacube-explorer/workflows/Scan/badge.svg)](https://github.com/opendatacube/datacube-explorer/actions?query=workflow%3AScan)
[![coverage](https://codecov.io/gh/opendatacube/datacube-explorer/branch/develop/graph/badge.svg)](https://codecov.io/gh/opendatacube/datacube-explorer)
[![Doc](https://readthedocs.org/projects/datacube-explorer/badge/?version=latest)](http://datacube-explorer.readthedocs.org/en/latest/)

## Overview

Datacube Explorer is part of the Open Data Cube ecosystem of software for geospatial data management and access.
It is what you need to answer "What's in my datacube?".

It is a web application for exploring which products and data are available,
and visualising what is available through time and space.

It also includes a STAC API for HTTP based programmatic access to an Open Data Cube
database, rather than the PostgreSQL access required by the Core ODC library.

![Explorer Screenshot](docs/images/screenshot.png)


## Usage (quick-start)

Assuming you already have an Open Data Cube instance, Explorer will use
its existing settings.

Install Explorer:

    pip install datacube-explorer[webserver]

Generate summaries for all of your products:

    cubedash-gen --init --all

Run Explorer locally:

    cubedash-run

It will now be viewable on [http://localhost:8090](https://localhost:8090)

## Developer Setup

These directions are for running from a local folder in development. But it will run from any typical Python WSGI server.

Firstly, install the Open Data Cube. Use of a [Data Cube conda environment](https://opendatacube.readthedocs.io/en/latest/installation/setup/common_install.html)
is recommended. You may need to also `conda install -c conda-forge postgis`

Test that you can run `datacube system check`, and that it's connecting
to the correct datacube instance.

### Dependencies

Now install the explorer dependencies:

    # These two should come from conda if you're using it, not pypi
    conda install fiona shapely

    pip install -e .

### Summary generation

Initialise and create product summaries:

    cubedash-gen --init --all

(This can take a long time the first time, depending on your datacube size.)

Other available options can be seen by running `cubedash-gen --help`.

### Run

A `cubedash-run` command is available to run Explorer locally:

    $ cubedash-run
        * Running on http://localhost:8080/ (Press CTRL+C to quit)

(see `cubedash-run --help` for list of options)

But Explorer can be run using any typical Python WSGI server, for example [gunicorn](https://gunicorn.org/):

    pip install gunicorn
    gunicorn -b '127.0.0.1:8080' -w 4 cubedash.startup_utils:create_app()

Products will begin appearing one-by-one as the summaries are generated in the
background.  If impatient, you can manually navigate to a product using
`/<product_name`. (Eg `/ls5_nbar_albers`)

### Code Style

All code is formatted using [black](https://github.com/ambv/black), and checked
with [pyflakes](https://github.com/PyCQA/pyflakes).

They are included when installing the test dependencies:

    pip install -e .[test]

Run `make lint` to check your changes, and `make format` to format your code
automatically.

You may want to configure your editor to run black automatically on file save
(see the Black page for directions), or install the pre-commit hook within Git:

### Pre-commit setup

A [pre-commit](https://pre-commit.com/) config is provided to automatically format
and check your code changes. This allows you to immediately catch and fix
issues before you raise a failing pull request (which run the same checks under
Travis).

Install pre-commit from pip, and initialise it in your repo:

    pip install pre-commit
    pre-commit install

Your code will now be formatted and validated before each commit. You can also
invoke it manually by running `pre-commit run`

**Note**: If you use Conda, install from conda-forge (This is *required* because the pip
version uses virtualenvs which are incompatible with Conda's environments)

    conda install pre_commit

## FAQ


### Can I use a different datacube environment?

Set ODC's environment variable before running the server:

    export ODC_ENVIRONMENT=staging

You can always see which environment/settings will be used by running `datacube system check`.

See the ODC documentation for [datacube configuration and environments](https://opendatacube.readthedocs.io/en/latest/installation/database/configuration.html)

### How can I set different timezone

Datacube-explorer default timezone is configured to: `Australia/Darwin`.

To configure the instance to a different timezone, the following configuration needs to be applied:

- `app.config` variable `CUBEDASH_DEFAULT_TIMEZONE` (via environment variable `CUBEDASH_SETTINGS`, which points to a `.env.py` file)

### Can I add custom scripts or text to the page (such as analytics)?

Create one of the following `*.env.html` files:

- Global include: for `<script>` and other tags at the bottom of every page.

      cubedash/templates/include-global.env.html

- Footer text include. For human text such as Copyright statements.

      echo "Server <strong>staging-1.test</strong>" > cubedash/templates/include-footer.env.html

(`*.env.html` is the naming convention used for environment-specific templates: they are ignored by
Git)

### How can I configure the deployment?

Refer to [deployment instructions](https://datacube-explorer.readthedocs.io/en/latest/deploying.html) and [app-config](https://datacube-explorer.readthedocs.io/en/latest/config.html)

### How do I modify the CSS/Javascript?

The CSS is compiled from [Sass](https://sass-lang.com/), and the JavaScript is compiled from
[Typescript](https://www.typescriptlang.org/).

Install [npm](https://www.npmjs.com/get-npm), and then install them both:

    npm install -g sass typescript

You can now run `make static` to rebuild all the static files, or
individually with `make style` or `make js`.

Alternatively, if using [PyCharm](https://www.jetbrains.com/pycharm), open a
Sass file and you will be prompted to enable a `File Watcher` to
compile automatically.

PyCharm will also compile the Typescript automatically by ticking
the "Recompile on changes" option in `Languages & Frameworks ->
Typescript`.

### How do I run the integration tests?

The integration tests run against a real PostgreSQL database, which is
automatically started and stopped using Docker. This requires Docker to
be available, but no further database setup is required.

Install the test dependencies: `pip install -e .[test]`

The run the tests with: `pytest integration_tests`

**Docker Compose alternative:**
See below for running the tests using `docker compose`.

**Without Docker alternative:**

Assuming you have a PostgreSQL server running locally, that you're setup to log into.

```
# Create a test database to use
$ createdb odc-explorer-testing

# Add the PostGIS extensions to it
$ psql odc-explorer-testing
odc-explorer-testing=# create extension if not exists postgis;
CREATE EXTENSION

# Export environment variables so that the tests know which Database to use
$ export ODC_DEFAULT_INDEX_DRIVER=postgres
$ export ODC_POSTGIS_INDEX_DRIVER=postgis
$ export ODC_DEFAULT_DB_URL=postgresql://localhost/odc-explorer-testing
$ export ODC_POSTGIS_DB_URL=postgresql://localhost/odc-explorer-testing
$ export CUBEDASH_BYPASS_DOCKER=True

# Finally, run the tests
$ uv run pytest integration_tests/

```


### How do I add test data for the automated tests?

Most of the automated tests for Datacube Explorer require sample data to run. This comprises
definitions of ODC *Metadata Types*, *Products* and *Datasets*.

These are contained within YAML files in the [`integration_tests/data`](https://github.com/opendatacube/datacube-explorer/tree/develop/integration_tests/data) directory.

Test data is loaded using a pytest fixture called `auto_odc_db`, which is activated per
test module, and will automatically populate the database using files referenced in module
global variables. Activate and use it similar to the following example:


    pytestmark = pytest.mark.usefixtures("auto_odc_db")

    METADATA_TYPES = ["metadata/qga_eo.yaml"]
    PRODUCTS = ["products/ga_s2_ard.odc-product.yaml"]
    DATASETS = ["s2a_ard_granule.yaml.gz"]


To add sample datasets required for the test case, create a `.yaml` file
with the product name and place all the sample datasets split by `---` in the yaml.

If the sample datasets file is large, compress it with `gzip <dataset_file>.yaml` and reference
that file instead.

## Roles for production deployments

Explorer uses the default roles created by datacube-core. If you created your datacube without permissions,
you should re-run `datacube system init` as a database superuser or as the user that originally created the
datacube, to create the required roles and grant them the required permissions.

Running `cubedash-gen --init` grants permissions to Explorer-specific tables and views to these roles:

- `agdc_user` (postgres driver) or `odc_user` (postgis driver):
  Read-only access to the explorer summary tables and views.  This role should be used by the web interface.
- `agdc_manage` (postgres driver) or `odc_manage` (postgis driver):
  Read-write access to the explorer summary tables and views.  This role should be used by the `cubedash-gen` command.
- `agdc_admin` (postgres driver) or `odc_admin` (postgis driver):
  Owner of explorer summary tables and views. The initial call of the `cubedash-gen --init` command to create the
  Explorer schema and table should be run as a database superuser. The first schema update after upgrading from
  Explorer 3.1.2 to later versions will also need to be run as a database superuser.  Subsequent calls to
  `cubedash-gen --init` to update the schema will be safe to run as the `agdc_admin` role.

Prior to version 3.1.2, Explorer used its own roles, `explorer_viewer` and `explorer_generator`, and `explorer_owner`.  These
roles are granted the above roles by `cubeddash-gen --init` if they already exist, ensuring cross-compatibility with
older versions of Explorer.

## Docker for Development and running tests

You need to have Docker and Docker Compose installed on your system.

To create your environment, run `make up` or `docker-compose up`.

You need an ODC database, so you'll need to refer to the [ODC docs](https://opendatacube.readthedocs.io/en/latest/) for help on indexing, but you can create the database by running `make initdb` or `docker-compose exec explorer datacube system init`. (This is not enough, you still need to add a product and index datasets.)

When you have some ODC data indexed, you can run `make index` to create the Explorer indexes.

Once Explorer indexes have been created, you can browse the running application at [http://localhost:5000](http://localhost:5000).

You can run tests with `make test-docker`.

And you can run a single test in Docker using a command like this: `docker-compose --file docker-compose.yml run explorer pytest integration_tests/test_dataset_listing.py`


## Docker-compose for Development and running tests
### Testing with app.config
edit `.docker/settings_docker.py` and setup application config. Then `docker-compose -f docker-compose.yml -f docker-compose.override.yml up` to bring up explorer docker with database, explorer with settings


## STAC API Extensions

The STAC endpoint implements the [filter](https://github.com/stac-api-extensions/filter), [fields](https://github.com/stac-api-extensions/fields), and [sort](https://github.com/stac-api-extensions/sort) extensions, all of which are bound to the STAC API - Item Search (`/search`) endpoint. All support both GET and POST request syntax.

Fields contained in the item properties must be prefixed with `properties.`, ex `properties.dea:dataset_maturity`.

The implementation of `fields` differs somewhat from the suggested include/exclude semantics in that it does not permit for invalid STAC entities, so the `id`, `type`, `geometry`, `bbox`, `links`, `assets`, `properties.datetime`, `collection`, and `stac_version` fields will always be included, regardless of user input.

The `sort` and `filter` implementations will recognise any syntactically valid version of a property name, which is the say, the STAC, eo3, and search field (as defined by the metadata type) variants of the name, with or without the `item.` or `properties.` prefixes. If a property does not exist for an item, `sort` will ignore it while `filter` will treat it as `NULL`.

The `filter` extension supports both `cql2-text` and `cql2-json` for both GET and POST requesets, and uses [pygeofilter](https://github.com/geopython/pygeofilter) to parse the cql and convert it to a sqlalchemy filter expression. `filter-crs` only accepts http://www.opengis.net/def/crs/OGC/1.3/CRS84 as a valid value.


## Release process

Create a release using the GitHub release interface.
