Metadata-Version: 2.4
Name: fielddash
Version: 0.1.1
Summary: Schema-driven dashboards for field data collection (Epicollect5).
Author: André Moura Lima
Author-email: Sergio Souza Costa <sergio.costa@ufma.br>, Ricardo Luvizotto Santos <ricardo.luvizotto@ufma.br>, Marianna Basso Jorge <mb.jorge@ufma.br>
License-Expression: MIT
Project-URL: Homepage, https://github.com/LambdaGeo/fielddash
Project-URL: Repository, https://github.com/LambdaGeo/fielddash
Project-URL: Documentation, https://lambdageo.github.io/fielddash/
Project-URL: Bug Tracker, https://github.com/LambdaGeo/fielddash/issues
Keywords: epicollect5,fieldwork,data-collection,streamlit,gis,survey,geoprocessing
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Science/Research
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: GIS
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: streamlit>=1.39
Requires-Dist: pandas>=2.0
Requires-Dist: plotly>=5.0
Requires-Dist: folium>=0.15
Requires-Dist: requests>=2.28
Requires-Dist: python-dotenv>=1.0
Requires-Dist: PyYAML>=6.0
Requires-Dist: openpyxl>=3.1
Provides-Extra: dev
Requires-Dist: pytest>=8.0; extra == "dev"
Provides-Extra: docs
Requires-Dist: mkdocs-material<10,>=9.5; extra == "docs"
Dynamic: license-file

# fielddash

**Documentation: <https://lambdageo.github.io/fielddash/>**

Schema-driven dashboards for **field data collection** (Epicollect5), built directly from the **form schema**. Every question is identified by Epicollect's stable `ref`, and its input type (`radio`, `checkbox`, `integer`, `location`, ...) automatically dictates the filter, chart, and map representation. Modifying, adding, or reordering questions in the form does not break the dashboard, and setting up a new fieldwork survey requires only a YAML configuration file.

Ready-to-use pages:
- **Overview**: summary indicators, map, submissions over time, collectors.
- **Highlights**: curated charts configured in your YAML.
- **Questions**: all questions grouped by section, with search, text-field toggle, and dynamic cross-tabulation.
- **Data**: searchable table with CSV and Excel export.

## Installation

Install it in an isolated environment rather than globally.

**One environment per project (recommended).** Works everywhere and is required for deployment and custom pages:

```bash
mkdir my-survey && cd my-survey
python -m venv .venv && source .venv/bin/activate   # Windows: .venv\Scripts\activate
pip install fielddash
fielddash init .
```

**Nothing installed first.** [pipx](https://pipx.pypa.io) or [uv](https://docs.astral.sh/uv/) can run `init` once; then create the environment from the generated `requirements.txt`:

```bash
pipx run fielddash init my-survey       # or: uvx fielddash init my-survey
cd my-survey
python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
```

**Command line only.** If you never edit code in the project, `pipx install fielddash` (or `uv tool install fielddash`) is enough for `init`, `fields` and `run`. The `streamlit_app.py` generated by `--deploy` imports `fielddash`, so it needs the per-project environment above.

For development: `git clone https://github.com/LambdaGeo/fielddash && cd fielddash && pip install -e ".[dev]"`.

## Quickstart

The example project lives in the repository; it is not part of the PyPI package.

```bash
git clone https://github.com/LambdaGeo/fielddash && cd fielddash
python -m venv .venv && source .venv/bin/activate
pip install -e .
fielddash run examples/waste/project.yaml
```

The example uses anonymized household survey data on solid waste management (Itaqui-Bacanga, São Luís – MA) and showcases a custom project page (`Recycling`).

## Setting Up a New Field Project

1. Scaffold the project folder:
   ```bash
   fielddash init my-survey            # project.yaml, requirements.txt, .env.example, .gitignore
   fielddash init my-survey --deploy   # + streamlit_app.py, secrets example
   fielddash init my-survey --source json   # offline project reading data/ (git-ignored)
   ```
   Then `cp .env.example .env` and fill in the project slug and the Epicollect app credentials (generated under *Apps* in your project's administration area):
   ```bash
   PROJECT_MY_SURVEY=project-slug
   MY_SURVEY_CLIENT_ID=...
   MY_SURVEY_CLIENT_SECRET=...
   ```
   *Public projects do not require credentials: simply omit `credentials` in the config.*

2. The generated `project.yaml` starts minimal (or write it by hand):
   ```yaml
   title: "My Field Survey"
   source:
     type: epicollect
     project: ${PROJECT_MY_SURVEY}    # slug, or the literal value
     credentials: MY_SURVEY
   ```

3. Inspect the form fields to select aliases, filters, and highlights:
   ```bash
   fielddash fields project.yaml
   ```

4. Complete the configuration (all keys below are optional) and run `fielddash run project.yaml`:
   ```yaml
   subtitle: "Research team, institution..."
   fields:            # alias -> ref suffix, column, or question label
     neighborhood: "5401ce"
     waste_dest: "48c35e"
   types:             # force type (e.g. treat free text as category)
     neighborhood: category
   ignore: [created_by, "3401cb"]   # hide fields (e.g. collector email)
   filters: [created_at, neighborhood]
   highlights:
     - {field: waste_dest, by: neighborhood, title: "Waste destination by neighborhood"}
   sections:          # tabs on the Questions page (default: form groups)
     - {title: "Profile", fields: [age, gender]}
   map: {field: location, popup: [neighborhood]}
   extensions: [pages]              # .py files or directories with custom pages
   timezone: America/Fortaleza
   cache_minutes: 5
   ```
   *(Portuguese keys from early versions — `titulo`, `fonte`, `campos`, ... — still load but are deprecated and emit a warning.)*

Running `fielddash run folder/` will scan all `.yaml` files in the directory and present a project selector in the sidebar.
To work offline or test without internet access, use `source: {type: json, data: ..., schema: ...}` (as shown in `examples/waste/project.yaml`).

## Using in a Custom Streamlit Script / Deployment

The dashboard can also be invoked as a Python function inside your own Streamlit app:

```python
# streamlit_app.py
import fielddash

fielddash.dashboard("projects/")          # or "projects/waste.yaml"
```

Relative paths are resolved relative to the current directory or the script folder.
Run with:
```bash
streamlit run streamlit_app.py
```

### Streamlit Community Cloud

1. Create the project with the deploy files and try it locally:
   ```bash
   mkdir my-survey && cd my-survey
   python -m venv .venv && source .venv/bin/activate   # Windows: .venv\Scripts\activate
   pip install fielddash
   fielddash init . --deploy
   cp .env.example .env          # fill in, then edit project.yaml
   streamlit run streamlit_app.py
   ```
2. Push the folder to a GitHub repository. `.env` and `.streamlit/secrets.toml` are git-ignored: credentials never go to the repo.
3. On [share.streamlit.io](https://share.streamlit.io) choose *Create app*, pick the repository and branch, and set `streamlit_app.py` as the main file.
4. Under *Advanced settings → Secrets* (later: *Settings → Secrets*) paste the content of `.streamlit/secrets.toml.example`, filled in:
   ```toml
   FIELD_ACCESS_PIN = "choose-a-code"
   PROJECT_MY_SURVEY = "project-slug"
   MY_SURVEY_CLIENT_ID = "..."
   MY_SURVEY_CLIENT_SECRET = "..."
   ```
5. Deploy, then share the app link, or `https://<your-app>.streamlit.app/?token=<code>` to skip the login screen.

Notes:
- Use `source: {type: epicollect}` for the cloud. A `type: json` project reads `data/`, which is git-ignored on purpose (it may hold personal data), so it only works locally.
- Without `FIELD_ACCESS_PIN` the app is public: anyone with the link sees the data.

`fielddash` checks environment variables and `.env` first, then falls back to `st.secrets` (also resolving `${VAR}` references in YAML). The data cache is shared among server sessions, so Epicollect receives at most one request per project every `cache_minutes`.

### Restricting access

A deployed Streamlit app is public by default. Add `fielddash.require_access()` before `fielddash.dashboard(...)` (`fielddash init --deploy` already does) and set `FIELD_ACCESS_PIN` in the secrets: visitors then see a login screen, or can use a link ending in `?token=<code>`. Without the variable the call does nothing.

### Self-Hosted Server
```bash
fielddash run projects/ --server.port 8501 --server.headless true
```
(e.g., as a systemd service behind an Nginx reverse proxy with HTTPS).

## Custom Pages

A project-specific page is a standard `.py` file listed in `extensions`:

```python
from fielddash.core.registry import page
from fielddash.ui.charts import render_field


@page("Recycling", order=40)
def render(ctx):
    ds = ctx.dataset
    render_field(ds, ctx.df, ds.field("recycle_habit"), by=ds.field("neighborhood"))
```

- `ctx.df` provides the data with all active sidebar filters applied.
- `ds.field("alias")` resolves the field (column, type, options) dynamically by its stable schema ref.
- Additional data sources (e.g. KoboToolbox, CSV) can be registered with `@source("type")`.

## Project Structure

```
fielddash/
  cli.py            CLI commands `fielddash init`, `run` and `fields`
  scaffold.py       templates and writer behind `fielddash init`
  access.py         `fielddash.require_access()`: optional PIN/token gate
  web.py            `fielddash.dashboard()`: config, cache, filters, navigation
  app.py            Streamlit entry point used by `fielddash run`
  core/schema.py    Epicollect schema parser -> list[Field], column naming, reconciliation
  core/normalize.py raw entries -> typed series (ordered categories, lists, coords, datetimes)
  core/config.py    YAML config parser (+ ${VAR} resolution from .env and secrets)
  sources/          data source plugins (@source): epicollect (API, pagination), json
  ui/               dynamic filters, charts, and maps based on field types
  views/            default pages (@page): Overview, Highlights, Questions, Data
examples/           example projects with anonymized datasets
tests/              pytest test suite (schema, index shifting, AppTest integration)
docs/               MkDocs site (`mkdocs serve`); docs/history/ holds the original development plan
```

## Running Tests

```bash
pytest
```

## Building the Documentation

```bash
pip install -e ".[docs]"
mkdocs serve            # http://127.0.0.1:8000
mkdocs build --strict   # what CI runs
```

## Releasing (maintainers)

1. Bump the version in `pyproject.toml`, `fielddash/__init__.py` and `CITATION.cff`.
2. Publish a GitHub Release: `.github/workflows/publish.yml` builds the package and uploads it to PyPI through Trusted Publishing (no token stored).

One-time setup on pypi.org → *Publishing* → *Add a new pending publisher*: project `fielddash`, owner `LambdaGeo`, repository `fielddash`, workflow `publish.yml`. PyPI never lets a published version be overwritten, so check with `python -m build && twine check dist/*` (and optionally TestPyPI) first.

## Authors

- **Sergio Costa** ([@LambdaGeo](https://github.com/LambdaGeo)) — Universidade Federal do Maranhão (UFMA)
- **André Moura Lima** ([@AndreMouraL](https://github.com/AndreMouraL)) — Universidade Federal do Maranhão (UFMA)
- **Ricardo Luvizotto Santos** — Universidade Federal do Maranhão (UFMA)
- **Marianna Basso Jorge** — Universidade Federal do Maranhão (UFMA)

## License

MIT — see [LICENSE](LICENSE).
