Metadata-Version: 2.1
Name: dockflow
Version: 1.3.0
Summary: Easily deploy Airflow for local development
Home-page: https://gitlab.com/spatialedge/public/dockflow
Author: Pieter van der Westhuizen
Author-email: pieter@spatialedge.co.za
License: Spatialedge Community License
Platform: UNKNOWN
Classifier: Programming Language :: Python :: 3
Classifier: Operating System :: OS Independent
Requires-Python: >=3.6
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: click>=8.1.3
Requires-Dist: docker>=4.2.2

# Dockflow CLI

![coverage report](https://gitlab.com/spatialedge/airflow/dockflow/badges/master/coverage.svg)
![pipeline status](https://gitlab.com/spatialedge/airflow/dockflow/badges/master/pipeline.svg)

This package requires docker to be installed and running on your machine.

## Getting started

1. Install dockflow:  
   `pip install dockflow`

2. Authenticate docker to pull from private GCR repository:  
   2.1. First make sure that your `gcloud` is authenticated using `gcloud auth application-default login --impersonate-service-account=<service-account>`.  
   2.2. Ensure you are in the correct project using `gcloud config set project <project name>`  
   2.3. Use `gcloud auth configure-docker <location>-docker.pkg.dev` to configure gcloud as the credential helper for the Artifact Registry domain associated with your repository’s location. For example, `gcloud auth configure-docker europe-west1-docker.pkg.dev`.

## Quickstart

1. Ensure that your docker file sharing settings allows access to your development directory.
2. Navigate to the root directory of your `dags` folder.
3. Use `dockflow config` and enter your container repo url excluding the version.
   - This will save the url in a config file.
   - Eg. `<location>-docker.<domain>/<project-id>/<repo-id>/<image>`
   - If you get permission errors, Ensure that you have artifact reader permission on this registry.
   - This should only be used if the container repo changes.
4. If the image version tag is not `composer-2.9.8-airflow-2.9.3` specify the tag using `dockflow start -iv <version>`. Alternatively, use the `--check-images` flag to list the available images at the provided container repo.
5. Use `dockflow start` (This will mount the dag folder and start Airflow).
6. Use the UI to add connections.
   - Admin -> Connections -> Create
   - If you need to connect to a GCP resource, provide values for the following fields:
     - Project Id, e.g. `research-se-de`
     - Credential configuration file, located at `/usr/local/airflow/gcp_credentials.json`
     - Impersonation chain, e.g. `local-file-ingestor@research-se-de.iam.gserviceaccount.com`
7. Use `dockflow refresh` to refresh the configs cache or to bundle configs.
8. Remember to use `dockflow stop` to shut down the instance to save local machine resources.
   - The state will be persisted in the same directory as the `dags` folder.
9. To stop and remove the container use `dockflow stop --rm`

## CloudSQL Proxy

```
docker run -d \
  -v <PATH_TO_KEY_FILE>:/config \
  -p 127.0.0.1:5432:5432 \
  --network='dockflow' \
  --name='cloudsql' \
  gcr.io/cloudsql-docker/gce-proxy:1.17 /cloud_sql_proxy \
  -instances=<INSTANCE_CONNECTION_NAME>=tcp:0.0.0.0:5432 -credential_file=/config
```


