Metadata-Version: 2.4
Name: tmg-etl-pipeline
Version: 0.0.9
Summary: TMG ETL pipeline
Home-page: https://github.com/telegraph/tmg-etl-pipeline
Author: TMG Data Platform team
Author-email: data.platform@telegraph.co.uk
License: Apache 2.0
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python
Classifier: Programming Language :: Python :: 3
Classifier: Operating System :: OS Independent
Requires-Python: >=3.6
License-File: LICENSE
Requires-Dist: google-cloud-secret-manager==2.23.2
Requires-Dist: PyYAML==6.0.2
Dynamic: author
Dynamic: author-email
Dynamic: classifier
Dynamic: description
Dynamic: home-page
Dynamic: license
Dynamic: license-file
Dynamic: requires-dist
Dynamic: requires-python
Dynamic: summary

TMG ETL pipeline
==================================

TMG ETL pipeline is the base structure of our ETL pipelines.
Any pipeline should inherit from TMGETLPipeline class and implement the necessary methods.


Quick Start
-----------

Installation
~~~~~~~~~~~~

Install this library in a `virtualenv`_ using pip. `virtualenv`_ is a tool to
create isolated Python environments. The basic problem it addresses is one of
dependencies and versions, and indirectly permissions.

.. _`virtualenv`: https://virtualenv.pypa.io/en/latest/

Supported Python Versions
^^^^^^^^^^^^^^^^^^^^^^^^^
Python >= 3.5

Mac/Linux
^^^^^^^^^

.. code-block:: console

    pip install virtualenv
    virtualenv <your-env>
    source <your-env>/bin/activate
    <your-env>/bin/pip install tmg-etl-pipeline

Example Usage
-------------

.. code:: python

    from tmg_etl_pipeline.etl import TMGETLPipeline


    class MyETL(TMGETLPipeline):

        def run(self):
            # your ETL logic goes here
            # access the configs
            # self.config['some-config-variable']
            # access the secrets
            # self.secrets['some-secret-variable']

        def cleanup(self):
            # your clean up code goes here

    etl = MyETL(app_name='your-app-name', config_path='path/to-the-config-file')
    etl.execute()
