Metadata-Version: 2.2
Name: balto-core
Version: 0.1.7
Summary: With balto, data analysts and engineers can build analytics the way engineers build applications.
Home-page: https://github.com/balto-data/balto
Author: Balto Data
Author-email: support@baltodatatool.com
Classifier: Development Status :: 5 - Production/Stable
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Operating System :: Microsoft :: Windows
Classifier: Operating System :: MacOS :: MacOS X
Classifier: Operating System :: POSIX :: Linux
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Requires-Python: >=3.8
Description-Content-Type: text/markdown
Requires-Dist: agate<1.10,>=1.7.0
Requires-Dist: Jinja2<4,>=3.1.3
Requires-Dist: mashumaro[msgpack]<4.0,>=3.9
Requires-Dist: click<9.0,>=8.0.2
Requires-Dist: networkx<4.0,>=2.3
Requires-Dist: protobuf<5,>=4.0.0
Requires-Dist: requests<3.0.0
Requires-Dist: snowflake-connector-python[secure-local-storage]~=3.0
Requires-Dist: pathspec<0.13,>=0.9
Requires-Dist: sqlparse<0.6.0,>=0.5.0
Requires-Dist: dbt-extractor<=0.6,>=0.5.0
Requires-Dist: dbt-semantic-interfaces<0.8,>=0.7.0
Requires-Dist: dbt-common<2.0,>=1.6.0
Requires-Dist: packaging>20.9
Requires-Dist: pytz>=2015.7
Requires-Dist: pyyaml>=6.0
Requires-Dist: daff>=1.3.46
Requires-Dist: typing-extensions>=4.4
Dynamic: author
Dynamic: author-email
Dynamic: classifier
Dynamic: description
Dynamic: description-content-type
Dynamic: home-page
Dynamic: requires-dist
Dynamic: requires-python
Dynamic: summary

<p align="center">
  <img src="https://raw.githubusercontent.com/balto-data/balto/main/etc/balto-banner.png" alt="balto logo" width="750"/>
</p>

# Welcome to Balto

Build durable data pipelines directly in Snowflake with no external orchestrators. Effortlessly scale your dbt projects from proof of concept to production grade data pipelines running millions of model executions per day. Orchestrate ML pipelines, data exports, and deletions directly in Snowflake with zero external dependencies. 

## Getting Started

Ready to get started? Check out the [installation instructions](https://baltodatatool.com). If you're already familiar with [dbt](https://github.com/dbt-labs/dbt-core), check out [our guide](https://baltodatatool.com) on what's new in Balto.

## Installation
Clone the repo and install from source, or install with pip:
```
pip install 'git+https://github.com/balto-data/balto.git@v0.0.10#egg=balto&subdirectory=core'
```

## Overview

![architecture](https://github.com/balto-data/balto/blob/main/etc/balto-diagram.svg)

Balto is comprised of two pieces. The Balto CLI, a drop-in replacement for the [dbt-core](https://github.com/dbt-labs/dbt-core) CLI, and the Balto orchestrator, which is implemented as a Snowflake Native App.  Balto orginated as a fork of dbt-core that has been rebuilt from the ground up to run as a Snowflake Native app. The following are the major design differences:

### Compilation

Balto simplifies the parsing process when building your project. Any database interactions are deferred during compilation, this allows us push all database interactions to the Balto Snowflake Native App installed in your account. This means if you use our [Github Integation](https://baltodatatool.com/cicd/), you can enable automated deployments without having to manage any external accounts or connections.

### Orchestration

Balto implements a batteries included orchestration engine inside the Snowflake App. When you deploy your
compiled projects via our Gitub integration (or via a call to our stored procedure), the Balto orchestrator handles deploying your compiled project from directly inside your Snowflake account.

Under the hood, Balto's orchestration engine uses Snowflake triggered tasks and streams to implement a completely event driven scheduler that allows it to scale to zero when no models are being executed.

### dbt macro executor

Balto includes a lightweight re-implementation of dbt's Jinja `macro` extension that allows it to execute your model inside a stored procedure that runs on the same warehouse the underlying queries get run on.

## Reporting bugs and contributing code

- Want to report a bug or request a feature? Let us know and open [an issue](https://github.com/balto-data/balto/issues/new/choose)
