Metadata-Version: 2.3
Name: tgedr-datasets
Version: 2.0.21
Summary: datasets to be consumed elsewhere
Requires-Dist: pandas~=2.3.0
Requires-Dist: tgedr-dataops==1.1.1
Requires-Dist: pyarrow~=23.0.0
Requires-Dist: pyyaml>=6.0
Requires-Dist: lxml>=6.0.2
Requires-Dist: html5lib>=1.1
Requires-Dist: bs4>=0.0.2
Requires-Dist: tgedr-pycommons>=1.0.1
Requires-Dist: yfinance==0.2.66
Requires-Dist: finnhub-python>=2.4.26
Requires-Dist: matplotlib>=3.10.8
Requires-Dist: datasets>=4.5.0
Requires-Dist: scikit-learn>=1.9.0
Requires-Python: >=3.11, <3.14
Description-Content-Type: text/markdown

# datasets

[![cicd](https://github.com/jtviegas/datasets/actions/workflows/cicd.yml/badge.svg)](https://github.com/jtviegas/datasets/actions/workflows/cicd.yml)
![Coverage](./coverage.svg)
[![PyPI](https://img.shields.io/pypi/v/tgedr-datasets)](https://pypi.org/project/tgedr-datasets/)

## motivation

This repository provides a production-grade Python library (`tgedr-datasets`) for building and running automated data pipelines that collect, validate, and store financial market data. It orchestrates the extraction of stock tickers, daily OHLCV prices (via Yahoo Finance), and news articles (via Alpha Vantage, NewsAPI, and Finnhub), transforms them into structured formats, validates them against ODCS data contracts, and loads them into Hugging Face datasets. 

## datalake pipelines


[![tickers](https://github.com/jtviegas/datasets/actions/workflows/tickers.yml/badge.svg)](https://github.com/jtviegas/datasets/actions/workflows/tickers.yml)
[![prices](https://github.com/jtviegas/datasets/actions/workflows/prices.yml/badge.svg)](https://github.com/jtviegas/datasets/actions/workflows/prices.yml) 
[![articles](https://github.com/jtviegas/datasets/actions/workflows/articles.yml/badge.svg)](https://github.com/jtviegas/datasets/actions/workflows/articles.yml)


### articles

[download @ hugging face](https://huggingface.co/datasets/jtviegas/ticker_analysis_articles)

![articles metrics](metrics/articles_metrics.png)

### prices

[download @ hugging face](https://huggingface.co/datasets/jtviegas/ticker_analysis_prices)

![prices metrics](metrics/prices_metrics.png)

### tickers

[download @ hugging face](https://huggingface.co/datasets/jtviegas/ticker_analysis_tickers)

![tickers metrics](metrics/tickers_metrics.png)

## development
- main requirements:
  - _uv_  
  - _bash_
- Clone the repository like this:

  ``` bash
  git clone git@github.com:jtviegas/datasets
  ```
- cd into the folder: `cd datasets`
- install requirements: `./helper.sh reqs`