Metadata-Version: 2.4
Name: micromole
Version: 0.0.1
Summary: Journaling and event sourcing micro-framework for the SB stack
Author: Maintainers
Author-email: solubrew@solutionsbrewer.com
License: MIT
Platform: Linux
Requires-Python: >=3.6
Requires-Dist: backoff==2.2.1
Requires-Dist: click==8.3.3
Requires-Dist: defusedxml; platform_system == "Windows"
Requires-Dist: futures; python_version < "3"
Requires-Dist: six; platform_system == "Linux" and python_version == "1.17.0"
Requires-Dist: pypiwin32; platform_system == "Windows"
Description: # MicroMole
        
        MicroMole is a Python package that provides tools to collect and store data from various sources such as local files, databases, website APIs, and more. It offers a unified interface for data ingestion, transformation, and persistence, enabling seamless integration of diverse data streams into your applications or pipelines. With support for multiple collectors and storage backends, MicroMole simplifies data management tasks for developers and data scientists.
        
        ## Features
        
        - **Data Collectors**: Modules for fetching data from local files (CSV, JSON, XML, etc.), databases (SQL, NoSQL), web APIs (REST, GraphQL), and other sources like streams or sensors.
        - **Storage Backends**: Support for storing collected data in files, databases, caches, or cloud storage with configurable options.
        - **Unified API**: Consistent methods for collection, processing, and storage across different sources.
        - **Data Transformation**: Built-in tools for cleaning, filtering, and transforming data during collection.
        - **Configuration-Driven**: Use YAML or JSON configs to define sources, transformations, and destinations.
        - **Error Handling and Logging**: Robust mechanisms for retries, validation, and detailed logging.
        - **Extensible**: Easily add custom collectors or storage adapters.
        - **Cross-Platform Compatibility**: Works on Windows, macOS, and Linux.
        
        ## Installation
        
        You can install MicroMole via pip:
        
        ```bash
        pip install micromole
        ```
        
        Alternatively, clone the repository and install from source:
        
        ```bash
        git clone https://github.com/<USER_OR_ORG>/micromole.git
        cd micromole
        pip install -e .
        ```
        
        ### Requirements
        
        - Python 3.<MIN_VERSION> or higher
        - Dependencies: requests, pandas, sqlalchemy, pyyaml (automatically installed via pip where applicable)
        
        ## Quick Start
        
        Import the module, configure a collector, and fetch data:
        
        ```python
        import micromole
        
        # Load configuration for a source
        config = micromole.load_config('path/to/api.yaml')
        
        # Initialize collector
        collector = micromole.Collector(config)
        
        # Collect data
        data = collector.fetch()
        
        # Store data
        micromole.store(data, 'path/to/storage.db')
        ```
        
        ## Usage
        
        ### Loading Configurations
        
        MicroMole uses YAML files to define data sources and storage. A sample YAML for a web API might look like:
        
        ```yaml
        source_type: api
        url: https://api.example.com/data
        method: GET
        headers:
          Authorization: Bearer <TOKEN>
        params:
          query: value
        transform:
          filter: key == 'active'
        storage:
          type: database
          engine: sqlite
          file: data.db
        ```
        
        Use `micromole.load_config(yaml_path)` to parse and validate the config.
        
        ### Collecting Data
        
        ```python
        # From local file
        file_config = micromole.load_config('file.yaml')
        file_collector = micromole.Collector(file_config)
        file_data = file_collector.fetch()
        
        # From database
        db_config = micromole.load_config('db.yaml')
        db_collector = micromole.Collector(db_config)
        db_data = db_collector.query('SELECT * FROM table')
        ```
        
        ### Storing Data
        
        ```python
        # Store to file
        micromole.store(data, 'output.csv', format='csv')
        
        # Store to database
        micromole.store(data, 'sqlite:///data.db', table='results')
        ```
        
        ### Supported Sources
        
        - Local Files: CSV, JSON, Excel, Parquet, etc.
        - Databases: SQLite, PostgreSQL, MySQL, MongoDB, etc.
        - Web APIs: RESTful services, with authentication support.
        - Others: Streams, FTP, cloud APIs (configurable via extensions).
        
        ## Examples
        
        ### Example 1: Collecting from API and Storing to Database
        
        ```python
        import micromole
        
        config = micromole.load_config('weather_api.yaml')
        collector = micromole.Collector(config)
        weather_data = collector.fetch()
        
        # Transform data
        transformed = micromole.transform(weather_data, filter=lambda x: x['temp'] > 20)
        
        # Store
        micromole.store(transformed, 'weather.db', table='daily')
        ```
        
        ### Example 2: Batch Collection from Multiple Sources
        
        ```python
        import micromole
        
        configs = ['file1.yaml', 'api2.yaml', 'db3.yaml']
        data_list = []
        
        for cfg in configs:
            config = micromole.load_config(cfg)
            collector = micromole.Collector(config)
            data = collector.fetch()
            data_list.append(data)
        
        # Combine and store
        combined = micromole.merge(data_list)
        micromole.store(combined, 'combined.json', format='json')
        ```
        
        ## Configuration Guide
        
        Each YAML config must include:
        
        - `source_type`: String identifier (e.g., 'file', 'api', 'database')
        - `connection`: Dictionary of parameters (e.g., path, url, credentials)
        - `transform`: Optional list of transformations (filter, map, etc.)
        - `storage`: Optional details for immediate storage after collection
        
        For advanced customization, refer to the [docs/config-reference.md](docs/config-reference.md).
        
        ## Contributing
        
        Contributions are welcome! Please follow these steps:
        
        1. Fork the repository.
        2. Create a feature branch (`git checkout -b feature/<FEATURE_NAME>`).
        3. Commit your changes (`git commit -am 'Add some feature'`).
        4. Push to the branch (`git push origin feature/<FEATURE_NAME>`).
        5. Open a Pull Request.
        
        See [CONTRIBUTING.md](CONTRIBUTING.md) for more details.
        
        ## License
        
        This project is licensed under the <LICENSE_TYPE> License - see the [LICENSE](LICENSE) file for details.
        
        ## Acknowledgments
        
        - Built with inspiration from open-source data ingestion communities.
        - Thanks to contributors of underlying libraries like requests, pandas, sqlalchemy.
Description-Content-Type: text/markdown
