Metadata-Version: 2.4
Name: tabular2mcap
Version: 0.1.11
Summary: Convert tabular data (CSV, Parquet, Excel, etc.) to MCAP format with support for ROS2 and JSON schemas
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: jinja2>=3.0.0
Requires-Dist: mcap>=1.3.0
Requires-Dist: mcap-ros1-support>=0.7.3
Requires-Dist: mcap-ros2-support>=0.5.5
Requires-Dist: pandas>=2.3.3
Requires-Dist: pydantic>=2.12.0
Requires-Dist: scipy>=1.15.3
Requires-Dist: tqdm>=4.67.1
Requires-Dist: av<16.0.0,>=15.0.0
Requires-Dist: opencv-python>=4.12.0.88
Requires-Dist: platformdirs>=4.5.0
Provides-Extra: parquet
Requires-Dist: pyarrow>=21.0.0; extra == "parquet"
Provides-Extra: feather
Requires-Dist: pyarrow>=21.0.0; extra == "feather"
Provides-Extra: orc
Requires-Dist: pyarrow>=21.0.0; extra == "orc"
Provides-Extra: excel
Requires-Dist: openpyxl>=3.0.0; extra == "excel"
Provides-Extra: excel-legacy
Requires-Dist: xlrd>=2.0.0; extra == "excel-legacy"
Provides-Extra: xml
Requires-Dist: lxml>=4.9.0; extra == "xml"
Provides-Extra: protobuf
Requires-Dist: mcap-protobuf-support>=0.5.2; extra == "protobuf"
Requires-Dist: foxglove-schemas-protobuf>=0.1.1; extra == "protobuf"
Provides-Extra: all-formats
Requires-Dist: pyarrow>=21.0.0; extra == "all-formats"
Requires-Dist: openpyxl>=3.0.0; extra == "all-formats"
Requires-Dist: xlrd>=2.0.0; extra == "all-formats"
Requires-Dist: lxml>=4.9.0; extra == "all-formats"
Dynamic: license-file

# Tabular2MCAP

[![Documentation](https://img.shields.io/badge/docs-mkdocs-2d62ff?style=flat&labelColor=f8f9fa&color=2d62ff)](https://alloyrobotics.github.io/tabular2mcap/)
[![Supported by Alloy](https://img.shields.io/badge/Supported%20by-Alloy-2d62ff?style=flat&labelColor=f8f9fa&color=2d62ff)](https://usealloy.ai)

Convert tabular data (CSV, Parquet, Excel, and more) to MCAP format with support for ROS2 and JSON schemas, enabling seamless integration with robotics workflows, data visualization in Foxglove Studio, and playback of sensor and navigation data.

> **Please ⭐ if this helps you today!**

## 🎥 Video Tutorial

Convert CSV, Excel, or Parquet to MCAP in minutes with this step‑by‑step walkthrough.

<p align="center">
  <a href="https://www.youtube.com/watch?v=jhVM6RGVs1M" target="_blank">
    <img src="https://img.youtube.com/vi/jhVM6RGVs1M/hqdefault.jpg"
         alt="Converting CSV to MCAP tutorial"
         width="640">
  </a>
</p>


## Features

- **Multi-Format Support**: Convert to ROS2, JSON, or Protobuf messages with support for standard message types (e.g., `sensor_msgs/msg/NavSatFix`, `geometry_msgs/msg/TransformStamped`, `foxglove.LocationFix`, and more)
- **Configuration-Driven**: YAML-based mapping with Jinja2 templates for flexible data transformation
- **Batch Processing**: Process multiple files and directories with a single command

## Quick Start

### Installation

```bash
# Basic installation (CSV, TSV, JSON, Pickle)
pip install tabular2mcap

# With optional formats
pip install tabular2mcap[parquet]      # Parquet files
pip install tabular2mcap[feather]      # Feather files
pip install tabular2mcap[orc]          # ORC files
pip install tabular2mcap[excel]        # Excel files (.xlsx)
pip install tabular2mcap[excel-legacy] # Excel files (.xls)
pip install tabular2mcap[xml]          # XML files
pip install tabular2mcap[protobuf]     # Protobuf output format
pip install tabular2mcap[all-formats]  # All formats
```

### Basic Usage

```bash
tabular2mcap \
  -i /path/to/data/directory \
  -o output.mcap \
  -c config.yaml \
  -f converter_functions.yaml
```

### Configuration System

The tool uses YAML configuration files to define how CSV files should be processed. These are:

- **`config.yaml`**: Defines file patterns and converter function mappings
- **`converter_functions.yaml`**: Contains Jinja2 templates for data transformation

#### Example Configurations

**JSON Format:**
```yaml
# config.yaml
writer_format: "json"
file_mappings:
  - file_pattern: 'data.csv'
    converter_functions:
      - function_name: "row_to_foxglove_location_fix"
        schema_name: "foxglove.LocationFix"
        topic_suffix: "LocationFix"
```

**ROS2 Format:**
```yaml
# config.yaml
writer_format: "ros2"
tabular_mappings:
  - file_pattern: '**/data.csv'
    converter_functions:
      - function_name: "row_to_nav_sat_fix"
        schema_name: "sensor_msgs/msg/NavSatFix"
        topic_suffix: "NavSatFix"
```

**Protobuf Format:**
```yaml
# config.yaml
writer_format: "protobuf"
tabular_mappings:
  - file_pattern: '**/data.csv'
    converter_functions:
      - function_name: "row_to_location_fix"
        schema_name: "foxglove.LocationFix"
        topic_suffix: "LocationFix"
```

> **Note:** Protobuf `repeated` fields are variable length (no fixed-size arrays like ROS2). Array templates show a single placeholder element; populate with actual data dynamically.


## Tutorial

For detailed instructions on adding support for new types of CSV data, see the [Tutorial](https://alloyrobotics.github.io/tabular2mcap/tutorial/) in the documentation.

## Supported By

<a href="https://usealloy.ai" target="_blank">
  <img src="https://cdn.prod.website-files.com/68c02115d6be7142be8a1553/68e70108f9c8d1c5629ce407_logo-padding-256px.png" alt="Alloy" style="background-color: white; padding: 0px; border-radius: 6px; box-shadow: 0 2px 4px rgba(0, 0, 0, 0.1); height: 56px" />
</a>

This project is supported by [Alloy](https://usealloy.ai) - Search all your robot data in plain english. Alloy provides a platform to unify image, time-series, and log search for robotics teams.

## License

GNU General Public License v3.0 - see [LICENSE](https://github.com/alloyrobotics/tabular2mcap/blob/main/LICENSE) for details.

## Contributing

See [CONTRIBUTING.md](CONTRIBUTING.md) for development setup, testing, and contribution guidelines.
