Metadata-Version: 2.4
Name: nexusflowmeter
Version: 0.1.4
Summary: Convert PCAP files to flow-based analysis (CSV, JSON, Excel)
Author-email: Ashwin <ashwinkulkarni737@gmail.com>
License: MIT License
        
        Copyright (c) 2025 Collgamer0008
        
        Permission is hereby granted, free of charge, to any person obtaining a copy
        of this software and associated documentation files (the "Software"), to deal
        in the Software without restriction, including without limitation the rights
        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
        copies of the Software, and to permit persons to whom the Software is
        furnished to do so, subject to the following conditions:
        
        The above copyright notice and this permission notice shall be included in all
        copies or substantial portions of the Software.
        
        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
        SOFTWARE.
        
Requires-Python: >=3.8
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: scapy
Requires-Dist: pandas
Requires-Dist: openpyxl
Dynamic: license-file

# NexusFlowMeter

[![PyPI version](https://badge.fury.io/py/nexusflowmeter.svg)](https://badge.fury.io/py/nexusflowmeter)
[![Python Version](https://img.shields.io/pypi/pyversions/nexusflowmeter.svg)](https://pypi.org/project/nexusflowmeter/)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
[![Downloads](https://pepy.tech/badge/nexusflowmeter)](https://pepy.tech/project/nexusflowmeter)

**NexusFlowMeter** is a high-performance network flow analysis tool that converts packet capture (PCAP) files into comprehensive flow-based insights. Unlike traditional packet-by-packet analysis, NexusFlowMeter aggregates network traffic into flows, providing deeper understanding of network behavior, communication patterns, and traffic characteristics.
```
  Contributions are welcome! Please feel free to submit pull requests, report bugs, or suggest features.
```

## Architecture
```mermaid
graph LR
    %% Input Layer
    subgraph "Input Layer"
        A[Single PCAP File] 
        B[Directory of PCAPs]
        C[Large PCAPs >1GB]
    end

    %% Input Processing
    subgraph "Input Processing"
        D[File Size Check]
        E[PCAP Discovery]
        F[File Splitting]
        G[Chunk Generation]
    end

    %% Core Engine
    subgraph "Core Engine"
        H[PCAPToFlowConverter]
        I[Streaming Reader: PcapReader]
        J[Non-Streaming Reader: rdpcap]
        K[Flow Key Extractor]
        L[Packet Parser]
    end

    %% Flow Management
    subgraph "Flow Management"
        M[FlowKey Generator]
        N[NetworkFlow Objects]
        O[Flow Statistics]
        P[TCP State Analysis]
        Q[Timing Analysis]
    end

    %% Parallel Processing
    subgraph "Parallel Processing"
        R[ProcessPoolExecutor]
        S[Worker 1]
        T[Worker 2]
        U[Worker N]
        V[Results Merger]
    end

    %% Protocol Analysis
    subgraph "Protocol Analysis"
        W[TCP Handler]
        X[UDP Handler]
        Y[ICMP Handler]
        Z[ARP Handler]
        AA[DNS Handler]
    end

    %% Data Aggregation
    subgraph "Data Aggregation"
        BB[Flow Merger]
        CC[Statistics Calculator]
        DD[DataFrame Converter]
        EE[Protocol Filter]
        FF[Flow Limiter]
    end

    %% Output Processing
    subgraph "Output Processing"
        GG[CSV Exporter]
        HH[JSON Exporter]
        II[Excel Exporter]
        JJ[Report Generator]
        KK[Summary Statistics]
    end

    %% Batch Modes
    subgraph "Batch Modes"
        LL[Separate Files Mode]
        MM[Merged Mode]
        NN[Source File Tracking]
    end

    %% Output Layer
    subgraph "Output Layer"
        OO[Individual CSV Files]
        PP[Merged CSV File]
        QQ[JSON Output]
        RR[Excel Files]
        SS[Analysis Reports]
        TT[Processing Logs]
    end

    %% Flow connections
    A --> D
    B --> E
    C --> F
    D --> H
    E --> H
    F --> G
    G --> R
    H --> I --> L
    H --> J --> L
    L --> K --> M --> N
    N --> O & P & Q
    G --> S & T & U
    S --> V
    T --> V
    U --> V
    L --> W & X & Y & Z & AA
    V --> BB
    O & P & Q --> CC
    CC --> DD --> EE --> FF
    FF --> GG & HH & II
    DD --> JJ
    CC --> KK
    B --> LL & MM
    MM --> NN
    LL --> OO
    MM --> PP
    GG --> OO & PP
    HH --> QQ
    II --> RR
    JJ --> SS
    KK --> TT

    %% Styling
    classDef input fill:#d9edf7,stroke:#31708f,stroke-width:1px;
    classDef process fill:#e6e6fa,stroke:#6a1b9a,stroke-width:1px;
    classDef core fill:#fff2cc,stroke:#b36b00,stroke-width:1px;
    classDef flow fill:#e8f5e9,stroke:#2e7d32,stroke-width:1px;
    classDef parallel fill:#fde0dc,stroke:#b71c1c,stroke-width:1px;
    classDef proto fill:#ede7f6,stroke:#4527a0,stroke-width:1px;
    classDef data fill:#f1f8e9,stroke:#33691e,stroke-width:1px;
    classDef outputProc fill:#f0f4c3,stroke:#827717,stroke-width:1px;
    classDef batch fill:#ffe0b2,stroke:#e65100,stroke-width:1px;
    classDef finalOut fill:#c8e6c9,stroke:#1b5e20,stroke-width:2px;

    class A,B,C input
    class D,E,F,G process
    class H,I,J,K,L core
    class M,N,O,P,Q flow
    class R,S,T,U,V parallel
    class W,X,Y,Z,AA proto
    class BB,CC,DD,EE,FF data
    class GG,HH,II,JJ,KK outputProc
    class LL,MM,NN batch
    class OO,PP,QQ,RR,SS,TT finalOut

```

## Installation

### From PyPI (Recommended)

```bash
pip install nexusflowmeter
```

### From Source

```bash
git clone https://github.com/Collgamer0008/NexusFlowMeter.git
cd NexusFlowMeter
pip install -r requirements.txt

```

## Dependencies
- ```scapy ``` - Packet manipulation library
- ```pandas``` - Data analysis and manipulation
- ```openpyxl``` - Excel file support (optional)

## NexusFlowMeter CLI Usage

```bash
 nexusflowmeter [-h] [-d DIRECTORY] [-c OUTPUT_DIR] [--merge] [--protocols PROTOCOLS] [--max-flows MAX_FLOWS]
                      [--output-format {csv,json,xlsx}] [--quick-preview QUICK_PREVIEW] [--split-by-protocol]
                      [--stream] [--chunk-size CHUNK_SIZE] [--flow-timeout FLOW_TIMEOUT] [--max-workers MAX_WORKERS]
                      [--verbose]
                      [pcap_file] [output_file]

Convert PCAP files to flow-based analysis (CSV, JSON, Excel)

positional arguments:
  pcap_file                               Input PCAP file path (for single file processing)
  output_file                             Output file path (for single file processing)

options:
  -h, --help                              show this help message and exit
  -d DIRECTORY, --directory DIRECTORY     Input directory containing PCAP files (for batch processing)
                                      
  -c OUTPUT_DIR, --output-dir OUTPUT_DIR  Output directory (for batch processing)
                        
  --merge                                 Merge all PCAP files into single output (directory processing only)
  --protocols PROTOCOLS                   Comma-separated list of protocols to include (tcp,udp,icmp,arp,dns,all)
                        
  --max-flows MAX_FLOWS                   Maximum number of flows to analyze
                        
  --output-format {csv,json,xlsx}, -of {csv,json,xlsx}
                                          Output file format (default: csv)

  --quick-preview QUICK_PREVIEW           Show first N flows before conversion (single file only)
                        
  --split-by-protocol                     Create separate files for each protocol (single file only)
  --stream                                Use streaming mode (PcapReader) instead of loading the whole file into memory
  --chunk-size CHUNK_SIZE                 Chunk size in MB for large files (default: 1024MB = 1GB)
                        
  --flow-timeout FLOW_TIMEOUT             Flow timeout in seconds (default: 60)
                        
  --max-workers MAX_WORKERS               Maximum parallel workers for chunk processing (default: 4)
                        
  --verbose                               Enable verbose logging

```


---

### Positional Arguments

| Argument      | Description |
|---------------|-------------|
| `pcap_file`   | Input PCAP file path (for single file processing) |
| `output_file` | Output file path (for single file processing) |
| `-d /folder_path/` | Input folder path containing PCAPs (for multiple file processing) |
| `-c /folder_path/` | Output folder path for extracted CSVs (for multiple file processing) |

Examples
```bash
    Single file conversion:
      nexusflowmeter capture.pcap flows.csv

    Directory processing:
      nexusflowmeter -d ./pcaps/ -c ./out/                    # Convert each PCAP in the folder to separate CSV
      nexusflowmeter -d ./pcaps/ -c ./out/ --merge            # Merge all PCAPs into one CSV
      nexusflowmeter -d ./pcaps/ -c ./out/ --output-format json  # Convert to JSON format
```

### Options

| Option | Description |
|--------|-------------|
| `-h, --help` | Show help message and exit |
| `-d DIRECTORY, --directory DIRECTORY` | Input directory containing PCAP files (for batch processing) |
| `-c OUTPUT_DIR, --output-dir OUTPUT_DIR` | Output directory (for batch processing) |
| `--merge` | Merge all PCAP files into single output (directory processing only) |
| `--protocols PROTOCOLS` | Comma-separated list of protocols to include (`tcp,udp,icmp,arp,dns,all`) |
| `--max-flows MAX_FLOWS` | Maximum number of flows to analyze |
| `--output-format {csv,json,xlsx}`, `-of {csv,json,xlsx}` | Output file format (default: `csv`) |
| `--quick-preview QUICK_PREVIEW` | Show first N flows before conversion (single file only) |
| `--split-by-protocol` | Create separate files for each protocol (single file only) |
| `--stream` | Use streaming mode (`PcapReader`) instead of loading the whole file into memory (recommended for memory constrained systems) |
| `--chunk-size CHUNK_SIZE` | Chunk size in MB for large files (default: `1024MB = 1GB`) |
| `--flow-timeout FLOW_TIMEOUT` | Flow timeout in seconds (default: `60`) |
| `--max-workers MAX_WORKERS` | Maximum parallel workers for chunk processing (default: `4`) |
| `--verbose` | Enable verbose logging |

Examples
```bash
    nexusflowmeter capture.pcap flows.json --output-format json  OR
    nexusflowmeter capture.pcap flows.json -of json


    nexusflowmeter capture.pcap flows --split-by-protocol
    nexusflowmeter capture.pcap flows.csv --quick-preview 5
    nexusflowmeter capture.pcap tcp_flows.xlsx --output-format xlsx --protocols tcp
```


## Flow Features Extracted  

NexusFlowMeter generates **34** per-flow statistics useful for traffic analysis, anomaly detection, and ML feature extraction.  

| Feature | Description |
|---------|-------------|
| **flow_id** | Unique identifier for the flow (`src_ip:src_port <-> dst_ip:dst_port protocol`) |
| **src_ip, dst_ip** | Source and destination IP addresses |
| **src_port, dst_port** | Source and destination ports |
| **protocol** | Transport or network protocol (TCP, UDP, ICMP, etc.) |
| **start_time, end_time** | Timestamps marking the beginning and end of the flow |
| **duration_seconds** | Duration of the flow |
| **total_packets** | Total number of packets in the flow |
| **forward_packets, backward_packets** | Packets in forward (src→dst) and backward (dst→src) directions |
| **is_bidirectional** | Indicates if packets exist in both directions |
| **forward_backward_ratio** | Ratio of forward to backward packet counts |
| **total_bytes** | Total bytes transferred in the flow |
| **forward_bytes, backward_bytes** | Bytes in forward and backward directions |
| **total_payload_bytes** | Total payload size excluding headers |
| **forward_payload_bytes, backward_payload_bytes** | Payload bytes split by direction |
| **packets_per_second** | Packet transmission rate |
| **bytes_per_second** | Byte transfer rate |
| **min_packet_size, max_packet_size** | Smallest and largest packet sizes observed |
| **avg_packet_size** | Average packet size |
| **avg_inter_arrival_ms** | Average packet inter-arrival time (milliseconds) |
| **min_inter_arrival_ms, max_inter_arrival_ms** | Minimum and maximum inter-arrival times |
| **tcp_flags** | Summary of TCP flags observed (e.g., SYN, FIN, ACK) |
| **syn_count, fin_count, rst_count, ack_count** | Count of specific TCP flags |
| **connection_state** | State of the TCP connection (e.g., Established, Closed) |

---

## What makes this tool unique? 

NexusFlowMeter includes several **productivity-focused options** which makes using this tool very convinient and easy.  

###  Directory & Batch Processing  
- **Feature**: Process an entire folder of PCAP files at once.  
- **Example**:  
  ```bash
  convert all pcaps in the folder to individual csvs
    nexusflowmeter -d ./pcaps/ -c ./out/
  
  convert all pcaps in the folder into a single file
    nexusflowmeter -d ./pcaps/ -c ./out/ --merge
  ```
###  Protocol Filtering  
- **Feature**: Focus only on specific protocols (TCP, UDP, ICMP, ARP, DNS, or all) i.e, extracted file contains flow with mentioned protocol only.  
- **Example**:  
  ```bash
  nexusflowmeter capture.pcap tcp_flows.csv --protocols tcp
  ```
###  Quick Preview  
- **Feature**: View the first N flows before committing to full conversion.  
- **Example**:  
  ```bash
  nexusflowmeter capture.pcap flows.csv --quick-preview 5
  ```
  Prints the first 5 flows in the console.
  
###  Split by Protocol  
- **Feature**: Automatically create separate output files for each protocol. 
- **Example**:  
  ```bash
  nexusflowmeter capture.pcap flows --split-by-protocol
  ```
  Generates files like flows_tcp.csv, flows_udp.csv, etc.

###  Streaming Mode  
- **Feature**: Processes packets using a stream reader instead of loading the entire PCAP into memory (1000 pkts/sec). Useful for large captures and memory contrained systems. 
- **Example**:  
  ```bash
  nexusflowmeter capture.pcap flows.csv --stream
  ```
  
###  Chunked Processing  
- **Feature**: Split very large PCAPs into chunks (in MB) for parallel or memory-efficient processing (default : 1024 MB). 
- **Example**:  
  ```bash
  nexusflowmeter capture.pcap flows.csv --chunk-size 512
  ```
  Processes the file in 512 MB chunks.
  ```bash
  nexusflowmeter capture.pcap flows.csv --chunk-size 512 --max-workers 5
  ```
  Processes the file in 512 MB chunks parallely , here 5 seperate processes.

  ###  Output Formats  
- **Feature**: Export results in CSV, JSON, or Excel (XLSX). 
- **Example**:  
  ```bash
   nexusflowmeter capture.pcap flows.json --output-format json
   nexusflowmeter capture.pcap flows.xlsx --output-format xlsx
  ```
