Metadata-Version: 2.1
Name: sap-ecs-log-forwarder
Version: 1.2.1
Summary: A package to consume events from an AWS SQS queue, Azure Storage Account Topic and GCP PubSub Topic, process log files, and forward them to a HTTP endpoint or file.
Home-page: https://www.sap.com/
License: SAP DEVELOPER LICENSE AGREEMENT
Keywords: SAP AWS Log Forwarder,SAP Azure Log Forwarder,SAP GCP Log Forwarder,SAP Log Forwarder
Author: SAP SE
Requires-Python: >=3.10,<4.0
Classifier: Development Status :: 5 - Production/Stable
Classifier: Intended Audience :: Developers
Classifier: License :: Other/Proprietary License
Classifier: Operating System :: MacOS :: MacOS X
Classifier: Operating System :: POSIX :: Linux
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Requires-Dist: aiohttp (>=3.13.2,<4.0.0)
Requires-Dist: azure-identity (>=1.25.1,<2.0.0)
Requires-Dist: azure-storage-blob (>=12.27.1,<13.0.0)
Requires-Dist: azure-storage-queue (>=12.14.1,<13.0.0)
Requires-Dist: boto3 (>=1.17.49,<2.0.0)
Requires-Dist: click (>=8.3.0,<9.0.0)
Requires-Dist: cryptography (>=44.0.1,<45.0.0)
Requires-Dist: google-cloud-pubsub (>=2.33.0,<3.0.0)
Requires-Dist: google-cloud-storage (>=3.6.0,<4.0.0)
Requires-Dist: requests (>=2.32.4,<3.0.0)
Requires-Dist: urllib3 (>=2.5.0,<3.0.0)
Description-Content-Type: text/markdown

# 1 Introduction
**SAP ECS Log Forwarder** is a reference implementation built using the native SDKs of major cloud providers, including AWS, Azure, and GCP. It provides a flexible and unified solution for customers to consume logs from the LogServ ObjectStore.

This service is a unified Python-based application that listens to object or blob creation events across multiple cloud platforms:

AWS SQS (triggered by S3 ObjectCreated notifications)
GCP Pub/Sub (triggered by Cloud Storage OBJECT_FINALIZE events)
Azure Queue (triggered by Storage BlobCreated events)

Upon receiving these events, the forwarder downloads the associated log files, decompresses them if required (e.g., gzip), processes the content line by line, and forwards each entry to configured destinations such as HTTP endpoints, local files, console, or Microsoft Sentinel.

The service includes advanced capabilities such as structured JSON logging, in-memory metrics, regex-based filtering, secure handling of encrypted credentials, retry mechanisms with jitter, and a configuration-driven CLI for easy setup and management.

# 2 Features
This package is designed to provide a simple, flexible, and secure way to collect and forward logs across cloud environments. It offers the following key benefits:

- Unified Configuration
Manage log collection from multiple cloud providers using a single configuration file, making setup and maintenance straightforward.
- Flexible Log Filtering
Easily control which logs are processed and forwarded by applying inclusion or exclusion rules based on your needs.
- Multiple Output Options
Send logs to various destinations such as APIs, local storage, console, or monitoring platforms, with the ability to use multiple outputs simultaneously.
- Secure Data Handling
Sensitive information such as credentials is securely stored in encrypted format to ensure data protection.
- Reliable Log Delivery
Built-in retry mechanisms ensure logs are delivered even in case of temporary failures or network issues.
- Customizable Log Routing
Apply different filtering rules for each destination, allowing precise control over what data goes where.
- Secure Communication Support
Supports secure connections when sending logs to external systems, ensuring data is transmitted safely.
- Structured Logging
Logs are processed in a consistent and structured format, making them easier to analyze and integrate with monitoring tools.
- Operational Visibility
Provides insights into processing activity through internal metrics and logging, helping track system performance.
- Easy Configuration Management
Comes with a command-line interface that simplifies setup, updates, and management of the log forwarding configuration.
- Cloud-Agnostic Design
Works seamlessly across AWS, Azure, and GCP, giving you flexibility to operate in multi-cloud environments.

# 3 Architecture

### 3.1 AWS

![Image for AWS on sap-ecs-log-forwarder](https://raw.githubusercontent.com/sap-ecs-log-forwarders/pypi/main/aws-log-forwarder.png)

 

### 3.2 Azure

![Image for Azure on sap-ecs-log-forwarder](https://raw.githubusercontent.com/sap-ecs-log-forwarders/pypi/main/azure-log-forwarder.png)

### 3.3 GCP

![Image for GCP on sap-ecs-log-forwarder](https://raw.githubusercontent.com/sap-ecs-log-forwarders/pypi/main/gcp-log-forwarder.png)

# 4 Configration Setup Flow (Correct Order)

```bash
Set config path → generate key → set key → set log path → add input → set auth → add output → run
```

# 5 Prerequisites Installation and Verification

### 5.1 Windows Installation

#### Step 1: Download Python
Visit the official Python website:
https://www.python.org/downloads/  
Download the latest available version of Python.

#### Step 2: Run Installer
- Double-click the downloaded `.exe` file  
- Select the option **"Add Python to PATH"**  
- Click **Install Now**

#### Step 3: Verify Installation
Open Command Prompt and execute:
```bash
python --version
pip --version
```

#### Step 4: Set Python Path (if not set automatically)
If Python is not recognized, follow these steps:

1. Open Environment Variables  
2. Click "Edit the system environment variables"  
3. Navigate to System Variables → Path → Edit  
4. Add the following paths:
```
C:\Users\<YourUser>\AppData\Local\Programs\Python\Python3x\
C:\Users\<YourUser>\AppData\Local\Programs\Python\Python3x\Scripts\
```

Restart the terminal and verify again.

#### Step 5: Ensure pip is Installed
```bash
pip --version
```

If pip is not installed:
```bash
python -m ensurepip --upgrade
```

---

### 5.2 Linux Installation (Ubuntu/Debian)

#### Step 1: Update Package List
```bash
sudo apt update
```

#### Step 2: Install Python
```bash
sudo apt install python3 -y
```

#### Step 3: Install pip
```bash
sudo apt install python3-pip -y
```

#### Step 4: Verify Installation
```bash
python3 --version
pip3 --version
```

#### Step 5: Optional – Configure python command
```bash
sudo update-alternatives --install /usr/bin/python python /usr/bin/python3 1
```

Verify:
```bash
python --version
```

---

### 5.3 macOS Installation

#### Step 1: Install Homebrew (if not installed)
```bash
/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"
```

#### Step 2: Install Python
```bash
brew install python
```

#### Step 3: Verify Installation
```bash
python3 --version
pip3 --version
```

#### Step 4: Set Python Path (if required)
```bash
nano ~/.zshrc
```

Add:
```bash
export PATH="/usr/local/bin:$PATH"
```

Apply changes:
```bash
source ~/.zshrc
```

#### Step 5: Optional Alias
```bash
echo "alias python=python3" >> ~/.zshrc
echo "alias pip=pip3" >> ~/.zshrc
source ~/.zshrc
```

---

### 5.4 Final Verification
```bash
python --version
pip --version
```

---

## 6 Activate Virtual Environment

### 6.1 Windows
```bash
cd your-project-folder
python -m venv venv
venv\Scripts\activate
```

Verify:
```bash
where python
```

Deactivate:
```bash
deactivate
```

---

### 6.2 Linux
```bash
sudo apt install python3-venv -y
python3 -m venv venv
source venv/bin/activate
```

Verify:
```bash
which python
```

Deactivate:
```bash
deactivate
```

---

### 6.3 macOS
```bash
python3 -m venv venv
source venv/bin/activate
```

Verify:
```bash
which python
```

Deactivate:
```bash
deactivate
```

---

### 6.4 Troubleshooting

If `venv` is not available:
```bash
python -m venv venv
```

On Linux:
```bash
sudo apt install python3-venv
```

For PowerShell execution issues:
```bash
Set-ExecutionPolicy RemoteSigned -Scope CurrentUser
```

---

## 7 Install sap-ecs-log-forwarder

```bash
pip install sap-ecs-log-forwarder
```

To install a specific version:
```bash
pip install sap-ecs-log-forwarder==<version>
```

Verify installation:
```bash
pip show sap-ecs-log-forwarder
```

List installed packages:
```bash
pip list
```
---

## 8 Collect Configuration Details

### 8.1 AWS

Dynamic Authentication:
- CLIENT_ID  
- CLIENT_SECRET  
- LOGIN_URL  
- AWS_CREDS_URL  
- BUCKET_NAME  
- AWS_REGION  

Static Authentication:
- AccessKeyId  
- SecretAccessKey    
- Region  
- Expiration  

### 8.2 GCP
- subscription  
- project_id  
- config.json (authentication)  

---

## 9 Define Output Configuration

### 9.1 Supported Output Handlers
- files
- http(API)
- console
- sentinel


### Example Configuration
```python
OUTPUT = [
    {
        "type": "http",
        "destination": "https://example.com/ingest",
        "authorization": {
            "type": "bearer",
            "token": "enc:your_encrypted_token",
            "encrypted": True
        },
        "tls": {
            "pathToClientCert": "/path/client.crt",
            "pathToClientKey": "/path/client.key",
            "pathToCACert": "/path/ca.crt",
            "insecureSkipVerify": False
        },
        "includeFilter": ["prod"],
        "excludeFilter": ["test"]
    },
    {
        "type": "files",
        "destination": "logs/",
        "compress": True,
        "includeFilter": [],
        "excludeFilter": []
    },
    {
        "type": "console"
    },
    {
      "type": "sentinel",
      "includeFilter": [],
      "excludeFilter": [],
      "sentinel_dce_tenant_id": "<Destination tenant ID>",
      "sentinel_dce_application_id": "<Destination Application / client ID>",
      "sentinel_dce_application_secret": "<Destination client secret>",
      "sentinel_dce_log_ingestion_url": "<Log ingestion base URL>",
      "sentinel_dce_dcr_immutable_id": "<Data Collection Rule immutable ID>",
      "sentinel_dce_dcr_stream_id": "Custom-SAPLogServ_CL"
    }
]
```

---
## 10 Set Configuration File Path

```bash
SAP_LOG_FORWARDER_CONFIG=<full_path_to_config.py>
```

### 10.1 Windows
```bash
set SAP_LOG_FORWARDER_CONFIG=C:\path\to\config.py
setx SAP_LOG_FORWARDER_CONFIG "C:\path\to\config.py"
echo %SAP_LOG_FORWARDER_CONFIG%
```

### 10.2 Linux
```bash
export SAP_LOG_FORWARDER_CONFIG=/path/to/config.py
echo $SAP_LOG_FORWARDER_CONFIG
```

### 10.3 macOS
```bash
export SAP_LOG_FORWARDER_CONFIG=/path/to/config.py
echo $SAP_LOG_FORWARDER_CONFIG
```

#### Verify Config Path
```bash
sap-ecs-config-cli config-path
```
---


## 11 For Set Configurations there are  Two approach

###  11.1 APPROACH 1 – CLI (Recommended)
 Apply Configuration with Encryption, This approach will generate the configration file automatically. 

#### Generate Encryption Key
```cmd
sap-ecs-config-cli gen-key 
```

Example output:
```bash
Generated key: GF6sgLZ23fwKYgasdw_aKmsdssdlrne4-xyz=
Export it: export FORWARDER_ENCRYPTION_KEY='GF6sgLZ23fwKYgasdw_aKmsdssdlrne4-xyz='
```

#### Set Environment Variable

Linux / macOS:
```bash
export FORWARDER_ENCRYPTION_KEY='GF6sgLZ23fwKYgasdw_aKmsdssdlrne4-xyz='
```

Windows (Command Prompt):
```bash
set FORWARDER_ENCRYPTION_KEY=GF6sgLZ23fwKYgasdw_aKmsdssdlrne4-xyz=
```

Windows (PowerShell):
```bash
$env:FORWARDER_ENCRYPTION_KEY="GF6sgLZ23fwKYgasdw_aKmsdssdlrne4-xyz="
```

---

#### Set Log File Path
```bash
sap-ecs-config-cli set-log-file --path /var/log/sap-log-forwarder/app.log
```

If the directory does not exist:
```bash
mkdir -p /var/log/sap-log-forwarder
```

Linux and macOS:
```bash
sudo mkdir -p /var/log/sap-log-forwarder
sudo chmod 777 /var/log/sap-log-forwarder
```

---

#### Add AWS Input
```bash
sap-ecs-config-cli input add \
  --provider aws \
  --name aws1 \
  --queue https://sqs.us-east-1.amazonaws.com/123/queue \
  --region us-east-1 \
  --bucket my-bucket
```

#### Add GCP Input
```bash
sap-ecs-config-cli input add \
  --provider gcp \
  --name gcp1 \
  --subscription projects/my-project/subscriptions/my-sub \
  --bucket my-bucket
```

---

#### Set AWS Authentication
```bash
sap-ecs-config-cli creds set-provider-auth --input-name aws2
```

---

#### Add Output (HTTP)
```bash
sap-ecs-config-cli output add \
  --input-name aws1 \
  --type http \
  --destination https://example.com/ingest
```

#### File Output
```bash
sap-ecs-config-cli output add \
  --input-name aws1 \
  --type files \
  --destination logs/ \
  --compress
```

#### Sentinel Output
```bash
sap-ecs-config-cli output add \
    --input-name aws1 \
    --type sentinel \
    --sentinel-dce-tenant-id <TENANT_ID> \
    --sentinel-dce-app-id <APP_ID> \
    --sentinel-dce-app-secret <APP_SECRET> \
    --sentinel-dce-ingestion-url <INGESTION_URL> \
    --sentinel-dce-dcr-immutable-id <DCR_IMMUTABLE_ID> \
    --include ".*ERROR.*" \
    --exclude ".*DEBUG.*"
```

#### Console Output
```bash
sap-ecs-config-cli output add \
  --input-name aws1 \
  --type console
```
---

### 11.2 APPROACH 2 – Manual Configuration

Create a `config.json` file and define all required configurations.

This file is common for all hyperscalers.

```json
{
"logLevel": "INFO",
"logFile": "/var/log/sap-log-forwarder/app.jsonl",
"inputs": [
{
"provider": "aws",
"name": "aws1",
"numberOfWorkers": 1,
"queue": "",
"region": "",
"bucket": "",
"includeFilter": ["\.log$", "sap", "dns"],
"excludeFilter": ["debug"],
"maxRetries": 5,
"retryDelay": 10,
"authentication": {
},
"outputs": [
    {
        "type": "http",
        "destination": "",
        "authorization": {
        "type": "bearer",
        "token": "enc:...",
        "encrypted": true
        },
        "tls": {
        "pathToClientCert": "/path/client.crt",
        "pathToClientKey": "/path/client.key",
        "pathToCACert": "/path/ca.crt",
        "insecureSkipVerify": false
        },
        "includeFilter": ["prod"],
        "excludeFilter": ["test"]
    },
    {
        "type": "files",
        "destination": "logs/",
        "compress": true,
        "includeFilter": [],
        "excludeFilter": []
    },
    {
        "type": "console"
    }
]
}
]
}
```
**Note:** If you need to modify anything in the configuration file, you can update it, save the changes, and then rerun the forwarder package.

---


## 12 Run Log Forwarder
```bash
sap-ecs-log-forwarder
```

---

## 13 Verify Output
- Ensure logs are processed correctly  
- Confirm data reaches the destination  
- Verify there are no errors in logs or console  

---

### Example: How to create config.json File using CLI

```bash
# Activate environment
source log-forwarder-env/bin/activate

# Set config path
export SAP_LOG_FORWARDER_CONFIG="/path/to/config.json"

# Create directory if required
mkdir -p /path/to

# Generate encryption key
sap-ecs-config-cli gen-key

# Set encryption key
export FORWARDER_ENCRYPTION_KEY='<YOUR_KEY>'

# Create logs directory
mkdir -p logs

# Set log file
sap-ecs-config-cli set-log-file --path ./logs/app.log

# Add AWS input
sap-ecs-config-cli input add \
  --provider aws \
  --name aws1 \
  --queue https://sqs.us-east-1.amazonaws.com/123/queue \
  --region us-east-1 \
  --bucket my-bucket

# Set AWS authentication
sap-ecs-config-cli creds set-provider-auth --input-name aws1

# Add HTTP output
sap-ecs-config-cli output add \
  --input-name aws1 \
  --type http \
  --destination https://example.com/ingest

# Optional file output
sap-ecs-config-cli output add \
  --input-name aws1 \
  --type files \
  --destination logs/ \
  --compress

# Optional HTTP authentication
sap-ecs-config-cli creds set-http-auth \
  --input-name aws1 \
  --output-index 0 \
  --auth-type bearer

# Run forwarder
sap-ecs-log-forwarder
```

---

## 14 Usage

To run the Log Forwarder, use the following command:

```sh
sap-ecs-log-forwarder
```

Service starts threads / async loops per input, logs metrics snapshot every 30s.

## 15 Structured Logging
Emits JSON lines to console (and to logFile if configured):
```json
{"ts":"2025-11-25T12:00:00","level":"INFO","message":"Wrote logs to logs/app.log.gz","thread":"aws-aws1"}
```

## 16 Metrics (logged)
Counters (examples):
- files_forward_success / files_forward_error
- http_forward_success / http_forward_error
- output_invocations
- aws_messages_processed / aws_retry
- gcp_messages_processed / gcp_retry
- azure_messages_processed / azure_retry

Snapshot logged periodically; can be extended to expose HTTP endpoint.

## 17 Filtering
- Include / exclude lists are regex patterns applied (case-insensitive) to object/blob/file names.
- Exclude overrides include.
- Output-level include/exclude filters are also supported per output.

## 18 Retries
Exponential backoff with added random jitter up to base retryDelay for download / processing failures.

## 19 HTTP Output
- Per-line POST with `Content-Type: application/json`.
- Auth types: bearer, api-key, basic.
- TLS options:
  - pathToClientCert + pathToClientKey for mTLS
  - pathToCACert for custom CA
  - insecureSkipVerify to skip verification (not recommended)

## 20 Extending
- Add new output: implement handler and register in `processor.py` (OUTPUT_HANDLERS).
- Add metrics: `metrics.inc("<name>")`.
- Add provider: create runner class following existing pattern and register in `consumer.PROVIDERS`.

## 21 Security Notes
- Ensure `FORWARDER_ENCRYPTION_KEY` is injected via secrets manager.
- Structured logs avoid sensitive fields.

### 21.1 AWS Dynamic Authentication (Temporary Credentials)
Instead of static IAM keys, you can configure dynamic AWS auth. The forwarder will:
- POST to your backend login endpoint with `client_id` and `client_secret`, receive `session-id-raven` cookie.
- GET temporary AWS credentials for the target bucket, and refresh them every ~15 minutes or before expiration.

CLI:
```bash
sap-ecs-config-cli creds set-provider-auth --input-name aws1
# Choose auth mode: dynamic
# Provide client_id, client_secret, login URL (default http://logserv.forwarder.host/api/v1/app/login)
# Provide AWS credentials URL (default http://logserv.forwarder.host/api/v1/aws/credentials)
```

Config (snippet):
```json
{
  "provider": "aws",
  "name": "aws1",
  "queue": "https://sqs.us-east-1.amazonaws.com/123/queue",
  "region": "us-east-1",
  "bucket": "my-s3-bucket",
  "authentication": {
    "clientId": "enc:...",
    "clientSecret": "enc:...",
    "loginUrl": "http://logserv.forwarder.host/api/v1/app/login",
    "awsCredsUrl": "http://logserv.forwarder.host/api/v1/aws/credentials",
    "encrypted": true
  }
}
```

Backend contract:
```bash
# Login
curl -X POST -H 'content-type: application/x-www-form-urlencoded' \
  -d client_id=xxx -d client_secret=xxx \
  http://logserv.forwarder.host/api/v1/app/login

# Fetch temporary creds (cookie required)
curl -H 'Cookie: session-id-raven=xxxx' \
  'http://logserv.forwarder.host/api/v1/aws/credentials?bucket=my-s3-bucket'
```

Expected response:
```json
{
  "data": {
    "AccessKeyId": "xxxx",
    "SecretAccessKey": "xxxxx",
    "SessionToken": "xxxxxx",
    "Expiration": "2025-11-27T00:20:33Z",
    "Region": "ap-south-1"
  }
}
```

Notes:
- Region from response overrides input region if provided; otherwise uses configured `region`.
- Static keys are used if dynamic fields are absent.
- Credentials refresh occurs on a schedule or 60s before expiration.

### 21.2 Azure Dynamic Authentication (Temporary SAS)
Instead of a static `sasToken`, you can configure dynamic Azure auth. The forwarder will:
- POST to your backend login endpoint with `client_id` and `client_secret`, receive `session-id-raven` cookie.
- GET temporary credentials for the storage account, extract `SASToken`, and refresh it every ~15 minutes (or before expiry).

CLI:
```bash
sap-ecs-config-cli creds set-provider-auth --input-name azure1
# Choose auth mode: dynamic
# Provide client_id, client_secret, login URL (default http://logserv.forwarder.host/api/v1/app/login)
# Provide credentials URL (default http://logserv.forwarder.host/api/v1/azure/credentials)
```

Config (snippet):
```json
{
  "provider": "azure",
  "name": "azure1",
  "queue": "my-queue",
  "storageAccount": "mystorageacct",
  "authentication": {
    "clientId": "enc:...",
    "clientSecret": "enc:...",
    "loginUrl": "http://logserv.forwarder.host/api/v1/app/login",
    "credsUrl": "http://logserv.forwarder.host/api/v1/azure/credentials",
    "encrypted": true
  }
}
```

Notes:
- The credentials endpoint must accept `storage-account-name` query parameter.
- Response must include `data.SASToken` and `data.Expiration` (RFC3339 UTC).
- If `sasToken` is set and dynamic fields absent, static SAS is used.
- For connection strings, the runner uses them directly and skips SAS refresh.

```bash
# macOS: test endpoints locally
curl -X POST -H "content-type: application/x-www-form-urlencoded" \
  -d "client_id=xxx" -d "client_secret=xxx" \
  http://logserv.forwarder.host/api/v1/app/login

curl -H "Cookie: session-id-raven=xxxxx" \
  "http://logserv.forwarder.host/api/v1/azure/credentials?storage-account-name=mystorageacct"
```

### 21.3 GCP Dynamic Authentication (Temporary Credentials)

**Will be available in a future release**

## 22 Systemd Deployment Guide
(For Linux servers only.)

###  Overview
Run the forwarder as a managed service with automatic restarts and persistent log files.

### 1. Create a Service Account (Optional)
```bash
sudo useradd -r -s /bin/false saplogfwd || true
```

### 2. Create Directories
```bash
sudo mkdir -p /opt/sap-log-forwarder
sudo mkdir -p /var/log/sap-log-forwarder
sudo chown -R saplogfwd:saplogfwd /opt/sap-log-forwarder /var/log/sap-log-forwarder
```

Place your project (editable install or wheel) under /opt/sap-log-forwarder or install via pip:
```bash
cd /opt/sap-log-forwarder
sudo pip install sap-ecs-log-forwarder
```

### 3. Environment File
Create /etc/sap-log-forwarder.env:
```bash
SAP_LOG_FORWARDER_CONFIG=/etc/sap-log-forwarder/config.json
FORWARDER_ENCRYPTION_KEY=YOUR_FERNET_KEY
PYTHONUNBUFFERED=1
LOG_LEVEL=INFO
```

(Generate key earlier and replace.)

### 4. Systemd Unit
Create /etc/systemd/system/sap-log-forwarder.service:
```ini
[Unit]
Description=SAP ECS Log Forwarder
After=network.target
Wants=network-online.target

[Service]
Type=simple
User=saplogfwd
Group=saplogfwd
WorkingDirectory=/opt/sap-log-forwarder
EnvironmentFile=/etc/sap-log-forwarder.env
ExecStart=/usr/bin/python -m sap_ecs_log_forwarder.consumer
Restart=always
RestartSec=5
# File logging (append). Systemd 240+ supports append:
StandardOutput=append:/var/log/sap-log-forwarder/forwarder.out.log
StandardError=append:/var/log/sap-log-forwarder/forwarder.err.log
# Alternatively (recommended) use journald and remove StandardOutput/StandardError lines.

# Resource limits (optional)
NoNewPrivileges=yes
ProtectSystem=full
ProtectHome=yes

[Install]
WantedBy=multi-user.target
```

Reload and start:
```bash
sudo systemctl daemon-reload
sudo systemctl enable --now sap-log-forwarder
```

Status:
```bash
systemctl status sap-log-forwarder
```

### 5. Log Rotation (If Using File Output)
Create /etc/logrotate.d/sap-log-forwarder:
```conf
/var/log/sap-log-forwarder/forwarder.*.log {
  daily
  rotate 14
  compress
  missingok
  notifempty
  create 0640 saplogfwd saplogfwd
  sharedscripts
  postrotate
    systemctl kill -s SIGUSR1 sap-log-forwarder || true
  endscript
}
```

(Forwarder ignores SIGUSR1 now; postrotate just optional. Compression handled by logrotate.)

### 6. Using Journald Instead
Remove StandardOutput/StandardError lines to keep logs in journal:
```bash
journalctl -u sap-log-forwarder -f
```
To export periodically, use:
```bash
journalctl -u sap-log-forwarder --since "1 hour ago" > /var/log/sap-log-forwarder/hour.log
```

### 7. Config File Location
Place config.json in WorkingDirectory (/opt/sap-log-forwarder/config.json). Edit via CLI:
```bash
sudo -u saplogfwd sap-ecs-config-cli input list
```

### 8. Permissions
Ensure /var/log/sap-log-forwarder writable by service user. If using file outputs (destination logs/):
```bash
sudo mkdir -p /opt/sap-log-forwarder/logs
sudo chown saplogfwd:saplogfwd /opt/sap-log-forwarder/logs
```

### 9. Updating
```bash
sudo systemctl stop sap-log-forwarder
sudo pip install --upgrade sap-ecs-log-forwarder
sudo systemctl start sap-log-forwarder
```

### 10. Common Checks
```bash
systemctl status sap-log-forwarder
journalctl -u sap-log-forwarder -n 50
ls -lh /var/log/sap-log-forwarder
```

### 11. Troubleshooting Restart Loops
- Inspect recent logs (`journalctl -u sap-log-forwarder -f`).
- Validate config.json structure.
- Verify FORWARDER_ENCRYPTION_KEY present (`systemctl show -p Environment sap-log-forwarder`).

### 12. Security Hardening (Optional)
Add directives:
```
PrivateTmp=yes
ProtectKernelTunables=yes
ProtectKernelModules=yes
RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX
```

Keep encryption key in a root-readable-only environment file (chmod 640).

### 13. Switching To Virtualenv
If installed in virtualenv:
```
ExecStart=/opt/sap-log-forwarder/venv/bin/python -m sap_ecs_log_forwarder.consumer
```

Ensure venv ownership matches service user.

### 14. Testing Manual Run
```bash
sudo -u saplogfwd FORWARDER_ENCRYPTION_KEY=YOUR_FERNET_KEY python -m sap_ecs_log_forwarder.consumer
```

Stop with:
```bash
sudo systemctl stop sap-log-forwarder
```

Service now runs with automatic restart (Restart=always) and logs persisted under /var/log/sap-log-forwarder.

## 23 Troubleshooting

### Quick Checklist
1. config.json present and valid (`jq '.' config.json`).
2. `FORWARDER_ENCRYPTION_KEY` exported.
3. Inputs listed (`sap-ecs-config-cli input list`).
4. Outputs configured (`sap-ecs-config-cli output list <input>`).
5. Per-input `logLevel` set (use `DEBUG` for diagnosis).
6. If using HTTP/TLS, verify cert/key/CA paths.

### General Issues
| Symptom | Cause | Fix |
|---------|-------|-----|
| Encrypted values not decrypted | Missing `FORWARDER_ENCRYPTION_KEY` | Export key before running consumer |
| No files processed | Check exclude/include filters | Adjust filters |
| Output skipped | Output-level include/exclude filters | Remove or correct filters |
| HTTP send fails | Bad URL, TLS paths, undecrypted auth | Verify destination, cert/key/CA paths, export key |
| File output missing | No destination or permission denied | Provide writable directory in output config |

### AWS (SQS + S3)
- No messages: Ensure S3 event notification configured (ObjectCreated -> SQS).
- Objects ignored: Bucket mismatch, eventName not `ObjectCreated:Put`, regex filters.
- Repeated retries (`aws_retry`): Missing `s3:GetObject` or corrupt gzip.
- Credential failure: Re-run `creds set-provider-auth` and export key.

### GCP (Pub/Sub + Storage)
- Subscription path invalid: Must be `projects/<project>/subscriptions/<name>`.
- PermissionDenied: Grant `roles/pubsub.subscriber` and `roles/storage.objectViewer`.
- Service account JSON parse failure: Re-enter credentials (file/paste mode).
- Events ignored: Not `OBJECT_FINALIZE` or filters reject.

### Azure (Queue + Blob)
- Decode errors: Ensure Event Grid -> Queue message schema.
- Missing blob URL: Recreate Event Grid subscription; `data.url` must exist.
- Unauthorized blob download: Provide SAS token or full connection string.
- Filters miss: Subject pattern `/blobServices/default/containers/<c>/blobs/<name>`; adjust regex.

### Performance
- Slow: Large object fully loaded; consider future streaming enhancement.
- Many retries: Increase `retryDelay`, verify network and permissions.
- Parallelism: Each input runs its own thread/loop; split workloads across inputs.

### Debug Commands (macOS)
```bash
env | grep FORWARDER_ENCRYPTION_KEY
sap-ecs-config-cli input list
sap-ecs-config-cli output list <input-name>
python -m sap_ecs_log_forwarder.consumer
```

### Metrics Reference
- aws_messages_processed / aws_retry
- gcp_messages_processed / gcp_retry
- azure_messages_processed / azure_retry
- output_invocations
- files_forward_success / files_forward_error
- http_forward_success / http_forward_error

### When Nothing Processes
1. Set input `logLevel` to DEBUG.
2. Temporarily clear include/exclude filters.
3. Upload a small test file that matches expected pattern.
4. Confirm metrics counters increment.
5. Reapply filters gradually.

## License
This application and its source code are licensed under the terms of the SAP Developer License Agreement. See the LICENSE file for more information.

## Changelog

**Version 1.2.0**

- Add possibility to run multiple workers to consume queues
  - Add at config file the key `$.inputs[].numberOfWorkers` the number of workers ( default 1, limited to 256 ) to enable concurrency.

**Version 1.1.0**

- Added output type `sentinel` to ingest data into a Microsoft Sentinel deployment through the solution "SAP LogServ (RISE), S/4 HANA Cloud Private Edition" from the Microsoft Sentinel Content Hub as described [here](https://community.sap.com/t5/enterprise-resource-planning-blog-posts-by-sap/sap-logserv-integration-with-microsoft-sentinel-for-sap-rise-customers-is/ba-p/14085387). 

**Version 1.0.2**

- Solving minor issue to avoid data loss when HTTP destination is unavailable

**Version 1.0.1**

- Security Patches

**Version 1.0.0**

- First proper release
