Metadata-Version: 2.5
Name: s2s-forwarder
Version: 0.0.1
Summary: Send log events to Splunk indexers using the Splunk-to-Splunk (S2S) protocol v4
Project-URL: Homepage, https://github.com/pasdesignal/s2s-forwarder
Project-URL: Repository, https://github.com/pasdesignal/s2s-forwarder
Project-URL: Issues, https://github.com/pasdesignal/s2s-forwarder/issues
Author-email: pasdesignal <pasdesignal@users.noreply.github.com>
License-Expression: Apache-2.0
License-File: LICENSE
Keywords: forwarder,log-shipping,logging,logging-handler,s2s,siem,splunk
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: System Administrators
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Topic :: System :: Logging
Classifier: Topic :: System :: Networking
Requires-Python: >=3.9
Requires-Dist: click
Provides-Extra: dev
Requires-Dist: pytest; extra == 'dev'
Description-Content-Type: text/markdown

# s2s-forwarder

A Python library for sending log events to Splunk indexers using the
Splunk-to-Splunk (S2S) protocol version 4 — the native protocol used by
Splunk Universal and Heavy Forwarders. No official Python library exists for
S2S v4 sending, so this community project was started to fill that gap for testing
and edge-case purposes.

## Install

Not yet published to PyPI. Install from a checkout:

```bash
pip install .                      # from the repo root
pip install -e .                   # editable, for development
pip install /path/to/s2s           # from elsewhere
```

That gives you the `s2s-forwarder` package and the `s2s-fwd` command.
Python 3.9+ - the only runtime dependency is `click`.

Verify:

```bash
python -c "import s2s_forwarder; print(s2s_forwarder.__version__)"
s2s-fwd --help
```

## Usage
### Within a python script as a simple event sender

```python
from s2s_forwarder import S2SClient, Event

with S2SClient("splunk.example.com", port=9997) as client:
    client.send(Event(
        raw="2026-08-19 10:30:00 something happened",
        index="main",
        source="s2s:appname",
        sourcetype="myapp",
    ))
```

Use the context manager. `close()` is required for correctness in this
protocol — see "Closing the connection" — and `with` guarantees it even if
your code raises. The explicit form (`connect()` … `close()`) also works.

`Event` fields: `raw` (required), `time`, `index`, `host`, `source`,
`sourcetype`. Unset fields fall back to sensible defaults.

**`send()` also accepts a list of `Event`s**, coalescing consecutive
events that share one channel into fewer wire sends. Call
`send_single(event)` directly if you want a guaranteed one-event-per-send
with no coalescing at all.

```python
client.send([Event(raw=line, source="app.log", sourcetype="myapp")
             for line in lines])
```

**`Event.time` is a fallback, not an override.** Splunk prefers a
timestamp it can parse out of `raw`, and only uses `Event.time` when `raw`
has none — verified live. So setting `time` on an event whose text already
carries a timestamp does nothing. To control event time explicitly, either
put the timestamp in `raw` (what a real forwarder does) or leave `raw`
without one and set `time`.

### Command line

```bash
# a single event
s2s-fwd --host splunk.example.com --index main --sourcetype syslog "something happened"

# one event per line from stdin - a bounded queue keeps memory flat even
# on a large file, and consecutive lines are coalesced into fewer sends
cat /var/log/app.log | s2s-fwd --host splunk.example.com --index main

# follow a live log
tail -f /var/log/app.log | s2s-fwd --host splunk.example.com --index main
```

`--source`, `--sourcetype`, `--index` and `--event-host` set event
metadata; `--sender-hostname` sets the name this sender identifies as
during the handshake. `--batch-size`/`--batch-linger` control how stdin
lines are coalesced (defaults: up to 100 lines, waiting up to 0.1s for
more before sending a partial batch — keeps a live `tail -f` responsive
instead of stalling until a full batch arrives). `s2s-fwd --help` lists
everything.

### asyncio

`AsyncS2SClient` mirrors `S2SClient` — same arguments, same defaults, same
semantics, same encoder. Both share one session implementation, and a test
asserts the two constructors stay identical, so neither the protocol rules
nor the policy built on them can drift between the clients.

```python
import asyncio
from s2s_forwarder import Event
from s2s_forwarder.aio import AsyncS2SClient

async def main():
    async with AsyncS2SClient("splunk.example.com") as client:
        await client.send(Event(raw="2026-08-19 10:30:00 something happened",
                                index="main", sourcetype="myapp"))

asyncio.run(main())
```

One client owns one connection, and S2S channel state is per-connection,
so `send()` serialises via an internal lock — concurrent callers are safe
but share the wire. For real parallelism use several clients, i.e. several
connections, which is what a forwarder does too:

```python
async def worker(n):
    async with AsyncS2SClient("splunk.example.com") as c:
        for line in lines[n]:
            await c.send(Event(raw=line, source=f"/var/log/app{n}.log"))

await asyncio.gather(*(worker(n) for n in range(4)))
```

### Python logging handler

Drop-in handler for the stdlib `logging`:

```python
import logging
from s2s_forwarder.logging_handler import SplunkS2SHandler

handler = SplunkS2SHandler("splunk.example.com", index="main",
                           sourcetype="myapp", tls=True, ca_cert="ca.pem")
logging.getLogger().addHandler(handler)

logging.info("service started")     # returns immediately
logging.shutdown()                  # flushes and closes cleanly
```

Records are written by a background thread, so `logging.info()` never
waits on the network. If Splunk is unreachable and the queue fills, the
handler **drops records and counts them** (`handler.stats`) rather than
blocking the application — stalling the program producing the telemetry is
worse than losing some of it. Send failures never propagate to the caller
and never kill the worker.

`close()` matters here: it triggers the per-channel padding described
above. `logging.shutdown()` calls it at normal interpreter exit, so the
protocol quirk stays invisible — but a hard `kill -9` still costs the last
record per channel.

Extra keyword arguments are passed to `S2SClient`, so TLS, mTLS and retry
work exactly as they do there.

### Containers: shipping a file-based log

The common pattern — a sidecar or entrypoint tailing an application's log
file and forwarding it:

```dockerfile
FROM python:3.12-slim

COPY . /src
RUN pip install --no-cache-dir /src && rm -rf /src

# certs are mounted at runtime, never baked into the image
ENV SPLUNK_HOST=inputs.example.splunkcloud.com \
    SPLUNK_INDEX=main \
    SPLUNK_SOURCETYPE=myapp

# tail -F (capital F) survives log rotation; -f does not
CMD tail -F /var/log/app/app.log | s2s-fwd \
      --host "$SPLUNK_HOST" \
      --index "$SPLUNK_INDEX" \
      --sourcetype "$SPLUNK_SOURCETYPE" \
      --source /var/log/app/app.log \
      --tls --ca-cert /certs/ca.pem --client-cert /certs/client.pem \
      --key-password "$(cat /certs/key.pw)" \
      --retry 20 --quiet
```

```bash
docker run -v /var/log/app:/var/log/app:ro -v ./certs:/certs:ro myimage
```

Points that matter in a container specifically:

- **`tail -F`, not `-f`.** Capital `F` reopens the file after rotation;
  lowercase `-f` keeps holding the old inode and silently stops producing
  lines.
- **`--retry`** — a long-running shipper will outlive indexer restarts and
  network blips. Without it the first failure ends the process.
- **`SIGTERM` is handled.** `docker stop`, `podman stop` and Kubernetes all
  terminate with `SIGTERM`, and Python's default handler kills the process
  *without* running cleanup — which would drop the last event on every
  channel, every restart. `s2s-fwd` and `SplunkS2SHandler` both install a
  handler so shutdown stays graceful. Give the container enough grace
  period to finish (`docker stop -t 15`).
- **Mount certificates at runtime**; don't bake credentials into an image.
- **`--source`** is worth setting explicitly, since the log's path inside
  the container is rarely the name you want in Splunk.
- **PID 1**: if you wrap the command in a shell, signals may not reach the
  process. Use an init (`docker run --init`) or `exec` the pipeline.

Note: for a containerised Python application, prefer the logging handler over
tailing a file — it keeps tracebacks intact as single events and avoids the
file entirely.

### No fishbucket — position tracking is on you

A Universal Forwarder checkpoints its read position per file (its
"fishbucket"), so a restart resumes from exactly where it left off instead
of either re-reading old lines or silently skipping whatever was written
during the outage. **This library is a sender, not a UF reimplementation,
and deliberately does not do this.** `tail -F | s2s-fwd` has neither
property: `tail -F` starts wherever `tail` starts — the current end of the
file by default — so a restart during downtime silently skips every line
written in the gap. Easy to miss if you're assuming UF-equivalent
behaviour.

If gap-free delivery across restarts matters for your use case,
`examples/resumable_tail.py` shows the minimal pattern: persist a byte
offset, seek to it on start, advance it after each `send()` returns. It's
not production-grade fishbucket parity (a real UF also tracks inode+size
to catch rotation more robustly), but it's a starting point.

### TLS

TLS is a transport wrap — the S2S bytes inside the tunnel are identical to
the plaintext case. Parameters mirror Splunk's `outputs.conf` names so an
existing forwarder config translates directly.

```python
client = S2SClient(
    "splunk.example.com",
    tls=True,
    ca_cert="/path/ca.pem",          # Splunk's sslRootCAPath
    client_cert="/path/client.pem",  # Splunk's clientCert (mutual TLS)
    key_password="…",                # Splunk's sslPassword
)
```

```bash
s2s-fwd --host splunk.example.com --tls --ca-cert /path/ca.pem --index main "event"
```

Certificate verification is **on by default**. Splunk's own forwarder
default (`sslVerifyServerCert = false`) is the opposite, but silently
trusting any certificate is a poor default for a library, so disabling it
is explicit: `verify=False` / `--no-verify`. For the common lab case of a
self-signed cert whose CN doesn't match the IP, keep verification on and
turn off just the hostname check (`check_hostname=False` /
`--no-check-hostname`).

### Error handling

```python
from s2s_forwarder import S2SError, S2SHandshakeError, S2SConnectionError
```

`S2SError` is the base
`S2SHandshakeError` means the handshake didn't complete
`S2SConnectionError` means the connection failed or was closed

### Library logging

s2s_forwarder logs its own operational events — connect, reconnect,
rotate, retry, and (at DEBUG) wire-level detail like channel registration
and closing padding — through the standard `logging` module, under the
`"s2s_forwarder"` logger. Following the same convention as `requests` and
`urllib3`, it attaches a `NullHandler` and stays completely silent until
an application configures that logger itself:

```python
import logging
logging.getLogger("s2s_forwarder").addHandler(logging.StreamHandler())
logging.getLogger("s2s_forwarder").setLevel(logging.INFO)  # or DEBUG
```

`s2s-fwd` wires this up for you via `-v`/`-vv`:

```bash
s2s-fwd --host splunk.example.com -v  "something happened"   # INFO
s2s-fwd --host splunk.example.com -vv "something happened"   # + DEBUG
```

Two recipes worth knowing about:

- **Local troubleshooting** — a rotating file handler on `"s2s_forwarder"`
  is the one diagnostic path guaranteed to work even when the Splunk
  connection itself is broken (`examples/local_troubleshooting_handler.py`).
- **Self-diagnostics in Splunk** — attach a *second* `SplunkS2SHandler`
  instance to `"s2s_forwarder"` itself with
  `forward_internal_logs=True`, so the library ships its own
  connect/reconnect/rotate activity into Splunk alongside a first handler
  shipping the application's normal logs
  (`examples/self_diagnostic_logging.py`). Close the two handlers
  explicitly, application handler first, rather than relying on
  `logging.shutdown()`: it closes handlers in *reverse* creation order,
  so the diagnostics handler would otherwise close before the handler
  it's observing has necessarily done anything worth observing yet.

**`SplunkS2SHandler` does not ship the library's own log records by
default** (`forward_internal_logs=False`). This matters because the
handler is normally attached to the root logger, which `s2s_forwarder`'s
records propagate to — so without the filter the handler is fed by the
thing it is feeding. Left unchecked that is not merely untidy: a failing
indexer makes each failed batch log a warning that becomes the next
batch, which also fails, indefinitely. The filter is the first line of
defence and a re-entrancy guard is the second;
`handler.stats["internal_filtered"]` and `["reentrant_drops"]` show each
firing.

When forwarding *is* enabled, the library's own records do **not** inherit
your application's metadata. They are routed to:

| | value | override |
|---|---|---|
| `index` | `_internal` | `internal_index=` |
| `sourcetype` | `s2s_forwarder` | `internal_sourcetype=` |
| `source` | `s2s_forwarder` | `internal_source=` |

This mirrors a real Universal Forwarder, which ships its own
`splunkd.log` and `metrics.log` to `_internal` rather than into whichever
index the data it forwards belongs to. It keeps sender diagnostics out of
your application searches, off your application index's retention, and in
the place a Splunk admin already looks for forwarder health. Set
`internal_index=None` to send them to the same index as everything else.

## What works so far

- S2S v4 handshake
- Compact-format event encoding
- Multiple channels — each distinct `(source, host, sourcetype)` — on one connection
- Single-event and high-volume multi-event sessions
- Batching (`send()` with a list, the logging handler, and CLI stdin all coalesce)
- Explicit error types for handshake and connection failures
- `s2s-fwd` CLI, including streamed and batched stdin
- TLS, including mutual TLS and certificate verification
- Opt-in reconnection and retry (see "Reconnection and retry")
- Auto load balancing across several indexers (see "Load balancing")
- asyncio client (`s2s_forwarder.aio.AsyncS2SClient`)
- `logging.Handler` for Python applications (see "Python logging")
- Library-internal operational logging, silent by default (see "Library logging")

## Not implemented yet

- **Acknowledgements** — see "Delivery semantics"
- **Compression** (`compression=1`)

## Delivery semantics

The library currently negotiates `ack=0`, matching a default-configured forwarder, so there is
no confirmation that any given event was durably received. A raised
`S2SConnectionError` tells you something broke, but events already "sent"
before it may or may not have landed.

Do not use this where losing any events is unacceptable.

### Reconnection and retry

Opt in with `retry=N` (or `--retry N`) to reconnect and re-send after a
connection failure, with exponential backoff:

```python
client = S2SClient("splunk.example.com", retry=10, retry_backoff=1.0)
...
print(client.stats)   # {'sent': 812, 'reconnects': 2, 'resends': 2}
```

Retry is **off by default** because, with no acknowledgement, a failure
gives no indication whether the bytes already arrived — so a re-sent event
may be duplicated. Enable it when keeping a long-running stream up matters
more than avoiding duplicates.

### Load balancing across indexers

Pass more than one indexer and the client spreads across them, the way a
forwarder does. `hosts` is polymorphic — you never have to normalise it:

```python
S2SClient("idx1")                          # one indexer
S2SClient("idx1:9998")                     # with a port
S2SClient("idx1,idx2")                     # comma-separated
S2SClient(["idx1", "idx2:9998"])           # a list
```

```bash
s2s-fwd --host idx1 --host idx2:9998 --index main "event"
s2s-fwd --host idx1,idx2 --index main "event"      # equivalent
```

Every `auto_lb_frequency` seconds (default **30**, matching Splunk's own
`autoLBFrequency`) the client closes its connection gracefully and opens a
new one. Selection is random but never re-picks the indexer already in
use, so each rotation actually moves; `lb_strategy="round_robin"` gives
deterministic cycling instead.

## Event breaking: what Splunk does with your data

This trips people up and is **not** a bug in this library — a real
Universal Forwarder behaves identically.

Splunk decides event boundaries itself, at the indexer, using its
documented `props.conf` defaults: it splits on newlines, then *re-merges*
those lines into events (`SHOULD_LINEMERGE=true`), starting a new event
only before a line carrying a **recognisable date**
(`BREAK_ONLY_BEFORE_DATE=true`), up to `MAX_EVENTS=256` lines.

So if you stream content with no recognisable dates, everything you send
is delivered completely and intact — but it may be indexed as a few large
merged events rather than one event per line.

If you want one event per line, configure it receiver-side in
`props.conf` for your sourcetype:

```ini
[my_sourcetype]
SHOULD_LINEMERGE = false
LINE_BREAKER = ([\r\n]+)
```

That is what Splunk itself recommends for well-structured data.

## Requirements

Python 3.9+ (ships with RHEL 9). No runtime dependencies beyond `click`
for the CLI.

## Disclaimer

This library implements the Splunk-to-Splunk (S2S) protocol v4 based
entirely on packet capture analysis and publicly available third-party
documentation. No Splunk source code was used or referenced. Splunk (Cisco)
does not publish the S2S specification and does not support third-party use
of this protocol.

This library is intended for development, testing, and research use. It
does not implement the S2S acknowledgement mechanism, consistent with the
default configuration of the majority of enterprise Splunk deployments.

Tested against **Splunk Enterprise 9.0.5** (plaintext and TLS on TCP 9997)
and against **Splunk Cloud** over mutual TLS with full certificate and
hostname verification, both negotiating `v4=true` / `pl=6`. Protocol
behaviour may change in future Splunk releases without notice.

## Development

```bash
pip install -e ".[dev]"
pytest
```

## License

Apache 2.0 — see `LICENSE`.


