Syslog Server#

Syslog server is central concentration for syslog messages. Additionally, it provides web interface for real time monitoring and filtering of log messages.

Syslog listening socket can be configured as TCP, UDP or TCP+SSL socket. Communication is based on RFC 5425, RFC 5426, RFC 6587. Once message is received, server stores message in predefined database.

Warning

Current implementation of Syslog server doesn’t support UDP communication

skinparam linetype ortho

folder Client {
    component SysLogHandler
}

folder "Syslog Server" {
    component SysLogServer
    component Backend
    component WebServer

    interface add
    interface query
    interface notify
}

folder Browser {
    component JS
    component GUI
}

database Sqlite

interface syslog
interface http
interface websocket

syslog - SysLogServer
Backend -- Sqlite
add - Backend
Backend - query
Backend - notify
http - WebServer
WebServer -- websocket

SysLogHandler --> syslog
SysLogServer -> add
query <- WebServer
notify -> WebServer

http <-- Browser
websocket <-> JS
JS -> GUI

Running#

Syslog Server is implemented as python hat.syslog.server package which can be run with hat-syslog script with additional command line arguments:

usage: hat-syslog [-h] [--conf PATH] [--log LEVEL] [--syslog-addr ADDR]
                  [--syslog-pem PATH] [--ui-addr ADDR] [--db-path PATH]
                  [--db-low-size N] [--db-high-size N] [--db-enable-archive]
                  [--db-disable-journal]

options:
  -h, --help            show this help message and exit
  --conf PATH           configuration defined by hat-syslog://server.yaml#
                        (default $XDG_CONFIG_HOME/hat/syslog.{yaml|yml|json})
  --log LEVEL           console log level
  --syslog-addr ADDR    syslog listening address (default tcp://0.0.0.0:6514)
  --syslog-pem PATH     pem file path - mandatory for ssl communication
  --ui-addr ADDR        UI listening address (default http://0.0.0.0:23020)
  --db-path PATH        sqlite database file path (default
                        $XDG_DATA_HOME/hat/syslog.db)
  --db-low-size N       number of messages kept in database after database
                        cleanup (default 1000000)
  --db-high-size N      number of messages that will trigger database cleanup
                        (default 10000000)
  --db-enable-archive   should messages, deleted during database cleanup, be
                        kept in archive files
  --db-disable-journal  disable sqlite jurnaling

This application is part of hat-syslog python package.

Configuration#

Syslog Server configuration written in form of single YAML or JSON file with structure defined by JSON Schema hat-syslog://server.yaml#. Path to configuration file is provided as command line argument during process startup. Additionally, configuration parameters provided in configuration file can be overridden by command line arguments. If configuration file could not be found, default values of configuration parameters are used.

Example of configuration:

---
type: syslog
log:
    version: 1
syslog_addr: 'tcp://0.0.0.0:6514'
ui_addr: 'http://0.0.0.0:23020'
dp_path: 'syslog.db'
db_low_size: 1_000_000
db_high_size: 10_000_000
db_enable_archive: false
db_disable_journal: false
...

Data backend#

All incoming syslog messages are stored in single sqlite database. Maximum number of syslog messages stored in this database can be configured by configuration parameter db_high_size (value 0 represents unlimited number of messages). Once number of messages exceed configured limit, database cleanup procedure is triggered. During cleanup procedure, oldest messages are removed from database until number of messages reaches configuration parameter db_low_size when cleanup procedure stops. Prior to message deletion, if configuration parameter db_enable_archive is set, new database with unique file name is created and all messages scheduled for removal are inserted into newly created database. Archive database has got same structure as original database and can be used in place of original database for accessing archived syslog messages.

Web UI#

Together with acquiring and storing syslog messages, Syslog Server provides web-based user interface for querying messages from database and observing changes in real time. Communication between web server and browser is based on juggler communication.

Server state#

Server state is used for providing continuously updated list of log entries to clients, based on applied filters.

State structure is defined by JSON schema hat-syslog://juggler.yaml#/definitions/state.

Request/response#

Juggler request/response communication is used for changing filter parameters.

Request data structures are defined by JSON schema hat-syslog://juggler.yaml#/definitions/request.

In case of successful request execution, response data is null.

JSON Schemas#

Configuration#

---
"$schema": "http://json-schema.org/schema#"
id: "hat-syslog://server.yaml#"
title: Syslog server
description: Syslog server configuration
type: object
required:
    - type
    - log
    - syslog_addr
    - ui_addr
    - db_path
    - db_low_size
    - db_high_size
    - db_enable_archive
    - db_disable_journal
properties:
    type:
        const: syslog
        description: configuration type identification
    version:
        type: string
        description: component version
    log:
        "$ref": "hat-json://logging.yaml#"
    syslog_addr:
        type: string
        description: |
            syslog listening address in form `<protocol>://<host>:<port>`
            where `<protocol>` can be `tcp`, `udp` or `ssl`
    syslog_pem:
        type: string
        description: |
            path to PEM file is mandatory if syslog address is ssl
    ui_addr:
        type: string
        description: |
            web server listening address in form `http://<host>:<port>`
    db_path:
        type: string
        description: |
            path to sqlite database file
    db_low_size:
        type: integer
        description: |
            number of messages kept in database after database cleanup
    db_high_size:
        type: integer
        description: |
            number of messages that will trigger database cleanup
    db_enable_archive:
        type: boolean
        description: |
            should messages, deleted during database cleanup, be kept in
            archive files
    db_disable_journal:
        type: boolean
        description: |
            disable sqlite jurnaling
...

Juggler#

---
"$schema": "http://json-schema.org/schema#"
id: "hat-syslog://juggler.yaml#"
definitions:
    state:
        type: object
        required:
            - filter
            - entries
            - first_id
            - last_id
        properties:
            filter:
                "$ref": "hat-syslog://juggler.yaml#/definitions/filter"
            entries:
                type: array
                items:
                    "$ref": "hat-syslog://juggler.yaml#/definitions/entry"
            first_id:
                type:
                    - 'null'
                    - integer
            last_id:
                type:
                    - 'null'
                    - integer
    request:
        filter:
            "$ref": "hat-syslog://juggler.yaml#/definitions/filter"
    filter:
        type: object
        required:
            - max_results
            - last_id
            - entry_timestamp_from
            - entry_timestamp_to
            - facility
            - severity
            - hostname
            - app_name
            - procid
            - msgid
            - msg
        properties:
            max_results:
                type:
                    - 'null'
                    - integer
            last_id:
                type:
                    - 'null'
                    - integer
            entry_timestamp_from:
                type:
                    - 'null'
                    - number
            entry_timestamp_to:
                type:
                    - 'null'
                    - number
            facility:
                oneOf:
                    - type: 'null'
                    - "$ref": "hat-syslog://juggler.yaml#/definitions/facility"
            severity:
                oneOf:
                    - type: 'null'
                    - "$ref": "hat-syslog://juggler.yaml#/definitions/severity"
            hostname:
                type:
                    - 'null'
                    - string
            app_name:
                type:
                    - 'null'
                    - string
            procid:
                type:
                    - 'null'
                    - string
            msgid:
                type:
                    - 'null'
                    - string
            msg:
                type:
                    - 'null'
                    - string
    entry:
        type: object
        required:
            - id
            - timestamp
            - msg
        properties:
            id:
                type: integer
            timestamp:
                type: number
            msg:
                "$ref": "hat-syslog://juggler.yaml#/definitions/msg"
    msg:
        type: object
        required:
            - facility
            - severity
            - version
            - timestamp
            - hostname
            - app_name
            - procid
            - msgid
            - data
            - msg
        properties:
            facility:
                oneOf:
                    - type: 'null'
                    - "$ref": "hat-syslog://juggler.yaml#/definitions/facility"
            severity:
                oneOf:
                    - type: 'null'
                    - "$ref": "hat-syslog://juggler.yaml#/definitions/severity"
            version:
                type: integer
            timestamp:
                type:
                    - 'null'
                    - number
            hostname:
                type:
                    - 'null'
                    - string
            app_name:
                type:
                    - 'null'
                    - string
            procid:
                type:
                    - 'null'
                    - string
            msgid:
                type:
                    - 'null'
                    - string
            data:
                type:
                    - 'null'
                    - string
            msg:
                type:
                    - 'null'
                    - string
    facility:
        enum:
            - KERNEL
            - USER
            - MAIL
            - SYSTEM
            - AUTHORIZATION1
            - INTERNAL
            - PRINTER
            - NETWORK
            - UUCP
            - CLOCK1
            - AUTHORIZATION2
            - FTP
            - NTP
            - AUDIT
            - ALERT
            - CLOCK2
            - LOCAL0
            - LOCAL1
            - LOCAL2
            - LOCAL3
            - LOCAL4
            - LOCAL5
            - LOCAL6
            - LOCAL7
    severity:
        enum:
            - EMERGENCY
            - ALERT
            - CRITICAL
            - ERROR
            - WARNING
            - NOTICE
            - INFORMATIONAL
            - DEBUG
...