Monitoring

This document covers the following topics:

Logging

Log messages are written to standard output. The level is set with loggingMode (environment variable NATSUBSRV_LOG).

Level Use
OFF (alias QUIET) No log output.
CRITICAL Only errors that terminate the server.
ERROR Failed requests and server errors.
WARN Situations that need attention but do not stop processing.
INFO (alias ON) Default. Startup, configuration, shutdown, and important events.
DEBUG Detailed processing information; request and response payloads truncated to the first and last 512 bytes.
TRACE Similar to DEBUG, but with complete request and response payloads.

Use INFO in production. DEBUG and TRACE produce large amounts of output and slow down processing — enable them only for analysis and switch them off afterwards.

Passwords are never written to the log. The password field of a request is always replaced with asterisks at every log level including TRACE, and this cannot be switched off. All other payload content is not masked, however, TRACE writes complete request and response payloads and therefore exposes the data your subprograms exchange. Use test data rather than production data when logging at TRACE.

Syslog

Set enableSyslog to true (environment variable NATSUBSRV_SYSLOG) to send log messages to the system's syslog service in addition to standard output. On Linux, you can then read them with journalctl or in /var/log/messages, depending on your system configuration.

Debug-level messages reach syslog only if loggingMode is set to DEBUG and your syslog daemon forwards that level. For rsyslog, adjust /etc/rsyslog.conf: 

*.debug;mail.none;authpriv.none;cron.none                /var/log/messages

Then, restart the service:

sudo systemctl restart rsyslog

Metrics

Set metrics to true (environment variable NATSUBSRV_METRICS) to enable metrics collection. Each worker then reports its values to the parent process every metricsInterval seconds (default: 15). Shorter intervals give a finer resolution at the cost of slightly more overhead.

Scope of this release: metrics collection is implemented in the server and can be switched on. This release does not include a mechanism for exporting the collected values to an external monitoring system, so the values cannot be read out of a running server. Please take this into account when planning your monitoring. See Retrieving Health and Telemetry Data.

The following metrics are provided:

Metric Type Description
total.connections Counter Client connections established since server start.
subprogram.calls Counter Natural subprogram invocations.
calls.per.subprogram Counter Calls per subprogram, tagged by library and subprogram name.
invalid.subprogram.calls Counter Failed or invalid subprogram calls.
invalid.calls.per.subprogram Counter Invalid calls per subprogram, tagged by library and subprogram name.
cpu.utime / cpu.stime Gauge Cumulative CPU user/system time in seconds across all workers.
cpu.utime.normalized / cpu.stime.normalized Gauge CPU user/system time as a percentage (0–100) over the interval.
memory.rss.kb Gauge Physical memory used by all workers, in kilobytes.
memory.vms.kb Gauge Virtual memory used by all workers, in kilobytes.
connection.dur.min.ms / max / avg Gauge Minimum, maximum, and average connection duration in milliseconds.
connection.dur.sample.count Counter Sample size for the connection duration average.
request.size.min.bytes / max / avg Gauge Minimum, maximum, and average request payload size in bytes.
request.size.sample.count Counter Sample size for the request size average.
listen.socket.pending.accepts Gauge Connections waiting to be accepted.
requests.per.interval Gauge Requests processed in the current interval.
amount.of.workers Gauge Currently active worker processes.
network.throughput.sent.bytes Counter Bytes sent over the network.
network.throughput.received.bytes Counter Bytes received over the network.

Counters increase continuously and are reset when the server restarts; gauges represent a current value.

Useful starting points: invalid.subprogram.calls and invalid.calls.per.subprogram to detect failing subprograms, listen.socket.pending.accepts together with connection.dur.avg.ms to decide whether numberOfWorkers needs to be increased, and memory.rss.kb to size the machine.

Traces

A trace records the individual processing steps of a single request and thereby shows where its time is spent.

Tracing is activated per request. The request must carry a valid W3C traceparent value, which the middleware adds. Additionally, metrics must be enabled — otherwise traces are created but not forwarded to the parent process.

tracesStoredPerWorker (environment variable NATSUBSRV_TRACES_PER_WORKER, default 1000) determines how many traces a worker buffers before the oldest entries are overwritten. Increase the value if traces are collected less frequently than they are produced.

The log level does not activate tracing, it only controls whether the measured times additionally appear as log messages. The log output is the only way to inspect the measured times. See Retrieving Health and Telemetry Data.

Retrieving Health and Telemetry Data

Natsubsrv does not offer an HTTP endpoint of its own. Its health status is made available by the surrounding components: the middleware (for example, the Natural OpenAPI Server) checks the availability of Natsubsrv and reports it through its own interfaces.

Metrics and traces: this release does not include a mechanism for exporting the collected metrics and traces to an external monitoring system. Collection in the server itself is in place — the metrics listed above are gathered and traces are recorded while the server is running — but the collected data cannot be read out from outside the server.

Base your operational monitoring on:

  • the health check of the middleware for availability,

  • log output (see Logging, optionally through Syslog) for errors, startup and shutdown events, and request-level detail,

  • the operating system: monitor the server process and its listening port, and take CPU and memory figures from the process itself.

The collected values cannot be read out and thus, enabling metrics yields no additional insight for monitoring in this release. The overhead of enabling it is small, so either setting is unproblematic.

On this page