Troubleshooting

This document covers the following topics:

Reporting a Problem

The more completely the following information is provided, the faster a problem can be analyzed:

  1. A minimal example. A subprogram that reproduces the problem with as few dependencies as possible — no database access, no calls to further subprograms, hard-coded test data instead of production data. Include the parameter definition (number, type, and length of the parameters) and an example request.

  2. Logs at TRACE level. Set loggingMode to TRACE, restart the server, reproduce the problem, and provide the complete log from the start onwards — it contains the version and the effective configuration. Note that TRACE writes complete request and response payloads — passwords are masked, other content is not — so use test data only, and switch the level back afterwards.

  3. The crash report, if it exists: crash.log from <appDataDir>/crash/. See Crash Reports.

  4. The configuration: the natsubsrv.json in use and any NATSUBSRV_* environment variables that are set. Remove passwords beforehand.

  5. Version and environment: Natsubsrv version and build number — from the startup log or from a ping requests — the Natural version, the Linux distribution and version, and the middleware in use with its version.

  6. Classification: Does the problem occur on every call or sporadically, and since when? What changed most recently — a Natural update, a configuration change, or an increase in load?

Common Problems

Symptom Cause Solution
NAT6010 — memory allocation failed Natural's user area is too small for the subprogram Increase usize
NAT3009 after idle periods The database context has expired Configure natRpc39IntervalInSeconds
NAT3048 in all workers but one The workers open Adabas with the same ETID Set ETID=$$ in the Natural startup parameters
NAT3148 at startup, server terminates The database holding FSEC/FUSER is not reachable yet Ensure the startup order, or configure nscStartupLogonRetryCount
Request rejected as too large The request exceeds the configured limit Increase maxMessageSize
Repeated warnings while closing connections The wait time is too short for the network latency Increase closeDrainTimeout
Requests time out under load, workers not fully utilized Too few workers, connections queue up Increase numberOfWorkers, check listen.socket.pending.accepts
Service fails immediately after starting The Natural environment could not be set up, or a required file is missing Inspect journalctl -u sag<id>npisrv — the start script names a missing natenv, npienv, or npisrv. See Running as a Service
Server runs, but ignores the configuration The configuration file was not found, so defaults and environment variables were used Pass config= with an absolute path; the log reports the file it looked for when none was found, and DEBUG shows the origin of every value
Server does not start, configuration error in the log An invalid value, or a directory that cannot be created or written The message names the affected parameter or directory
Server does not start, license error The license file is missing or expired Install a valid Natural API Server license
Middleware cannot connect Wrong port, server still starting up, or a firewall in between Send a ping requests from the middleware host to tell the two sides apart; compare the configured ports

 

If the startup logon is being retried, the log shows the reason and each attempt:

[warning] NSC LOGON failed with NAT3148 (attempt 1 of 3), retrying in 5 second(s).
[warning] NSC LOGON failed with NAT3148 (attempt 2 of 3), retrying in 5 second(s).
[error]   NSC LOGON still failing with NAT3148 after 3 attempt(s), giving up. The database holding FSEC/FUSER could not be reached.

Crash Reports

If the server or one of its workers terminates on a fatal signal (for example SIGSEGV or SIGABRT), Natsubsrv captures the crash data and stores it under <appDataDir>/crash/. The data is first written in a compact binary form and is converted into the readable report crash.log afterwards — for a worker by the parent process, for the server itself at its next start.

Provide crash.log when reporting a problem. The file is appended to, so it can contain several crashes; keep the whole file rather than an excerpt.

On this page