System Logging Facilities

The information is organized under the following headings:


Windows Logging Facility

The logging facility on Windows operating systems is the Event Viewer.

Start of instruction setTo trace log messages of the installed products with the Event Viewer

  1. Enable the logging registry key of the Client/Server Layer by setting Enable to "1"

    The default registry key setting of the Client/Server Layer after installation of System Management Hub is [HKEY_LOCAL_MACHINE\SOFTWARE\Software AG\System Management Hub\CSLayerServer\Syslog] "Enable"="0".

  2. Set the logmask values to use the Windows logging service by setting Logmask.

    Logmask specifies which messages go to the system log file.

    For example: Logmask = WEF stands for a warning with error and fatal messages logged.

    Logmask can be a combination of the following:

    I: info message
    						W: warning message
    						E:  error message
    						F: fatal message
    						J: job message

    The default registry key setting of the Event Layer after installation of System Management Hub is [HKEY_LOCAL_MACHINE\SOFTWARE\Software AG\System Management Hub\EventLayer\Syslog] "Logmask"="FE".

Following is the location of the log files:

  • Location of Client/Server Layer log files
    "Agent_Stderr_Directory"="Software AG_directory\InstanceManager\log"
  • Location of Event Layer log files
    "Logging_Path"="Software AG_directory\InstanceManager\log\events"

UNIX Logging Facility

The logging facility on UNIX operating systems is called syslog.

The information is organized under the following headings:

Overview of Syslog

UNIX has a centralized system logging process that runs the program /etc/syslogd or /etc/syslog. Individual programs that need to have information logged send the information to syslog. The big advantage of syslog is that the message file holds all reported messages. By using scripts, a system administrator can detect complex error scenarios and act accordingly.

The syslog daemon is native to all distributions of UNIX and Linux, but is not available under Windows. It can be configured to write messages depending on their facility and severity to different log files. Additionally, it can discard them totally or forward them to a chained syslog daemon.

Following are several limitations to the system log facility:

  • The message size is limited to 1K

  • No assumption is made upon the formatting or contents of the messages. The protocol is simply designed to transport these event messages. Since the format of the message is not standardized, there is a large variety of message contents.

  • Traditional syslog works well over a non-congested network.

  • Syslog protocol is UDP-based, so it is unreliable and does not guarantee you the delivery of the messages. Expect some packet loss when the network is congested, the sender or receiver is busy, the WAN network is slow, or there is heavy syslog traffic.

  • A machine that is not configured correctly sends syslog messages to a syslog daemon representing itself as another machine. This means the status of the supposed sender of the messages is not accurately reflected in the received messages.

Syslog Fields

The full format of a syslog message has three distinct parts - priority, header, and message.

The Priority Field

This is a number that represents the facility and the severity of the message.

With the facility and the severity values, you can apply certain filters on the events in the syslog daemon.

Note:
The syslog daemon does not generate the severity and the facility values. They are generated by the applications on which the event is generated. If the message received by the syslog daemon has a generated code by the application that is different from the recommended code for the specified situations, then the syslog daemon receives the message without changing the code.

Facility of Syslog

The facility is the application or operating system component that generates a log message. It is defined by the syslog protocol and provides a basic clue from what part of a system the message originated.

Traditionally, under UNIX, there are facilities like KERN (the OS kernel itself), LPD (the line printer daemon) and so on. There are also the LOCAL_0 to LOCAL_7 facilities (see code number 16 from the facility codes below) that are traditionally reserved for administrator and application use.

A device on which syslog is enabled allows you to configure any value as the facility to distinguish between different classes of syslog messages. That is why the facility filed is very helpful to define rules that split messages (for example, to different log files based on the facility level).

Refer to the following table for the facility codes and their description:

Code Description
0 kernel messages
1 user-level messages
2 mail system
3 system daemons
4 security/ authorization messages
5 messages generated internally by syslogd
6 line printer subsystem
7 network news subsystem
8 UUCP subsystem
9 clock daemon
10 security/ authorization messages
11 FTP daemon
12 NTP subsystem
13 log audit
14 log alert
15 clock daemon
16 local use 0 reserved for site-specific use

Severity of Syslog

Codes

The severity is the emergency of the generated messages.

Refer to the following table for the severity codes and their description.

Code Description
0 Emergency: system is unusable.
1 Alert: action must be taken immediately.
2 Critical: critical conditions.
3 Error: error conditions.
4 Warning: warning conditions.
5 Notice: normal but significant condition.
6 Informational: informational messages.
7 Debug: debug-level messages.

The Header Field

This field contains the following data:

  • A timestamp value for the date and time at which the message was generated

  • A value for the hostname (the IP address of the device)

The Message Field

The message field is the third part of the syslog packet. It contains some additional information of the process that generated the message and the text of the message. The message field consists of two parts:

  • tag

    The tag value is the name of the program or process that generated the message.

  • content

    The content value contains the details of the message.

Interface Configuration

The registry string value HKEY_LOCAL_MACHINE\SOFTWARE\Software AG\System Management Hub\EventLayer\Syslog Logmask decides which Joblog messages are written to syslog.

Character Argus Severity Syslog severity default
F ARGUS_SV_SEVERE_ERROR LOG_CRIT X
E ARGUS_SV_ERROR LOG_ERR X
W ARGUS_SV_WARNING LOG_WARNING  
I ARGUS_SV_INFO LOG_INFO  
J Open Job Messages form all severities Appropriate LOG_ severity  

If F, E, W, I or J are part of the Logmask value, the Joblog is also written to syslog. All messages are written under LOG_USER.

The syslogd daemon writes the syslog. When syslogd starts up, it reads its configuration file (usually /etc/syslog.conf) to determine what kinds of events to log and to what location.

For detailed information about your current configuration file call man syslog.conf. It describes the parameter of your syslog.conf file.

There is no standard for the message output file and no utility known that reads automatically the right file.

Follow these steps:

  1. Open the /etc/syslog.conf and find out there messages are written.

  2. Open the plain text syslog output file in any editor or run cat, more, less, etc., as user root or with root permissions.

  3. Browse to Software AG.

Samples of Configuration Files

Linux

Suse 9.1 /etc/syslog.conf example:

# /etc/syslog.conf - Configuration file for syslogd(8)
#
# For info about the format of this file, see "man syslog.conf".
#

#
# print most on tty10 and on the xconsole pipe
#
kern.warning;*.err;authpriv.none /dev/tty10
kern.warning;*.err;authpriv.none|/dev/xconsole
*.emerg *

# enable this, if you want that root is informed
# immediately, for example,  of logins
#*.alert root


#
# all email-messages in one file
#
mail.*-/var/log/mail
mail.info-/var/log/mail.info
mail.warning-/var/log/mail.warn
mail.err /var/log/mail.err

#
# all news-messages
#
# these files are rotated and examined by "news.daily"
news.crit-/var/log/news/news.crit
news.err-/var/log/news/news.err
news.notice-/var/log/news/news.notice
# enable this, if you want to keep all news messages
# in one file
#news.*-/var/log/news.all

#
# Warnings in one file
#
*.=warning;*.=err               -/var/log/warn
*.crit                           /var/log/warn

#
# save the rest in one file
#
*.*;mail.none;news.none         -/var/log/messages

#
# enable this, if you want to keep all messages
# in one file
#*.*                            -/var/log/allmessages

#
# Some foreign boot scripts require local7
#
local0,local1.*                 -/var/log/localmessages
local2,local3.*                 -/var/log/localmessages
local4,local5.*                 -/var/log/localmessages
local6,local7.*                 -/var/log/localmessages

Critical errors, errors and warnings here are written to /var/log/warn. The rest are written to /var/log/messages.

Example:

cat /var/log/warn
…
Sep 28 10:24:43 pclinarg1 Software AG - argevsrv[9997]: Test - Warning
Sep 28 10:24:43 pclinarg1 Software AG - argevsrv[9997]: Test - Error
Sep 28 10:24:43 pclinarg1 Software AG - argevsrv[9997]: Test - Crit Error

cat /var/log/messages
…
Sep 28 10:24:43 pclinarg1 Software AG - argevsrv[9997]: Test - Info
Sep 28 10:24:43 pclinarg1 Software AG - argevsrv[9997]: Test - Warning
Sep 28 10:24:43 pclinarg1 Software AG - argevsrv[9997]: Test - Error
Sep 28 10:24:43 pclinarg1 Software AG - argevsrv[9997]: Test - Crit Error
Sep 28 10:54:27 pclinarg1 Software AG - argevsrv[10449]: start argevsrv demon

Linux (Suse 9.1) Syslog.conf Man Pages

  • NAME
    syslog.conf - syslogd(8) configuration file

  • Description
    The syslog.conf file is the main configuration file for the syslogd(8) that logs system messages on UNIX and Linux systems. This file specifies rules for logging. For special features see the sysklogd(8) manpage.

    Every rule consists of two fields, a selector field and an action field. These two fields are separated by one or more spaces or tabs. The selector field specifies a pattern of facilities and priorities belonging to the specified action.

    Lines starting with a hash mark (#'') and empty lines are ignored.

    This release of syslogd is able to understand an extended syntax. One rule can be divided into several lines if the leading line is terminated with an backslash (\'').

  • Selectors
    The selector field itself again consists of two parts, a facility and a priority, separated by a period (.''). Both parts are case insensitive and can also be specified as decimal numbers, but don't do that, you have been warned. Both facilities and priorities are described in syslog(3). The names mentioned below correspond to the similar LOG_-values in /usr/include/syslog.h.

    The facility is one of the following keywords: 'auth', 'authpriv', 'cron', 'daemon', 'kern', 'lpr', 'mail', 'mark', 'news', 'security' (same as 'auth'), 'syslog', 'user', 'ucp' and 'local0' through 'local7'. The keyword security is not used anymore and mark is only for internal use. Therefore, it is not used in applications. Anyway, if you want to specify and redirect these messages here, you can so. The facility specifies the subsystem that produced the message, that is, all mail programs log with the mail facility (LOG_MAIL) if they log using syslog.

    The priority is one of the following keywords, in ascending order: 'debug', 'info', 'notice', 'warning', 'warn' (same as 'warning'), err, error (same as 'err'), 'crit', 'alert', 'emerg', 'panic' (same as 'emerg'). The keywords 'error', 'warn' and 'panic' are deprecated and must not be used anymore. The priority defines the severity of the message

    The behavior of the original BSD syslogd is that all messages of the specified priority and higher are logged according to the given action. This syslogd(8) behaves the same, but has some extensions.

    In addition to the above mentioned names the syslogd(8) understands the following extensions: An asterisk (*'') stands for all facilities or all priorities, depending on where it is used (before or after the period). The keyword none stands for no priority of the given facility.

    You can specify multiple facilities with the same priority pattern in one statement using the comma (,'') operator. You can specify as much facilities as you want. Remember that only the facility part from such a statement is taken. A priority part is skipped.

    Multiple selectors can be specified for a single action using the semicolon (;'') separator. Remember that each selector in the selector field is capable to overwrite the preceding ones. Using this behavior you can exclude some priorities from the pattern.

    This syslogd(8) has a syntax extension to the original BSD source, that makes its use more intuitively. You can precede every priority with an equation sign (='') to specify only this single priority and not any of the above. You can also precede the priority with an exclamation mark (!'') to ignore all that priorities, either exact this one or this and any higher priority. If you use both extensions than the exclamation mark must occur before the equation sign, just use it intuitively.

  • Actions
    The action field of a rule describes the abstract term logfile''. A logfile'' need not to be a real file. The syslogd(8) provides the following actions.

    • Regular File
      Typically messages are logged to real files. The file has to be specified with full pathname, beginning with a slash /''.

      You can prefix each entry with the minus -'' sign to omit syncing the file after every logging. Note that you may lose information if the system crashes right behind a write attempt. Nevertheless, this may give you back some performance, especially if you run programs that use logging in a very verbose manner.

    • Named Pipes
      This version of syslogd(8) has support for logging output to named pipes (fifos). A fifo or named pipe can be used as a destination for log messages by prepending a pipe symbol (|'') to the name of the file. This is handy for debugging. Note that the fifo must be created with the mkfifo(1) command before syslogd(8) is started.

    • Terminal and Console
      If the file you specified is a tty, special tty-handling is done, same with /dev/console.

    • Remote Machine
      This syslogd(8) provides full remote logging, that is, it is able to send messages to a remote host running syslogd(8) and to receive messages from remote hosts. The remote host does not forward the message again, but just logs them locally. To forward messages to another host, use the hostname with the at sign (@'').

      With this feature you can control all syslog messages on one host, if all other machines log remotely to that host.

    • List of Users
      Usually, critical messages are also directed to root'' on that machine. You can specify a list of users that shall get the message by simply writing the login. You can specify more than one user by separating them with commas (,''). If they are logged in they get the message.

    • Everyone logged on

      Emergency messages often go to all users currently online to notify them that something strange is happening with the system. To specify this wall(1)-feature use an asterisk (*'').

  • Examples
    Here are some example, partially taken from a real existing site and configuration.

    # Store critical stuff in critical
    #
    *.=crit;kern.none            /var/adm/critical
    
    That stores all messages with the priority crit in the file /var/adm/critical, except for any kernel message.
    
    
    # Kernel messages are first, stored in the kernel
    # file, critical messages and higher ones also go
    # to another host and to the console
    #
    kern.*                       /var/adm/kernel
    kern.crit                    @finlandia
    kern.crit                    /dev/console
    kern.info;kern.!err          /var/adm/kernel-info
    

    The first rule direct any message that has the kernel facility to the file /var/adm/kernel.

    The second statement directs all kernel messages of the priority crit and higher to the remote host finlandia. This is useful, because if the host crashes and the disks get irreparable errors, you may not be able to read the stored messages. If they are on a remote host, too, you can still try to find out the reason for the crash.

    The third rule directs these messages to the actual console, so the person who works on the machine can get them, too.

    The fourth line tells the syslogd to save all kernel messages that come with priorities from info up to warning in the file /var/adm/kernel-info. Everything from err and higher is excluded.

    # The tcp wrapper loggs with mail.info, we display
    # all the connections on tty12
    #
    mail.=info                   /dev/tty12
    

    This directs all messages that uses mail.info (in source LOG_MAIL | LOG_INFO) to /dev/tty12, the 12th console. For example the TCP wrapper tcpd(8) uses this as it is default.

    # Store all mail concerning stuff in a file
    #
    mail.*;mail.!=info           /var/adm/mail
    

    This pattern matches all messages that come with the mail facility, except for the info priority. These are stored in the file /var/adm/mail.

    # Log all mail.info and news.info messages to info
    #
    mail,news.=info              /var/adm/info
    

    This command extracts all messages that come (either with mail.info or with news.info) and stores them in the file /var/adm/info.

    # Log info and notice messages to messages file
    #
    *.=info;*.=notice;\
         mail.none  /var/log/messages
    

    This lets the syslogd log all messages that come with either the info or the notice facility into the file /var/log/messages, except for all messages that use the mail facility.

    # Log info messages to messages file
    #
    *.=info;\
         mail,news.none       /var/log/messages
    
    

    This statement causes the syslogd to log all messages that come with the info priority to the file /var/log/messages.In this case, messages coming with the mail or the news facility are not stored.

    # Emergency messages are displayed using wall
    #
    *.=emerg                     *
    

    This rule tells the syslogd to write all emergency messages to all currently logged on users. This is the wall action.

    # Messages of the priority alert is directed
    # to the operator
    #
    *.alert                      root,joey
    

    This rule directs all messages with a priority of alert or higher to the terminals of the operator, that is, of the users root'' and joey'' if they're logged on.

    *.*																		@finlandia

    This rule redirects all messages to a remote host called finlandia. This is useful especially in a cluster of machines where all syslog messages are stored on one machine only.

  • Configuration File Syntax Differences
    Syslogd uses a slightly different syntax for its configuration file than the original BSD sources. Originally all messages of a specific priority and above were forwarded to the log file. The modifiers ='', !'' and -'' were added to make the syslogd more flexible and to use it in a more intuitive manner.

    The original BSD syslogd does not understand spaces as separators between the selector and the action field.

    FILES
    /etc/syslog.conf
           Configuration file for syslogd
    
  • BUGS
    The effects of multiple selectors are sometimes not intuitive. For example mail.crit, *.err'' selects mail'' facility messages at the level of err'' or higher, not at the level of crit'' or higher.

Solaris Example

Sun Solaris 8 64 /etc/syslog.conf example:

#ident  "@(#)syslog.conf        1.5     98/12/14 SMI"   /* SunOS 5.0 */
#
# Copyright (c) 1991-1998 by Sun Microsystems, Inc.
# All rights reserved.
#
# syslog configuration file.
#
# This file is processed by m4 so be careful to quote (`') names
# that match m4 reserved words.  Also, within ifdef's, arguments
# containing commas must be quoted.
#
*.err;kern.notice;auth.notice                   /dev/sysmsg
*.err;kern.debug;daemon.notice;mail.crit        /var/adm/messages

*.alert;kern.err;daemon.err                     operator
*.alert                                         root

*.emerg                                         *

# if a non-loghost machine chooses to have authentication messages
# sent to the loghost machine, un-comment out the following line:
#auth.notice                    ifdef(`LOGHOST', /var/log/authlog, @loghost)

mail.debug                      ifdef(`LOGHOST', /var/log/syslog, @loghost)

#
# non-loghost machines uses the following lines to cause "user"
# log messages to be logged locally.
#
ifdef(`LOGHOST', ,
user.err                                        /dev/sysmsg
user.err                                        /var/adm/messages
user.alert                                      `root, operator'
user.emerg                                      *
)