Rsyslog

From WhyAskWhy.org Wiki
Jump to: navigation, search


Overview

From the rsyslog homepage:

Rsyslog is a rocket-fast system for log processing. It offers high-performance, great security features and a modular design. While it started as a regular syslogd, rsyslog has evolved into a kind of swiss army knife of logging, being able to

accept inputs from a wide variety of sources, transform them, and output the results to diverse destinations.

Rsyslog has a strong enterprise focus but also scales down to small systems. It supports, among others, MySQL, PostgreSQL, failover log destinations, syslog/tcp transport, fine grain output format control, high precision timestamps, queued operations and the ability to filter on any message part.


Filter Conditions

Rsyslog supports three different types of filter conditions [1]:

  • RainerScript-based filters
  • "traditional" severity and facility based selectors
  • property-based filters


RainerScript-based filters

if $programname == 'prog1' then {
   action(type="omfile" file="/var/log/prog1.log")
   if $msg contains 'test' then
     action(type="omfile" file="/var/log/prog1test.log")
   else
     action(type="omfile" file="/var/log/prog1notest.log")
}


Selectors

#
# Split it up so that it is easy to write scripts to parse these files.
#
#mail.info                      -/var/log/mail.info
#mail.warn                      -/var/log/mail.warn
mail.err                        /var/log/mail.err

# Catch-all mail log
mail.*                          -/var/log/mail.log

# Prevent mail log entries from being duplicated to /var/log/syslog
& stop


Property-based filters

[2] 

:syslogtag, isequal, "mysqld:" -/var/log/mysqld.log
& ~
:syslogtag, isequal, "mysqld_safe:" -/var/log/mysqld.log
& ~
:syslogtag, startswith, "/etc/mysql/debian-start" -/var/log/mysqld.log
& ~


Real-world example

This conf block combines RainerScript and Property-based filters on a Prosody server to:

  • Exclude auth_dovecot messages which include base64 encoded usernames/passwords
  • Send all debug messages to a separate log
  • Send all messages of severity level 6 or below OR those mentioning full JIDs to the "normal" log file


# Catch log messages generated by the Prosody daemon
if $programname == 'prosody' then {

    #
    # Skip logging of SASL AUTH log entries (base64 encoded username/passwords)
    #
    # Ubuntu 12.04 (ryslog 5.8.x) compatible syntax:
    #:msg, contains, ":auth_dovecot: sending" ~
    #
    # Ubuntu 14.04 (rsyslog 7.4.x) compatible syntax:
    :msg, contains, ":auth_dovecot: sending" stop

    # Debug mode is enabled in the Prosody config, so since we're not filtering
    # further at this point all of those entries will go here
    action(type="omfile" file="/var/log/prosody-debug.log")

    # Catch all severity levels 6 and lower along with Resource bound messages
    # and place them in the general prosody log file
    if $syslogseverity <= 6 or $msg contains 'Resource bound:' then {

        action(type="omfile" file="/var/log/prosody.log")
    }

    stop
}


and here is the same filter written entirely in Rainer-script: 


if $programname == 'prosody' then {

    #
    # Skip logging of SASL AUTH log entries (base64 encoded username/passwords)
    #
    # Ubuntu 12.04 (ryslog 5.8.x) compatible syntax:
    #:msg, contains, ":auth_dovecot: sending" ~
    #
    # Ubuntu 14.04 (rsyslog 7.4.x) compatible syntax:
    if $msg contains ":auth_dovecot: sending" then {

        stop
    }
    else {

        # Debug mode is enabled in the Prosody config, so since we're not filtering
        # further at this point all of those entries will go here
        action(type="omfile" file="/var/log/prosody-debug.log")

        # Catch all severity levels 6 and lower along with Resource bound messages
        # and place them in the general prosody log file
        if $syslogseverity <= 6 or $msg contains 'Resource bound:' then {

            action(type="omfile" file="/var/log/prosody.log")
        }

        stop

    }
}


Discarding messages after they are logged (aka, prevent logging of repeat/duplicate messages)

The discard operator

The discard operator (aka, "discard action" operator) is the ~ character. Using it tells rsyslog that you want to discard log messages matched by filters. That operator is used in conjunction with the & character which allows for having multiple actions per selector. This allows for combinations where you direct matching log messages to a specific file and prevent the same message from being logged a second time. The & character is necessary (I found out the hard way) to "glue" the discard action to the filter above, otherwise it stops all log messages from being logged from that point forward. This is particularly important with selector or property-based filters.

The RHEL documentation has this to say [3]:

The discard action is mostly used to filter out messages before carrying on any further processing. It can be effective if you want to omit some repeating messages that would otherwise fill the log files. The results of discard action depend on where in the configuration file it is specified, for the best results place these actions on top of the actions list. Please note that once a message has been discarded there is no way to retrieve it in later configuration file lines.

The stop directive

As of v7-stable the discard operator has been marked as deprecated and a warning similar to this one is logged: 

 rsyslogd-2307: warning: ~ action is deprecated, consider using the 'stop' statement instead [try http://www.rsyslog.com/e/2307 ]

From v7 onward, the stop RainerScript directive is recommended instead. From my testing I have found that it works well with RainerScript, Selector or Property-based filters. If using it with RainerScript it doesn't appear that you need to use & to tie the directive to the previous filter like you do with selector or property-based filters. As with the discard operator (~), with those two filter approaches you run the risk of stopping all logging after the use of the directive if you don't tie it back to a preceding filter.

Example: Using the discard operator

From an Ubuntu 12.04 LTS box that runs v5.x. Here we're using a property-based filter. 

# /etc/rsyslog.d/20-ufw.conf

# Log kernel generated UFW log messages to file
:msg,contains,"[UFW " /var/log/ufw.log

# Uncomment the following to stop logging anything that matches the last rule.
# Doing this will stop logging kernel generated UFW log messages to the file
# normally containing kern.* messages (eg, /var/log/kern.log)
& ~


Example: Using the stop operator

From an Ubuntu 14.04 LTS box that runs v7.4.x. Here we're also using a property-based filter. 

# /etc/rsyslog.d/20-ufw.conf

:msg,contains,"[UFW " /var/log/ufw.log

# Uncomment the following to stop logging anything that matches the last rule.
# Doing this will stop logging kernel generated UFW log messages to the file
# normally containing kern.* messages (eg, /var/log/kern.log)
& stop


References

  1. ↑ Rsyslog Filter Conditions>
  2. ↑ Logging MySQL events to file in addition to syslog on Debian>
  3. ↑ Red Hat Enterprise Linux 7 - System Administrator's Guide - 18.2. Basic Configuration of Rsyslog

General

Deprecated Features

Actions

Filters

Regular Expressions

Properties

Receiving from and Sending to remote Syslog boxes

Retrieved from "http://wiki.whyaskwhy.org/index.php?title=Rsyslog&oldid=697"