Difference between revisions of "Rsyslog"

From WhyAskWhy.org Wiki
Jump to: navigation, search
m (Spacing tweak)
m (More references, additional tweaks to explanation of discard/stop directives)
Line 76: Line 76:
  
  
== Discarding messages after they are logged (aka, stopping logging of repeat/duplicate messages) ==
+
== Discarding messages after they are logged (aka, prevent logging of repeat/duplicate messages) ==
  
 
=== The discard operator ===
 
=== The discard operator ===
  
The ''discard operator'' (aka, "discard action" operator) is the <code>~</code> character. Using it tells rsyslog that you want to discard log messages matched by filters. That operator is used in conjunction with the <code>&</code> character which allows for having multiple actions per selector. This allows for combinations where you direct matching log messages to a specific file and prevent the same message from being logged a second time. The <code>&</code> character is necessary (I found out the hard way) to "glue" the discard action to the filter above, otherwise it stops ''all'' log messages from being logged from that point forward. This is particularly important with  
+
The ''discard operator'' (aka, "discard action" operator) is the <code>~</code> character. Using it tells rsyslog that you want to discard log messages matched by filters. That operator is used in conjunction with the <code>&</code> character which allows for having multiple actions per selector. This allows for combinations where you direct matching log messages to a specific file and prevent the same message from being logged a second time. The <code>&</code> character is necessary (I found out the hard way) to "glue" the discard action to the filter above, otherwise it stops ''all'' log messages from being logged from that point forward. This is particularly important with '''selector''' or '''property-based''' filters.
 +
 
 +
The RHEL documentation has this to say <ref name="rhel7-rsyslog-basic-config" />:
 +
 
 +
<blockquote>The discard action is mostly used to filter out messages before carrying on any further processing. It can be effective if you want to omit some repeating messages that would otherwise fill the log files. The results of discard action depend on where in the configuration file it is specified, for the best results place these actions on top of the actions list. Please note that once a message has been discarded there is no way to retrieve it in later configuration file lines.</blockquote>
  
 
=== The <code>stop</code> directive ===
 
=== The <code>stop</code> directive ===
Line 129: Line 133:
 
<ref name="rsyslog-filter-conditions">[http://www.rsyslog.com/doc/rsyslog_conf_filter.html Rsyslog Filter Conditions]></ref>
 
<ref name="rsyslog-filter-conditions">[http://www.rsyslog.com/doc/rsyslog_conf_filter.html Rsyslog Filter Conditions]></ref>
 
<ref name="rsyslog-mysql-log-messages">[http://serverfault.com/questions/323112/logging-mysql-events-to-file-in-addition-to-syslog-on-debian Logging MySQL events to file in addition to syslog on Debian]></ref>
 
<ref name="rsyslog-mysql-log-messages">[http://serverfault.com/questions/323112/logging-mysql-events-to-file-in-addition-to-syslog-on-debian Logging MySQL events to file in addition to syslog on Debian]></ref>
 +
<ref name="rhel7-rsyslog-basic-config">[https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/System_Administrators_Guide/s1-basic_configuration_of_rsyslog.html Red Hat Enterprise Linux 7 - System Administrator's Guide - 18.2. Basic Configuration of Rsyslog]</ref>
  
 
</references>
 
</references>
Line 144: Line 149:
 
* [[wikipedia:Syslog#Header|Syslog Wikipedia entry]]
 
* [[wikipedia:Syslog#Header|Syslog Wikipedia entry]]
  
=== Rsyslog actions ===
+
=== Deprecated Features ===
 +
 
 +
* [http://www.rsyslog.com/doc/v8-stable/compatibility/v7compatibility.html#omruleset-and-discard-action-are-deprecated omruleset and discard (~) action are deprecated]
 +
* [http://kb.monitorware.com/problems-with-and-stop-t11760.html]
 +
* [http://kb.monitorware.com/kbeventdb-detail-id-7171.html rsyslogd-2307 - Deprecated functionality is used]
 +
 
 +
=== Actions ===
  
 
* [http://www.rsyslog.com/doc/master/configuration/actions.html (master branch)]
 
* [http://www.rsyslog.com/doc/master/configuration/actions.html (master branch)]
Line 153: Line 164:
  
 
* [http://www.rsyslog.com/discarding-unwanted-messages/ Discarding unwanted messages]
 
* [http://www.rsyslog.com/discarding-unwanted-messages/ Discarding unwanted messages]
 +
* [http://www.rsyslog.com/writing-specific-messages-to-a-file-and-discarding-them/ Writing specific messages to a file and discarding them]
 
* [http://www.rsyslog.com/doc/rsyslog_conf_filter.html Rsyslog Filter Conditions]
 
* [http://www.rsyslog.com/doc/rsyslog_conf_filter.html Rsyslog Filter Conditions]
 
* [http://blog.endpoint.com/2014/09/rsyslog-new-filtering-features.html Rsyslog property based filtering features ]
 
* [http://blog.endpoint.com/2014/09/rsyslog-new-filtering-features.html Rsyslog property based filtering features ]

Revision as of 14:08, 16 February 2015


Overview

From the rsyslog homepage:

Rsyslog is a rocket-fast system for log processing. It offers high-performance, great security features and a modular design. While it started as a regular syslogd, rsyslog has evolved into a kind of swiss army knife of logging, being able to

accept inputs from a wide variety of sources, transform them, and output the results to diverse destinations.

Rsyslog has a strong enterprise focus but also scales down to small systems. It supports, among others, MySQL, PostgreSQL, failover log destinations, syslog/tcp transport, fine grain output format control, high precision timestamps, queued operations and the ability to filter on any message part.


Filter Conditions

Rsyslog supports three different types of filter conditions [1]:

  • RainerScript-based filters
  • "traditional" severity and facility based selectors
  • property-based filters


RainerScript-based filters

if $programname == 'prog1' then {
   action(type="omfile" file="/var/log/prog1.log")
   if $msg contains 'test' then
     action(type="omfile" file="/var/log/prog1test.log")
   else
     action(type="omfile" file="/var/log/prog1notest.log")
}


Selectors

#
# Split it up so that it is easy to write scripts to parse these files.
#
#mail.info                      -/var/log/mail.info
#mail.warn                      -/var/log/mail.warn
mail.err                        /var/log/mail.err

# Catch-all mail log
mail.*                          -/var/log/mail.log

# Prevent mail log entries from being duplicated to /var/log/syslog
& stop


Property-based filters

[2]

:syslogtag, isequal, "mysqld:" -/var/log/mysqld.log
& ~
:syslogtag, isequal, "mysqld_safe:" -/var/log/mysqld.log
& ~
:syslogtag, startswith, "/etc/mysql/debian-start" -/var/log/mysqld.log
& ~


Discarding messages after they are logged (aka, prevent logging of repeat/duplicate messages)

The discard operator

The discard operator (aka, "discard action" operator) is the ~ character. Using it tells rsyslog that you want to discard log messages matched by filters. That operator is used in conjunction with the & character which allows for having multiple actions per selector. This allows for combinations where you direct matching log messages to a specific file and prevent the same message from being logged a second time. The & character is necessary (I found out the hard way) to "glue" the discard action to the filter above, otherwise it stops all log messages from being logged from that point forward. This is particularly important with selector or property-based filters.

The RHEL documentation has this to say [3]:

The discard action is mostly used to filter out messages before carrying on any further processing. It can be effective if you want to omit some repeating messages that would otherwise fill the log files. The results of discard action depend on where in the configuration file it is specified, for the best results place these actions on top of the actions list. Please note that once a message has been discarded there is no way to retrieve it in later configuration file lines.

The stop directive

As of v7-stable the discard operator has been marked as deprecated and a warning similar to this one is logged:

 rsyslogd-2307: warning: ~ action is deprecated, consider using the 'stop' statement instead [try http://www.rsyslog.com/e/2307 ]

From v7 onward, the stop RainerScript directive is recommended instead. From my testing I have found that it works well with RainerScript, Selector or Property-based filters. If using it with RainerScript it doesn't appear that you need to use & to tie the directive to the previous filter like you do with selector or property-based filters. As with the discard operator (~), with those two filter approaches you run the risk of stopping all logging after the use of the directive if you don't tie it back to a preceding filter.

Example: Using the discard operator

From an Ubuntu 12.04 LTS box that runs v5.x. Here we're using a property-based filter.

# /etc/rsyslog.d/20-ufw.conf

# Log kernel generated UFW log messages to file
:msg,contains,"[UFW " /var/log/ufw.log

# Uncomment the following to stop logging anything that matches the last rule.
# Doing this will stop logging kernel generated UFW log messages to the file
# normally containing kern.* messages (eg, /var/log/kern.log)
& ~


Example: Using the stop operator

From an Ubuntu 14.04 LTS box that runs v7.4.x. Here we're also using a property-based filter.

# /etc/rsyslog.d/20-ufw.conf

:msg,contains,"[UFW " /var/log/ufw.log

# Uncomment the following to stop logging anything that matches the last rule.
# Doing this will stop logging kernel generated UFW log messages to the file
# normally containing kern.* messages (eg, /var/log/kern.log)
& stop


References

General

Deprecated Features

Actions

Filters

Regular Expressions

Properties

Receiving from and Sending to remote Syslog boxes