logtrawl - simple syslog file analyzer
logtrawl [-h] [-c logtrawl.conf] [-a alerts.conf] [-i ignores.conf] logfile logfile ...
logtrawl is a simple log file anaylizer and reporting tool based on the concept of strings that will trigger alerts and strings that will be ignored. Any line that is not matched in one of these classes will be reported as "unmatched" after lines that match the alerts.
All matching is done using Perl compatible regular expressions. To facilitate the processing of syslog files, the header portion of each line is stripped off, typically including the timestamp and host name. Matching then begins with the daemon name.
The file containing regular expressions for lines that will trigger alerts. See MATCHING for a description of the file format. If the file name does not begin with a slash, it is looked for relative to the include_directory configuration variable.
The main logtrawl configuration file. See CONFIGURATION for a description of the configuration file.
Exit after giving a short usage summary.
The file containing regular expressions for lines that will be ignored. See MATCHING for a description of the file format. If the file name does not begin with a slash, it is looked for relative to the include_directory configuration variable.
logtrawl can be given a configuration file to set some of its internal operations. The format of the configuration file is
variable: value
Blank lines and lines beginning with # are ignored. Any spaces between the colon and the vlue will be stripped.
The following variables may be set:
A regular expression that describes the format of the log messages. The default message format is
[A-Z][a-z][a-z]\s+\d+\s+\d\d:\d\d:\d\d\s+([^\s]+)\s+(.*)
This matches, for example,
Jan 5 14:32:16 loghost daemon[32]: message
The message format should include two sets of capturing parentheses. The first set should grab the hostname and the second set should grab the remainder of the message. This portion is then used as the text against which alert and ignore regular expressions are tested.
Sets the directory that will be searched when a matching file uses an include statement that does not reference a full path name. This directory is also used if an alert or ignore file name is not a full path name.
The alert and ignore regular expression files determine which log messages will be elevted to alerts and which will be ignored in the output. The format of the files is identical. Blank lines and lines beginning with # are ignored (except for include statements; see INCLUDES). Everything else is treated as a Perl compatible regular expression that is anchored at the beginning of the line. Pattern matching is only performed on that portion of the log message that is returned after the log header is stripped off (see the message_format configuration variable).
A simple match that would alert (or ignore) on any log message from sendmail would be:
sendmail\[\d+\]:
In order to fine tune the alerts and ignores, the pattern can be made more complex. If the log message matches any of the patterns in the alert or ignore file it is considered to have matched for that file. The following patterns would only match for sendmail log messages corresponding to email that has been permanently rejected or temporarily delayed:
sendmail\[d+\]:.* dsn=5\.\d\.\d, stat=.*
sendmail\[d+\]:.* dsn=4\.\d\.\d, stat=.*
These could be combined into one line as such:
sendmail\[d+\]:.* dsn=(4|5)\.\d\.\d, stat=.*
It is usually better for performance to follow the more explicit first form rather than to try to combine regular expressions. Internally, logtrawl uses the Regexp::Assemble module to produce optimized regular expressions, which typically results in more efficient regular expressions than can be produced by mere mortals.
The alert and ignore regular expression files may also contain file include statements in either of the following forms:
#include /path/to/file
#include file
In both cases, the contents of the referenced file are read and inserted into the matching expression as if they were present in the original file. Included files may, themselves, include other files.
If the included file name begins with a slash, it is treated as a full path, otherwise it mus be present in the include_directory.
No checks are made on the regular expressions found in any of the files. It is perfectly possible to specify a regular expression that will, when used against standard log messages, consume all available resources and hasten the eventual heat death of the universe.
This software was written by John "Rowan" Littell and is hereby placed into the public domain.