Merged logs
Why?
CTDB, running at NOTICE
debug level, can generate a non-trivial amount of log output. If there is a problem in a cluster-wide operation, such as database recovery, it can be difficult to make sense of the overall cluster flow while flipping between logs.
Therefore, it can be useful to create a merged log when debugging cluster-wide operations.
How?
Use syslog
To create a merged log, CTDB must be using syslog
logging:
[logging] location = syslog
rather than file
logging.
While CTDB's current file
logging format (different to Samba's file logging format, which contains separate header lines, so can't be easily sorted) can be sorted, it can be very useful to have log messages from smbd
(and perhaps the cluster filesystem) in the same file, since they form part of the overall debugging context.
High resolution timestamps
To be able to reliably merge logs, a high resolution timestamp format (such as 2020-06-18T09:41:24.305332+10:00
) must be used.
In current rsyslog
versions this can be configured via:
module(load="builtin:omfile" Template="RSYSLOG_FileFormat")
Although this is the default, many platforms use:
module(load="builtin:omfile" Template="RSYSLOG_TraditionalFileFormat")
If this is done in the main configuration file (usually /etc/rsyslog.conf
) then it appears impossible to override it in an include file. The main configuration file needs to be changed.
In older versions it was possible to use the following in an include file (such as /etc/rsyslog.d/local.conf
:
$ActionFileDefaultTemplate RSYSLOG_FileFormat
If high resolution timestamps are in use then logs files from all nodes call be copied to a central location and a merged log file can be created:
sort *.messages >messages.merged
A compromise
If logs are only available with a traditional timestamp format (such as Jun 18 09:25:38
) then the following method provides a reasonable approach:
sort -s -k 1,3 *.messages >messages.merged
The use of the -s
option seems to force sorting only on the key fields, leaving files with the same timestamp in their original order. Note that messages from different nodes won't be interleaved correctly but an approximation can be better than nothing.