CTDB, running at
NOTICE debug level, can generate a non-trivial amount of log output. If there is a problem in a cluster-wide operation, such as database recovery, it can be difficult to make sense of the overall cluster flow while flipping between logs.
Therefore, it can be useful to create a merged log when debugging cluster-wide operations.
To create a merged log, CTDB must be using
[logging] location = syslog
While CTDB's current
file logging format (different to Samba's file logging format, which contains separate header lines, so can't be easily sorted) can be sorted, it can be very useful to have log messages from
smbd (and perhaps the cluster filesystem) in the same file, since they form part of the overall debugging context.
High resolution timestamps
To be able to reliably merge logs, a high resolution timestamp format (such as
2020-06-18T09:41:24.305332+10:00) must be used.
rsyslog versions this can be configured via:
Although this is the default, many platforms use:
If this is done in the main configuration file (usually
/etc/rsyslog.conf) then it appears impossible to override it in an include file. The main configuration file needs to be changed.
In older versions it was possible to use the following in an include file (such as
If high resolution timestamps are in use then logs files from all nodes call be copied to a central location and a merged log file can be created:
sort *.messages >messages.merged
If logs are only available with a traditional timestamp format (such as
Jun 18 09:25:38) then the following method provides a reasonable approach:
sort -s -k 1,3 *.messages >messages.merged
The use of the
-s option seems to force sorting only on the key fields, leaving files with the same timestamp in their original order. Note that messages from different nodes won't be interleaved correctly but an approximation can be better than nothing.