Merged logs

Why?

CTDB, running at NOTICE debug level, can generate a non-trivial amount of log output. If there is a problem in a cluster-wide operation, such as database recovery, it can be difficult to make sense of the overall cluster flow while flipping between logs.

Therefore, it can be useful to create a merged log when debugging cluster-wide operations.

How?

Use syslog

To create a merged log, CTDB must be using syslog logging:

 [logging]
       location = syslog

rather than file logging.

While CTDB's current file logging format (different to Samba's file logging format, which contains separate header lines, so can't be easily sorted) can be sorted, it can be very useful to have log messages from smbd (and perhaps the cluster filesystem) in the same file, since they form part of the overall debugging context.

High resolution timestamps

To be able to reliably merge logs, a high resolution timestamp format (such as 2020-06-18T09:41:24.305332+10:00) must be used.

In current rsyslog versions this can be configured via:

 module(load="builtin:omfile" Template="RSYSLOG_FileFormat")

Although this is the default, many platforms use:

 module(load="builtin:omfile" Template="RSYSLOG_TraditionalFileFormat")

If this is done in the main configuration file (usually /etc/rsyslog.conf) then it appears impossible to override it in an include file. The main configuration file needs to be changed.

In older versions it was possible to use the following in an include file (such as /etc/rsyslog.d/local.conf:

 $ActionFileDefaultTemplate   RSYSLOG_FileFormat

If high resolution timestamps are in use then logs files from all nodes call be copied to a central location and a merged log file can be created:

 sort *.messages >messages.merged 

A compromise

If logs are only available with a traditional timestamp format (such as Jun 18 09:25:38) then the following method provides a reasonable approach:

 sort -s -k 1,3 *.messages >messages.merged

The use of the -s option seems to force sorting only on the key fields, leaving files with the same timestamp in their original order. Note that messages from different nodes won't be interleaved correctly but an approximation can be better than nothing.