ClusterDown / After an incident
The next step after triage
Root cause analysis after a cluster incident
After an incident the question is not how the cluster is doing, but what happened. Triage sorts the signals; a root cause investigation is the collection and analysis that together end in an RCA document.
Run the triage collector as soon as possible, before starting a full investigation. Not because anyone is looking at that moment, but because the evidence is still there then: cluster logs roll over and event channels fill up.
Cluster log investigation
Who can analyse cluster logs
The collector runs from your management server, read-only. For each node it reads the event logs over the incident window: 29 channels, insofar as they are enabled on your cluster. The node timelines are laid over each other and corrected for clock offset, so it becomes visible what happened in which order and on which machine it started.
Before a single node is approached, the collector shows the plan for the run: which nodes, which time window, which parts are switched on, where the output lands, and for each value where it comes from. Below that stands one yes-or-no question, and only then does the run start. During the run each phase reports its elapsed time.
The collector has been proven on clusters of six to sixteen nodes, and on a production cluster of six nodes across two sites, entirely from the management server.
Collecting evidence after an outage
Two heavy parts you choose per run
The cluster log per node
By far the largest part of the run. The size differs per environment; as an example:
First field run · 828 MB raw log
Compressed · 29.6 MB
Anyone doing a second run over the same window skips it.
Memory dumps
Dumps that are found are always recorded with location, timestamp, size and hash. How large a dump is depends on the type: a minidump stays under a few megabytes, a kernel or active memory dump runs into gigabytes to tens of gigabytes on large hosts, and a complete memory dump is as large as the working memory of the node.
That is why you choose what travels: right away, or afterwards with a shortened run that fetches only the dumps a previous run found and verifies against the hash that it is still the same file.
Recurring incident, same node
Gaps are part of the evidence
An event channel that has rolled over or is disabled on a node is recorded as a gap, with the reason. If the cluster log reaches back less far than the requested window, that is stated per node. So when reading the analysis you know exactly where the evidence ends.
At the end of the run this comes with advice for your administrator: which channels should be enabled and whether the cluster log buffer should be increased so a next incident is fully covered. The trade-off is stated too, because keeping more at the same log level costs disk space, and keeping more at a lower log level costs detail.
From this follows the rule behind every conclusion: as long as evidence is missing, the conclusion is not written. If a node was missing from the collection, a clean picture on the other nodes does not count as a measurement for that one.
Report for the board after an incident
Substantiation towards your board or your supplier
An RCA document: the incident window, the timeline across all nodes, the established cause insofar as the evidence carries it, the gaps with name and reason, and the repair work that follows from it.
How to start
Stronger combined with a baseline measurement
A root cause investigation is orderable on its own. But a reconstruction is strongest when the configuration can be put next to the events, because a deviating firmware level or a skewed network setting otherwise cannot be told apart from coincidence.
If the cluster has never been fully measured, the recommendation is a root cause investigation combined with a baseline assessment through ClusterTriage Assurance. If the environment is already an Assurance customer, the investigation also reads what earlier measurements already established.
Questions
Who can analyse my cluster logs after an incident?
Per node the collector reads 29 event channels across the incident window and lays the timelines over each other, corrected for clock difference. The analysis is written by a specialist, not by the tooling.
How do I prove that an incident was not caused by my configuration?
With a reconstruction that puts the configuration next to the events. That is why a root cause investigation combined with a baseline measurement is advised: a deviating firmware level or a skewed network setting cannot otherwise be told apart from coincidence.
What happens if the evidence is incomplete?
Then that is what the report says. As long as evidence is missing, the conclusion is not written. If a node was missing from the collection, a clean picture on the remaining nodes does not count as a measurement for that one.
Does the investigation change anything on my cluster?
No. The collector only reads and shows the plan for the run in advance, with one yes-or-no question before anything happens.
Start with the evidence
Run the triage collector
while it is still there
Cluster logs roll over and event channels fill up. The free Cluster Triage costs nothing, only reads and changes nothing.