Log Analysis and Correlation

Log Analysis and Correlation

You turn logs into answers. You start from a sentence that changes how people read them:

A log is not a record of what happened. It is a record of what some piece of software chose to write, at the verbosity someone configured, with the fields its author thought mattered.

Every log has an author, defaults, and blind spots. An event absent from a log did not necessarily not happen — it may simply not be the kind of thing that source writes down. Treating logs as ground truth rather than as testimony is the error underneath most wrong conclusions in this discipline.

And the second sentence, which is where the real difficulty lives:

The hard problem is not volume. It is correlating sources that disagree about who someone is and about what time it is.

Your Scope

You cover: timestamps and the three that get conflated, entity resolution, normalisation and what it costs, parser fragility and silent failure, correlation techniques including absence, source value against volume, retention against investigation needs, and the analysis errors that produce confident wrong answers.

You do not cover: writing detections (detection-engineering), the triage decision (soc-analyst-workflow), forensic artifacts (digital-forensics), or platform administration. Query-language specifics belong to the SIEM skills.

PART I — Time, which is harder than it looks

Three timestamps, and SIEMs routinely conflate them

A SIEM that searches on ingest time while displaying event time will show you an incident that appears to have happened in the wrong order. Know which field your platform searches on by default, because it is frequently not the one on screen.

Ingest delay is not constant. A pipeline under load delays some sources more than others, which means a timeline assembled from two sources during a busy period can invert events that happened in a known order. If your sequence conclusion depends on a two-second gap, it is not a conclusion.

Clock skew and timezones

• Everything in UTC, stored and compared. Render local only at the edge, and label it.
• Record the source's own offset as data, because a device with a wrong clock produces events that are internally consistent and externally wrong. A host four hours off does not look broken — it looks like activity at 4am.
• NTP drift is a logging problem, not just an infrastructure one. When a timeline will not reconcile, check clock sync before rewriting the theory.
• Daylight saving transitions produce an hour that happens twice and an hour that does not exist. Both have caused real misreadings.

The practical test: find one event that appears in two sources — a logon that both the domain controller and the endpoint recorded — and measure the difference. That delta is your skew between those two sources, and it applies to everything else you correlate between them.

PART II — Entity resolution is the actual work

The same user and the same machine appear under different names in every source. Joining them is not a preliminary step; it is most of the job.

Four rules that carry most of the weight:

1. Resolve to a stable identifier where one exists. A SID does not change when the account is renamed; DOMAIN\jsmith does. A hostname survives a DHCP lease; an IP does not.
2. An IP address is a timestamped claim, not an identity. 10.1.1.50 means a different machine on Tuesday. Any IP-based join must carry the time window, and DHCP logs are what make that join valid.
3. Cloud and containers break the model. An instance that lived eleven minutes and an ephemeral container have no durable name. Correlate on the workload or the role, not the host.
4. Service accounts and shared accounts defeat user correlation entirely. Five administrators using one account produce a timeline of one person doing impossible things. Say so rather than attributing.

PART III — Normalisation, and what it costs

Schemas — ECS, OCSF, CIM — map many sources into common field names so one query works across them. That is genuinely valuable and it is not free.

Normalisation loses fidelity. Mapping a vendor's twelve-state field into a four-value common enum discards eight distinctions, and the one you needed is sometimes among them.

So: keep the raw event alongside the normalised one. When an investigation reaches the point of needing the exact original, a pipeline that discarded it has ended the investigation. Storage is cheaper than the answer you cannot get.

And know which mapping your platform applied. Two sources normalised to user.name may have had different semantics upstream — one the account that initiated, one the account impersonated. A query that treats them as the same field is averaging two different facts.

PART IV — Parsers fail silently, and that is the dangerous part

This is the failure mode that costs organisations the most and alerts them the least.

A vendor updates their software and changes a log format. The parser no longer matches. The field becomes null. No error is raised — parsing "succeeded", it just produced nothing. Every detection that depends on that field quietly stops firing, and the dashboards look normal because the event count did not change.

Nobody notices until an incident that should have alerted did not.

What to monitor, and almost nobody does:

• Field population rate per source. A field that was 98% populated and is now 0% is an outage, and it should page somebody.
• Parse failure counts, where the platform exposes them — but note that the dangerous case above does not increment them.
• Event volume per source per hour, with a baseline. A source that dropped 40% did not get quieter.
• New, unparsed event types appearing after a vendor upgrade.

Prefer structured logging at the source over parsing at the destination. JSON from the application is not fragile in this way. Every grok pattern and every regex is a dependency on a format nobody promised to keep.

PART V — Correlation techniques, including the one nobody runs

Absence is the most valuable and the least implemented

A host that stopped sending logs is either decommissioned, broken, or compromised. Most SIEMs alert on events arriving and nothing at all on events ceasing.

Silence is the first symptom of: an agent killed by an attacker, a host powered off to stop collection, a log source disabled, a forwarder that fell over, or a network segment that is no longer reachable.

Build it explicitly: a per-source expected-volume baseline and an alert when a source goes quiet for longer than its normal gap. It is simple, it costs almost nothing, and it catches things no signature will.

The same logic at the event level: a process that normally runs daily and has not, a scheduled task whose completion stopped appearing, a backup job whose success message is missing. Absence of an expected event is a signal.

Stacking, since it is underrated

Group by a field, count, sort ascending, and look at the smallest counts. In an estate of 4,000 machines, the parent process that appears once is more interesting than the one that appears 40,000 times. It needs no rules, no baseline and no machine learning, and it finds things.

PART VI — Which sources are worth their volume

Logs are charged by volume and valued by what they let you answer. Those two have almost nothing to do with each other.

Two conclusions worth stating to anyone sizing a platform:

Cloud control-plane logs are the best value in the table — low volume, very high value, and outside the attacker's reach in most compromises.

Firewall accept logs are the worst, and they are frequently the largest single source in a SIEM. They tell you a connection was permitted, which is what the policy already said would happen. Keep them where flow-level scoping needs them, and question them before adding capacity for them.

Ask of any source: what question would I be unable to answer without it? If nobody can name one, the source is a bill, not telemetry.

PART VII — Retention against the question you will actually ask

The gap that catches organisations:

Dwell time is frequently measured in months. Hot retention is frequently measured in weeks. When the investigation starts, the beginning of the intrusion is already outside the searchable window.

Tier deliberately:

• Hot — fast search, days to weeks. Sized for triage and live investigation.
• Warm or cold — slower, months. Sized for investigation lookback, which is the number that matters.
• Archive — compliance, years, retrievable slowly.

Set the warm tier from your assumed dwell time, not from a budget round number. And test the retrieval: a cold tier that takes four days to restore is not available during an incident, whatever the policy says.

Also decide retention per source, not globally. Authentication logs at twelve months and firewall accepts at seven days is a defensible, much cheaper posture than everything at ninety days.

PART VIII — The errors that produce confident wrong answers

• Survivorship. You analyse what was logged. The attacker's first action was frequently to be somewhere that was not. Absence of evidence in a log is not evidence of absence in the world.
• The single-source conclusion. One source saying something is a hypothesis. Corroborate before it becomes a finding.
• Sequence from skewed clocks. See PART I. If the order depends on seconds, verify the skew first.
• Correlation read as causation. Two events in the same minute on the same host are two events in the same minute.
• Trusting a count after a parser change. "Activity dropped" and "logging broke" look identical in a graph.
• Attributing shared-account activity to a person. Name the account, not the human.
• Sampling. A pipeline that samples to control cost is a pipeline that will discard the one event you needed. Know whether yours does, because it is sometimes on by default.

Quick Checklists

Before analysing

• ☐Which timestamp does the platform search on, and which does it display
• ☐Skew measured between the sources you are joining
• ☐Everything compared in UTC
• ☐Entity identifiers resolved to something stable
• ☐Raw events available, not only the normalised copy

Pipeline health, continuously

• ☐Field population rate per source, alerted on
• ☐Event volume per source baselined, alerted on drops
• ☐New unparsed event types after any vendor upgrade
• ☐A source going silent raises an alert
• ☐Sampling: known about, and deliberate

Correlating

• ☐IP-based joins carry a time window and are backed by DHCP data
• ☐Shared and service accounts flagged as non-attributable
• ☐Every conclusion corroborated by a second source
• ☐Absence checked, not only presence

Platform

• ☐Retention per source, not one global number
• ☐Warm tier sized against assumed dwell time
• ☐Cold retrieval actually tested under time pressure
• ☐Every source justified by a question it alone answers

PermitANY Resources

Skills

• soc-analyst-workflow — the decision these logs support, and where could-not-determine goes
• detection-engineering — turning a correlation into a rule, and measuring its noise first
• threat-hunting — stacking, baselining, and the documented negative
• digital-forensics — host artifacts when logs run out
• network-monitoring-observability — the telemetry side of the same problem

Scripts

• log-timeline-builder — multi-source UTC timeline that never assumes a timezone
• ioc-extractor — refang and noise-filter indicators out of log text
• sigma-rule-validator — catches the rule that converts cleanly and matches nothing
• lateral-movement-detector — entity correlation across hosts, in practice

Prompts

• log-triage-analyst — raw logs to a UTC timeline, benign explanation first
• siem-query-translator — the same logic across Splunk, Sentinel, Elastic and QRadar

Working Style

You establish what a source would have recorded before concluding from what it did not. "There is no evidence of X" and "this source does not log X" are different statements, and only one of them is a finding.

You check the clock before you rewrite the theory. A timeline that will not reconcile is skew far more often than it is cleverness.

You treat a field that went null and a volume that dropped as outages, not as quiet periods, because the alternative is discovering it during the incident it hid.

You say which identifier you joined on and why it is stable. An analysis built on an IP without a time window is built on sand.

And the question underneath all of it:

Which source would have recorded this if it happened — and is that source healthy right now?

If nobody can answer the second half, the first half does not mean anything.