Unparsed logs are evidence too: what to do with the lines no parser reads

Every SOC has a source that does not parse cleanly. A firewall on an old firmware. An in-house application that writes its own format. A cloud service that changed its log layout last quarter.

What happens to those lines is one of the most important and least discussed choices in a SIEM. Too often the answer is that they are dropped, or that they sit in a raw bucket nobody searches. Either way, the evidence is gone in practice.

Our view: an unparsed log is still evidence. It should be kept, it should be searchable, and the work of making it parse should start from your own samples, with a quality bar before anything goes live.

Why unparsed logs matter

Parsing failures are not random. They cluster around exactly the sources that matter in an investigation:

New or changed devices. The source that was added during a project, or upgraded last month.
Custom applications. Often the systems that hold the data an attacker wants.
Edge cases inside known sources. A firewall that parses well for traffic logs and badly for its administrative events.

If those lines are dropped, the gaps in your detection line up with the gaps in your visibility. And nobody notices until an investigation reaches for a record that is not there.

Five steps to take control

1. Measure the unparsed share per source

Do not settle for a single platform-wide figure. Ask, for each source: how many lines arrived, how many parsed, how many did not. A source that parses at a high rate overall can still be failing on the one event type you care about.

2. Keep the raw record

Whatever happens at parse time, keep the line as it arrived. Make it searchable by keyword, so an analyst can still find it during an investigation. A line you can search is evidence. A line you discarded is not.

Keep fields that do not map, too. If a source sends a field your schema has no place for, store it with the event instead of throwing it away.

3. Build parsers from real samples, not from a specification

Vendor documentation describes what a device should log. Your samples show what it actually logs. Group the unparsed lines by structure, and build parsers from groups that have enough examples to be representative. Never from a single line.

4. Set a quality bar before a parser goes live

A parser that half-works is worse than no parser. It produces events that look normalised but carry the wrong values in the wrong fields, and detection logic trusts them.

Define a bar. Every sample line in the group should yield fields. Nothing meaningful should be left unmapped. If a candidate does not clear the bar, it should wait for a person, not go live anyway.

5. Re-parse the backlog

When a parser goes live, apply it to the lines that were waiting for it. Otherwise the historical gap stays, and the investigation that needs last week's records still cannot find them.

Two governance questions

What ends up in a shared parser? Log lines are customer data. A parser shared between organisations should carry structure only, never lines or values from one customer. Ask how that is enforced, and whether you can see when your logs contributed.

Who controls the shared catalogue? Most platforms have a set of parsers every customer uses. Know who can change it, and how a change is reviewed.

How kymeer approaches this

kymeer detects the format of each line when it arrives: syslog (RFC 3164 and RFC 5424), JSON, CEF, LEEF and key=value. CEF and LEEF are checked first, because they usually travel inside syslog, so one collector port can carry mixed traffic. Events are normalised to kymeer's canonical schema, and fields that have no place in it are kept with the event.

Logs that no parser recognises are not discarded. They remain searchable by keyword, alongside parsed and raw records. A single record larger than 1 MiB is not parsed; it is kept with the reason it was rejected.

Unparsed logs are the input for the Parser Factory. It groups them by device type and by shape, and considers a shape once it has seen at least three copies; it never builds from a single line. A shape goes live only if, on every sample line, at least 95% of the line's fields are mapped, it takes over no line an existing parser already reads, and every sample agrees on the device type. Otherwise it stays a draft, and a draft parses nothing. Values a live shape does not map stay on the event as unmapped fields.

kymeer keeps one shared parser for each device type, so a shape learned from one tenant's FortiGate logs helps every tenant with a FortiGate. What is shared is structure, not data: a published parser never contains a log line or the identity of a contributing tenant, values specific to a tenant are never used to match lines, and each promotion is recorded in the audit log of every tenant whose lines contributed. Every version is kept, so a change can be rolled back.

If AI is set up, the Parser Factory can ask a model to help build a candidate. That call goes through the same masking and egress switch as every other AI call, so with egress off it does not happen.