How to evaluate a SOC platform with your own evidence

A benchmark tells you how a platform performed on someone else's data, on someone else's hardware, for someone else's questions. It does not tell you whether your analysts will reach a defensible answer faster on your sources.

That is the question a SOC lead or CISO actually has to answer. So the evaluation should be built around it. Not around a demo script, and not around a number on a slide.

Our view is simple. An evaluation is only worth running if the success criteria are written down before it starts, and the result is measured against how your team works today.

Why the usual evaluation goes wrong

Most platform trials fail in predictable ways.

The criteria arrive late. The team runs the trial, then decides what "good" meant. Whatever the platform did well becomes the criterion.
The data is wrong. A curated sample set, or a handful of sources that parse cleanly, stands in for the messy estate you actually run.
The vendor drives. A sales engineer runs the queries. Your analysts watch. You learn how well the vendor knows its own product.
The baseline is missing. Without a record of how long a use case takes today, any new timing looks impressive.

Each of these produces a confident decision on weak evidence. Fixing them costs very little.

A six-step method

Treat the evaluation like a small, controlled study.

Scope. Choose one tenant, the named users who will take part, and the sources that matter most. Not every source you own. The two or three that carry your hardest investigations.
Connect. Bring those sources in the way you would in production. Include at least one source that is known to be awkward to parse. That is where platforms differ.
Select. Pick two or three use cases that reflect real work. A hunt across a week of authentication logs. An investigation that starts from a real alert. A report you owe an auditor every month.
Baseline. Before anyone touches the new platform, record how each use case runs today. How many tools. How long to the first useful piece of evidence. What was missing.
Run. Your analysts work the same use cases on the new platform. Observers watch and take notes. They do not drive.
Decide. Compare the results with the criteria you agreed in step one. Nothing else.

What to measure

Measure the workflow, not the demo. These four measures travel well across teams and tools:

Time to first useful evidence. From opening the alert or question to the first record that moves the decision forward.
Tool switches. How many times the analyst leaves one screen or product for another.
Preserved pivots. Whether the path from the broad question to the specific record is kept as the analyst goes deeper, or has to be rebuilt.
Fields available. Whether the fields the decision depends on are actually there, parsed and searchable, for your sources.

Add one measure that is specific to you. For a bank, it might be the time to assemble the evidence for a regulator. For a manufacturer, it might be whether plant traffic shows up at all.

Write the criteria down first

A short criteria sheet, agreed by the people who will make the decision, prevents most arguments later. For each use case, write:

the question the analyst must answer
the sources it depends on
today's baseline for each measure
the result that would count as better
the result that would count as a failure

Sign it before the trial starts. If a criterion changes mid-trial, record why.

Test the access model, not only the features

A SOC platform holds some of the most sensitive data you have. Before rollout, check who can see and change what:

Visible identities. An administrator for one tenant should see only the identities inside that tenant.
Assignable roles. Invitation and role-change screens should offer only the roles appropriate to that tenant.
Audited changes. Every access change should leave a record with the actor, the tenant, the outcome and the time.

If you plan to buy through a managed service provider, test the model you will actually use. A provider that runs your SOC sees different things from a provider that only hosts your platform.

What not to accept as evidence

Performance figures measured on data that is not yours.
Screenshots or test-tenant values presented as outcomes.
Return-on-investment estimates that do not start from your baseline.

None of these is dishonest in itself. None of them answers your question.

How kymeer approaches this

A kymeer evaluation follows the same six steps: scope, connect, select, baseline, run and decide. Your kymeer partner connects your priority sources through agents, collectors or connectors. Your analysts work the use cases; we watch, and we do not drive. The results are compared with criteria agreed in writing before day one.

We do not quote performance figures. Performance, detection efficacy and return on investment depend on your sources and use cases, so they are measured during your evaluation, on your data.

Awkward sources are part of the test. kymeer detects syslog, JSON, CEF, LEEF and key=value per line, and logs that no parser recognises are kept rather than discarded, so you can see exactly what parsed and what did not.

For trials, kymeer has a proof-of-concept tenant type. It behaves either like a tenant run by a managed service provider or like one run by the customer, so you can test the access model you plan to use in production.

Read the full method in Evaluating kymeer, or talk to us about planning one.