We measured eight large language models on 600 realistic SIEM alert scenarios spanning twenty categories. Each model made a decision on every scenario; the decisions were compared against ground truth labeled by human analysts. Accuracy, latency and cost must be read together — any one alone misleads.
Most benchmarks report one accuracy number. socbench reports what that accuracy costs — in money, in seconds, and in the alerts a model waves through.
Every model saw the same 600 alerts drawn from live SIEM data across twenty categories, and returned a verdict on each: true positive, false positive, or needs confirmation. Ground truth was labelled by working analysts, not by another model.
600 scenarios · 20 categories · 8 models · ~20,000 evaluations
leaderboard
all 8 models · sorted by strict accuracyStrict scoring demands an exact match. Merging “true positive” with “needs confirmation” — operationally the same call — lifts every model, some far more than others.
Rank here is accuracy alone. Five of these eight are beaten on cost by a model that is both cheaper and more accurate; six are beaten the same way on latency.
Strict scoring requires an exact match: TP=TP, FP=FP, CCN=CCN. Lenient scoring merges TP and CCN — operationally both point the same way: "don't close it, act on it." The distance between them measures the model's hesitation between a real threat and a case that still needs confirmation; a long line means an indecisive model.
How to read this.These scores are specific to this scenario set; they are not a measure of general model capability. They measure one thing only: whether the model makes the same call an L1 analyst would, matching the human label. You can sort by clicking any column header.
| Model | Provider | Strict | Lenient | C / P / W | Latency | Total | $ / correct |
|---|
Slow thinking doesn't pay off. It backfires.
Top-left is the corner you want: cheap and accurate. Bottom-right: expensive and weak. The most expensive model costs 32× as much as the cheapest and is only 8 points more accurate — hence the logarithmic horizontal axis.
In a SOC queue, 100 seconds per alert rules out real-time triage. This chart asks whether there's a trade-off between speed and accuracy. The point cloud is shapeless: no trade-off.
Scenarios are weighted to mirror the composition of real SOC queues. One in five is an ambiguous case that needs customer confirmation; one in eight is an outright false positive — so a model that calls everything "malicious" cannot score well.
| Verdict | What it means | Count | Share |
|---|---|---|---|
| TP | Genuine malicious activity; critical cases trigger automatic escalation to L2 | 383 | |
| CCN | Authorization or scope must be confirmed — ask the customer | 122 | |
| FP | False positive; the alert can be closed | 68 | |
| ID | Not enough information to reach a verdict | 27 |
Each scenario was built as an envelope close to what an analyst would see on screen: raw log fields, asset context, and the title of the rule that fired. The model was not asked for free text — it had to pick one of four options.
Labels were assigned by human analysts and were not shown to the models at any stage. Critical TPs were flagged as requiring automatic L2 escalation.
Each scenario was run across roughly 30 variants; the reported scores are the aggregate of those runs. A single run is not evidence — only repetition separates model-to-model differences from noise.
Calculated from provider list prices as of the measurement date. Qwen 3.6 ran on our own hardware; the figure shown is the equivalent API price, not our electricity cost.
600 scenarios · 8 models · 20 categories · ~20,000 evaluations.
This is an independent measurement study; no model provider sponsored it. Scores are specific to this scenario set and should not be read as a measure of general model capability.