Detection Engineering: Sigma, Logs & ATT&CK Coverage

A rule that has never fired means either nothing bad happened – or the rule is broken, the log source is gone, or its field was renamed last month. Detection engineering lets you tell those cases apart. In this lesson you turn one raw Windows event into a Sigma rule, test it automatically against good and bad samples, confirm it in your own lab, and then build a harder detection that counts failed SSH logins over time.

• 12lessons, from telemetry to a full capstone project
• 2detections built and tested end to end in this lesson
• 5layers in the detection model you learn here
• ~20 minreading time for this lesson

What detection engineering is – and what it is not

A detection is a piece of logic that looks at security telemetry and decides that something deserves human attention. A rule is one way to express that logic, for example a Sigma rule or a SIEM query. An alert is what the rule produces when it matches. Telemetry is the raw material: event logs, endpoint sensor data, authentication records, DNS and proxy logs, cloud audit trails.

Detection engineering is the work of designing, building, testing, deploying and maintaining detections with the discipline software teams apply to code: requirements (what behaviour, and why?), version control (who changed this rule, and when?), tests (does it still match what it should?), reviews, releases and measurements.

Three neighbouring jobs often sit in the same team. SOC triage handles alerts that already exist. Threat hunting searches, from a hypothesis, for activity no alert covers yet. Detection engineering builds the alerting logic itself and owns its quality: coverage, precision, robustness and health. Triage tells engineers which rules are noisy, hunters tell them where the gaps are, and engineers give both better alerts. This course is written from the engineer's chair, with the analyst who receives your alert always in mind.

Why rules without engineering fail

Most organisations that own a SIEM also own a graveyard of rules: written once, working on the day they were written, then slowly decaying. The failure modes are consistent, and each is invisible until someone looks:

• The silent rule. An agent update renamed CommandLine to process.command_line. The query still runs and never matches again – and "no alerts" looks exactly like "no attacks".
• The noisy rule. It fires hundreds of times a day on a backup tool, analysts close it unread, and the real incident is closed the same way.
• The untested rule. A typo or a case-sensitive comparison on mixed-case data; it has never seen an event that should match.
• The coverage claim. Rules are tagged with ATT&CK IDs, so a slide colours the techniques green. Nobody checked whether the rules would fire.

None of the "better" column needs expensive tooling: Git, a text editor, Sigma and a few small scripts are enough, and that is what this lesson uses.

The mental model: five layers from telemetry to response

When a detection fails, the cause sits in one of five layers. Asking "which layer is broken?" turns a vague "the rule does not work" into a short list of checks.

• 1 SourceIs the activity recorded at all?
• 2 TransportDoes the record reach the platform, on time?
• 3 ShapeIs it parsed into the fields the rule expects?
• 4 LogicDoes the rule express the behaviour correctly?
• 5 ResponseCan an analyst act on the alert?

Only one of the five layers is the rule itself. Beginners spend their time on layer 4; experienced engineers spend much of theirs on layers 1 to 3, because a perfect rule on missing or misshaped data detects nothing.

Worked example: from log line to tested Sigma rule

The behaviour: a user creates a scheduled task from the command line. Attackers use scheduled tasks to make code run again after a reboot – MITRE ATT&CK technique T1053.005 Scheduled Task/Job: Scheduled Task. Administrators and installers create tasks too, so the rule must be useful without drowning anyone.

1. Write the hypothesis. One sentence that names the behaviour, the actor and why it matters: "A scheduled task is created with schtasks.exe from a normal user session on a Windows workstation; this can indicate persistence (T1053.005)."
2. Find the telemetry. Process creation with full command lines. That is Sysmon event ID 1, or Windows Security event 4688 once "Include command line in process creation events" is enabled. Note the alternatives too: Security event 4698 ("A scheduled task was created", needs object-access auditing) and event 106 in the Task Scheduler operational log record the task itself, regardless of which tool created it.
3. Read a real event from your lab and note the exact field names and values.
4. Write the Sigma rule with metadata, ATT&CK tags and an honest false-positive section.
5. Lint and convert the rule to see the query your platform will run.
6. Test the logic against stored positive and negative events, automatically.
7. Generate a harmless event in your lab and confirm the alert end to end.
8. Commit the rule together with its tests, so the next change cannot silently break it.

Step 3 – the raw event

On a lab Windows machine with Sysmon installed, creating a test task produces a process-creation event. Shipped as JSON and trimmed to the fields that matter, it looks like this (example event from a lab host; your agent may name or nest fields differently):

{
  "EventID": 1,
  "UtcTime": "2026-10-31 09:14:07.512",
  "Computer": "WS-LAB-01.lab.example.com",
  "Image": "C:\\Windows\\System32\\schtasks.exe",
  "OriginalFileName": "schtasks.exe",
  "CommandLine": "schtasks /create /tn \"DE-Test-001\" /tr \"cmd.exe /c exit 0\" /sc once /st 23:59 /f",
  "ParentImage": "C:\\Windows\\System32\\cmd.exe",
  "User": "LAB\\alice",
  "IntegrityLevel": "Medium"
}

Four fields carry the detection. Image is the path of the program. OriginalFileName comes from the file's embedded version information, so it still says schtasks.exe after the binary is copied to another name. CommandLine shows the /create action. User separates a person's session from NT AUTHORITY\SYSTEM, which installers and management agents typically use.

Step 4 – the Sigma rule

Sigma is an open, vendor-neutral YAML format for describing detections. You write the rule once against generic field names, and a converter translates it into the query language of your SIEM. Here is the complete rule:

title: Scheduled Task Created via Schtasks by a Non-System User
id: 6f1d2c4e-8a37-4b52-9e0f-3c1a7d5b9e21
status: experimental
description: |
    Detects schtasks.exe creating a scheduled task from a user context.
    Attackers use scheduled tasks for persistence and execution;
    administrators and installers use them too, so expect tuning.
references:
    - https://attack.mitre.org/techniques/T1053/005/
author: Your Name
date: 2026-10-31
tags:
    - attack.persistence
    - attack.execution
    - attack.t1053.005
logsource:
    category: process_creation
    product: windows
detection:
    selection_img:
        - Image|endswith: '\schtasks.exe'
        - OriginalFileName: 'schtasks.exe'
    selection_cli:
        CommandLine|contains: '/create'
    filter_main_system:
        User|contains:
            - 'AUTHORI'    # NT AUTHORITY (English)
            - 'AUTORI'     # NT-AUTORITÄT and similar localised names
    condition: all of selection_* and not 1 of filter_main_*
falsepositives:
    - Software installers and updaters running in a user context
    - Administrators creating tasks interactively
level: medium

The detection block is the heart of Sigma. Each named block is a search identifier. Keys in the same map are combined with AND; items in a list with OR. So selection_img means "Image ends with \schtasks.exe OR OriginalFileName is schtasks.exe". |endswith and |contains are modifiers that change how a value is compared. The condition combines the blocks: all selections must match, no main filter may.

The metadata is not decoration. The id is a UUID that stays fixed for the life of the rule, so alert history can be tied to it. status: experimental says the rule has not yet proven itself. The tags feed coverage reporting (Lesson 6), and falsepositives tells the analyst which benign causes are already known.

Step 5 – lint and convert

The reference command-line tool is sigma-cli, built on the pySigma library. Backends (target query languages) and processing pipelines (field mappings) are installed as plugins:

# one-time setup in a virtual environment
python3 -m venv .venv && . .venv/bin/activate
pip install sigma-cli pyyaml
sigma plugin install splunk
sigma plugin install sysmon

# check the rule for structural problems, then convert it
sigma check rules/
sigma convert -t splunk -p sysmon rules/windows/process_creation/proc_creation_win_schtasks_user_create.yml

The conversion prints the query your SIEM will actually run. Example output (the exact formatting differs between backend versions):

EventID=1 Channel="Microsoft-Windows-Sysmon/Operational" (Image="*\\schtasks.exe" OR OriginalFileName="schtasks.exe") CommandLine="*/create*" NOT User IN ("*AUTHORI*", "*AUTORI*")

Always read the converted query once: it shows what the pipeline added (here the Sysmon event ID and channel) and exposes logic mistakes fast.

Step 6 – test the logic automatically

A rule is only as trustworthy as the events it has been tested against. Store a few sample events next to each rule: positive samples that must match and negative samples that must not. The layout of the repository looks like this:

detections/
├── rules/windows/process_creation/
│   └── proc_creation_win_schtasks_user_create.yml
├── tests/proc_creation_win_schtasks_user_create/
│   ├── positive/01_user_create.json        # the lab event from step 3
│   ├── positive/02_renamed_binary.json     # copied binary, OriginalFileName still schtasks.exe
│   ├── negative/01_system_updater.json     # installer running as NT AUTHORITY\SYSTEM
│   └── negative/02_query_only.json         # schtasks /query - no task created
└── tools/test_rule.py

In production you test the converted query against a test index (Lesson 9). To see the idea in its purest form, here is a small Python harness that evaluates the rule logic directly; it implements only the Sigma features this rule uses:

#!/usr/bin/env python3
"""Mini test harness for Sigma rule logic - a teaching subset, not a full Sigma engine."""
import fnmatch, json, pathlib, re, sys
import yaml  # pip install pyyaml

def match_value(actual, mods, expected):
    if actual is None:
        return expected is None
    a, e = str(actual).lower(), str(expected).lower()  # Sigma matching is case-insensitive
    if 'contains' in mods:
        return e in a
    if 'startswith' in mods:
        return a.startswith(e)
    if 'endswith' in mods:
        return a.endswith(e)
    return a == e  # exact match (wildcards are not handled in this subset)

def match_map(event, sel):
    for key, expected in sel.items():          # every field in a map must match (AND)
        field, *mods = key.split('|')
        values = expected if isinstance(expected, list) else [expected]
        check = all if 'all' in mods else any  # a list of values means OR, unless |all
        if not check(match_value(event.get(field), mods, v) for v in values):
            return False
    return True

def match_selection(event, sel):
    if isinstance(sel, list):                  # a list of maps means OR
        return any(match_map(event, s) for s in sel)
    return match_map(event, sel)

def evaluate(rule, event):
    det = dict(rule['detection'])
    cond = det.pop('condition')
    res = {name: match_selection(event, sel) for name, sel in det.items()}

    def quantifier(m):
        names = list(res) if m.group(2) == 'them' else \
            [n for n in res if fnmatch.fnmatch(n, m.group(2))]
        vals = [res[n] for n in names]
        return str(all(vals) if m.group(1) == 'all' else any(vals))

    expr = re.sub(r'\b(1|all) of (them|\w+\*?)', quantifier, cond)
    expr = re.sub(r'\b[A-Za-z_]\w*\b',
                  lambda m: m.group(0) if m.group(0) in ('and', 'or', 'not', 'True', 'False')
                  else str(res[m.group(0)]), expr)
    return eval(expr, {'__builtins__': {}})

def main(rule_path, test_dir):
    rule = yaml.safe_load(open(rule_path, encoding='utf-8'))
    failures = 0
    for expect in ('positive', 'negative'):
        for f in sorted(pathlib.Path(test_dir, expect).glob('*.json')):
            hit = evaluate(rule, json.loads(f.read_text(encoding='utf-8')))
            ok = hit == (expect == 'positive')
            failures += not ok
            print(f"{'PASS' if ok else 'FAIL'}  {expect:<9}{f.name:<26}matched={hit}")
    print(f"{failures} failure(s)")
    sys.exit(1 if failures else 0)

if __name__ == '__main__':
    main(sys.argv[1], sys.argv[2])

Run it from the repository root:

python3 tools/test_rule.py \
  rules/windows/process_creation/proc_creation_win_schtasks_user_create.yml \
  tests/proc_creation_win_schtasks_user_create

What you should see:

PASS  positive 01_user_create.json       matched=True
PASS  positive 02_renamed_binary.json    matched=True
PASS  negative 01_system_updater.json    matched=False
PASS  negative 02_query_only.json        matched=False
0 failure(s)

Now break the rule on purpose: change the condition to all of selection_*, dropping the filter. Run the harness again and the installer event is caught – exactly the regression the tests exist to stop:

PASS  positive 01_user_create.json       matched=True
PASS  positive 02_renamed_binary.json    matched=True
FAIL  negative 01_system_updater.json    matched=True
PASS  negative 02_query_only.json        matched=False
1 failure(s)

The script exits with status 1 on any failure, so a CI system can block the pull request. That is the whole idea of detection-as-code in one screen.

Step 7 – a harmless lab event, end to end

Stored events prove the logic, not that your sensor, pipeline and SIEM produce those events today. So generate one on a lab machine you own, with a task that does nothing and is removed immediately:

:: Windows lab VM, normal user session (cmd.exe)
schtasks /create /tn "DE-Test-001" /tr "cmd.exe /c exit 0" /sc once /st 23:59 /f
schtasks /query /tn "DE-Test-001"
schtasks /delete /tn "DE-Test-001" /f

If the start time has passed, schtasks warns that the task may not run; irrelevant here, because the creation event is what you test. Within minutes the rule should alert for WS-LAB-01 and alice. Record date, rule version, host and delay. The /query and /delete commands did not alert – your negative test, live.

Step 8 – commit rule and tests together

git add rules/windows/process_creation/proc_creation_win_schtasks_user_create.yml \
        tests/proc_creation_win_schtasks_user_create/
git commit -m "Add schtasks user task creation detection (T1053.005) with tests"

From now on, every change to this rule runs the four tests. Lesson 5 turns this into a full CI pipeline.

Under the hood: how a Sigma rule becomes a query

Sigma itself never touches your data. It is a description. Something has to turn that description into a query for a specific platform, and understanding the layers of that conversion explains most of the strange behaviour you will meet later.

Layer by layer

Parsing. The converter reads the YAML into a rule object and checks its structure. A typo such as |endwith fails here, which is why linting belongs in CI.

Modifiers become value patterns. endswith turns the value into the wildcard pattern *\schtasks.exe, contains into */create*. A backslash is literal unless it precedes a wildcard or another backslash, so '\schtasks.exe' means exactly one backslash. Plain string comparisons are case-insensitive by default – which is why /CREATE in the renamed-binary test still matches.

The processing pipeline. Sigma uses a generic taxonomy (Image, CommandLine, User); your platform may use other names. A pipeline maps the generic names to yours, adds log source conditions (for example "Sysmon channel, event ID 1") and can transform values:

The User row shows why this matters. Sysmon stores LAB\alice as one string; ECS splits it into user.domain and user.name. A filter looking for AUTHORI in the user name silently stops working when NT AUTHORITY lands in the domain field. The rule did not change; the shape of the data did. Lesson 3 is about getting this layer right.

The backend. The last stage writes the target syntax – Splunk SPL, Lucene, ES|QL, KQL and others – including quoting, escaping and how lists and NOT are expressed. If a backend does not support a feature such as regular expressions or correlations, conversion fails loudly. Loud failure is good; it is what CI is for.

A second, deeper example: counting failed SSH logins

Many behaviours only become visible across several events: a burst of failed logins, one source trying many user names, failures followed by a success. Sigma version 2 adds correlation rules for this. The scenario: a Linux lab server, and you want to know when one source produces a burst of SSH password failures.

The raw lines and the parsing problem

OpenSSH writes authentication results to the system log. On a recent distribution with ISO timestamps, /var/log/auth.log (or the journal) contains lines like these (example lines from a lab host):

2026-10-31T10:01:12+00:00 lab-srv sshd[2154]: Failed password for invalid user admin from 192.0.2.44 port 51122 ssh2
2026-10-31T10:02:03+00:00 lab-srv sshd[2160]: Failed password for alice from 192.0.2.44 port 51130 ssh2
2026-10-31T10:05:30+00:00 lab-srv sshd[2180]: Accepted password for bob from 192.0.2.77 port 49812 ssh2

This is free text. Before anything can count "failures per source", a parser must extract source address and user into fields. Note the trap in the first line: for unknown accounts sshd inserts invalid user before the name, so a naive parser records the user as invalid.

The Sigma rules

A correlation rule refers to an ordinary base rule by its name. The base rule finds single failures; the correlation counts them. Both live in one file, separated by ---:

title: SSH Password Authentication Failure
name: ssh_failed_password
status: experimental
logsource:
    product: linux
    service: sshd
detection:
    keywords:
        - 'Failed password for'
    condition: keywords
level: informational
---
title: Burst of SSH Password Failures From One Source
status: experimental
correlation:
    type: event_count
    rules:
        - ssh_failed_password
    group-by:
        - src_ip          # produced by your parser; the name depends on your schema
    timespan: 5m
    condition:
        gte: 5
tags:
    - attack.credential-access
    - attack.t1110
level: medium

A second correlation of type value_count asks a sharper question: how many different user names did one source try?

correlation:
    type: value_count
    rules:
        - ssh_failed_password
    group-by:
        - src_ip
    timespan: 5m
    condition:
        field: user
        gte: 3

Many attempts against one account look like password guessing (T1110.001); few attempts each against many accounts look like password spraying (T1110.003). Your group-by and counted field decide which behaviour the rule can see. Backend support for correlations is still uneven, so check your target converts them.

Test it: parse, window, count

Again, a small script makes the logic visible. It parses the failure lines, keeps a five-minute sliding window per source address, and reports when either threshold is crossed:

#!/usr/bin/env python3
"""Parse sshd 'Failed password' lines and apply two sliding-window correlations."""
import re, sys
from collections import defaultdict, deque
from datetime import datetime, timedelta

LINE = re.compile(r'^(?P<ts>\S+) (?P<host>\S+) sshd\[\d+\]: Failed password for '
                  r'(?:invalid user )?(?P<user>\S+) from (?P<ip>\S+) port \d+')
WINDOW, MAX_FAILS, MAX_USERS = timedelta(minutes=5), 5, 3

windows = defaultdict(deque)   # source ip -> deque of (time, user) inside the window
alerted = set()

for raw in open(sys.argv[1], encoding='utf-8'):
    m = LINE.match(raw)
    if not m:
        continue                                  # not a failed-password line
    t = datetime.fromisoformat(m['ts'])
    w = windows[m['ip']]
    w.append((t, m['user']))
    while w and t - w[0][0] > WINDOW:             # drop events older than 5 minutes
        w.popleft()
    users = {u for _, u in w}
    if len(w) >= MAX_FAILS and (m['ip'], 'count') not in alerted:
        alerted.add((m['ip'], 'count'))
        print(f"{t:%H:%M:%S} ALERT event_count  src={m['ip']} fails={len(w)} in 5m")
    if len(users) >= MAX_USERS and (m['ip'], 'users') not in alerted:
        alerted.add((m['ip'], 'users'))
        print(f"{t:%H:%M:%S} ALERT value_count  src={m['ip']} users={sorted(users)}")

The sample log auth_sample.log has eleven lines: one successful key login, seven failures from 192.0.2.44 against six names (one of them repeated eight minutes later), two failures for bob from 192.0.2.77 followed by a successful password login. Running python3 ssh_burst.py auth_sample.log prints:

10:02:03 ALERT value_count  src=192.0.2.44 users=['admin', 'alice', 'test']
10:03:58 ALERT event_count  src=192.0.2.44 fails=5 in 5m

What went wrong if you see…

• No alerts at all – the regex does not match your timestamp format (older syslog writes Oct 31 10:01:12). Fix the parser, never the threshold.
• A user called invalid – the optional (?:invalid user )? group is missing, so all unknown accounts count as one user.
• Alerts hours late – correlation runs on ingest time, or a forwarder buffers. Correlate on event time.
• Nothing for 192.0.2.77 – correct for these thresholds, but "failures, then success, same source" is a valuable pattern of its own. Sigma's temporal_ordered correlation expresses it; Lesson 7 builds it.

To generate test lines without touching sshd, use logger -t sshd --id=4242 "Failed password for invalid user de-test from 192.0.2.44 port 50000 ssh2". That tests parsing and counting, not that sshd logs this way. For an end-to-end test, make a few wrong password attempts from a second lab VM against a lab test account – and tell whoever watches the lab SIEM first.

Troubleshooting: when a rule does not fire, or fires too much

How experienced detection engineers work

Senior engineers rarely start with the rule. A typical first week for one new detection:

1. Day 1 – Question and dataWrite the hypothesis and confirm the telemetry exists on representative hosts. If it does not, the deliverable is a logging change, not a rule.
2. Day 1–2 – BaselineRun a broad query over 14–30 days of history: how often would it have fired, where, from which parents? False positives are cheapest to find here.
3. Day 2 – Rule and testsWrite the rule and its test events; put the baseline numbers in the pull request.
4. Day 3 – Lab validationGenerate harmless events, confirm the alert, measure the delay.
5. Day 3–7 – Shadow modeDeploy at a low level or to a separate queue and count dispositions.
6. After a week – Promote or reworkRaise status and level, or rework the logic with real numbers.

Their heuristics: detect behaviour rather than names, prefer fields an attacker cannot easily change, keep filters as narrow as the false positive they remove, and never ship a rule an analyst could not act on. A technique with no telemetry gets a logging ticket, not a heroic query on the wrong data.

Defender's view: making detections hard to evade

A detection is as useful as what an attacker must change to avoid it. Hashes and IP addresses are trivial to change; behaviour is expensive. The schtasks rule already survives renaming and has a narrow filter. Three more improvements, each with its own test:

• Command-line variants. Many Windows tools accept - as well as / before options. Sigma's windash modifier (CommandLine|windash|contains: '/create') expands the value into those variants. Add a positive test with the dash form.
• A second telemetry path. The PowerShell cmdlet Register-ScheduledTask creates a task without starting schtasks.exe, so the process rule cannot see it. A companion rule on Security event 4698 or Task Scheduler event 106 catches task creation whatever the tool. Defence in depth applies to detections too.
• Telemetry that cannot quietly disappear. Enable the audit settings the rules depend on, and alert when a source goes silent.

:: Windows: enable process creation auditing and command lines (lab or via GPO)
auditpol /set /subcategory:"Process Creation" /success:enable
reg add "HKLM\Software\Microsoft\Windows\CurrentVersion\Policies\System\Audit" ^
    /v ProcessCreationIncludeCmdLine_Enabled /t REG_DWORD /d 1 /f

:: verify
auditpol /get /subcategory:"Process Creation"

Test hardening like a rule: run the lab command, confirm the event contains a command line, and keep that check in your scheduled tests.

Measuring coverage honestly

Tagged is not covered. Real coverage is a ladder, and only the upper rungs mean "we would notice":

Report coverage per rung, weighted by the techniques that matter for your threats. Lessons 6 and 10 build exactly that.

Check yourself: detection engineering basics

You can now…

• Define detection engineering – and separate it from triage and hunting.
• Locate a failure – in one of five layers: source, transport, shape, logic, response.
• Write a Sigma rule – with selections, filters, a condition, ATT&CK tags and false-positive notes.
• Test rule logic – with positive and negative events and a failing exit code.
• Validate end to end – with a harmless lab event that is created and removed.
• Build a correlation – count failures and distinct users in a sliding window.
• Harden a detection – variants, a second telemetry path, audit settings.
• Report coverage honestly – by rung, not by tag count.

Glossary