diff --git a/doc/userguide/devguide/extending/app-layer/transactions.rst b/doc/userguide/devguide/extending/app-layer/transactions.rst index 03bf2c417d..0593652d10 100644 --- a/doc/userguide/devguide/extending/app-layer/transactions.rst +++ b/doc/userguide/devguide/extending/app-layer/transactions.rst @@ -120,6 +120,12 @@ This is controlled by implementing progress states. In Suricata, those will be e progresses. A state will start at 0. The higher its value, the closer the transaction would be to completion. Due to how the engine tracks detection across states, there is an upper limit of 48 to the state progress (it must be < 48). +Entering a state does not imply that the state's data is present: a state is entered when the unit it names starts +being parsed, and the data (buffers, extracted fields) exists only once the unit has been fully parsed. Keywords +reading that data must tolerate the unit still arriving - buffer keywords bail out when the buffer's GetData +callback reports the buffer unavailable, and keywords reading extracted fields must gate on the field being +present rather than assume the state implies the data. + The engine interacts with transactions' state using a set of callbacks the parser registers. State is defined per flow direction (``STREAM_TOSERVER`` / ``STREAM_TOCLIENT``). In Summary - Transactions and State diff --git a/doc/userguide/firewall/firewall-design.rst b/doc/userguide/firewall/firewall-design.rst index 351cd19f25..9148065b79 100644 --- a/doc/userguide/firewall/firewall-design.rst +++ b/doc/userguide/firewall/firewall-design.rst @@ -138,7 +138,7 @@ by one or more secondary actions. Example:: - accept:flow,pass:flow,alert tls:client_hello_done ... tls.sni; ... + accept:flow,pass:flow,alert tls:client_hello ... tls.sni; ... In this example the first action ``accept:flow`` is the primary firewall action. When the rule matches, the flow will be accepted. The secondary actions ``pass:flow`` and ``alert`` are @@ -232,19 +232,27 @@ Available states: Request (``to_server``) side: -* ``client_in_progress`` -* ``client_hello_done`` -* ``client_cert_done`` -* ``client_handshake_done`` +* ``client_started`` +* ``client_hello`` +* ``client_cert`` +* ``client_data`` * ``client_finished`` +The ``client_hello`` state names the ClientHello message being parsed: +it is entered when the message starts, so the fragments of a hello split +over several TLS records are all decided at ``client_hello``. The hello +data (SNI, version, ...) is available once the final fragment completes +the message, which moves the state to ``client_cert``; rules evaluated +on the earlier fragments see the buffers still empty, the same way a +split HTTP request line is in ``request_line`` before the line is +complete. + Response (``to_client``) side: -* ``server_in_progress`` +* ``server_started`` * ``server_hello`` -* ``server_cert_done`` -* ``server_hello_done`` -* ``server_handshake_done`` +* ``server_cert`` +* ``server_data`` * ``server_finished`` ssh @@ -260,13 +268,14 @@ a rule accept not just the hook it matches in, but also the hooks before it. Example:: - accept:flow tls: $EXTERNAL_NET 443 (flow:established; sid:4;) Then on the TLS level this will be a TLS SNI firewall. +:doc:`tls-state-migration` covers migrating rules and config +keys from the pre-phase state names. -Again all the states need to be accepted. Only in the ``client_hello_done`` state will -there be additional constraints:: +Again all the states need to be accepted. Only in the ``client_hello`` +state will there be additional constraints:: - accept:hook tls:client_in_progress $HOME_NET any -> $EXTERNAL_NET any (sid:100;) + accept:hook tls:client_started $HOME_NET any -> $EXTERNAL_NET any (sid:100;) # allow the good sites - accept:hook tls:client_hello_done $HOME_NET any -> $EXTERNAL_NET any (tls.sni; \ - pcre:"/^(suricata.io|oisf.net)$/; sid:101;) - accept:hook tls:client_cert_done $HOME_NET any -> $EXTERNAL_NET any (sid:102;) - accept:hook tls:client_handshake_done $HOME_NET any -> $EXTERNAL_NET any (sid:103;) - accept:hook tls:client_finished $HOME_NET any -> $EXTERNAL_NET any (sid:104;) - - accept:hook tls:server_in_progress $EXTERNAL_NET any -> $HOME_NET any (sid:200;) + accept:flow tls:client_hello $HOME_NET any -> $EXTERNAL_NET any (tls.sni; \ + pcre:"/^(suricata.io|oisf.net)$/R"; sid:101;) + # drop every ClientHello that carried an SNI the rule above does + # not allow; the SNI buffer is required, so a fragment still in + # flight (no SNI buffer yet) cannot match + drop:flow tls:client_hello $HOME_NET any -> $EXTERNAL_NET any (alert; tls.sni; \ + pcre:!"/^(suricata.io|oisf.net)$/R"; sid:102;) + # hello packets that carry no verdict of their own: the in-flight + # fragments + accept:hook tls:client_hello $HOME_NET any -> $EXTERNAL_NET any (sid:103;) + # no accept at the certificate state: a hello without an SNI matches + # no SNI rule and is decided by the default policy of the states it + # walked over (the implicit drop:flow) - the same outcome the base + # release gave it at the hello-completion milestone. If you want to + # accept SNI-less hellos, add an explicit accept here and audit the + # relaxation. + accept:hook tls:client_data $HOME_NET any -> $EXTERNAL_NET any (sid:105;) + accept:hook tls:client_finished $HOME_NET any -> $EXTERNAL_NET any (sid:106;) + + accept:hook tls:server_started $EXTERNAL_NET any -> $HOME_NET any (sid:200;) accept:hook tls:server_hello $EXTERNAL_NET any -> $HOME_NET any (sid:201;) - accept:hook tls:server_cert_done $EXTERNAL_NET any -> $HOME_NET any (sid:202;) - accept:hook tls:server_hello_done $EXTERNAL_NET any -> $HOME_NET any (sid:203;) - accept:hook tls:server_handshake_done $EXTERNAL_NET any -> $HOME_NET any (sid:204;) + accept:hook tls:server_cert $EXTERNAL_NET any -> $HOME_NET any (sid:202;) + accept:hook tls:server_data $EXTERNAL_NET any -> $HOME_NET any (sid:204;) accept:hook tls:server_finished $EXTERNAL_NET any -> $HOME_NET any (sid:205;) +The ``client_hello`` state names the ClientHello being parsed: it is +entered at the first byte of the message, and its completion moves +the state to ``client_cert``. The SNI buffer is owned by the hello +state, so ``tls.sni`` rules must live in ``client_hello`` even though +the record +that completes the hello is decided at ``client_cert``: a flow that +matches the SNI accept is carried over by the flow-scoped accept (101) +and a flow whose SNI is not allowed is dropped by 102 on that same +record - the drop is rule-scoped and alerts, where the implicit +default policy would drop silently. The plain ``accept:hook`` at 103 +carries the fragments in flight (the SNI buffer is not available +before the message completes, so it also cannot gate them). A +ClientHello that carries no SNI at all matches no SNI rule either - +and with no accept at the certificate state it is decided by the +default policy of the states it walked over (the implicit +drop:flow), the same outcome the base release gave it at the +hello-completion milestone. A flow whose hello is accepted by 101 is +carried by the flow-scoped accept: the flow action is checked before +any further firewall rule of a packet is evaluated, so the later +states of that flow never reach the implicit default. +``client_started`` covers the packets before the hello. + TLS SNI with auto-accept logic ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Rule that has the same effect as the 11 TLS rules above:: - accept:flow tls: $EXTERNAL_NET any (tls.sni; \ - pcre:"/^(suricata.io|oisf.net)$/; sid:101;) + accept:flow tls: $EXTERNAL_NET any (tls.sni; \ + pcre:"/^(suricata.io|oisf.net)$/R"; sid:101;) Explanation: ``accept:flow`` accepts all of the TLS flow from the moment the rule -has matched. The ``tls:client_in_progress`` hook is auto-accepted by the use of the -``<`` modifier in the hook ``tls: $EXTERNAL_NET any \ + accept:flow,pass:flow tls: $EXTERNAL_NET any \ (tls.sni; pcre:"/^(suricata.io|oisf.net)$/; sid:101;) -Explanation: as soon as this rule fully matches at the ``tls:client_hello_done`` hook, +Explanation: as soon as this rule fully matches at the ``tls:client_hello`` hook, a ``pass`` is applied to the flow effectively bypassing the threat detection engine. diff --git a/doc/userguide/firewall/index.rst b/doc/userguide/firewall/index.rst index c1386a371b..a225e8c50d 100644 --- a/doc/userguide/firewall/index.rst +++ b/doc/userguide/firewall/index.rst @@ -6,3 +6,4 @@ Firewall Mode firewall-design firewall-example firewall-stats + tls-state-migration diff --git a/doc/userguide/firewall/tls-state-migration.rst b/doc/userguide/firewall/tls-state-migration.rst new file mode 100644 index 0000000000..a3111829f9 --- /dev/null +++ b/doc/userguide/firewall/tls-state-migration.rst @@ -0,0 +1,204 @@ +Migrating from the old TLS state names +====================================== + +In this release the TLS state names changed from completion +milestones to active phases, following the HTTP1 unit states: a state +names the handshake message (or data exchange) the state progression +is working through - it is entered when that unit starts being parsed +and left when the unit completes. The phase data (SNI, certificates) is +filled only once the message has been fully parsed; rules evaluated +while it is still arriving (a fragmented message) see no data at +all, so their content keywords cannot match. The state names used +in rules +(``accept:hook tls:``, ``alert tls:``) and in the +``firewall.policies.app.tls`` config keys were renamed accordingly, and +the flight-completion state was removed. + +Old and new names +----------------- + +.. list-table:: + :header-rows: 1 + + * - old state (milestone) + - new state (phase) + - note + * - ``client_in_progress`` + - ``client_started`` + - covers the packets before the hello + * - ``client_hello_done`` + - ``client_hello`` + - the phase is entered at the start of the hello message + * - ``client_cert_done`` + - ``client_cert`` + - entered when the client hello completes, before any certificate + byte: the records up to the completing certificate message carry + no certificate data - add a plain ``accept:hook tls:client_cert`` + (or a drop-based cert check plus that accept) for them + * - ``client_handshake_done`` + - ``client_data`` + - + * - ``client_finished`` + - ``client_finished`` + - unchanged + * - ``server_in_progress`` + - ``server_started`` + - + * - ``server_hello`` + - ``server_hello`` + - unchanged + * - ``server_cert_done`` + - ``server_cert`` + - entered when the server hello completes, before any certificate + byte: the records up to the completing certificate message carry + no certificate data - add a plain ``accept:hook tls:server_cert`` + (or a drop-based cert check plus that accept) for them + * - ``server_hello_done`` + - ``server_data`` + - the flight-completion milestone no longer exists + * - ``server_handshake_done`` + - ``server_data`` + - both map to the data phase + * - ``server_finished`` + - ``server_finished`` + - unchanged + +A plain accept at ``client_cert`` or ``server_cert`` - including the +pairing of a drop-based cert check with that accept - also decides the +record that completes the hello, and with it a ClientHello that +carries no SNI at all: the SNI buffer is only filled once the message +is fully parsed, so no ``tls.sni`` rule (including a negated one) can +match it. The base release dropped that hello at the +``client_hello_done`` milestone through the implicit default policy; a +certificate-state accept accepts it. Keep the certificate state free +of an accept to preserve the base outcome - the implicit default +policy decides it, identically to the milestone - or add an explicit +rule for the SNI-less case if it must be accepted. The SNI example in +the firewall user guide and the upgrade chapter carry the same +warning. + +Semantics that changed with the rename +-------------------------------------- + +- ``client_hello`` names the ClientHello being parsed: it is entered + at the first byte of the message, so every fragment of a hello split + over several TLS records is decided at ``client_hello`` (with the + SNI and the other hello buffers filled only once the final fragment + completes the message). Completion moves the state to ``client_cert`` + before a single certificate byte arrives, and the record that + completes the hello is decided there; a Certificate message also + names the ``client_cert`` phase at its own start, which keeps a state + progression that begins mid-connection or after a failed hello + coherent. Rules on hello data + (``tls.sni``) still live in ``client_hello`` - that state owns the + buffer - but their accept must carry the flow: the verdict of the + completing record is rendered at ``client_cert``, so an + ``accept:hook`` at ``client_hello`` alone does not let the flow + through. A hello that fails to parse keeps the state progression in + ``client_hello``: the hand-over to the certificate phase is + suppressed, but the fields the parser already filled (version, session + id, an SNI extension parsed before the failure) stay in the connp and + remain matchable - the phase data may be incomplete rather than + absent, as in the previous model. ``server_hello`` behaves + identically for the ServerHello. +- ``server_data`` is entered on the first server (to_client) app-data + record, or as soon as the server certificate is parsed (the + subject/issuer shortcut that establishes the data phase + mid-handshake) - not when the server flight ends. A session that + completes the handshake but sends no server app data and no + parseable certificate never reaches ``server_data``; app data no + longer re-enters a completion state. The shortcut is per direction: a + direction's own certificate data decides that direction only. In + particular, in mutual TLS the server certificate (which arrives + before the CertificateRequest) does not push the client-side + progression through ``client_cert`` - the client certificate message + is still evaluated at ``client_cert``. +- The ``server_hello_done`` milestone is gone. The server record + progression between hello and data is ServerHello, the certificate + messages, then the ChangeCipherSpec/Finished flight; the flight + carries no inspectable data, which is why no state exists for it. + A rule that wanted to act at "flight complete, before data" now + belongs to the tail of ``server_cert`` or the head of + ``server_data`` depending on intent - the rename hint + (``server_hello_done`` -> ``server_data``) is approximate in the + same way; at the old milestone the state progression was usually + still in ``server_cert``. +- The firewall applies one decision per state: after an ``accept:hook`` + rule matches at a state, further rules at the same state are not + evaluated. Remapping two old rules onto one new state therefore leaves + the second one dead - keep at most one rule per state. +- The TLS event carrying the ``ja3``/``ja3s`` fingerprint fields is a + session level event: it flushes as soon as everything it logs is + final, that is once a certificate of the state progression has + decoded (or the progression advanced beyond the certificate phase, + showing it carries none) and, when the server requested a client + certificate, the same for the client certificate. While the server + flight has not yet + revealed whether a client certificate is required - the + CertificateRequest normally rides after the server Certificate - the + event waits for that to be decided (a message at or past the + CertificateRequest position, a change cipher spec, the first + application data record, or the close of the handshake; a TLS 1.3 + flight is encrypted past the ServerHello, so application data is + its mid-flow signal), so an mTLS client certificate and its decode + failures are still included, as they were when the old logger flushed + at server flight end. A session that never reaches that point + publishes the event at connection close. Its packet attribution + matches the old model's, because the old logger gated on the same + certificate material. +- TLS handshake records often share a TCP segment (e.g. Certificate, + CertificateRequest and ServerHelloDone in one flight). Every state + walked over by such a packet is evaluated in order, but the packet + decision renders at the state the progress reaches: rules of the + earlier states in the segment can only accept weakly (mid-walk). +- The state names are also exposed by the ``app-layer-state`` keyword + and in the ``ts_progress``/``tc_progress`` output fields. The name + table changed with this rename: rules using ``app-layer-state`` + against the old TLS state names no longer match, and consumers + filtering on the progress fields see the new names. +- SSLv2: a v2 client hello moves the client-side progression to + ``client_hello``, a v2 server hello moves the server-side + progression to ``server_hello``, and a v2 client certificate message + moves the client-side progression to ``client_cert`` (the v2 + certificate payload itself is not extracted, so cert keywords stay + empty there). The old + ``server_cert_done`` and ``server_hello_done`` milestones are gone. + TLS records embedded in a v2 message are parsed by the regular v3 + state machine. A rule keyed on the post-hello v2 flight belongs to + the state the progression is in when it arrives. +- The per-direction state progressions are monotonic: once a + progression reaches ``*_data`` or ``*_finished``, a renegotiated + handshake does not re-enter the hello/cert phases on that + progression; a second ClientHello does not + re-evaluate the ``client_hello`` rules or re-expose the SNI. +- A ruleset that accepted the pre-hello state and then matched the SNI + on the completing record now drops the first fragment of every + fragmented hello: the fragment is decided at ``client_hello`` (not at + the pre-hello state as before), it carries no data to match, and an + unmatched packet falls on the implicit default policy (drop in + firewall mode). If you accept fragmented handshake messages today, + add a plain ``accept:hook tls:client_hello`` / ``tls:server_hello`` + rule for the fragments; only the SNI/other-data rules keep their + content match. + +Migration steps +--------------- + +1. **Rules.** Search the ruleset for the old names + (``client_in_progress``, ``client_hello_done``, ``client_cert_done``, + ``client_handshake_done``, ``server_in_progress``, ``server_hello_done``, + ``server_cert_done``, ``server_handshake_done``) and update them to + the new names. A rule with an unknown state fails to load with a + ``does not support hook`` error and is **not enforced**, so check the + load output after updating. +2. **Firewall config keys.** Config keys use the **dash** form + (``firewall.policies.app.tls.client-hello-done``); rules use the + underscore form (``tls:client_hello_done``). A key that no longer + matches a state is reported at load: a key known to be a renamed + state is an error naming the replacement (fatal when + ``init-failure-fatal`` is set), other unknown keys are a warning + and are ignored. + Update the keys, or the state falls back to the implicit default + policy (drop in firewall mode). +3. **Monitoring.** The ``ts_progress`` field in alert events now carries + the new state names; update any correlation on the old values. diff --git a/doc/userguide/upgrade.rst b/doc/userguide/upgrade.rst index e7013b1641..a6426b73cc 100644 --- a/doc/userguide/upgrade.rst +++ b/doc/userguide/upgrade.rst @@ -50,6 +50,45 @@ Major Changes ``app-layer.protocols.pgsql`` section was absent from suricata.yaml, the parser would be enabled. It's now disabled by default. Simply enabling its EVE output will no longer suffice, either. +- The TLS state names used in firewall rules (``accept:hook + tls:``, ``alert tls:``) and in the + ``firewall.policies.app.tls`` config keys changed from completion + milestones to active phases (e.g. ``client_hello_done`` becomes + ``client_hello``, ``client_in_progress`` becomes ``client_started``). + A rule with an old state name fails to load and is **not enforced**, + and a config key that matches a renamed state is reported at load + and aborts init when ``engine.init-failure-fatal`` is set (a warning + otherwise). The certificate phase states (``client_cert``, + ``server_cert``) are entered when the hello completes, before any + certificate data exists: migrated certificate content rules must be + paired with a plain accept at the cert state (or a drop-based cert + check plus that accept). See + :doc:`firewall/tls-state-migration` for the old-to-new mapping. +- The firewall's per-packet evaluation of app-layer states is now + bounded by the state the packet's parse progress reaches. This + applies to every app-layer protocol, not only the renamed TLS + states: a state above the packet's current progress can no longer + decide the packet (for example, a default policy configured at a + later state no longer drops a packet that stops at an earlier + state), and on a rule no-match the states between the last rule and + the packet's progress are now evaluated. Existing rulesets that + relied on a policy at a later state to drop packets in an earlier + state need to move that policy to the state the packet actually + reaches. +- A ClientHello that carries no SNI extension is decided at the + certificate state (the state the hello-completing record reaches) + instead of at the hello-completion milestone: it matches no + ``tls.sni`` rule (the buffer only exists once the message is fully + parsed) and, without an accept at the certificate state, falls on the + implicit default policy (drop:flow) - the same drop the base release + gave it at the milestone. A plain accept at the certificate state + (the remedy for migrated certificate rules) changes that: it also + decides the hello-completing record, so a SNI-less hello is accepted + by it. If your ruleset accepts SNI-less hellos, that is a relaxation + over the base behaviour - pair the certificate-state accept with an + explicit rule that catches the SNI-less case if you want to keep the + drop (the reference SNI example in :doc:`firewall/firewall-example` + keeps the drop by having no certificate-state accept). Logging Changes ~~~~~~~~~~~~~~~ @@ -72,6 +111,11 @@ Logging Changes under ``properties`` object, instead of as ``{key: value}`` pairs as a part of the ``properties`` object itself. +- The ``ts_progress``/``tc_progress`` values in alerts for the renamed + TLS firewall states now use the new phase names (e.g. + ``client_hello_done`` is now ``client_hello``). See + :doc:`firewall/tls-state-migration` for the old-to-new mapping. + Removals ~~~~~~~~