DetectEngineInspectGenericList returned NO_MATCH for every keyword miss,
also when the tx had moved past the phase the engine is registered at.
Unlike the buffer engines it never produced a terminal no match, so a
firewall LTE rule using such a keyword (tls.version on a hook) stayed
provisional and was accepted for the rest of the session.
Make a miss terminal at P + 1. This engine also backs the lua
on-request/on-response hooks and the tls_cert_*, dnp3/dnp3_ind and
ftpdata/ftp mode keywords, so a plain IDS rule using them keeps its
verdict once the tx has moved past the registered phase.
Export the parser's end state computation as AppLayerParserGetTxEndState()
here, so the version keyword can use it; the remaining call sites are
converted later.
Ticket: #8944.
The TLS state rename changed the name tables the rule parser and the
firewall policy config resolve against, and the state ids double as
the ts_progress values in alert events, but no unit test pinned the
tables themselves. Only the downstream sv behavior did.
Add unit tests asserting the current phase names resolve to their
expected ids in both directions (and back), that the pre-phase
milestone names no longer resolve at all, and that the completion
values keep the 5-value tracks.
Also pin the decode-failure suppression at the parser level: a
failed hello does not advance, a clean hello does (the suppression
is per message), and a failed certificate keeps the track out of
the cert phase. The tests allocate and free the full state through
the parser lifecycle functions instead of manual partial cleanup.
A certificate message whose first certificate is undecodable but
whose rest is valid is pinned as well: the bad leaf is skipped
(the valid certificate is kept in the chain and the certificate
data comes from it, compared against a direct extraction from the
fixture) and the track stays out of the cert phase. The
incomplete-data condition is pinned as sticky across a following
handshake message: the per-message suppression resets, the
condition does not, and the track stays out of the cert and data
phases.
Ticket: #9047.
The server states were completion milestones: server_hello landed
when the server hello record was fully parsed and server_cert_done
when the certificate message was done, so the states named moments
at which the phase data was complete instead of the phase being
processed. The flight-completion state (server_hello_done) named
a moment at which no data is published and no decision can act.
The states now denote the record phase being processed:
server_hello is entered when the server hello message begins
(its header is read once, the message may span several
records) and server_cert when the certificate message begins,
so the state names the message being parsed and a message that
fails to decode leaves the track in the state that names it.
The record that completes a hello hands the track over to the
certificate phase before a single certificate byte arrives. The
enter/leave is the shared handshake phase mechanism, so the client
track follows: client_hello is entered at the start of the client
hello as well, and the completing client hello is decided at
client_cert.
The flight-done state is removed and renumbers the tail
(server_data, server_finished), leaving both tracks with the
same five-phase shape and an absorbing finished state.
Hello and certificate decode failures suppress the phase advance
of their own message only: the decoders keep their pre-existing
return contract (the message bytes are consumed and the error
events are raised as before) and set a per-connp suppression flag
that is reset at the start of the next handshake message and
checked at the phase advance. A decode failure is not an app-layer
parse error: the flow is neither dropped nor blinded, and the next
message that decodes cleanly still advances.
An undecodable certificate in a chain is skipped, not fatal: one
decoder event is raised for the bad certificate and the remaining
certificates are still decoded - the rest of the chain is kept and
the certificate data is filled from the first fully decodable
certificate. The certificate phase is suppressed for such a
message (the phase data is incomplete). The incomplete-data
condition is sticky per connp: it is set while the certificate
message is being decoded, cleared by a fully decoded certificate
message only, and consulted by the handshake-done heuristic that
would otherwise establish the data phase from the certificate
data - a later handshake message does not complete the data, so
the tracks do not cross the cert phase on partial data.
The certificate field extraction is commit-on-success: the fields
are extracted into stashes and published to the connp only when
the whole certificate yields every field, so a certificate that
fails a late extraction leaves no partial data behind and the next
certificate can fill the fields. An extraction failure raises its decoder event once per message: the message no longer aborts at the first failing certificate, and the event is not repeated for the later failing entries.
The tls.cert_chain_len match is gated on the chain being final (the
per-connp flag set by a fully decoded certificate message), so it
does not match while the track sits in server_cert with a chain that
is still arriving.
TOCLIENT keyword registrations and the tc log threshold follow the
renamed constants (values 1/2 unchanged; the threshold is
triggered by the same writes as before, so the log fires on the
same packets). The ja3s lua registration moves to server_data: lua keeps a numeric
hook, so it must follow the renamed state the old hook value pointed
at (server hello done was 3, server data is 3). The ja3s content is
final once the ServerHello parses, which is why the C keywords
(ja3s.string, ja3s.hash) stay registered at server_hello. Two string-based
generic buffer lookups and the user guide follow the renamed
states.
Ticket: #9047.
The client states were completion milestones: client_hello_done
landed only when the hello record was fully parsed, so during the
in-hello window (possibly multiple packets) the flow sat in
client_in_progress and no firewall state named the phase being
processed.
The states now denote the record phase being processed:
client_hello is entered when the hello message is fully parsed
(it may span several records) and client_cert when the
certificate message is fully parsed, so the phase data (SNI,
certificates) is available when its state is entered; a message
that fails to parse does not advance the track. The milestone
writes are removed and the name table, TOSERVER keyword
registrations (values unchanged) and the firewall user guide
follow the new names.
Ticket: #9047.
In TLS 1.2 and 1.3 the Finished message precedes the application
data on both directions. The DATA milestone write is keyed on the
parsed certificate subjects and runs on every record, so the first
record after the handshake - for a session that ends with a fatal
alert, the alert itself - moved the tracks backwards (client 4->3,
server 5->4). The completion states were therefore not absorbing:
the finished hook windows were transient in decryption mode, and a
keyword registered at progress >= 3 would have its eof revoked and
be suppressed by the stored CANT_MATCH flag.
The setters now only move a state forward, so post-terminal writes
are no-ops and the finished states absorb.
Ticket: #9047.
SSLv2Decode() consumed six fixed CLIENT_HELLO body bytes (version +
cipher_spec_length + session_id_length) guarded only by input_len >= 6,
which checks the buffer remainder but not the record itself. When an
attacker sends a record whose declared record_length is < 7 (e.g. 1),
bytes_processed advances past record_lengths_length + record_length,
leading an integer underflow.
Address by adding a record_length >= 7 check to the CLIENT_HELLO case.
Also add another check: bail if bytes_processed > record_length +
record_lengths_length.
Ticket: #8853.
TLS parsers use x509-parser crate which parses X.509 certificates that
use ASN.1 DER encoding that can allow arbitrary byte sequences. An
attacker could inject null byte in a certificate anywhere to stump the
common language parsers terminating the string at a null byte leading to
a bypass of a possibly malicious certificate.
So far, the rust TLS parser for "subjectaltname" used a pattern that involved:
-> Get ASN.1 DER encoded raw data from the x509-parser crate
-> Convert this raw data to a decoded string (Rust)
-> Convert the Rust string to CString
-- The problem lies here. CString only accepts proper strings/byte
buffers and converts it into an owned C-compatible, null-terminated
string. However, if any null byte occurs in the string passed to the
CString then it panics.
In the rust TLS parser, this panic is handled by returning NULL.
This means that the parser will error out during the decoding of the
certificate. However, Suricata must be able to detect the null byte
injection attack being an IDS/IPS.
Hence, replace all such string patterns w.r.t. TLS SAN with a byte
array.
Bug 7887
TLS parsers use x509-parser crate which parses X.509 certificates that
use ASN.1 DER encoding that can allow arbitrary byte sequences. An
attacker could inject null byte in a certificate anywhere to stump the
common language parsers terminating the string at a null byte leading to
a bypass of a possibly malicious certificate.
So far, the rust TLS parser for "issuerdn" used a pattern that involved:
-> Get ASN.1 DER encoded raw data from the x509-parser crate
-> Convert this raw data to a decoded string (Rust)
-> Convert the Rust string to CString
-- The problem lies here. CString only accepts proper strings/byte
buffers and converts it into an owned C-compatible, null-terminated
string. However, if any null byte occurs in the string passed to the
CString then it panics.
In the rust TLS parser, this panic is handled by returning NULL.
This means that the parser will error out during the decoding of the
certificate. However, Suricata must be able to detect the null byte
injection attack being an IDS/IPS.
Hence, replace all such string patterns w.r.t. TLS IssuerDN with a byte
array.
Bug 7887
TLS parsers use x509-parser crate which parses X.509 certificates that
use ASN.1 DER encoding that can allow arbitrary byte sequences. An
attacker could inject null byte in a certificate anywhere to stump the
common language parsers terminating the string at a null byte leading to
a bypass of a possibly malicious certificate.
So far, the rust TLS parser for "Subject" used a pattern that involved:
-> Get ASN.1 DER encoded raw data from the x509-parser crate
-> Convert this raw data to a decoded string (Rust)
-> Convert the Rust string to CString
-- The problem lies here. CString only accepts proper strings/byte
buffers and converts it into an owned C-compatible, null-terminated
string. However, if any null byte occurs in the string passed to the
CString then it panics.
In the rust TLS parser, this panic is handled by returning NULL.
This means that the parser will error out during the decoding of the
certificate. However, Suricata must be able to detect the null byte
injection attack being an IDS/IPS.
Hence, replace all such string patterns w.r.t. TLS Subject with a byte
array.
Bug 7887
SSL connp maintains all the state and certificate data that was
parsed/decoded successfully and it must retain that for later usage.
There should be just one place to free this object which is SSLStateFree
for both the directions. By freeing the connp data during parsing error,
there is room for memory errors.
This works so far because the field parsed after this cannot error out so
if there's an error parsing this, it anyway does not exist. However, this
is incorrect and leaves scope for mistakes.
Remove this extra free and treat SAN like all other TLS keywords.
Bug 7996
The functions around TriggerRawStreamReassembly are misnomers in the
current layout of the code. The functions were named appropriately when
they were created as per the structural and logical layout of the code
at the time.
These functions in today's code are being used to track, update and
trigger progress of inspection in the raw stream only. Hence, rename them
to TriggerRawStreamInspection.
Ticket: 6695
"server_handshake" which logs the following:
1. TLS version used during handshake
2. The chosen cipher suite, excluding GREASE
3. TLS extensions, excluding GREASE
This commit is designed in preparation of enabling the handshake object
to log it's own contents rather than being done on the C side.
Moving the tls versions functionality to rust has a couple of uses:
1. Allows both rust and C side to use these fields
2. Moves more of the tls related logic to rust
3. C side can still use these values because of cbindgen
Ticket: 6695
With the introduction of the HandshakeParams object we're able to
utilise the theory further by using it as the object to track the ALPNs.
The HandshakeParams object is now responsible for holding all ALPNS. The
user of this HandshakeParams object i.e. JA4, can use whichever fields
are needed. So only when we generate a JA4 hash do we use the first ALPN
and require to format it. Other users of HandshakeParams may opt to use
all ALPN's i.e. during TlsAlpnGetData().
Ticket: 6695
This introduction splits the use of the handshake parameters into their
own object, HandshakeParams, which is populated by the TLS decoder. The
JA4 object is now very simple. It's a simple String object (the JA4
Hash) which is generated during new().
This introduction is part of a larger idea, which is to enable
outputting these raw parameters without JA3/JA4. These handshake
parameters are the components used to generate the JA4 hash, thus it
makes sense for it to be a user of HandshakeParams.
The default value for TLS encryption is generic and not precise.
The new keyword, "track-only" substitues the old keyword, being
more on point what action is actually taken when the TLS flow
reaches an encrypted state.
If not all data is ACK'd during the FIN session shutdown, the last calls
to the parser can be with a non-NULL data pointer, but a input length of
0. This wasn't considered by the EOF check, which then lead to it being
seen as an error. No event was raised, but the tls error stats were
incremented.
Bug: #7554.
To optimize detection, and logging, to avoid going through
all the live transactions when only a few were modified.
Two boolean fields are added to the tx data: updated_tc and ts
The app-layer parsers are now responsible to set these when
needed, and the logging and detection uses them to skip
transactions that were not updated.
There may some more optimization remaining by when we set
both updated_tc and updated_ts in functions returning
a mutable transaction, by checking if all the callers
are called in one direction only (request or response)
Ticket: 7087
Introduce a common function for mapping names to IDs that performs
bounds checking.
Note: For event IDs in the enum that are larger than a uint8_t, -1
will be returned instead of -4. -4 has special meaning during
signature parsin that means requirements were not met. -4 has no
special handling prior to requirements, or the meaning has been lost.
Victor Julien
Philippe Antoine
Shivani Bhardwaj
Richard McConnell
Jason Ish
Lukas Sismis