Split into 2 sub-states:
- stream, which has the "HTTP" requests and responses, including DOH2
- global, which has the settings and other global or control handling
Introduce a simpler progress tracking for the global sub state:
- HTTP2ProgGlobalStart and HTTP2ProgGlobalComplete.
The stream sub state uses the same state machine as before.
Ticket: #8386.
Support for per transaction sub states: different state machines per
transaction type.
Skip engines not belonging to our substate.
Store tx_type in DetectTransaction.
Each protocol supporting sub states will register states from 1 and up.
To support:
Ticket: #8386.
Allow registration of per substate progress name mappings.
Implement logic for getting sub-state names, id's.
Add transaction type and end of progress values to AppLayerTxData.
Introduce helpers to keep logic clean.
Ticket: 8725
So that multiple HTTP2 DATA frames with EndOfStream flag set,
do not make the buffer grow, while processing it each time,
resulting in quadratic complexity
Ticket: 8629
When we are in async-oneside mode, we see only one direction
of the traffic, and should not wait for the other direction
before cleaning up a transaction.
new_tx() now refuses to create a transaction when the list is already at
SMB_MAX_TX, returning None instead of a transaction. Every creation path --
the new_*_tx helpers and their callers across smb1/smb2/dcerpc/session/
files/ioctl -- propagates that, so no single input can create more than the
limit, including a compound SMB2 request that chains many PDUs in one
record. When the list is full the parser puts the flow into an error state
and stops processing it.
This replaces the previous force-completion of old transactions, which did
not reliably bound the list and could leave transactions unreclaimable on
asymmetric flows. The now-unused tx_index_completed bookkeeping is removed.
Issue: 8629
Ticket: 8694
Otherwise, a flow full of small compression bombs is too slow
to process.
When the threshold is reached, decompression is skipped for the
rest of the flow.
Ticket: 8649
Fully resets all the fields before tackling an ecapsulated message
to avoid evasion, due to the encoding of the upper file
leaking into the next one...
Body lengths were stored as u16, so a SIP body of 65536 bytes truncated
body_len to 0 and the RequestBody/ResponseBody frame was never created,
letting body content evade inspection. Widen the framing fields to u32.
Ticket #8582
dcerpc parser creates a new tx with id 0 and compensates for the 1 based
index handling throughout the code by overriding that value in a trait
implementation. Make this consistent with other applayer parsers.
Task 8720
Tx ID handling did not take the required + 1 into account.
From a report:
RDP can skip cleanup because its id convention does not match the
generic Rust iterator. The generic iterator in applayer.rs returns
tx.id() - 1, and cleanup trusts that id when calling StateTransactionFree
in app-layer-parser.c. RDP registers that iterator in rdp.rs, but
RdpTransaction::id() returns the stored id unchanged in rdp.rs, while
free_tx also compares against the raw stored id in rdp.rs. For a single
freeable RDP tx with stored id 1, the iterator returns C id 0; cleanup
calls free_tx(0), nothing is removed, then has_next == false allows
min_id to advance to total_txs in app-layer-parser.c. That leaves the
tx live but now below min_id, so later cleanup will not revisit it.
This patch brings the handling in line with the other parsers.
Bug: #8717.
Mark the direction into RDP transactions at creation time,
so the tx carries SKIP_INSPECT for the direction it is
never seen in, matching DHCP and the other single-direction
parsers. This lets cleanup free completed transactions and
stops a tx from being inspected (and alerting) twice, once per
direction.
RDP bounds its transactions to connection setup and stops
parsing once bypass_parsing is set.
Issue: 8621
DHCP is a stateless parser where each datagram is its own standalone,
single-direction transaction. It was creating transactions with
AppLayerTxData::new(), which leaves both SKIP_INSPECT bits clear, so the
engine treats every tx as still needing inspection in both directions.
For a flow that only ever carries one direction (broadcast DHCP, or a
relay seeing one side), the never-observed direction's inspect bit can
never be set, so AppLayerParserTransactionsCleanup() never frees the tx.
The per-flow transaction Vec then grows without bound and every packet
re-scans the whole list, giving O(n^2) CPU and unbounded memory on a
busy DHCP aggregation point.
Use AppLayerTxData::for_direction() like every other stateless parser
(DNS, SNMP, NTP, IKE, KRB5, MQTT, QUIC, SIP, WebSocket, bittorrent-dht)
so the tx carries SKIP_INSPECT for the direction it will never be seen
in. This lets cleanup free completed transactions and also stops the tx
from being inspected (and alerting) twice, once per direction.
Issue: 8621
Ticket: 8659
If a client sends a SOTR/RETR command before doing a PASV/PORT
command, ther server may reply
425 Use PORT or PASV first.\r\n
and allow the client to continue sendinf other commands
So, Suricata should not put the whole flow into error, it just
sets an event, and continues to parse further
Extend the xor transform to accept a variable-position key using
the 'extract <nbytes> <offset>' syntax. The engine reads <nbytes> bytes
at buffer position <offset> at transform time.
The optional 'offset' parameter specifies where XOR decoding
starts in the buffer; bytes before that position are left unchanged.
Syntax:
xor:"<hex_key>"
xor:extract <nbytes> <offset>
xor:offset <N>,"<hex_key>"
xor:offset <N>,extract <nbytes> <offset>
Variable-key helpers (VariableKeyLocation, variable_key_bytes,
parse_key_location, strip_keyword_prefix) are imported from the new
varkey module.
The variable key bytes are copied out of the inspection buffer before
decoding, so an in-place transform cannot corrupt the key when the key
region overlaps the decoded range.
Issue: 8671
Introduce rust/src/detect/transforms/varkey.rs to centralise helpers
used by transforms that read key material directly from the inspection
buffer at transform time.
Exported items:
VariableKeyLocation { offset: u16, nbytes: u8 }
variable_key_bytes bounds-checked slice into an inspection buffer
strip_keyword_prefix strip 'keyword<ws>' from option strings
parse_key_location parse '<nbytes> <offset>' into VariableKeyLocation
All four items are pub so any transform can import them. Unit tests
are included in the module.
Issue: 8671
SIP and SSDP share method names like NOTIFY and SUBSCRIBE,
causing SSDP traffic to be misidentified as SIP.
Add a probing parser callback that checks for "SIP/" in the
payload before accepting a pattern match.
Example of a misidentified flow before the fix:
{"timestamp":"2014-02-27T19:44:43.164211+0100","flow_id":986757542077835,"event_type":"flow","src_ip":"192.168.1.1","src_port":9489,"dest_ip":"239.255.255.250
","dest_port":1900,"ip_v":4,"proto":"UDP","app_proto":"sip","flow":{"..."}}
After the fix:
{"timestamp":"2014-02-27T19:44:43.164211+0100","flow_id":986757542077835,"event_type":"flow","src_ip":"192.168.1.1","src_port":9489,"dest_ip":"239.255.255.250
","dest_port":1900,"ip_v":4,"proto":"UDP","app_proto":"failed","flow":{"..."}}
Ticket #8355
to deal with the failure due to cbindgen updates and mismatches in
generated bindings.
detect-bytemath.c:61: error: "DETECT_BYTEMATH_ENDIAN_DEFAULT" redefined [-Werror]
61 | #define DETECT_BYTEMATH_ENDIAN_DEFAULT (uint8_t) BigEndian
|
In file included from rust.h:34,
from detect-bytemath.c:32:
./../rust/gen/rust-bindings.h:5071: note: this is the location of the previous definition
5071 | #define DETECT_BYTEMATH_ENDIAN_DEFAULT BigEndian
|
At some point, bindgen include ordering changed such that
AppLayerGetFileState was being bindgen'd as opaque, as the definition of
StreamBufferingConfig was not available when bindgen hit
AppLayerGetFileState, and bindgen processes in order.
Move the util includes before the app-layer includes to fix the ordering
problem, but still keep util includes grouped.
The sys diff is large as many things have been re-ordered.
Per RFC 2132 the BOOTP sname and file fields can hold extra DHCP
options when option 52 is present, but the parser ignored them. After
parsing the main options we now look up option 52 and walk sname or
file as additional option streams, appending what we find to the same
options vector so the logger and detection keywords see the
overloaded values too.
Bug: #8538.
Ticket: 8518
Keywords that work for HTTP2 headers match now as soon as possible
A push promise is now considered like a headers frame with regards
to the progress (no dedicated "reserved" progress/state)
http.protocol and http.stat_msg keywords are now registered at
earliest progress, since these are synthetic like "HTTP/2" and
not really seen on the wire.
http.request_line and http.response_line match only on data,
and not on headers, since we must wait the end of headers
to be sure to have the full line
http2.size_update now matches at headers progress as it should
http2.frametype, http2.errorcode, http2.priority now match like
http2.window, when the tx is complete from both sides, as a
half-closed client may still send priority, rst_stream
or window_update frames
This adds an LLMNR protocol logger that reuses existing DNS functions,
following the same approach as the mDNS logger:
- No grouped logging
- Rdata is logged in a field that is named after the rdata type
- Types are logged in lower case
- Flags are logged as an array
Ticket #8366
This adds a parser for LLMNR protocol over both UDP and TCP.
The parser reuses the existing DNS functions since LLMNR shares
the same wire format as DNS, but implements its own protocol-specific
validation:
- LLMNR-specific flag checks (C, TC, T bits)
- Opcode validation (only standard query opcode 0 is valid)
- An Event is set if Z-flag is set, since it's must be zero per RFC4795
LLMNR transactions inherit DNS behavior where each packet creates a new state
with one transaction.
Ticket #8366
Ticket: 8513
Suricata decides at 2 levels if a http2 flow is doing a compression
bomb.
There is a direct computation when one chunk of TCP data
is being parsed.
In this case, do not take the ratio into account, just use the size
of the decompressed data, so that if we get a big chunk of TCP data
like 1 MiB, and a not so high ratio of 200, we do not trigger
the debug assertion in util-file.c about 64MiB
The other case stays unchanged : when accumulating over the lifetile of
a flow with multiple txs, take into account the compression ratio,
so that a flow of many txs, having a super high (brotli) compression
ratio, ends up classified as a compression bomb.
(For example, having 100 txs each turning a 100 byte input into a 700 KiB
one)
Ticket: 8513
During decompression, fail early if we have a big decompression
ratio, and enough data.
Track this data also during a tx lifetime, and even a flow/state
lifetime, so that we set event and fail also if the compression
bomb is split over multiple packets
Ticket: 7929
Builds a vector of the reason code in a tx to do so,
except if we use the default "any", where we do not append
to the vector, but just run detection while iterating
Other parsers (SMB, POP3, ENIP) already raise a too_many_transactions
event when a flow exceeds its per-flow transaction limit. FTP had
the limit wired up through app-layer.protocols.ftp.max-tx but just
dropped the excess transaction on the floor with a "FTP does not set
events yet..." TODO, so no event was raised when the limit was
reached.
Add the FtpEventTooManyTransactions variant, a stock rule at
sid 2232002, and raise the event from FTPTransactionCreate when the
live-tx count goes past ftp_config_maxtx.
Issue: 8489
When we got a bind without the first fragment flag, and did not
set any_frag in the signature, the signature always matched,
whatever the uuid value
Ticket: 8457
Issue: 7672
The subslice transform creates a slice of the input buffer.
Specify the subslice desired -- nbytes and truncate are optional:
subslice: offset <,nbytes> <,truncate>
offset: Specifies the starting offset for the new subslice. When
negative, expresses how far from the end of the input buffer to begin.
nbytes: Specifies the size of the subslice. When negative, specifies
that the subslice will end that many bytes from the end of the input
buffer. Nbytes must be non-zero.
When nbytes is not specified, the size of the subslice will be the size
of the input buffer - offset.
truncate: Specify behavior when offset + nbytes exceeds buffer length.
When present, trims nbytes such that offset + nbytes equals buffer
length. When not present, an empty buffer is produced.
Examples:
subslice: 1; - The subslice will be a copy of the input
buffer but omits the input buffer's first byte
"This is Suricata" -> "his is Suricata"
subslice: 0, 13; - The slice is created from the first 13 bytes
of the input buffer
"This is Suricata" -> "This is Suric"
subslice: 10, -5; - The subslice is created starting at offset 10
and continues to 5 bytes before the end of the input buffer
"This is Suricata" -> "r"
subslice: -3; - The subslice will be the last 3 bytes of the
input buffer.
"This is Suricata" -> "ata"
Store the NTP reference ID as raw network-order bytes so it can be
exposed as a sticky buffer and matched with payload keywords. The
reference ID is often a 4 character string, or an IP address and not
just an integer identifier.
Updates the log reference ID to be a string of colon separated hex
digits as this matches what tshark does.
Ticket: #8488
NotRequest and NotResponse event type have never been used.
Unsolicited is also unused, however there is a rule for it so removing
support would be a breaking change at this time.
Since snmp txs are created at progress 1 which is the completion,
this does not change detection, except allowing firewall mode
to use these keywords with generic hook like
snmp:request_complete
Ticket: 8395
Ticket: 8395
For protocols where tx are complete at start, that is
the `tx_get_progress` function returns 1 inconditionnally,
we can use the progress 1, not losing time on detection in ips
mode, and getting ready for firewall mode
For the other protocols (where a tx needs both a request and response
for instance), keep progress 0 as was the case before
Wrap the EVE callback handler with a Rust friendly variant that allows
the user to register a callback as a closure which is provided an
already wrapped JsonBuilder object.
Ticket: #8477
Ticket: 8413
RFC 6455 Section 5.4 states
Control frames (see Section 5.5) MAY be injected in the middle of
a fragmented message.
Control frames are identified by opcodes where the most significant
bit of the opcode is 1.
unicode-segmentation (used by rustyline) was updated from 1.12.0 to
1.13.1 earlier and requires a newer version of Rust. Pin to 1.12.0 which
works with Rust 1.75.0.
So that we can log and detect on interface uuids, even if
AlterContext packets change one of the contexts, but other contexts
are still used after it in request/responses
Ticket: 8378
Ticket: 8375
logger already read this variable but it was never set, and
the rust compiler did not warn about this
Allows to log the interface uuid when ctx_id > 0
Wireshark filter for related pcaps
dcerpc.cn_ctx_id > 0 && dcerpc.pkt_type == 0
And consistently use the CARGO_ENV for all cargo calls.
This allows a user to point to rust binaries that are not on their
path just during ./configure, for example:
PATH=/usr/lib/rust-1.89/bin:$PATH ./configure ...
The full path to rustc and rustdoc will now be used, as discovered in
the PATH without the user have to carry the path over to the "make"
command. This was already done for cargo.
Related to ticket: #8381
Ticket: 8289
If stream.reassembly.depth is unlimited,
an attacker controlling the 2 sides of a communication going through Suricata
can send a transition with an infinite number of headers, until suricata OOMs
Solution is to offer a configuration option to bound the number
of HTTP2 frames we store in a HTTP2 transaction, and produce an
anomaly if this bound is crossed
Ticket: 8292
When we have buffered something in ctx.decoded_line,
we already looked for '\n' in it, so we do not need to run it again
Otherwise, callers that supply mime_smtp_find_url_strings with
a few bytes at a time without "\n", have a quadratic
complexity
Victor Julien
Philippe Antoine
Jason Ish
Jeff Lucovsky
Giuseppe Longo
Shivani Bhardwaj
Jeff Lucovsky
William Ling
Samaresh Kumar Singh
Giuseppe Longo
Juliana Fajardini
Philippe Antoine