Split into 2 sub-states:
- stream, which has the "HTTP" requests and responses, including DOH2
- global, which has the settings and other global or control handling
Introduce a simpler progress tracking for the global sub state:
- HTTP2ProgGlobalStart and HTTP2ProgGlobalComplete.
The stream sub state uses the same state machine as before.
Ticket: #8386.
Support for per transaction sub states: different state machines per
transaction type.
Skip engines not belonging to our substate.
Store tx_type in DetectTransaction.
Each protocol supporting sub states will register states from 1 and up.
To support:
Ticket: #8386.
Allow registration of per substate progress name mappings.
Implement logic for getting sub-state names, id's.
Add transaction type and end of progress values to AppLayerTxData.
Introduce helpers to keep logic clean.
Track the transaction id for each queued SMTP command so replies can update the
transaction that created the command instead of always using the current
transaction.
Ticket: #8393
Add directionality to completion states, and replace tx->done by checking for
both directions being complete.
This means that the transaction is now not complete until the server responds
to the clients of data marker, previously the tx was completed when the client
send end of data without waiting for the server response.
This keeps smtp:response_complete from being exposed before the server response
is parsed.
Ticket: #8393
Add minimal SMTP progress states to support envelope validation before
moving to data.
Update SMTP, file and email keywords to hook into the appropriate
states.
Purposefully kept minimal for now as to not break the current idea of an
SMTP transaction, which is probably not ideal for firewall mode.
Ticket: #8393
Extend the app-layer-protocol keyword to accept a pipe-separated list of
protocol values, so a single rule can match any of several protocols:
app-layer-protocol:[!]<proto1>|<proto2>[|...][,<qualifier>]...;
A non-negated list matches when the flow's protocol equals any listed value
(OR); a negated list matches when it equals none of them (NOR). The
single-value form and the trailing mode qualifier are unchanged.
Matching keeps the historical AppProtoEquals() equivalences by default
(dns/doh2, http/http1/http2, dcerpc/smb, ...). An `exact` qualifier selects
strict identity matching with no equivalences and no http umbrella; it
combines with a direction mode in any order. Because a flow is never the
generic ALPROTO_HTTP, `http,exact` is rejected at load.
Values are expanded once at rule load into an effective match-set bitmask, so
the per-packet match is a single bitmask test. Single-value rules remain
prefilterable; multi-value rules are excluded from prefiltering and an
explicit prefilter on them is rejected. Conflicting keyword combinations
(duplicate or overlapping negations, mixed positive/negated) are rejected at
load. Engine-analysis reports the effective match set.
Ticket: 7705
Ticket: 8725
So that multiple HTTP2 DATA frames with EndOfStream flag set,
do not make the buffer grow, while processing it each time,
resulting in quadratic complexity
Ticket: 8629
When we are in async-oneside mode, we see only one direction
of the traffic, and should not wait for the other direction
before cleaning up a transaction.
new_tx() now refuses to create a transaction when the list is already at
SMB_MAX_TX, returning None instead of a transaction. Every creation path --
the new_*_tx helpers and their callers across smb1/smb2/dcerpc/session/
files/ioctl -- propagates that, so no single input can create more than the
limit, including a compound SMB2 request that chains many PDUs in one
record. When the list is full the parser puts the flow into an error state
and stops processing it.
This replaces the previous force-completion of old transactions, which did
not reliably bound the list and could leave transactions unreclaimable on
asymmetric flows. The now-unused tx_index_completed bookkeeping is removed.
Issue: 8629
Ticket: 8694
Otherwise, a flow full of small compression bombs is too slow
to process.
When the threshold is reached, decompression is skipped for the
rest of the flow.
Ticket: 8649
Fully resets all the fields before tackling an ecapsulated message
to avoid evasion, due to the encoding of the upper file
leaking into the next one...
Ticket: 8592
Fixes: 5ddd808e9b ("ftp: don't halt the flow when raising
too_many_transactions")
In the case we receive a big chunk of TCP data, we end up
creating much more than max-tx transactions, and have
quadratic complexity on this packet, even if all
these transactions get cleaned up at the end of the processing
of this packet.
Shortly after minimizing the new authors check to one workflow, github
released an action update that required more permissions due to an
attack scenario, which didn't really apply to us as we didn't run code
from the remotes fork.
However, to avoid extending permissions, rework the authors check to
pull the OISF repo, checkout the PR fork as history, then do the new
authors check. This safely allows the check to run in the context of our
repo without the fork being able to inject code execution.
EBPFLoadFile() unconditionally raised RLIMIT_MEMLOCK to infinity and
aborted the eBPF/XDP load if the call failed.
Since Linux 5.11 BPF map memory is memcg-accounted and is no longer
charged against RLIMIT_MEMLOCK (https://lwn.net/Articles/829307/), so
raising the limit is unnecessary on those kernels.
Ticket: 8719
Body lengths were stored as u16, so a SIP body of 65536 bytes truncated
body_len to 0 and the RequestBody/ResponseBody frame was never created,
letting body content evade inspection. Widen the framing fields to u32.
Ticket #8582
dcerpc parser creates a new tx with id 0 and compensates for the 1 based
index handling throughout the code by overriding that value in a trait
implementation. Make this consistent with other applayer parsers.
Task 8720
Fix Flow Manager error when flows were not being timed out
in case flow.hash-size < 10.
This happened because calculation of variable rows_per_sec in
flow-manager.c rounded the value to zero.
This commit ensures that variable rows_per_sec is at least 1.
Ticket: 8710
Tx ID handling did not take the required + 1 into account.
From a report:
RDP can skip cleanup because its id convention does not match the
generic Rust iterator. The generic iterator in applayer.rs returns
tx.id() - 1, and cleanup trusts that id when calling StateTransactionFree
in app-layer-parser.c. RDP registers that iterator in rdp.rs, but
RdpTransaction::id() returns the stored id unchanged in rdp.rs, while
free_tx also compares against the raw stored id in rdp.rs. For a single
freeable RDP tx with stored id 1, the iterator returns C id 0; cleanup
calls free_tx(0), nothing is removed, then has_next == false allows
min_id to advance to total_txs in app-layer-parser.c. That leaves the
tx live but now below min_id, so later cleanup will not revisit it.
This patch brings the handling in line with the other parsers.
Bug: #8717.
In firewall mode, a accept:hook or accept:tx needs to lead to a accept
packet when the action is applied to the last TX. For this the code
relied of the `DetectTransaction::is_last` field, where the assumption
was that there would always be an inspection on the last transaction.
This assumption was wrong however, as transactions can be skipped for a
few reasons: not updated, fully inspected, unidirectional for the other
direction. This would cause the accept not be applied to the packet,
leading to a default drop.
The reason this wasn't noticed before is that until now the work had
focused on protocols that used sequential transactions (http/tls),
and/or short lived sequential unidir transactions (dns)
This patch addresses the issue by making a simple assumption: if the
last available transaction in the main detection loop is skipped, it
means it has been accepted before. Therefore we can apply the "accept
packet" logic in this case.
Bug: #8698.
Fix how error messages are printed in a case error occurs in the ebpf
flow table code. The log flag g_flowv4_ok was inverted,
so the error messages were never printed.
The fix follows g_flowv6_ok flag code structure.
Ticket: 8704
Victor Julien
Shivani Bhardwaj
Jason Ish
Philippe Antoine
Yash Datre
Jeff Lucovsky
Amir Boussejra
Giuseppe Longo
Adam Kiripolsky
dependabot[bot]