Split into 2 sub-states:
- stream, which has the "HTTP" requests and responses, including DOH2
- global, which has the settings and other global or control handling
Introduce a simpler progress tracking for the global sub state:
- HTTP2ProgGlobalStart and HTTP2ProgGlobalComplete.
The stream sub state uses the same state machine as before.
Ticket: #8386.
Support for per transaction sub states: different state machines per
transaction type.
Skip engines not belonging to our substate.
Store tx_type in DetectTransaction.
Each protocol supporting sub states will register states from 1 and up.
To support:
Ticket: #8386.
Allow registration of per substate progress name mappings.
Implement logic for getting sub-state names, id's.
Add transaction type and end of progress values to AppLayerTxData.
Introduce helpers to keep logic clean.
Track the transaction id for each queued SMTP command so replies can update the
transaction that created the command instead of always using the current
transaction.
Ticket: #8393
Add directionality to completion states, and replace tx->done by checking for
both directions being complete.
This means that the transaction is now not complete until the server responds
to the clients of data marker, previously the tx was completed when the client
send end of data without waiting for the server response.
This keeps smtp:response_complete from being exposed before the server response
is parsed.
Ticket: #8393
Add minimal SMTP progress states to support envelope validation before
moving to data.
Update SMTP, file and email keywords to hook into the appropriate
states.
Purposefully kept minimal for now as to not break the current idea of an
SMTP transaction, which is probably not ideal for firewall mode.
Ticket: #8393
Extend the app-layer-protocol keyword to accept a pipe-separated list of
protocol values, so a single rule can match any of several protocols:
app-layer-protocol:[!]<proto1>|<proto2>[|...][,<qualifier>]...;
A non-negated list matches when the flow's protocol equals any listed value
(OR); a negated list matches when it equals none of them (NOR). The
single-value form and the trailing mode qualifier are unchanged.
Matching keeps the historical AppProtoEquals() equivalences by default
(dns/doh2, http/http1/http2, dcerpc/smb, ...). An `exact` qualifier selects
strict identity matching with no equivalences and no http umbrella; it
combines with a direction mode in any order. Because a flow is never the
generic ALPROTO_HTTP, `http,exact` is rejected at load.
Values are expanded once at rule load into an effective match-set bitmask, so
the per-packet match is a single bitmask test. Single-value rules remain
prefilterable; multi-value rules are excluded from prefiltering and an
explicit prefilter on them is rejected. Conflicting keyword combinations
(duplicate or overlapping negations, mixed positive/negated) are rejected at
load. Engine-analysis reports the effective match set.
Ticket: 7705
Ticket: 8629
When we are in async-oneside mode, we see only one direction
of the traffic, and should not wait for the other direction
before cleaning up a transaction.
Ticket: 8694
Otherwise, a flow full of small compression bombs is too slow
to process.
When the threshold is reached, decompression is skipped for the
rest of the flow.
Ticket: 8592
Fixes: 5ddd808e9b ("ftp: don't halt the flow when raising
too_many_transactions")
In the case we receive a big chunk of TCP data, we end up
creating much more than max-tx transactions, and have
quadratic complexity on this packet, even if all
these transactions get cleaned up at the end of the processing
of this packet.
EBPFLoadFile() unconditionally raised RLIMIT_MEMLOCK to infinity and
aborted the eBPF/XDP load if the call failed.
Since Linux 5.11 BPF map memory is memcg-accounted and is no longer
charged against RLIMIT_MEMLOCK (https://lwn.net/Articles/829307/), so
raising the limit is unnecessary on those kernels.
Ticket: 8719
Fix Flow Manager error when flows were not being timed out
in case flow.hash-size < 10.
This happened because calculation of variable rows_per_sec in
flow-manager.c rounded the value to zero.
This commit ensures that variable rows_per_sec is at least 1.
Ticket: 8710
In firewall mode, a accept:hook or accept:tx needs to lead to a accept
packet when the action is applied to the last TX. For this the code
relied of the `DetectTransaction::is_last` field, where the assumption
was that there would always be an inspection on the last transaction.
This assumption was wrong however, as transactions can be skipped for a
few reasons: not updated, fully inspected, unidirectional for the other
direction. This would cause the accept not be applied to the packet,
leading to a default drop.
The reason this wasn't noticed before is that until now the work had
focused on protocols that used sequential transactions (http/tls),
and/or short lived sequential unidir transactions (dns)
This patch addresses the issue by making a simple assumption: if the
last available transaction in the main detection loop is skipped, it
means it has been accepted before. Therefore we can apply the "accept
packet" logic in this case.
Bug: #8698.
Fix how error messages are printed in a case error occurs in the ebpf
flow table code. The log flag g_flowv4_ok was inverted,
so the error messages were never printed.
The fix follows g_flowv6_ok flag code structure.
Ticket: 8704
buffer_type_id is a u32
We have a list per buffer of a list of AppProto
But the list of AppProto can grow maximum for the case of file.data
to all the alprotos supporting files.
Ticket: 8001
In almost every case, if there is a alstate there is also a pstate. So
remove the conditional pstate handling, and replace it by
unconditionally using the pointer. Add debug validation to make sure the
assumption is and stays correct.
Explicitly handle the one exception in AppLayerParserParse, which
follows an error before the pstate is allocated, or when pstate
allocation itself fails.
Completes commit 7e725c650d
autofp-scheduler with value ftp-hash ends up using
FlowGetIpPairProtoHash which ignores the ports for ftp-looking
flows so that the ftp and ftp-data flow get processed by the
same thread.
As for the other cases, we want to use every other parameter
to compute the flow hash, inclusing the live device
Victor Julien
Shivani Bhardwaj
Jason Ish
Philippe Antoine
Yash Datre
Amir Boussejra
Adam Kiripolsky
Philippe Antoine