A sub-state's buffers are final at a progress value the parser knows but
the API did not expose: AppLayerParserGetSubStateCompletion() had the http2
values hard-coded and asserted for every other protocol. Register the value
with the sub-state and resolve it from there.
The per-tx tx_type_eop_ts/tc fields become redundant: the tx type is the
registered sub-state, so AppLayerParserGetTxEndState() returns its
completion. http2 no longer sets the fields.
Ticket: #9149.
A `<hook` rule that is not final yet stays pending while a buffer of its
hook can still grow. Document that a match at a higher hook still decides
the flow, and that the pending rule's later no-match does not apply the
default policy of its own state over it. Cover the sub-state case: the
buffer can grow above the hook after the tx moved past it.
Ticket: #8944.
An LTE (`<hook`) rule that is not final yet returns a partial match, which
is handled as an `accept:hook` by stopping the tx walk. A rule hooked at a
higher state was then never inspected, even though it inspects the buffers
of every state below its hook, so a match there covers the pending rule.
When the transaction reached its end state the pending rule concluded as a
no-match and applied the default policy of its own hook: a flow an explicit
rule accepted was dropped by a policy that belongs to a different rule.
A full `accept:hook` match passes over the accepted hook and the ones below
it and the walk continues with the higher hooks. Do the same for the
pending rule's partial match once the tx has moved past its hook. The
states above the hook are not swept while the rule is pending: they are
decided when it resolves, so a higher state's default cannot preempt the
pending rule's own state. A resolved `accept:hook` match does sweep them;
that difference is deliberate, and a rule above the pending rule that
accepts the flow still ends the walk first. While the tx is still at or
below the hook the walk keeps stopping: the rule may resolve right there,
and a rule above it that contributes no coverage must not run instead.
The pending rule is also not finalized by an engine whose phase the tx
moved past: in a sub-state the buffers can still grow above the hook (a
http2 trailer HEADERS frame updates the header buffer), so its data is only
final once the sub-state completed. The rule stays revisitable until then
and the default policies of the states it covers stay deferred.
TD rules positioned after the pending rule are inspected again, as they are
after a full `accept:hook` match: they can alert or drop where the old walk
never reached them.
Ticket: #8944.
SCFTPFreeResponseLine gated the release of the code buffer on a null
test of the response pointer, so the code buffer had no check of its
own. Neither check could fire: parse_response_line sets both pointers
from Box::into_raw on a boxed slice, which never returns null, and C
code obtains an FTPResponseLine only from SCFTPParseResponseLine. The
comment on the code field claimed the pointer may be null; a line with
no status code gets an empty buffer with code_length 0 instead.
Remove both null checks, note why they are unnecessary, and correct the
field comment.
Ticket: 8803
parse_response_line allocates the code buffer with Box::into_raw on a
boxed slice, but SCFTPFreeResponseLine released it by rebuilding a
Vec<u8> with capacity code_length. Vec::from_raw_parts accepts any
global-allocator pointer whose size and alignment match, and both paths
compute the same layout for a byte slice, so the old free was sound. It
differed only from how the response buffer is released.
Release the code buffer through Box::from_raw on a slice pointer, so
both buffers are freed the same way they were allocated.
Ticket: 8803
IP Only rules are only evaluated once per direction per flow for a
standard rule and twice at the most for a bidirectional rule. Using
threshold or detection_filter with track by_flow does not make much
sense for higher count as the rule may never fire.
Issue a warning for this case in engine analysis mode for now.
Ticket: 9170
For calculating the address space to be matched against, the CIDR
netmask is taken to account by the IP Only Engine. However, this netmask
was not set properly for dotted netmasks. Fix it.
Ticket: 9166
In case a keyword should work in firewall mode, with firewall rules
only. The engine errors out if threat detection rules use the given keyword.
Part of
Ticket #9041
On PcapFileDispatch() failure, pv->current_file was left pointing at freed file
vars, which were then cleaned up again on thread teardown.
Generally unreachable, as no pcap-file pipeline slot fails on real input.
Ticket: #9162
Including decode.h pulls inline functions that use SCLogDebug and SCLogError
into the bindgen input, but util-debug.h hides those macros from bindgen,
breaking "make update-bindings". Only forward declarations are needed.
Action scopes for packet hook default policies were validated per hook
class, so `flow` was accepted for every packet hook. The pre_flow hook is
a special case as it is evaluated before Suricata obtains a flow record.
Firewall rules on the pre_flow hook were not validated at all, so `flow`
and `tx` scopes were accepted there as well.
This commit adds extra validation to scope parsing for the pre_flow
hook default policy and validates pre_flow firewall rules against the
same scopes.
Ticket: 8967
The packet default policy runtime only applied the action scope for
`accept`. For `drop:flow` and `reject:flow` it dropped the packet and left
the flow untouched, so the flow scope had no effect.
Ticket: 8969
Covers the killed-during-initial-pause case (mgmt thread returns so the
main loop runs once; non-mgmt returns false and skips it, THV_PAUSED
cleared on both paths) and the unpause-while-spinning path, driven from
a helper thread.
Ticket: #9168.
Every thread is created paused (THV_PAUSE) and the main thread unpauses
them all before the main loop. A thread that has not yet been scheduled
spins in TmThreadsWaitForUnpause() until it sees the unpause. Under
heavy thread oversubscription (many short offline runs in parallel) a
management thread can be descheduled across the whole run: by the time
it runs again the shutdown has set THV_KILL while it is still in the
initial pause. TmThreadsWaitForUnpause() then returned false, so the
thread skipped its main loop entirely and never did its final shutdown
work - for the stats thread that means the last stats event is never
written to eve.json. This intermittently fails suricata-verify tests
that assert on the final stats.
Mgmt threads (tv->type == TVT_MGMT) now return so the main loop still
runs once: it tests THV_KILL and exits after the pass, but the thread
gets the single pass it needs for final work (e.g. the last stats
output). THV_PAUSED is cleared on that path. RX, worker and other
non-mgmt threads keep the previous behaviour: their loops do not test
THV_KILL at the top - the RX loop exits only on the PktAcqLoop return
value and suricata_ctl_flags, and a pcap-file pass would dispatch
packets into a pipeline that is being torn down.
The log-maintenance thread breaks before its first sleep slice when it
is already being killed: on that pass no rotation or flush can happen
(their wait counts need a full second of slices), so the fixed 500 ms
sleep would be pure shutdown latency - the slice exists to react to
shutdown quickly, and a first wake into a KILL must not wait out a full
slice. A kill arriving after the first slice keeps the normal
behaviour, including the final rotation check.
Ticket: #9168.
Victor Julien
Jason Ish
Jeff Lucovsky
Shivani Bhardwaj
Juliana Fajardini
Philippe Antoine
Lukas Sismis