aiden
/
suricata
mirror of https://github.com/OISF/suricata
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Merge acdb78a229 into 9a37bb659b

Browse Source
pull/16392/merge
Victor Julien 13 hours ago committed by GitHub
parent 9a37bb659b acdb78a229
commit 959d8b89d1
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
15 changed files with 1348 additions and 111 deletions
Show Stats Download Patch File Download Diff File

109
doc/userguide/firewall/firewall-design.rst
Unescape Escape View File

@ -284,6 +284,42 @@ ruleset was actually::
This logic only applies to the ``app:filter`` table.
While such a rule is still pending - it ran the states before its hook without
matching, but a buffer of its hook can still grow - a rule hooked at a higher
state is still evaluated once the transaction has moved past the pending rule's
hook; while the transaction is at or before that hook the walk stops, so no
higher state's policy can decide the flow early. A rule of a higher id at the
pending rule's own hook is inspected in the same walk: a rule which cannot be
decided by this update must not stall the hook for the rules which can. The price
is that such a rule's action - a ``drop:flow``, say - decides the flow before the
pending accept resolves. ``ruletype-firewall-616`` pins it; ``ruletype-firewall-623`` pins
the other half, that nothing at all decides an update in which the transaction has not yet
reached the pending rule's hook.
A higher-hook rule inspects
the buffers of every state below its own hook, so its match covers the pending
rule and its action decides the flow: when it matches, the pending rule's later
no-match does not apply the default policy of its own state.
A sub-state's buffers can also grow after the transaction moved past the rule's
hook: a http2 trailer HEADERS frame updates the header lists above the
``request_headers`` hook. Such a rule is not kept open for that update; it is
brought in as a candidate, so the growth is covered without deferring the states
below the hook. The fast pattern at the hook decides there: above it a
group is retired once its rules are, so a rule whose buffer is final at that
point is not run again. A group whose buffer can still be rewritten is not: all
of its rules go into the list at that update, and with them the default policies
of the states they cover. A rule that would stay open because a *stream* match
can still arrive cannot use the notation at all, see the limitations below.
The bound is the transaction's end state. For a parser with per-direction sub-
states that can be later than the direction's own close - a http2 stream
completes when both sides closed, while its request buffers are final once the
client sent END_STREAM - so the bound can lag the point where a miss really is
final. What that costs is one more update at which a group is tested before its
retirement settles; for the http2 cases measured it changes no decision and no
packet, and it keeps one bound for every protocol instead of a per-direction
one per parser.
Firewall pipeline
-----------------
@ -434,3 +470,76 @@ reaches them::
A ``<`` hook rule at the protocol's first state (progress 0) is accepted and is
equivalent to the plain hook form, as there are no prior states to auto-accept.
The pending window
------------------
Two words for two sets of rules, both rebuilt on every update of a transaction:
* the *group* of a hook: the LTE rules registered for one progress state of the
transaction, held as a list of rule ids in one prefilter engine.
* the *pending window*: the rules of that group the fast pattern did not add as
candidates at the hook state. It is a set of rules, not a range of progress states;
the states only say when the set is filled and when it is dropped.
An LTE rule has no say below its hook, and at its hook the fast pattern decides which
of them are worth inspecting. The rules the pattern did not add hold the hook open:
they are not candidates, so the walk cannot decide them, but the default policy cannot
resolve the hook either.
An LTE rule with a packet condition on a variable that a rule can set (``flowbits``,
``xbits``) gets no fast pattern. Whether it is still in the running is only known once
the rules before it in the walk have run, so it is a candidate on every update and the
walk inspects it in order.
::
progress 0 ............ H (hook) C (a miss becomes final)
| | |
window | empty | filled | emptied here
(a set of | | |
rules) | | |
v v v
policy while the window is not empty the hook is left unresolved, so a match
anywhere else takes it and only an empty outcome falls through to the
default policy
Two moments, for the rules of a group hooked at ``H``:
* at ``H``: the pattern runs; matches go to the candidate list, the rules it did not
add are the window and the hook stays pending.
* at ``C``: a miss is final for the group's rules whose buffers are complete at
``H``. The window contributes its last rule only, so that any other match still
wins the state and the policy decides when nothing does. A group that also holds
a rule whose pattern can still grow above ``C`` is not retired there, so the
retirement cannot apply a default policy through it. Tracking such a rule keeps
its hook, so a rule which can never resolve does not hold it open for the whole
transaction. When no candidate takes an id above them, the walk takes the pending
rules at its end, so they are accounted for either way.
``C`` is ``H + 1`` for the app buffers of a protocol which declares no sub-states: an
app hooked rule can only hold patterns of buffers whose engine sits at its hook, so such
a buffer cannot grow once the transaction has moved past it. One case does grow: an
app buffer of a protocol declaring sub-states, because a frame can rewrite it above the
hook -- an http2 trailer HEADERS frame updates the header list buffers. The rules of such
a group are not retired to their last one: no pattern covers that update, so all of them
are inspected there. A rule
with no fast pattern at all has no update which brings it, so a group which holds one is
inspected whole there as well. A
transaction that ends at or below ``H`` never opens a window: its end state decides,
and a window opens only for a tx that keeps going past ``H``.
LIMITATIONS
-----------
* A rule using the auto-accept notation (``<hook``) cannot match the raw stream: a
``content`` or ``pcre`` on the stream (including a bare ``content`` before any sticky
buffer) makes the rule fail to load. The stream is the one buffer that keeps delivering
data above any hook, so a miss at the hook would not be final, and the retirement above
depends on it being so. Use the buffer keyword of the state you mean.
* A message trailer rewrites the header lists above their hook, in HTTP/1
(``http.header``) and in HTTP/2 (a trailer HEADERS frame). A rule hooked below them
whose pattern only appears in the trailer is therefore retired at the first update
past the hook, and that state's default policy decides before the trailer is read -
for http2 that update is the first DATA frame. Main is under the same constraint.
This is a known gap, open with the design discussion of the retirement bound.

6
doc/userguide/rules/http-keywords.rst
Unescape Escape View File

@ -254,6 +254,12 @@ keywords.
.. note:: The ``http.host`` keyword normalizes the host header contents. If a
host name has uppercase characters, those would be changed to lowercase.
.. note:: For HTTP/2 the value comes from the ``:authority`` pseudo-header, or from
the ``Host`` header field when the request has no ``:authority``. Both are taken from
the header section of the request only: a trailer cannot change the host. Pseudo-header
fields are not allowed in trailers (RFC 9113 8.1) and routing fields such as ``Host``
stay out of them (RFC 9110 6.5.1).
Normalization Example::
GET /index.html HTTP/1.1

125
rust/src/http2/detect.rs
Unescape Escape View File

@ -425,14 +425,50 @@ enum Http2Header<'a> {
fn http2_frames_get_header_value<'a>(
tx: &'a HTTP2Transaction, direction: Direction, name: &str,
) -> Option<Http2Header<'a>> {
let mut found = 0;
let mut vec = Vec::new();
let mut single = None;
let frames = if direction == Direction::ToServer {
&tx.frames_ts
} else {
&tx.frames_tc
};
http2_frames_slice_get_header_value(frames, name)
}
/// The header section of a message: the frames before the first DATA frame or
/// before a HEADERS frame that follows a completed header block (END_HEADERS),
/// which can only be the trailer section. The headers of an h2c upgrade arrive
/// as one HEADERS frame per field without END_HEADERS and all stay in.
fn http2_header_section(frames: &[HTTP2Frame]) -> &[HTTP2Frame] {
let mut block_done = false;
for (i, frame) in frames.iter().enumerate() {
match frame.data {
HTTP2FrameTypeData::DATA => {
return &frames[..i];
}
HTTP2FrameTypeData::HEADERS(_) => {
if block_done {
return &frames[..i];
}
if frame.header.flags & parser::HTTP2_FLAG_HEADER_END_HEADERS != 0 {
block_done = true;
}
}
HTTP2FrameTypeData::CONTINUATION(_)
if frame.header.flags & parser::HTTP2_FLAG_HEADER_END_HEADERS != 0 =>
{
block_done = true;
}
_ => {}
}
}
frames
}
fn http2_frames_slice_get_header_value<'a>(
frames: &'a [HTTP2Frame], name: &str,
) -> Option<Http2Header<'a>> {
let mut found = 0;
let mut vec = Vec::new();
let mut single = None;
for frame in frames {
if let Some(blocks) = http2_header_blocks(frame) {
for block in blocks.iter() {
@ -638,9 +674,13 @@ fn http2_normalize_host(ve: Http2Header) -> Http2Header {
fn http2_get_host(tx: &HTTP2Transaction) -> Option<Http2Header<'_>> {
// RFC 9113 states
// The recipient of an HTTP/2 request MUST NOT use the Host header field to determine the target URI if ":authority" is present.
let r = http2_frames_get_header_value(tx, Direction::ToServer, ":authority");
// Neither can come from a trailer: RFC 9113 8.1 forbids pseudo-header fields
// there, and RFC 9110 6.5.1 keeps routing fields like Host out of trailers.
// Reading them from every frame would let a trailer rewrite the target.
let section = http2_header_section(&tx.frames_ts);
let r = http2_frames_slice_get_header_value(section, ":authority");
if r.is_none() {
http2_frames_get_header_value(tx, Direction::ToServer, "host")
http2_frames_slice_get_header_value(section, "host")
} else {
r
}
@ -1178,6 +1218,81 @@ mod tests {
}
}
}
fn hdr(name: &str, value: &str, flags: u8) -> HTTP2Frame {
let block = parser::HTTP2FrameHeaderBlock {
name: name.as_bytes().to_vec().into(),
value: value.as_bytes().to_vec().into(),
error: parser::HTTP2HeaderDecodeStatus::HTTP2HeaderDecodeSuccess,
sizeupdate: 0,
};
HTTP2Frame {
header: parser::HTTP2FrameHeader {
length: 0,
ftype: parser::HTTP2FrameType::Headers as u8,
flags,
reserved: 0,
stream_id: 1,
},
data: HTTP2FrameTypeData::HEADERS(parser::HTTP2FrameHeaders {
padlength: None,
priority: None,
blocks: vec![block],
}),
}
}
fn data() -> HTTP2Frame {
HTTP2Frame {
header: parser::HTTP2FrameHeader {
length: 0,
ftype: parser::HTTP2FrameType::Data as u8,
flags: 0,
reserved: 0,
stream_id: 1,
},
data: HTTP2FrameTypeData::DATA,
}
}
#[test]
fn test_http2_get_host_from_header_section_only() {
let eos = parser::HTTP2_FLAG_HEADER_EOS;
let end = parser::HTTP2_FLAG_HEADER_END_HEADERS;
// a body ends the header section, so the closing HEADERS frame is a
// trailer and cannot rewrite the target
let mut tx = HTTP2Transaction::new();
tx.frames_ts.push(hdr(":authority", "evil.example", end));
tx.frames_ts.push(data());
tx.frames_ts
.push(hdr(":authority", "allowed.example", end | eos));
match http2_get_host(&tx) {
Some(Http2Header::Single(v)) => assert_eq!(v, "evil.example".as_bytes()),
other => panic!("trailer must not reach http.host, got {other:?}"),
}
// same without a body: the second block follows a completed one
let mut tx = HTTP2Transaction::new();
tx.frames_ts.push(hdr("host", "evil.example", end));
tx.frames_ts.push(hdr("host", "allowed.example", end | eos));
match http2_get_host(&tx) {
Some(Http2Header::Single(v)) => assert_eq!(v, "evil.example".as_bytes()),
other => panic!("trailer must not reach http.host, got {other:?}"),
}
// an h2c upgrade sends one HEADERS frame per field with no END_HEADERS:
// all of them are the header section
let mut tx = HTTP2Transaction::new();
tx.frames_ts.push(hdr(":authority", "evil.example", 0));
tx.frames_ts.push(hdr(":authority", "allowed.example", end));
match http2_get_host(&tx) {
Some(Http2Header::Multiple(v)) => {
assert_eq!(v, "evil.example, allowed.example".as_bytes())
}
other => panic!("upgrade frames must all count, got {other:?}",),
}
}
}
#[no_mangle]

10
rust/src/http2/http2.rs
Unescape Escape View File

@ -909,10 +909,6 @@ impl HTTP2State {
let mut tx = HTTP2Transaction::new();
tx.tx_data = AppLayerTxData::for_direction(dir);
tx.tx_data.0.tx_type = HTTP2TxType::HTTP2TxTypeGlobal as u8;
// a parser with tx types must fill both eop fields: the end state
// helper asserts on 0
tx.tx_data.0.tx_type_eop_ts = HTTP2TxGlobalProgress::HTTP2ProgGlobalComplete as u8;
tx.tx_data.0.tx_type_eop_tc = HTTP2TxGlobalProgress::HTTP2ProgGlobalComplete as u8;
self.tx_id += 1;
tx.tx_id = self.tx_id;
tx.progress = HTTP2Progress::GLOBAL(HTTP2GlobalProgress::complete());
@ -999,10 +995,6 @@ impl HTTP2State {
tx.update_file_flags(tx.tx_data.0.file_flags);
tx.tx_data.0.file_tx = STREAM_TOSERVER | STREAM_TOCLIENT; // might hold files in both directions
tx.tx_data.0.tx_type = HTTP2TxType::HTTP2TxTypeStream as u8;
// a parser with tx types must fill both eop fields: the end state
// helper asserts on 0
tx.tx_data.0.tx_type_eop_ts = HTTP2TxProgress::HTTP2ProgComplete as u8;
tx.tx_data.0.tx_type_eop_tc = HTTP2TxProgress::HTTP2ProgComplete as u8;
self.transactions.push_back(tx);
return Some(self.transactions.back_mut().unwrap());
}
@ -1911,12 +1903,14 @@ pub unsafe extern "C" fn SCRegisterHttp2Parser() {
HTTP2TxType::HTTP2TxTypeStream as u8,
Some(HTTP2TxProgress::ffi_id_from_name),
Some(HTTP2TxProgress::ffi_name_from_id),
HTTP2TxProgress::HTTP2ProgComplete as u8,
);
SCAppLayerParserRegisterGetTxSubStateFuncs(
ALPROTO_HTTP2,
HTTP2TxType::HTTP2TxTypeGlobal as u8,
Some(HTTP2TxGlobalProgress::ffi_id_from_name),
Some(HTTP2TxGlobalProgress::ffi_name_from_id),
HTTP2TxGlobalProgress::HTTP2ProgGlobalComplete as u8,
);
SCLogDebug!("Rust http2 parser registered.");

8
rust/sys/src/sys.rs
Unescape Escape View File

@ -1517,12 +1517,8 @@ pub struct AppLayerTxData {
#[doc = " detection engine progress tracking for use by detection engine\n Reflects the \"progress\" of prefilter engines into this TX, where\n the value is offset by 1. So if for progress state 0 the engines\n are done, the value here will be 1. So a value of 0 means, no\n progress tracked yet.\n"]
pub detect_progress_ts: u8,
pub detect_progress_tc: u8,
#[doc = " Type of transaction. Meaning is defined by the parser. Used to\n select a state machine. 0 means it is not used."]
#[doc = " Type of transaction. Meaning is defined by the parser. Used to\n select a state machine. 0 means it is not used. A non-zero value is\n also the sub-state id registered by the parser, whose completion is\n the tx's end state."]
pub tx_type: u8,
#[doc = " End of TX progress values\n\n toserver end of tx progress value"]
pub tx_type_eop_ts: u8,
#[doc = " toclient end of tx progress value"]
pub tx_type_eop_tc: u8,
pub de_state: *mut DetectEngineState,
pub events: *mut AppLayerDecoderEvents,
pub txbits: *mut GenericVar,
@ -1570,7 +1566,7 @@ extern "C" {
#[doc = " \\brief register state<>name funcs for a substate"]
pub fn SCAppLayerParserRegisterGetTxSubStateFuncs(
alproto: AppProto, sub_state: u8, GetIdByNameFunc: AppLayerParserGetStateIdByNameFn,
GetNameByIdFunc: AppLayerParserGetStateNameByIdFn,
GetNameByIdFunc: AppLayerParserGetStateNameByIdFn, completion: u8,
);
}
extern "C" {

44
src/app-layer-parser.c
Unescape Escape View File

@ -150,6 +150,10 @@ typedef struct AppLayerParserProtoCtx_
/* max value of a sub state
* only set for FLOW_PROTO_DEFAULT */
uint8_t max_sub_state;
/* progress value at which the sub-state's buffers are final, indexed by
* the sub state id
* only set for FLOW_PROTO_DEFAULT */
uint8_t sub_state_completion[APP_LAYER_MAX_SUB_STATES];
} AppLayerParserProtoCtx;
typedef struct AppLayerParserCtx_ {
@ -618,13 +622,19 @@ void AppLayerParserRegisterGetEventInfoById(uint8_t ipproto, AppProto alproto,
void SCAppLayerParserRegisterGetTxSubStateFuncs(AppProto alproto, const uint8_t sub_state,
AppLayerParserGetStateIdByNameFn GetIdByNameFunc,
AppLayerParserGetStateNameByIdFn GetNameByIdFunc)
AppLayerParserGetStateNameByIdFn GetNameByIdFunc, const uint8_t completion)
{
SCEnter();
/* validate input */
BUG_ON(sub_state == 0);
BUG_ON(GetIdByNameFunc == NULL);
BUG_ON(GetNameByIdFunc == NULL);
if (completion == 0 || sub_state >= APP_LAYER_MAX_SUB_STATES) {
/* 0 would make the first state look final, so it must not reach the
* end state helper */
FatalError("invalid sub state %u (completion %u) for %s", sub_state, completion,
AppProtoToString(alproto));
}
AppLayerParserProtoCtx *p = &alp_ctx.ctxs[alproto][FLOW_PROTO_DEFAULT];
/* double registration not allowed */
@ -640,6 +650,7 @@ void SCAppLayerParserRegisterGetTxSubStateFuncs(AppProto alproto, const uint8_t
m->GetStateNameById = GetNameByIdFunc;
m->next = p->sub_state_mappings;
p->sub_state_mappings = m;
p->sub_state_completion[sub_state] = completion;
p->max_sub_state = MAX(p->max_sub_state, sub_state);
SCLogDebug("alproto %u:%s, sub_state:%u max:%u %p:%p", alproto, AppProtoToString(alproto),
@ -1180,15 +1191,16 @@ uint8_t AppLayerParserGetTxEndState(uint8_t ipproto, AppProto alproto, void *tx,
if (txd == NULL || txd->tx_type == 0) {
return (uint8_t)AppLayerParserGetStateProgressCompletionStatus(alproto, flags);
}
const uint8_t eop = (flags & STREAM_TOSERVER) ? txd->tx_type_eop_ts : txd->tx_type_eop_tc;
if (unlikely(eop == 0)) {
/* a parser with tx types must fill both end-of-phase fields; without
* the fallback 0 would make the first state look final in release
* builds */
/* the tx type is the sub-state the parser registered; its completion is
* the tx's end state */
const uint8_t completion = AppLayerParserGetSubStateCompletion(alproto, txd->tx_type);
if (unlikely(completion == 0)) {
/* a parser with tx types must register the sub-state's completion:
* without the fallback 0 would make the first state look final */
DEBUG_VALIDATE_BUG_ON(1);
return (uint8_t)AppLayerParserGetStateProgressCompletionStatus(alproto, flags);
}
return eop;
return completion;
}
/**
@ -1308,18 +1320,16 @@ uint8_t AppLayerParserGetSubStateCompletion(const AppProto alproto, const uint8_
if (alproto == ALPROTO_DOH2)
return AppLayerParserGetSubStateCompletion(ALPROTO_HTTP2, sub_state);
BUG_ON(alp_ctx.ctxs[alproto][FLOW_PROTO_DEFAULT].max_sub_state == 0);
const AppLayerParserProtoCtx *p = &alp_ctx.ctxs[alproto][FLOW_PROTO_DEFAULT];
/* TODO hard coded for now */
BUG_ON(alproto != ALPROTO_HTTP2);
if (sub_state == 1) {
return 4;
} else if (sub_state == 2) {
return 1;
} else {
BUG_ON(1);
/* no max_sub_state guard here on purpose: an unregistered protocol lands on
* the fallback below, which is what a recovery path wants */
if (likely(sub_state < APP_LAYER_MAX_SUB_STATES && p->sub_state_completion[sub_state] != 0)) {
return p->sub_state_completion[sub_state];
}
/* a tx type without a registered sub-state is a parser bug: the caller
* falls back to the protocol's completion status */
DEBUG_VALIDATE_BUG_ON(1);
return 0;
}

15
src/app-layer-parser.h
Unescape Escape View File

@ -80,6 +80,9 @@ uint8_t AppLayerParserGetTxEndState(uint8_t ipproto, AppProto alproto, void *tx,
/** progress values need to stay under this. */
#define APP_LAYER_MAX_PROGRESS 48
/** maximum number of sub-states a parser can register (ids 1 to N-1) */
#define APP_LAYER_MAX_SUB_STATES 8
/***** transaction handling *****/
int AppLayerParserSetup(void);
@ -211,14 +214,10 @@ typedef struct AppLayerTxData {
uint8_t detect_progress_tc;
/// Type of transaction. Meaning is defined by the parser. Used to
/// select a state machine. 0 means it is not used.
/// select a state machine. 0 means it is not used. A non-zero value is
/// also the sub-state id registered by the parser, whose completion is
/// the tx's end state.
uint8_t tx_type;
/// End of TX progress values
///
/// toserver end of tx progress value
uint8_t tx_type_eop_ts;
/// toclient end of tx progress value
uint8_t tx_type_eop_tc;
DetectEngineState *de_state;
AppLayerDecoderEvents *events;
@ -304,7 +303,7 @@ void AppLayerParserRegisterGetStateFuncs(uint8_t ipproto, AppProto alproto,
/** \brief register state<>name funcs for a substate */
void SCAppLayerParserRegisterGetTxSubStateFuncs(AppProto alproto, const uint8_t sub_state,
AppLayerParserGetStateIdByNameFn GetIdByNameFunc,
AppLayerParserGetStateNameByIdFn GetNameByIdFunc);
AppLayerParserGetStateNameByIdFn GetNameByIdFunc, uint8_t completion);
void AppLayerParserRegisterTxDataFunc(uint8_t ipproto, AppProto alproto,
AppLayerTxData *(*GetTxData)(void *tx));

484
src/detect-engine-prefilter.c
Unescape Escape View File

@ -49,6 +49,9 @@
#include "detect-engine.h"
#include "detect-engine-prefilter.h"
#ifdef UNITTESTS
#include "util-unittest.h"
#endif
#include "detect-engine-mpm.h"
#include "detect-engine-frame.h"
#include "detect-engine-uint.h"
@ -65,6 +68,22 @@ static int PrefilterStoreGetId(DetectEngineCtx *de_ctx,
static const PrefilterStore *PrefilterStoreGetStore(const DetectEngineCtx *de_ctx,
const uint32_t id);
enum FwLteGrows {
LTE_GROWS_NONE, /**< every pattern of the group is final above the hook */
LTE_GROWS_LIST, /**< the buffer can be rewritten above the hook: inspect */
};
struct PrefilterNonPFDataTx {
uint32_t size;
/** some rule of the group has a fast pattern in a buffer that keeps growing
* above the hook, so its miss is not final there. */
enum FwLteGrows lte_grows;
uint32_t array[];
};
static bool PrefilterLteWindowAdd(
DetectEngineThreadCtx *det_ctx, const struct PrefilterNonPFDataTx *data);
static inline void QuickSortSigIntId(SigIntId *sids, uint32_t n)
{
if (n < 2)
@ -89,20 +108,35 @@ static inline void QuickSortSigIntId(SigIntId *sids, uint32_t n)
QuickSortSigIntId(l, (uint32_t)(sids + n - l));
}
/** LTE windows exist for firewall rules only: the `<` hook form is refused for any
* other rule, so `tx_max_progress` is set only when the engine groups LTE rules.
* Keep the dependency on the engine mode explicit rather than implicit. */
static bool PrefilterEngineIsFwLteWindow(const PrefilterEngine *engine)
{
/* tx_max_progress doubles as "no bound" and "hooked at progress 0": a rule at the
* very start has no state below its hook, so the plain run the base does already
* covers it and a window would only add a per-packet store. */
if (engine->ctx.app.tx_max_progress == 0)
return false;
DEBUG_VALIDATE_BUG_ON(!EngineModeIsFirewall());
return EngineModeIsFirewall();
}
/**
* \brief run prefilter engines on a transaction
*
* \retval true if an LTE pending window engine was skipped because the tx has
* reached the rules hook: the fast pattern decides there, and its miss
* is not final yet, so default policies must wait.
*/
void DetectRunPrefilterTx(DetectEngineThreadCtx *det_ctx,
const SigGroupHead *sgh,
Packet *p,
const uint8_t ipproto,
const uint8_t flow_flags,
const AppProto alproto,
void *alstate,
bool DetectRunPrefilterTx(DetectEngineThreadCtx *det_ctx, const SigGroupHead *sgh, Packet *p,
const uint8_t ipproto, const uint8_t flow_flags, const AppProto alproto, void *alstate,
DetectTransaction *tx)
{
/* reset rule store */
det_ctx->pmq.rule_id_array_cnt = 0;
det_ctx->fw_lte_window_cnt = 0;
bool lte_at_hook = false;
SCLogDebug("packet %" PRIu64 " tx %p id %" PRIu64 " progress %d tx->detect_progress %02x",
PcapPacketCntGet(p), tx->tx_ptr, tx->tx_id, tx->tx_progress, tx->detect_progress);
@ -143,6 +177,39 @@ void DetectRunPrefilterTx(DetectEngineThreadCtx *det_ctx,
/* if engine needs tx state to be higher, break out. */
if (engine->ctx.app.tx_min_progress > tx->tx_progress)
break;
/* A pending window engine covers LTE rules that have their hook at
* `bound`. At that state the fast pattern decides which of them match,
* so the list is left to it; above it the group is retired on its last
* rule, unless a rule of it can still grow there. */
const uint8_t bound = engine->ctx.app.tx_max_progress;
bool run = true;
if (PrefilterEngineIsFwLteWindow(engine) && tx->tx_progress >= bound) {
run = false;
/* only the pending windows carry a bound, so this is the
* non-prefilter data */
const struct PrefilterNonPFDataTx *data = engine->pectx;
if (tx->tx_progress == bound && tx->tx_progress < tx->tx_end_state) {
/* the rules the pattern did not add may still match: keep track of
* the group so the walk can act as if it had them in its list */
if (PrefilterLteWindowAdd(det_ctx, data)) {
lte_at_hook = true;
} else {
PrefilterAddSids(&det_ctx->pmq, data->array, data->size);
}
} else if (data->size > 0) {
/* The last rule can only retire the group if no other rule of it can
* match anymore. A protocol which declares sub-states rewrites a buffer
* above its hook - an http2 trailer HEADERS frame - so such a group is
* still live and its rules go in the list whole; a rule whose pattern
* can only arrive from the stream cannot load at an LTE hook (see
* DetectFirewallRuleValidate). */
if (data->lte_grows == LTE_GROWS_LIST) {
PrefilterAddSids(&det_ctx->pmq, data->array, data->size);
} else {
PrefilterAddSids(&det_ctx->pmq, &data->array[data->size - 1], 1);
}
}
}
if (tx->tx_progress > engine->ctx.app.tx_min_progress) {
SCLogDebug("tx->tx_progress %u > engine->ctx.app.tx_min_progress %d",
tx->tx_progress, engine->ctx.app.tx_min_progress);
@ -163,10 +230,12 @@ void DetectRunPrefilterTx(DetectEngineThreadCtx *det_ctx,
#ifdef DEBUG
uint32_t old = det_ctx->pmq.rule_id_array_cnt;
#endif
PREFILTER_PROFILING_START(det_ctx);
engine->cb.PrefilterTx(det_ctx, engine->pectx, p, p->flow, tx_ptr, tx->tx_id,
tx->tx_data_ptr, flow_flags);
PREFILTER_PROFILING_END(det_ctx, engine->gid);
if (run) {
PREFILTER_PROFILING_START(det_ctx);
engine->cb.PrefilterTx(det_ctx, engine->pectx, p, p->flow, tx_ptr, tx->tx_id,
tx->tx_data_ptr, flow_flags);
PREFILTER_PROFILING_END(det_ctx, engine->gid);
}
SCLogDebug("engine %p min_progress %d %s:%s: results %u", engine,
engine->ctx.app.tx_min_progress, AppProtoToString(engine->alproto), pname,
det_ctx->pmq.rule_id_array_cnt - old);
@ -210,6 +279,7 @@ void DetectRunPrefilterTx(DetectEngineThreadCtx *det_ctx,
QuickSortSigIntId(det_ctx->pmq.rule_id_array, det_ctx->pmq.rule_id_array_cnt);
PACKET_PROFILING_DETECT_END(p, PROF_DETECT_PF_SORT1);
}
return lte_at_hook;
}
/** \brief invoke post-rule match "prefilter" engines
@ -597,11 +667,6 @@ struct PrefilterNonPFData {
struct PrefilterNonPFDataSig array[];
};
struct PrefilterNonPFDataTx {
uint32_t size;
uint32_t array[];
};
/** \internal
* \brief wrapper for use in APIs */
static void PrefilterNonPFDataFree(void *data)
@ -617,6 +682,113 @@ static void PrefilterTxNonPF(DetectEngineThreadCtx *det_ctx, const void *pectx,
PrefilterAddSids(&det_ctx->pmq, data->array, data->size);
}
static int PrefilterNonPFDataSortU32(const void *a, const void *b)
{
const uint32_t x = *(const uint32_t *)a, y = *(const uint32_t *)b;
return x < y ? -1 : (x > y ? 1 : 0);
}
/** \brief keep track of a pending window that went silent at its bound. The number
* of groups bound at a state is set by the ruleset, so the array grows to
* fit: a window that could not be tracked would leave the walk unable to
* tell which rule is still pending.
*/
static bool PrefilterLteWindowAdd(
DetectEngineThreadCtx *det_ctx, const struct PrefilterNonPFDataTx *data)
{
for (uint32_t w = 0; w < det_ctx->fw_lte_window_cnt; w++) {
/* the engines of one group can share a set of rules, and the window is
* merged against itself: a duplicate only costs the merge */
if (det_ctx->fw_lte_windows[w] == data) {
return true;
}
}
if (det_ctx->fw_lte_window_cnt == det_ctx->fw_lte_window_size) {
const uint32_t size =
det_ctx->fw_lte_window_size == 0 ? 8 : det_ctx->fw_lte_window_size * 2;
void *ptmp = SCRealloc((void *)det_ctx->fw_lte_windows, size * sizeof(void *));
if (ptmp == NULL) {
/* out of memory: tell the caller to keep the group as candidates
* instead of killing the engine on the packet path */
return false;
}
det_ctx->fw_lte_windows = (const void **)ptmp;
det_ctx->fw_lte_window_size = size;
}
det_ctx->fw_lte_windows[det_ctx->fw_lte_window_cnt++] = data;
return true;
}
void DetectPrefilterLteWindowFree(DetectEngineThreadCtx *det_ctx)
{
SCFree((void *)det_ctx->fw_lte_windows);
det_ctx->fw_lte_windows = NULL;
det_ctx->fw_lte_window_size = 0;
det_ctx->fw_lte_window_cnt = 0;
}
static uint32_t LteArrayLowerBound(const uint32_t *a, const uint32_t n_, const uint32_t v)
{
uint32_t lo = 0, hi = n_;
while (lo < hi) {
const uint32_t mid = lo + (hi - lo) / 2;
if (a[mid] < v)
lo = mid + 1;
else
hi = mid;
}
return lo;
}
static uint32_t LteCandidateLowerBound(
const RuleMatchCandidateTx *cands, const uint32_t cand_cnt, const uint32_t v)
{
uint32_t lo = 0, hi = cand_cnt;
while (lo < hi) {
const uint32_t mid = lo + (hi - lo) / 2;
if (cands[mid].id < v)
lo = mid + 1;
else
hi = mid;
}
return lo;
}
/** \brief find the lowest rule of a silenced LTE pending window that is not a
* candidate: the fast pattern did not add it, but the tx is not final at
* the hook, so it may still match. A walk that had all rules in it would
* inspect this one before any rule with a higher iid.
* Both arrays are in iid order, so the search enters them at the cursor.
*/
static uint32_t LteWindowFirstMissing(const struct PrefilterNonPFDataTx *data,
const RuleMatchCandidateTx *cands, const uint32_t cand_cnt, const uint32_t min_iid)
{
/* both arrays are sorted, so enter them where min_iid lies rather than at the
* start: a caller that retries past rejected rules would otherwise rewalk
* everything below the cursor on every retry. */
uint32_t i = LteArrayLowerBound(data->array, data->size, min_iid);
while (i < data->size) {
const uint32_t iid = data->array[i];
const uint32_t j = LteCandidateLowerBound(cands, cand_cnt, iid);
if (j == cand_cnt || cands[j].id != iid)
return iid;
i++;
}
return UINT32_MAX;
}
uint32_t DetectPrefilterLtePendingIid(
DetectEngineThreadCtx *det_ctx, const uint32_t cand_cnt, const uint32_t min_iid)
{
uint32_t lowest = UINT32_MAX;
for (uint32_t i = 0; i < det_ctx->fw_lte_window_cnt; i++) {
const struct PrefilterNonPFDataTx *data = det_ctx->fw_lte_windows[i];
lowest =
MIN(lowest, LteWindowFirstMissing(data, det_ctx->tx_candidates, cand_cnt, min_iid));
}
return lowest;
}
#ifdef NONPF_PKT_STATS
static thread_local uint64_t prefilter_pkt_nonpf_called = 0;
static thread_local uint64_t prefilter_pkt_nonpf_mask_fail = 0;
@ -750,6 +922,11 @@ struct TxNonPFData {
* update (engine progress -1) so a provisional miss can be revisited as
* the transaction advances. */
bool run_always;
/** LTE pending window bound: the engine goes silent once the tx reaches this
* hook, as the fast pattern decides there. 0 means always add the sids. */
uint8_t lte_hook;
/** like #lte_hook, but for the group: #FwLteGrows of its rules */
enum FwLteGrows lte_grows;
uint32_t sigs_cnt;
struct PrefilterNonPFDataSig *sigs;
const char *engine_name; /**< pointer to name owned by DetectEngineCtx::non_pf_engine_names */
@ -758,7 +935,8 @@ struct TxNonPFData {
static uint32_t TxNonPFHash(HashListTable *h, void *data, uint16_t _len)
{
struct TxNonPFData *d = data;
return (d->alproto + d->sub_state + d->progress + d->dir + d->sig_list + d->run_always) %
return (d->alproto + d->sub_state + d->progress + d->dir + d->sig_list + d->run_always +
d->lte_hook) %
h->array_size;
}
@ -768,7 +946,7 @@ static char TxNonPFCompare(void *data1, uint16_t _len1, void *data2, uint16_t le
struct TxNonPFData *d2 = data2;
return d1->alproto == d2->alproto && d1->sub_state == d2->sub_state &&
d1->progress == d2->progress && d1->dir == d2->dir && d1->sig_list == d2->sig_list &&
d1->run_always == d2->run_always;
d1->run_always == d2->run_always && d1->lte_hook == d2->lte_hook;
}
static void TxNonPFFree(void *data)
@ -778,9 +956,31 @@ static void TxNonPFFree(void *data)
SCFree(d);
}
/** \internal \brief does the rule's fast pattern keep growing above its hook?
* An app hooked rule can only hold patterns from buffers whose engine sits at
* the hook progress (`SigValidateCheckBuffers`), so such a buffer is complete
* once the tx is past the hook and a miss there is final. One case escapes
* that: a protocol which declares sub-states rewrites a buffer above its hook
* - an http2 trailer HEADERS frame updates the header buffers. A raw stream
* pattern cannot get here: DetectFirewallRuleValidate() refuses an LTE rule
* with anything on the payload list. */
static enum FwLteGrows FwLtePatternGrowsAboveBound(const Signature *s)
{
if (s->init_data->mpm_sm == NULL) {
/* no fast pattern at all: nothing can bring the rule back into the
* candidates above the bound, so the group cannot be retired there */
return LTE_GROWS_LIST;
}
if (AppLayerParserSupportsSubStates(s->alproto)) {
return LTE_GROWS_LIST; /* a frame rewrites the buffer above the hook */
}
return LTE_GROWS_NONE;
}
static int TxNonPFAddSig(DetectEngineCtx *de_ctx, HashListTable *tx_engines_hash,
const AppProto alproto, const uint8_t sub_state, const int dir, const uint8_t progress,
const int sig_list, const char *name, const Signature *s, const bool run_always)
const int sig_list, const char *name, const Signature *s, const bool run_always,
const uint8_t lte_hook)
{
const uint32_t max_sids = DetectEngineGetMaxSigId(de_ctx);
@ -791,12 +991,16 @@ static int TxNonPFAddSig(DetectEngineCtx *de_ctx, HashListTable *tx_engines_hash
.progress = progress,
.sig_list = sig_list,
.run_always = run_always,
.lte_hook = lte_hook,
.sigs_cnt = 0,
.sigs = NULL,
.engine_name = NULL,
};
const enum FwLteGrows grows = lte_hook != 0 ? FwLtePatternGrowsAboveBound(s) : LTE_GROWS_NONE;
struct TxNonPFData *e = HashListTableLookup(tx_engines_hash, &lookup, 0);
if (e != NULL) {
e->lte_grows = MAX(e->lte_grows, grows);
bool found = false;
// avoid adding same sid multiple times
for (uint32_t y = 0; y < e->sigs_cnt; y++) {
@ -824,6 +1028,8 @@ static int TxNonPFAddSig(DetectEngineCtx *de_ctx, HashListTable *tx_engines_hash
add->progress = progress;
add->sig_list = sig_list;
add->run_always = run_always;
add->lte_hook = lte_hook;
add->lte_grows = grows;
add->sigs = SCCalloc(max_sids, sizeof(struct PrefilterNonPFDataSig));
if (add->sigs == NULL) {
SCFree(add);
@ -939,11 +1145,17 @@ static int SetupNonPrefilter(DetectEngineCtx *de_ctx, SigGroupHead *sgh)
if (s == NULL)
continue;
SCLogDebug("checking sid %u for non-prefilter", s->id);
/* pf_hook: the rules hook state is covered by a fast pattern or a keyword
* prefilter engine, so the rule needs no non-prefilter entry there. */
bool pf_hook = false;
if (s->init_data->mpm_sm != NULL && (s->flags & SIG_FLAG_MPM_NEG) == 0)
continue;
pf_hook = true;
if (s->init_data->prefilter_sm != NULL)
continue;
pf_hook = true;
if ((s->flags & (SIG_FLAG_PREFILTER | SIG_FLAG_MPM_NEG)) == SIG_FLAG_PREFILTER)
pf_hook = true;
const bool lte = (s->flags & SIG_FLAG_FW_HOOK_LTE) != 0;
if (pf_hook && !lte)
continue;
SCLogDebug("setting up sid %u for non-prefilter", s->id);
@ -994,6 +1206,14 @@ static int SetupNonPrefilter(DetectEngineCtx *de_ctx, SigGroupHead *sgh)
* hook 0 it could have a preceding rule that makes sure this sig isn't triggered, but then
* for hook 1 we would need to be called. */
if (s->flags & SIG_FLAG_FW_HOOK_LTE) {
/* With a fast pattern the pattern engine is the only one that may drop
* rules that do not match, so the hook state is left to it: the window
* covers the states below the hook and the engine bound takes care of
* the hook and the states above it. Without one the rule needs to be in
* the list at its hook, as it has always been. A hook of 0 has no state
* below it to defer, and gating it would leave the rule with neither a
* window nor an engine. */
const bool gated = pf_hook && s->app_progress_hook > 0;
for (uint8_t state = 0; state < s->app_progress_hook; state++) {
SCLogDebug("handle HOOK %u LTE", state);
const int dir = (s->flags & SIG_FLAG_TOSERVER) ? 0 : 1;
@ -1007,11 +1227,17 @@ static int SetupNonPrefilter(DetectEngineCtx *de_ctx, SigGroupHead *sgh)
const int sm_list =
DetectEngineAppHookToSmlist(s->alproto, sub_state, state, direction);
if (TxNonPFAddSig(de_ctx, tx_engines_hash, s->alproto, sub_state, dir, state,
sm_list, pname, s, false) != 0) {
sm_list, pname, s, false, gated ? s->app_progress_hook : 0) != 0) {
goto error;
}
tx_non_pf = true;
}
if (gated) {
/* skip the buffer loop: it would add the rule at the hook
* regardless of what the pattern engine made of it */
SCLogDebug("sid %u: LTE pending window only", s->id);
continue;
}
}
for (uint32_t x = 0; x < s->init_data->buffer_index; x++) {
@ -1082,7 +1308,7 @@ static int SetupNonPrefilter(DetectEngineCtx *de_ctx, SigGroupHead *sgh)
const bool run_always = buf != NULL && buf->run_always;
const uint8_t sub_state = app->sub_state;
if (TxNonPFAddSig(de_ctx, tx_engines_hash, app->alproto, sub_state, app->dir,
app->progress, sig_list, buf->name, s, run_always) != 0) {
app->progress, sig_list, buf->name, s, run_always, 0) != 0) {
goto error;
}
tx_non_pf = true;
@ -1105,7 +1331,7 @@ static int SetupNonPrefilter(DetectEngineCtx *de_ctx, SigGroupHead *sgh)
uint8_t sub_state = s->init_data->hook.t.app.sub_state;
if (TxNonPFAddSig(de_ctx, tx_engines_hash, s->alproto, sub_state, dir,
s->init_data->hook.t.app.app_progress, s->init_data->hook.sm_list, pname, s,
false) != 0) {
false, 0) != 0) {
goto error;
}
tx_non_pf = true;
@ -1196,22 +1422,35 @@ static int SetupNonPrefilter(DetectEngineCtx *de_ctx, SigGroupHead *sgh)
if (data == NULL)
goto error;
data->size = t->sigs_cnt;
data->lte_grows = t->lte_grows;
for (uint32_t i = 0; i < t->sigs_cnt; i++) {
data->array[i] = t->sigs[i].sid;
}
if (t->lte_hook) {
/* DetectPrefilterLtePendingIid merges the window against the sorted
* candidates, so the window has to be in iid order. */
qsort(data->array, data->size, sizeof(data->array[0]), PrefilterNonPFDataSortU32);
}
if (PrefilterAppendTxEngineSubState(de_ctx, sgh, PrefilterTxNonPF, t->alproto, t->sub_state,
engine_progress, (void *)data, PrefilterNonPFDataFree, t->engine_name) < 0) {
SCFree(data);
goto error;
}
if (t->run_always) {
/* A stateful keyword must be revisited as the tx advances. Keep
* its real progress (so the progress bookkeeping runs and the
* proto-agnostic -1 invariant holds) and carry an explicit flag. */
if (t->run_always || t->lte_hook) {
PrefilterEngineList *tail = sgh->init->tx_engines;
while (tail->next != NULL)
tail = tail->next;
tail->run_always = true;
if (t->run_always) {
/* A stateful keyword must be revisited as the tx advances. Keep
* its real progress (so the progress bookkeeping runs and the
* proto-agnostic -1 invariant holds) and carry an explicit flag. */
tail->run_always = true;
}
if (t->lte_hook) {
/* LTE pending window: stop adding the sids once the tx reached
* the rules hook, where the fast pattern takes over. */
tail->tx_max_progress = t->lte_hook;
}
}
}
HashListTableFree(tx_engines_hash);
@ -1388,6 +1627,7 @@ int PrefilterSetupRuleGroup(DetectEngineCtx *de_ctx, SigGroupHead *sgh)
e->local_id = local_id++;
e->alproto = el->alproto;
e->ctx.app.tx_min_progress = el->tx_min_progress;
e->ctx.app.tx_max_progress = el->tx_max_progress;
e->ctx.app.sub_state = el->sub_state;
e->run_always = el->run_always;
e->cb.PrefilterTx = el->PrefilterTx;
@ -1905,3 +2145,189 @@ void PostRuleMatchWorkQueueAppend(
det_ctx->post_rule_work_queue.len++;
SCLogDebug("det_ctx->post_rule_work_queue.len %u", det_ctx->post_rule_work_queue.len);
}
#ifdef UNITTESTS
static struct PrefilterNonPFDataTx *LteWindowTestData(const uint32_t *iids, uint32_t cnt)
{
struct PrefilterNonPFDataTx *data = SCCalloc(1, sizeof(*data) + cnt * sizeof(data->array[0]));
if (data == NULL)
return NULL;
data->size = cnt;
for (uint32_t i = 0; i < cnt; i++)
data->array[i] = iids[i];
return data;
}
/** \test the pending rule of a LTE window the fast pattern did not add */
static int PrefilterLteWindowTest01(void)
{
const uint32_t iids[] = { 2, 5, 9, 13 };
struct PrefilterNonPFDataTx *data = LteWindowTestData(iids, 4);
FAIL_IF(data == NULL);
/* the pattern added nothing: the lowest rule of the window is pending */
FAIL_IF(LteWindowFirstMissing(data, NULL, 0, 0) != 2);
/* a rule of the window matched, but it is not the lowest one */
RuleMatchCandidateTx cands[1] = { { .id = 5 } };
FAIL_IF(LteWindowFirstMissing(data, cands, 1, 0) != 2);
/* the lowest rule matched, so the miss is the next one */
cands[0].id = 2;
FAIL_IF(LteWindowFirstMissing(data, cands, 1, 0) != 5);
RuleMatchCandidateTx more[3] = { { .id = 2 }, { .id = 5 }, { .id = 9 } };
FAIL_IF(LteWindowFirstMissing(data, more, 3, 0) != 13);
/* the window is fully matched: nothing is pending */
RuleMatchCandidateTx all[4] = { { .id = 2 }, { .id = 5 }, { .id = 9 }, { .id = 13 } };
FAIL_IF(LteWindowFirstMissing(data, all, 4, 0) != UINT32_MAX);
/* the candidates also hold rules of other groups, before, between and after */
RuleMatchCandidateTx mix[5] = { { .id = 1 }, { .id = 2 }, { .id = 6 }, { .id = 9 },
{ .id = 40 } };
FAIL_IF(LteWindowFirstMissing(data, mix, 5, 0) != 5);
SCFree(data);
PASS;
}
/** \test empty and single entry windows */
static int PrefilterLteWindowTest02(void)
{
struct PrefilterNonPFDataTx *empty = LteWindowTestData(NULL, 0);
FAIL_IF(empty == NULL);
FAIL_IF(LteWindowFirstMissing(empty, NULL, 0, 0) != UINT32_MAX);
SCFree(empty);
const uint32_t iids[] = { 7 };
struct PrefilterNonPFDataTx *one = LteWindowTestData(iids, 1);
FAIL_IF(one == NULL);
FAIL_IF(LteWindowFirstMissing(one, NULL, 0, 0) != 7);
FAIL_IF(LteWindowFirstMissing(one, NULL, 0, 8) != UINT32_MAX);
RuleMatchCandidateTx cand = { .id = 7 };
FAIL_IF(LteWindowFirstMissing(one, &cand, 1, 0) != UINT32_MAX);
SCFree(one);
PASS;
}
/** \test DetectPrefilterLtePendingIid over several windows */
static int PrefilterLteWindowTest03(void)
{
DetectEngineThreadCtx det_ctx = { 0 };
Signature sigs[6] = { { 0 } };
for (uint32_t i = 0; i < 6; i++)
sigs[i].iid = i;
const uint32_t a[] = { 1, 3, 5 }, b[] = { 0, 2, 4 };
struct PrefilterNonPFDataTx *wa = LteWindowTestData(a, 3);
struct PrefilterNonPFDataTx *wb = LteWindowTestData(b, 3);
FAIL_IF(wa == NULL || wb == NULL);
FAIL_IF(wa == wb);
RuleMatchCandidateTx cands[2] = { { .id = 0 }, { .id = 3 } };
const void *windows[2] = { wa, wb };
det_ctx.tx_candidates = cands;
det_ctx.fw_lte_windows = windows;
det_ctx.fw_lte_window_size = 2;
det_ctx.fw_lte_window_cnt = 2;
/* window b has rule 0, so its first miss is 2; window a misses at 1 */
FAIL_IF(DetectPrefilterLtePendingIid(&det_ctx, 2, 0) != sigs[1].iid);
/* the caller takes the next one when a rule turns out to be out of scope */
FAIL_IF(DetectPrefilterLtePendingIid(&det_ctx, 2, 2) != 2);
FAIL_IF(DetectPrefilterLtePendingIid(&det_ctx, 2, 3) != 4);
FAIL_IF(DetectPrefilterLtePendingIid(&det_ctx, 2, 6) != UINT32_MAX);
/* nothing pending: every rule of every window made the list */
det_ctx.fw_lte_window_cnt = 0;
FAIL_IF(DetectPrefilterLtePendingIid(&det_ctx, 2, 0) != UINT32_MAX);
SCFree(wa);
SCFree(wb);
PASS;
}
/** \test more windows than the initial array holds: they all stay tracked */
static int PrefilterLteWindowTest04(void)
{
DetectEngineThreadCtx det_ctx = { 0 };
Signature sigs[16] = { { 0 } };
for (uint32_t i = 0; i < 16; i++)
sigs[i].iid = i;
const uint32_t pending[] = { 4, 9 };
const uint32_t matched[] = { 12 };
struct PrefilterNonPFDataTx *w0 = LteWindowTestData(pending, 2);
struct PrefilterNonPFDataTx *w[16] = { 0 };
FAIL_IF(w0 == NULL);
PrefilterLteWindowAdd(&det_ctx, w0);
for (uint32_t i = 0; i < 16; i++) {
/* distinct sets: the array has to grow past its initial 8 entries */
w[i] = LteWindowTestData(matched, 1);
FAIL_IF(w[i] == NULL);
PrefilterLteWindowAdd(&det_ctx, w[i]);
}
FAIL_IF(det_ctx.fw_lte_window_cnt != 17);
FAIL_IF(det_ctx.fw_lte_window_size != 32);
/* a window already tracked is not tracked twice: the engines of a group share it */
PrefilterLteWindowAdd(&det_ctx, w[3]);
FAIL_IF(det_ctx.fw_lte_window_cnt != 17);
/* the windows that are fully matched give no pending rule, window 0 misses at 9 */
RuleMatchCandidateTx cands[2] = { { .id = 4 }, { .id = 12 } };
det_ctx.tx_candidates = cands;
FAIL_IF(DetectPrefilterLtePendingIid(&det_ctx, 2, 0) != sigs[9].iid);
DetectPrefilterLteWindowFree(&det_ctx);
FAIL_IF(det_ctx.fw_lte_windows != NULL);
FAIL_IF(det_ctx.fw_lte_window_cnt != 0);
for (uint32_t i = 0; i < 16; i++) {
SCFree(w[i]);
}
SCFree(w0);
PASS;
}
/** \test a large window entered at any cursor answers as a linear scan would
* (the search enters both arrays where the cursor lies, not at index 0)
*/
static int PrefilterLteWindowTest05(void)
{
enum { WIN = 256, CAND = 128 };
uint32_t iids[WIN];
for (uint32_t i = 0; i < WIN; i++)
iids[i] = i * 2 + 1;
struct PrefilterNonPFDataTx *data = LteWindowTestData(iids, WIN);
FAIL_IF_NULL(data);
RuleMatchCandidateTx cands[CAND];
for (uint32_t i = 0; i < CAND; i++)
cands[i].id = i * 4 + 1;
for (uint32_t cursor = 0; cursor <= WIN * 2; cursor += 7) {
uint32_t want = UINT32_MAX;
for (uint32_t i = 0; i < WIN; i++) {
if (iids[i] < cursor)
continue;
bool in_cands = false;
for (uint32_t j = 0; j < CAND; j++)
in_cands |= cands[j].id == iids[i];
if (!in_cands) {
want = iids[i];
break;
}
}
FAIL_IF(LteWindowFirstMissing(data, cands, CAND, cursor) != want);
}
SCFree(data);
PASS;
}
void DetectPrefilterRegisterTests(void)
{
UtRegisterTest("PrefilterLteWindowTest01", PrefilterLteWindowTest01);
UtRegisterTest("PrefilterLteWindowTest02", PrefilterLteWindowTest02);
UtRegisterTest("PrefilterLteWindowTest03", PrefilterLteWindowTest03);
UtRegisterTest("PrefilterLteWindowTest04", PrefilterLteWindowTest04);
UtRegisterTest("PrefilterLteWindowTest05", PrefilterLteWindowTest05);
}
#endif /* UNITTESTS */

21
src/detect-engine-prefilter.h
Unescape Escape View File

@ -80,13 +80,16 @@ int PrefilterAppendPostRuleEngine(DetectEngineCtx *de_ctx, SigGroupHead *sgh,
DetectEngineThreadCtx *det_ctx, const void *pectx, Packet *p, Flow *f),
void *pectx, void (*FreeFunc)(void *pectx), const char *name);
void DetectRunPrefilterTx(DetectEngineThreadCtx *det_ctx,
const SigGroupHead *sgh,
Packet *p,
const uint8_t ipproto,
const uint8_t flow_flags,
const AppProto alproto,
void *alstate,
void DetectPrefilterLteWindowFree(DetectEngineThreadCtx *det_ctx);
/** \brief the iid of the lowest pending LTE rule at or above \a min_iid that the
* fast pattern did not add, UINT32_MAX if there is none
*/
uint32_t DetectPrefilterLtePendingIid(
DetectEngineThreadCtx *det_ctx, uint32_t cand_cnt, uint32_t min_iid);
bool DetectRunPrefilterTx(DetectEngineThreadCtx *det_ctx, const SigGroupHead *sgh, Packet *p,
const uint8_t ipproto, const uint8_t flow_flags, const AppProto alproto, void *alstate,
DetectTransaction *tx);
void PrefilterFreeEnginesList(PrefilterEngineList *list);
@ -114,4 +117,8 @@ void PostRuleMatchWorkQueueAppend(
void PrefilterPktNonPFStatsDump(void);
#ifdef UNITTESTS
void DetectPrefilterRegisterTests(void);
#endif
#endif

33
src/detect-engine.c
Unescape Escape View File

@ -663,8 +663,13 @@ static void AppendFrameInspectEngine(DetectEngineCtx *de_ctx,
}
if (mpm_list == u->sm_list) {
SCLogDebug("%s is mpm", DetectEngineBufferTypeGetNameById(de_ctx, u->sm_list));
prepend = true;
new_engine->mpm = true;
/* defensive only: the frame path has no mpm finality handling in the walk, so
* the flag would just reorder this engine ahead of the others. Keep the LTE
* rule in progress order like the stream and app engines. */
if ((s->flags & SIG_FLAG_FW_HOOK_LTE) == 0) {
prepend = true;
new_engine->mpm = true;
}
}
new_engine->type = u->type;
@ -772,9 +777,16 @@ static void AppendAppInspectEngine(DetectEngineCtx *de_ctx,
bool prepend = false;
if (mpm_list == t->sm_list) {
SCLogDebug("%s is mpm", DetectEngineBufferTypeGetNameById(de_ctx, t->sm_list));
prepend = true;
*head_is_mpm = true;
new_engine->mpm = true;
/* An LTE rule must keep its engines in progress order: the pass through
* engines cover the states below the hook, and a not yet reached hook has
* to stay a partial match instead of breaking out as a no match. The mpm
* flag is left off as well: it would make a no match above the hook final
* (`mpm_before_progress`), while LTE rules rely on the buffers own eof. */
if ((s->flags & SIG_FLAG_FW_HOOK_LTE) == 0) {
prepend = true;
*head_is_mpm = true;
new_engine->mpm = true;
}
}
new_engine->alproto = t->alproto;
@ -2127,10 +2139,12 @@ int DetectEngineReloadIsIdle(void)
/** \internal
* \brief is the engine's data final for this tx?
*
* Past the engine's phase the answer is yes; at or beyond the tx end state it
* is final too, even for engines registered at the completion state (no P+1).
* AppLayerParserGetStateProgress() returns the end progress for disrupted
* flows, so progress == end stays a valid finality signal there.
* Past the engine's phase the answer is yes: a rule hooked below is a candidate
* in the list at those updates, so it is inspected there without needing the
* engine kept open. At or beyond the tx end state it is final too, even for
* engines registered at the completion state (no P+1). AppLayerParserGetStateProgress() returns
* the end progress for disrupted flows, so progress == end stays a valid
* finality signal there.
*/
static bool DetectTxCompleted(
Flow *f, void *txv, uint8_t flags, const DetectEngineAppInspectionEngine *engine)
@ -3843,6 +3857,7 @@ static void DetectEngineThreadCtxFree(DetectEngineThreadCtx *det_ctx)
SCFree(det_ctx->replace);
RuleMatchCandidateTxArrayFree(det_ctx);
DetectPrefilterLteWindowFree(det_ctx);
AlertQueueFree(det_ctx);

40
src/detect-parse.c
Unescape Escape View File

@ -2777,14 +2777,43 @@ SigMatchData* SigMatchList2DataArray(SigMatch *head)
extern int g_skip_prefilter;
/** \internal
* \brief does a packet condition of the rule read a variable that a rule
* matching earlier on the same packet can set
*
* The list is the variable readers, not only those a firewall rule can use today.
*/
static bool SigPacketMatchReadsVars(const Signature *s)
{
for (const SigMatch *sm = s->init_data->smlists[DETECT_SM_LIST_MATCH]; sm != NULL;
sm = sm->next) {
switch (sm->type) {
case DETECT_FLOWBITS:
case DETECT_XBITS:
case DETECT_HOSTBITS:
case DETECT_FLOWINT:
case DETECT_FLOWVAR:
case DETECT_PKTVAR:
case DETECT_LUA:
return true;
}
}
return false;
}
static void SigSetupPrefilter(DetectEngineCtx *de_ctx, Signature *s)
{
SCEnter();
SCLogDebug("s %u: set up prefilter/mpm", s->id);
DEBUG_VALIDATE_BUG_ON(s->init_data->mpm_sm != NULL);
if (s->flags & SIG_FLAG_FW_HOOK_LTE) {
SCLogDebug("no prefilter for SIG_FLAG_FW_HOOK_LTE sig");
/* An LTE rule that the prefilter does not add at its hook is pending, and the walk
* decides at its start whether that rule is still in the running
* (DetectFwLtePendingRule()). A variable that a rule earlier in the same walk sets
* is only known when the walk reaches the rule, so a rule that reads one gets no
* prefilter: it is a candidate on every update and inspected in order. */
if ((s->flags & SIG_FLAG_FW_HOOK_LTE) && SigPacketMatchReadsVars(s)) {
SCLogDebug("%u: no prefilter for a SIG_FLAG_FW_HOOK_LTE sig reading variables", s->id);
SCReturn;
}
@ -2910,6 +2939,13 @@ static bool DetectFirewallRuleValidate(const DetectEngineCtx *de_ctx, const Sign
}
}
if (s->flags & SIG_FLAG_FW_HOOK_LTE) {
if (s->init_data->smlists[DETECT_SM_LIST_PMATCH] != NULL) {
SCLogError("rule %u: the auto-accept notation ('<hook') cannot match the raw "
"stream: its match must come from a buffer of that hook, so that a "
"miss at the hook is final",
s->id);
return false;
}
if (!(((s->action & ACTION_ACCEPT) != 0) &&
(s->action_scope == ACTION_SCOPE_FLOW || s->action_scope == ACTION_SCOPE_TX ||
s->action_scope == ACTION_SCOPE_HOOK))) {

254
src/detect.c
Unescape Escape View File

@ -1476,6 +1476,11 @@ static int DetectRunTxInspectRule(ThreadVars *tv, DetectEngineCtx *de_ctx,
file_no_match = 1;
}
/* implied DETECT_ENGINE_INSPECT_SIG_NO_MATCH */
/* LTE rules are in the prefilter fast pattern but carry no
* engine->mpm, so this arm cannot fire for them. Their candidacy
* comes from the pattern instead, and above the retirement bound
* the whole group goes into the list, so a pre-progress miss is
* still retested there. */
if (engine->mpm && mpm_before_progress) {
inspect_flags |= DE_STATE_FLAG_SIG_CANT_MATCH;
inspect_flags |= BIT_U32(engine->id);
@ -1741,6 +1746,16 @@ struct DetectFirewallAppTxState {
* rules cover it (their own hook and every prior hook). Progress values
* stay under APP_LAYER_MAX_PROGRESS. */
bool fw_lte_cover_active;
/** An LTE rule is pending at the tx's current hook: no default policy may
* decide the packet yet, as the rule may still match as data arrives. */
bool fw_defer_defaults;
/** LTE rule the fast pattern could not add at its hook while the tx is not
* final there. A walk that has all rules in it reaches this one before any
* rule of a higher iid, and breaks the walk at it. */
const Signature *fw_lte_pending;
/** iid of the pending rule the walk credited in fw_lte_cover, or UINT32_MAX: only the
* head of the chain borrowed a count, so only it has to give one back. */
uint32_t fw_lte_credited_iid;
};
/** \internal
@ -1864,6 +1879,66 @@ static inline void DetectFwCountAppendedLteRule(DetectEngineThreadCtx *det_ctx,
det_ctx->tx_candidates[can_idx].fw_lte_header_ok = true;
}
/** \internal
* \brief would a full transaction walk still consider this rule on this packet
*
* Runs the filters `DetectRunTxInspectRule()` applies before a rule can hold a
* hook, so a rule the walk would drop here cannot defer defaults or mask a higher
* id. Ordered by cost: scope, then the proto check that stream mpm and negated mpm
* signatures can fail, then the packet engines.
*/
static bool DetectFwLteRuleStillLive(ThreadVars *tv, DetectEngineCtx *de_ctx,
DetectEngineThreadCtx *det_ctx, Packet *p, Flow *f, const Signature *s)
{
if (!DetectRunInspectRuleHeader(p, f, s, s->flags)) {
return false;
}
if (!(AppProtoEquals(s->alproto, f->alproto) || s->alproto == ALPROTO_UNKNOWN)) {
return false;
}
return DetectEnginePktInspectionRun(tv, det_ctx, s, f, p, NULL);
}
/** \internal
* \brief the lowest LTE rule the fast pattern did not add at this state
* \note a rule that is out of the flows scope is no barrier: the walk would
* inspect it, get a header no match and carry on, so the search moves to the
* next pending rule
* \note the filters a walk runs before a rule can hold the hook apply here too, or
* a rule it would drop on this packet defers defaults and masks higher ids.
* Such a rule is never in the candidates, so this is not work already done.
* \note a rule resolved in an earlier pass cannot come out of here either. Resolved
* state lives in the tx store and every store item becomes a candidate in
* `DetectRunTx()`, while the search only yields iids missing from the candidate
* array. The other way in, a fast pattern in a state below the hook, is refused
* at load: `SigValidateCheckBuffers()` requires every same direction buffer
* engine of an app hooked rule to sit at the hook progress, so none of its
* engines can run and CANT before the hook. `DetectFwBuildLteCoverage()` tests
* the store flags because it walks the candidates; this walks their complement.
* \note a stale engine bit cannot decide a firewall outcome on either path: the state
* clear for a new file leaves the engine bits set but drops the resolution
* flags, which only ever moves a rule back into "not yet resolved".
*/
static const Signature *DetectFwLtePendingRule(ThreadVars *tv, DetectEngineCtx *de_ctx,
DetectEngineThreadCtx *det_ctx, Packet *p, Flow *f, const uint32_t cand_cnt,
const uint32_t min_iid)
{
uint32_t next = min_iid;
while (true) {
const uint32_t iid = DetectPrefilterLtePendingIid(det_ctx, cand_cnt, next);
if (iid == UINT32_MAX) {
return NULL;
}
const Signature *s = de_ctx->sig_array[iid];
if ((s->flags & (SIG_FLAG_FW_HOOK_LTE | SIG_FLAG_FIREWALL)) ==
(SIG_FLAG_FW_HOOK_LTE | SIG_FLAG_FIREWALL) &&
DetectFwLteRuleStillLive(tv, de_ctx, det_ctx, p, f, s)) {
return s;
}
next = iid + 1;
}
}
/** \internal
* \brief remove an LTE rule that resolved definitively from the hook coverage.
*/
@ -2000,10 +2075,21 @@ static struct DetectFirewallPolicy DetectFirewallApplyDefaultAppPolicy(
static enum DetectTxFirewallFlowControl DetectFirewallApplyDefaultPolicies(
DetectEngineThreadCtx *det_ctx, const struct DetectFirewallPolicies *policies,
DetectTransaction *tx, Packet *p, const AppProto alproto, const uint8_t direction,
const uint8_t start_hook, const uint8_t end_hook)
const uint8_t start_hook, const uint8_t end_hook, const bool defer)
{
DEBUG_VALIDATE_BUG_ON(start_hook > end_hook);
if (defer) {
SCLogDebug("LTE rule pending at the current hook: no default policy yet");
/* The rule is still in the running, so this packet is not the one to
* decide the flow: keep it, same as a rule that was inspected but is
* not final yet. */
if (tx->is_last) {
DetectRunAppendDefaultAccept(det_ctx, p);
}
return DETECT_TX_FW_FC_OK;
}
const bool need_verdict =
tx->is_last && (end_hook == tx->tx_end_state || end_hook == tx->tx_progress);
SCLogDebug("need_verdict:%s is_last:%s end_hook:%u tx->tx_end_state:%u tx->progress: %u",
@ -2147,9 +2233,10 @@ static enum DetectTxFirewallFlowControl DetectRunTxPreCheckFirewallPolicy(
s->app_progress_hook, tx->detect_progress, tx->detect_progress_orig);
/* if this rule was after the state we expected meaning that there are
* no rules for that state. Invoke the default policies. */
enum DetectTxFirewallFlowControl r = DetectFirewallApplyDefaultPolicies(det_ctx,
det_ctx->de_ctx->fw_policies, tx, p, s->alproto, direction,
tx->detect_progress_orig, s->app_progress_hook - 1);
enum DetectTxFirewallFlowControl r =
DetectFirewallApplyDefaultPolicies(det_ctx, det_ctx->de_ctx->fw_policies, tx, p,
s->alproto, direction, tx->detect_progress_orig,
s->app_progress_hook - 1, fw_state->fw_defer_defaults);
if (r != DETECT_TX_FW_FC_OK) {
/* both SKIP and BREAK mean: no more fw rules to inspect.
* SKIP applies to just this TX.
@ -2322,9 +2409,9 @@ static void DetectRunTxFirewallApplyAccept(DetectEngineThreadCtx *det_ctx, Packe
const uint8_t last_hook = fw_state->last_fw_rule
? tx->tx_progress
: MIN(tx->tx_progress, s->app_progress_hook + 1);
enum DetectTxFirewallFlowControl r =
DetectFirewallApplyDefaultPolicies(det_ctx, det_ctx->de_ctx->fw_policies, tx, p,
s->alproto, direction, s->app_progress_hook + 1, last_hook);
enum DetectTxFirewallFlowControl r = DetectFirewallApplyDefaultPolicies(det_ctx,
det_ctx->de_ctx->fw_policies, tx, p, s->alproto, direction,
s->app_progress_hook + 1, last_hook, fw_state->fw_defer_defaults);
if (r == DETECT_TX_FW_FC_BREAK) {
fw_state->fw_skip_app_filter = true;
return;
@ -2354,7 +2441,8 @@ static void DetectRunTxFirewallApplyAccept(DetectEngineThreadCtx *det_ctx, Packe
* \retval 0 ok, continue as normal. No policies applied.
*/
static int DetectTxFirewallNoRulesApplyPolicies(DetectEngineThreadCtx *det_ctx, Packet *p, Flow *f,
DetectTransaction *tx, const AppProto alproto, const uint8_t flow_flags, const int rule_cnt)
DetectTransaction *tx, const AppProto alproto, const uint8_t flow_flags, const int rule_cnt,
const bool defer)
{
/* if there are no rules / rule candidates, handling invoking the default
* policy. */
@ -2388,7 +2476,7 @@ static int DetectTxFirewallNoRulesApplyPolicies(DetectEngineThreadCtx *det_ctx,
enum DetectTxFirewallFlowControl r =
DetectFirewallApplyDefaultPolicies(det_ctx, det_ctx->de_ctx->fw_policies, tx, p,
alproto, flow_flags & (STREAM_TOSERVER | STREAM_TOCLIENT),
tx->detect_progress_orig, tx->tx_progress);
tx->detect_progress_orig, tx->tx_progress, defer);
SCLogDebug("r %u", r);
if (r == DETECT_TX_FW_FC_BREAK)
return 1;
@ -2453,7 +2541,9 @@ static void DetectRunTxFirewallRuleFullMatch(DetectEngineThreadCtx *det_ctx, con
*
* A default accept is appended. TD has a chance to override this accept.
*
* \retval 1 accept partial, caller must break loop
* \retval 1 partial accept handled: the caller breaks, unless the rule is an
* LTE rule the tx has moved past its hook - then it skips the rule's
* hook and the ones below it and continues with the higher hooks
* \retval 0 ok, caller must continue as normal
*/
static int DetectRunTxFirewallRulePartialMatch(
@ -2475,6 +2565,63 @@ static int DetectRunTxFirewallRulePartialMatch(
return 0;
}
/** \internal
* \brief behave as a walk with all rules in it would before it reaches iid `below`
*
* "below" and "above" are the iid space: the position in the sorted signature list,
* which is the order the walk inspects in. Not the order of the rules in the file.
*
* The rules the fast pattern did not add are pending, and more than one of them can
* sit below the same candidate: the walk has to reach each in iid order, not only the
* lowest, or a pending accept loses its priority over the rules above it.
*
* \retval 1 the walk has to end here
* \retval 0 the walk continues, *pending is the first rule above `below` or none
*/
static int DetectFwLtePendingThrough(ThreadVars *tv, DetectEngineCtx *de_ctx,
DetectEngineThreadCtx *det_ctx, const Signature **pending, Packet *p, Flow *f,
DetectTransaction *tx, struct DetectFirewallAppTxState *fw_state, const uint32_t cand_cnt,
const uint32_t below, uint32_t *credited_iid)
{
/* a pending rule is a firewall rule: the walk's skip state applies to it too,
* or running it here would end the walk and cost the candidates above it
* their inspection, threat detection included. `DetectRunTxPreCheckFirewall-
* Policy()` is what a listed rule would go through. */
const bool skip_firewall = fw_state->fw_skip_app_filter ||
(p->flow->flags & FLOW_ACTION_ACCEPT) != 0 ||
(tx->tx_data_ptr->flags & APP_LAYER_TX_ACCEPT) != 0;
while (*pending != NULL && (*pending)->iid < below) {
bool skip = skip_firewall;
/* skip_fw_hook ends at the first rule it does not skip; a rule already
* skipped by the accept state is not that rule */
if (fw_state->skip_fw_hook && !skip) {
if ((*pending)->app_progress_hook <= fw_state->skip_before_progress)
skip = true;
else
fw_state->skip_fw_hook = false;
}
if (!skip && DetectRunTxFirewallRulePartialMatch(det_ctx, *pending, tx, p) == 1) {
return 1;
}
if ((*pending)->iid == *credited_iid) {
/* out of the running: the count its hooks borrowed at walk start goes
* back. fw_lte_cover is read by DetectFwOtherLteCoversHook(), not only
* when deciding about a pending rule. */
const Signature *done = *pending;
const uint16_t last = MIN(done->app_progress_hook, APP_LAYER_MAX_PROGRESS - 1);
for (uint16_t h = 0; h <= last; h++) {
DEBUG_VALIDATE_BUG_ON(det_ctx->fw_lte_cover[h] == 0);
if (det_ctx->fw_lte_cover[h] > 0) {
det_ctx->fw_lte_cover[h]--;
}
}
*credited_iid = UINT32_MAX;
}
*pending = DetectFwLtePendingRule(tv, de_ctx, det_ctx, p, f, cand_cnt, (*pending)->iid + 1);
}
return 0;
}
/** \internal
* \brief handle the no-match case for a firewall rule
*
@ -2490,7 +2637,8 @@ static int DetectRunTxFirewallRuleNoMatch(DetectEngineThreadCtx *det_ctx, const
DetectTransaction *tx, struct DetectFirewallAppTxState *fw_state, Packet *p,
const uint8_t flow_flags)
{
if (fw_state->fw_last_for_progress && (s->flags & SIG_FLAG_FIREWALL)) {
if (fw_state->fw_last_for_progress && (s->flags & SIG_FLAG_FIREWALL) &&
!fw_state->fw_defer_defaults) {
SCLogDebug("%" PRIu64 ": %s default policy for progress %u", PcapPacketCntGet(p),
flow_flags & STREAM_TOSERVER ? "toserver" : "toclient", s->app_progress_hook);
/* if this rule was the last for our progress state, and it didn't match,
@ -2519,9 +2667,9 @@ static int DetectRunTxFirewallRuleNoMatch(DetectEngineThreadCtx *det_ctx, const
const uint8_t last_hook = fw_state->last_fw_rule
? tx->tx_progress
: MIN(tx->tx_progress, s->app_progress_hook + 1);
enum DetectTxFirewallFlowControl r =
DetectFirewallApplyDefaultPolicies(det_ctx, det_ctx->de_ctx->fw_policies, tx, p,
s->alproto, flow_flags, s->app_progress_hook + 1, last_hook);
enum DetectTxFirewallFlowControl r = DetectFirewallApplyDefaultPolicies(det_ctx,
det_ctx->de_ctx->fw_policies, tx, p, s->alproto, flow_flags,
s->app_progress_hook + 1, last_hook, fw_state->fw_defer_defaults);
if (r == DETECT_TX_FW_FC_BREAK) {
/* the sweep dropped the packet: same as a current-hook
* policy drop, the caller must stop the walk */
@ -2616,10 +2764,13 @@ static void DetectRunTx(ThreadVars *tv,
total_rules += (tx.de_state ? tx.de_state->cnt : 0);
/* run prefilter engines and merge results into a candidates array */
/* set if an LTE rule has its hook at the current tx progress while the
* tx is not final there: the fast pattern decides candidacy there. */
bool lte_at_hook = false;
if (sgh && sgh->tx_engines) {
PACKET_PROFILING_DETECT_START(p, PROF_DETECT_PF_TX);
DetectRunPrefilterTx(det_ctx, sgh, p, ipproto, flow_flags, alproto,
alstate, &tx);
lte_at_hook = DetectRunPrefilterTx(
det_ctx, sgh, p, ipproto, flow_flags, alproto, alstate, &tx);
PACKET_PROFILING_DETECT_END(p, PROF_DETECT_PF_TX);
SCLogDebug("%p/%"PRIu64" rules added from prefilter: %u candidates",
tx.tx_ptr, tx.tx_id, det_ctx->pmq.rule_id_array_cnt);
@ -2727,8 +2878,31 @@ static void DetectRunTx(ThreadVars *tv,
* tracked where the candidates are added: the list mixes fw and TD
* rules across tables in iid order, so a scan cannot stop early. */
struct DetectFirewallAppTxState fw_state = { 0 };
if (have_fw_rules && fw_lte_candidates) {
fw_state.fw_lte_credited_iid = UINT32_MAX;
if (have_fw_rules && lte_at_hook) {
/* The lowest of those rules is still in the running: it covers the hooks
* as if it were in the list, and it ends the walk before any rule that
* takes a higher id. It is not inspected here, so a matcher that fails
* definitively on a buffer that can still grow (which would retire it in
* a full walk) leaves it pending: the defaults wait for the hook. */
fw_state.fw_lte_pending =
DetectFwLtePendingRule(tv, de_ctx, det_ctx, p, f, array_idx, 0);
}
/* The fast pattern decided candidacy at the rules hook, so the rules it did
* not add are missing from the list. They are not final at this state, so the
* default policies have to wait for them. Only for those that are actually in
* the running: a window whose misses are all out of the flows scope defers no
* policy, or the one of this hook would be dropped on the floor. */
fw_state.fw_defer_defaults = have_fw_rules && fw_state.fw_lte_pending != NULL;
if (have_fw_rules && (fw_lte_candidates || fw_state.fw_lte_pending != NULL)) {
DetectFwBuildLteCoverage(det_ctx, p, f, &fw_state, array_idx);
if (fw_state.fw_lte_pending != NULL) {
const Signature *ps = fw_state.fw_lte_pending;
const uint16_t last = MIN(ps->app_progress_hook, APP_LAYER_MAX_PROGRESS - 1);
for (uint16_t h = 0; h <= last; h++)
det_ctx->fw_lte_cover[h]++;
fw_state.fw_lte_credited_iid = ps->iid;
}
}
SCLogDebug("%s: tx_progress %u tx %p have_fw_rules %s array_idx %u detect_progress_orig %u "
@ -2741,7 +2915,7 @@ static void DetectRunTx(ThreadVars *tv,
/* if there are no firewall rules to consider, handle invoking the default
* policies. */
const int r = DetectTxFirewallNoRulesApplyPolicies(
det_ctx, p, f, &tx, alproto, flow_flags, array_idx);
det_ctx, p, f, &tx, alproto, flow_flags, array_idx, fw_state.fw_defer_defaults);
if (r == 1) {
SCLogDebug("done");
return;
@ -2751,7 +2925,8 @@ static void DetectRunTx(ThreadVars *tv,
}
/* run rules: inspect the match candidates */
for (uint32_t i = 0; i < array_idx; i++) {
uint32_t i = 0;
for (; i < array_idx; i++) {
RuleMatchCandidateTx *can = &det_ctx->tx_candidates[i];
const Signature *s = det_ctx->tx_candidates[i].s;
uint32_t *inspect_flags = det_ctx->tx_candidates[i].flags;
@ -2760,6 +2935,22 @@ static void DetectRunTx(ThreadVars *tv,
flow_flags & STREAM_TOSERVER ? "toserver" : "toclient", tx.tx_progress,
tx.detect_progress, tx.detect_progress_orig, s->app_progress_hook);
if (have_fw_rules && fw_state.fw_lte_pending != NULL &&
s->iid > fw_state.fw_lte_pending->iid) {
/* A walk that has all rules in it reaches the pending rules before
* this candidate: what they do there is what would have happened. */
SCLogDebug(
"sid %u is beyond pending LTE rule %u", s->id, fw_state.fw_lte_pending->id);
if (DetectFwLtePendingThrough(tv, de_ctx, det_ctx, &fw_state.fw_lte_pending, p, f,
&tx, &fw_state, array_idx, s->iid,
&fw_state.fw_lte_credited_iid) == 1) {
break;
}
/* the chain can empty here, and the policy paths read the bool rather than
* the pointer: it has to follow what it was derived from */
fw_state.fw_defer_defaults = fw_state.fw_lte_pending != NULL;
}
if (have_fw_rules) {
const enum DetectTxFirewallFlowControl fw_r = DetectRunTxPreCheckFirewallPolicy(
det_ctx, p, &tx, flow_flags & (STREAM_TOSERVER | STREAM_TOCLIENT), s, i,
@ -2863,7 +3054,21 @@ static void DetectRunTx(ThreadVars *tv,
} else if (r == 0) {
SCLogDebug("sid %u partial match", s->id);
if (DetectRunTxFirewallRulePartialMatch(det_ctx, s, &tx, p) == 1) {
break;
/* partial matches are only produced for LTE rules today;
* keep the check as defence in case that changes */
if ((s->flags & SIG_FLAG_FW_HOOK_LTE) == 0 ||
tx.tx_progress <= s->app_progress_hook) {
break;
}
/* The tx moved past the pending rule's hook: a rule hooked
* above it still has to run, as a match there covers the
* pending rule and its policy decides the flow. Skip this
* hook and the ones below it, like an accept:hook does.
* The states above the hook are not swept while the rule is
* pending: they are decided when it resolves, so a higher
* state's default cannot preempt the pending rule's own. */
fw_state.skip_fw_hook = true;
fw_state.skip_before_progress = s->app_progress_hook;
}
} else if (r == -1) {
if ((s->flags & SIG_FLAG_FIREWALL) != 0 &&
@ -2924,6 +3129,15 @@ static void DetectRunTx(ThreadVars *tv,
}
}
if (have_fw_rules && i == array_idx && fw_state.fw_lte_pending != NULL) {
/* no candidate took an id above a pending rule, so the loop never stepped
* through them. A full walk would still have run them: do their treatment
* here, at the end of the walk. */
(void)DetectFwLtePendingThrough(tv, de_ctx, det_ctx, &fw_state.fw_lte_pending, p, f,
&tx, &fw_state, array_idx, UINT32_MAX, &fw_state.fw_lte_credited_iid);
fw_state.fw_defer_defaults = fw_state.fw_lte_pending != NULL;
}
det_ctx->tx_id = 0;
det_ctx->tx_id_set = false;
det_ctx->p = NULL;

19
src/detect.h
Unescape Escape View File

@ -1422,6 +1422,18 @@ typedef struct DetectEngineThreadCtx_ {
* Cleared by the firewall coverage build on first use in a walk. */
uint32_t fw_lte_cover[APP_LAYER_MAX_PROGRESS];
/** LTE rules still pending at this state: the tx reached the bound of their
* window engines, so the fast pattern decided their candidacy and the rules it
* did not add are absent from the candidates while they can still match. One
* entry per engine bound here: a rule with hook P has P of them, one per state
* below the hook, and another per sub-state, direction and buffer list it
* registered in. The walk merges them with the candidates to find the lowest
* rule that is still in the running. Grows with the engines, freed on thread
* deinit. */
const void **fw_lte_windows; /**< PrefilterNonPFDataTx, opaque here */
uint32_t fw_lte_window_cnt;
uint32_t fw_lte_window_size;
MpmThreadCtx mtc; /**< thread ctx for the mpm */
/* work queue for post-rule matching affecting prefilter */
PostRuleMatchWorkQueue post_rule_work_queue;
@ -1621,6 +1633,10 @@ typedef struct PrefilterEngineList_ {
* with Tx Engine. Set to -1 for all states. */
int8_t tx_min_progress;
/** Upper bound, exclusive: skip the engine once the tx reached it.
* 0 means no bound. */
uint8_t tx_max_progress;
uint8_t frame_type;
SignatureMask pkt_mask; /**< mask for pkt engines */
@ -1670,6 +1686,9 @@ typedef struct PrefilterEngine_ {
* with Tx Engine. Set to -1 for all states. */
int8_t tx_min_progress;
uint8_t sub_state;
/** Upper bound, exclusive: skip the engine once the tx reached it.
* 0 means no bound. Only set for the non-prefilter engines. */
uint8_t tx_max_progress;
} app;
uint8_t frame_type;
} ctx;

2
src/runmode-unittests.c
Unescape Escape View File

@ -28,6 +28,7 @@
#include "detect-parse.h"
#include "detect-engine.h"
#include "detect-engine-alert.h"
#include "detect-engine-prefilter.h"
#include "detect-engine-address.h"
#include "detect-engine-proto.h"
#include "detect-engine-port.h"
@ -208,6 +209,7 @@ static void RegisterUnittests(void)
DetectProtoTests();
DetectPortTests();
DetectEngineAlertRegisterTests();
DetectPrefilterRegisterTests();
SCAtomicRegisterTests();
AppLayerUnittestsRegister();
StreamingBufferRegisterTests();

289
src/tests/detect.c
Unescape Escape View File

@ -4805,6 +4805,292 @@ static int DetectFwLteCoverageTest03(void)
PASS;
}
/** \test a pending <hook accept rule holds first-match priority over a later
* same-hook rule for the whole tx, and a TD rule at the hook still runs.
*
* Add order sets the candidate (iid) order, newest first: the TD rule first,
* then the scaffold, the early rule and finally the late rule, so the late
* rule is walked first and holds the hook priority while pending. */
static int DetectFwLteMpmWalkTest01(void)
{
EngineModeSetFirewall(ENGINE_HOST_IS_ROUTER);
ThreadVars th_v;
memset(&th_v, 0, sizeof(th_v));
StatsThreadInit(&th_v.stats);
Flow f;
TcpSession ssn;
memset(&f, 0, sizeof(f));
memset(&ssn, 0, sizeof(ssn));
FLOW_INITIALIZE(&f);
f.protoctx = (void *)&ssn;
f.flags |= FLOW_IPV4;
f.proto = IPPROTO_TCP;
f.alproto = ALPROTO_HTTP1;
AppLayerParserThreadCtx *alp_tctx = AppLayerParserThreadCtxAlloc();
FAIL_IF_NULL(alp_tctx);
StreamTcpInitConfig(true);
DetectEngineCtx *de_ctx = DetectEngineCtxInit();
FAIL_IF_NULL(de_ctx);
de_ctx->flags |= DE_QUIET;
FAIL_IF(DetectFirewallInitDefaultPolicies(de_ctx) != 0);
/* TD rule at the hook: must still run while the accept rules are pending */
Signature *s_td = DetectEngineAppendSig(de_ctx,
"alert http1:request_headers any any -> any any "
"(http.host.raw; content:\"early.example.com\"; sid:1103;)");
FAIL_IF_NULL(s_td);
/* accept the packet hook for all tcp: suppresses the default drop and lets
* the app-layer rules decide */
Signature *s_scaf1 = DetectFirewallRuleAppendNew(
de_ctx, "accept:hook tcp:all any any -> any any (flow:not_established; sid:1021;)");
FAIL_IF_NULL(s_scaf1);
Signature *s_scaf2 = DetectFirewallRuleAppendNew(
de_ctx, "accept:hook tcp:all any any -> any any (flow:established; sid:1022;)");
FAIL_IF_NULL(s_scaf2);
/* content present from the first segment */
Signature *s_early = DetectFirewallRuleAppendNew(de_ctx,
"accept:flow,alert http1:<request_headers any any -> any any "
"(http.host.raw; content:\"early.example.com\"; sid:1102;)");
FAIL_IF_NULL(s_early);
/* the header only arrives in the second segment */
Signature *s_late = DetectFirewallRuleAppendNew(de_ctx,
"accept:flow,alert http1:<request_headers any any -> any any "
"(http.header; content:\"X-Late: a\"; sid:1101;)");
FAIL_IF_NULL(s_late);
SigGroupBuild(de_ctx);
DetectEngineThreadCtx *det_ctx = NULL;
DetectEngineThreadCtxInit(&th_v, (void *)de_ctx, (void *)&det_ctx);
FAIL_IF_NULL(det_ctx);
/* segment 1: request line + host, the header block is open */
const uint8_t *seg1 = (const uint8_t *)"GET / HTTP/1.1\r\nHost: early.example.com\r\n";
Packet *p1 = UTHBuildPacket((uint8_t *)seg1, (uint16_t)strlen((const char *)seg1), IPPROTO_TCP);
FAIL_IF_NULL(p1);
p1->flow = &f;
p1->flowflags |= FLOW_PKT_TOSERVER;
p1->flowflags |= FLOW_PKT_ESTABLISHED;
p1->flags |= PKT_HAS_FLOW | PKT_STREAM_EST;
p1->app_update_direction = STREAM_TOSERVER;
int r = AppLayerParserParse(NULL, alp_tctx, &f, ALPROTO_HTTP1, STREAM_TOSERVER, seg1,
(uint16_t)strlen((const char *)seg1));
FAIL_IF(r != 0);
SigMatchSignatures(&th_v, de_ctx, det_ctx, p1);
/* 1101 is pending: 1102 must not have accepted the flow yet */
FAIL_IF(f.flags & FLOW_ACTION_ACCEPT);
/* segment 2: the late header completes the block */
const uint8_t *seg2 = (const uint8_t *)"X-Late: a\r\n\r\n";
Packet *p2 = UTHBuildPacket((uint8_t *)seg2, (uint16_t)strlen((const char *)seg2), IPPROTO_TCP);
FAIL_IF_NULL(p2);
p2->flow = &f;
p2->flowflags |= FLOW_PKT_TOSERVER;
p2->flowflags |= FLOW_PKT_ESTABLISHED;
p2->flags |= PKT_HAS_FLOW | PKT_STREAM_EST;
p2->app_update_direction = STREAM_TOSERVER;
r = AppLayerParserParse(NULL, alp_tctx, &f, ALPROTO_HTTP1, STREAM_TOSERVER, seg2,
(uint16_t)strlen((const char *)seg2));
FAIL_IF(r != 0);
SigMatchSignatures(&th_v, de_ctx, det_ctx, p2);
/* 1101 matched on completion: the flow is accepted by the pending rule */
FAIL_IF(!(f.flags & FLOW_ACTION_ACCEPT));
/* the pending rule won; the early rule never matched. The TD rule ran
* once, on completion. */
uint32_t late = 0, early = 0, td = 0;
for (uint16_t i = 0; i < det_ctx->alert_queue_size; i++) {
if (det_ctx->alert_queue[i].s == s_late)
late++;
else if (det_ctx->alert_queue[i].s == s_early)
early++;
else if (det_ctx->alert_queue[i].s == s_td)
td++;
}
FAIL_IF(late != 1);
FAIL_IF(early != 0);
FAIL_IF(td != 1);
UTHFreePackets(&p2, 1);
UTHFreePackets(&p1, 1);
DetectEngineThreadCtxDeinit(&th_v, (void *)det_ctx);
DetectEngineCtxFree(de_ctx);
AppLayerParserThreadCtxFree(alp_tctx);
StreamTcpFreeConfig(true);
FLOW_DESTROY(&f);
StatsThreadCleanup(&th_v.stats);
EngineModeSetIDS();
PASS;
}
/** \test a pending LTE rule the walk would discard must not hold the hook.
* Pins the filters in DetectFwLteRuleStillLive(): a rule whose header and protocol
* checks pass but whose packet engines fail is no barrier, as the full walk would
* drop it before reaching the rules above it. */
static int DetectFwLtePendingLiveTest01(void)
{
EngineModeSetFirewall(ENGINE_HOST_IS_ROUTER);
ThreadVars th_v;
memset(&th_v, 0, sizeof(th_v));
StatsThreadInit(&th_v.stats);
DetectEngineCtx *de_ctx = DetectEngineCtxInit();
FAIL_IF_NULL(de_ctx);
de_ctx->flags |= DE_QUIET;
FAIL_IF(DetectFirewallInitDefaultPolicies(de_ctx) != 0);
/* live in every respect on the packet below */
Signature *s_ok = DetectFirewallRuleAppendNew(de_ctx,
"accept:flow,alert http1:<request_headers any any -> any any "
"(http.host; content:\"abc\"; sid:10;)");
FAIL_IF_NULL(s_ok);
/* same rule, but a packet predicate that cannot hold on this packet: it is not
* established. A variable reader would not do: it never gets a prefilter, so it
* cannot be in the window (ruletype-firewall-621/622). */
Signature *s_pkt = DetectFirewallRuleAppendNew(de_ctx,
"accept:flow,alert http1:<request_headers any any -> any any "
"(http.host; content:\"abc\"; flow:established; sid:11;)");
FAIL_IF_NULL(s_pkt);
/* port does not match, so the rule header already rejects it */
Signature *s_hdr = DetectFirewallRuleAppendNew(de_ctx,
"accept:flow,alert http1:<request_headers any any -> any 9999 "
"(http.host; content:\"abc\"; sid:12;)");
FAIL_IF_NULL(s_hdr);
SigGroupBuild(de_ctx);
DetectEngineThreadCtx *det_ctx = NULL;
DetectEngineThreadCtxInit(&th_v, (void *)de_ctx, (void *)&det_ctx);
FAIL_IF_NULL(det_ctx);
Flow f;
memset(&f, 0, sizeof(f));
FLOW_INITIALIZE(&f);
f.proto = IPPROTO_TCP;
f.alproto = ALPROTO_HTTP1;
Packet p;
memset(&p, 0, sizeof(p));
p.proto = IPPROTO_TCP;
p.dp = 80;
p.sp = 1234;
p.flow = &f;
p.flowflags |= FLOW_PKT_TOSERVER;
FAIL_IF(!DetectFwLteRuleStillLive(&th_v, de_ctx, det_ctx, &p, &f, s_ok));
/* the case the scan used to miss: header and protocol pass, packet engines fail */
FAIL_IF(DetectFwLteRuleStillLive(&th_v, de_ctx, det_ctx, &p, &f, s_pkt));
FAIL_IF(DetectFwLteRuleStillLive(&th_v, de_ctx, det_ctx, &p, &f, s_hdr));
/* a rule for another protocol is not a barrier either */
const AppProto keep = s_ok->alproto;
s_ok->alproto = ALPROTO_TLS;
FAIL_IF(DetectFwLteRuleStillLive(&th_v, de_ctx, det_ctx, &p, &f, s_ok));
s_ok->alproto = keep;
FAIL_IF(!DetectFwLteRuleStillLive(&th_v, de_ctx, det_ctx, &p, &f, s_ok));
DetectEngineThreadCtxDeinit(&th_v, (void *)det_ctx);
DetectEngineCtxFree(de_ctx);
FLOW_DESTROY(&f);
StatsThreadCleanup(&th_v.stats);
EngineModeSetIDS();
PASS;
}
/** \test a pending LTE rule that the walk would drop on a packet predicate must
* not defer the default policy of its hook (ticket #9149).
*
* The only accept rule of the state is blocked by `flow:not_established`, so no
* rule is in the running at the hook: the http1 app default (drop:flow) has to
* decide on the first packet. Counting the rule as pending instead defers the
* policy, which is a fail-open window opened by a rule that cannot be inspected
* at all. */
static int DetectFwLtePendingLivePolicyTest01(void)
{
EngineModeSetFirewall(ENGINE_HOST_IS_ROUTER);
ThreadVars th_v;
memset(&th_v, 0, sizeof(th_v));
StatsThreadInit(&th_v.stats);
Flow f;
TcpSession ssn;
memset(&f, 0, sizeof(f));
memset(&ssn, 0, sizeof(ssn));
FLOW_INITIALIZE(&f);
f.protoctx = (void *)&ssn;
f.flags |= FLOW_IPV4;
f.proto = IPPROTO_TCP;
f.alproto = ALPROTO_HTTP1;
AppLayerParserThreadCtx *alp_tctx = AppLayerParserThreadCtxAlloc();
FAIL_IF_NULL(alp_tctx);
StreamTcpInitConfig(true);
DetectEngineCtx *de_ctx = DetectEngineCtxInit();
FAIL_IF_NULL(de_ctx);
de_ctx->flags |= DE_QUIET;
FAIL_IF(DetectFirewallInitDefaultPolicies(de_ctx) != 0);
/* accept the packet hooks, so only the app state is under test */
Signature *scaf1 = DetectFirewallRuleAppendNew(
de_ctx, "accept:hook tcp:all any any -> any any (flow:not_established; sid:1021;)");
FAIL_IF_NULL(scaf1);
Signature *scaf2 = DetectFirewallRuleAppendNew(
de_ctx, "accept:hook tcp:all any any -> any any (flow:established; sid:1022;)");
FAIL_IF_NULL(scaf2);
/* The only LTE rule of the state. The pattern is absent, so the fast pattern
* leaves it in the pending window, and a packet predicate blocks it on this
* packet: only the window search can tell it is not live. */
Signature *s_blk = DetectFirewallRuleAppendNew(de_ctx,
"accept:flow,alert http1:<request_headers any any -> any any "
"(http.host; content:\"absent.example.com\"; flow:not_established; "
"sid:1101;)");
FAIL_IF_NULL(s_blk);
SigGroupBuild(de_ctx);
DetectEngineThreadCtx *det_ctx = NULL;
DetectEngineThreadCtxInit(&th_v, (void *)de_ctx, (void *)&det_ctx);
FAIL_IF_NULL(det_ctx);
/* the header block stays open, so the hook is not crossed on this packet */
const uint8_t seg[] = "GET / HTTP/1.1\r\nHost: open.example.com\r\n";
Packet *p = UTHBuildPacket((uint8_t *)seg, (uint16_t)strlen((const char *)seg), IPPROTO_TCP);
FAIL_IF_NULL(p);
p->flow = &f;
p->flowflags |= FLOW_PKT_TOSERVER;
p->flowflags |= FLOW_PKT_ESTABLISHED;
p->flags |= PKT_HAS_FLOW | PKT_STREAM_EST;
p->app_update_direction = STREAM_TOSERVER;
FAIL_IF(AppLayerParserParse(NULL, alp_tctx, &f, ALPROTO_HTTP1, STREAM_TOSERVER, seg,
(uint16_t)strlen((const char *)seg)) != 0);
SigMatchSignatures(&th_v, de_ctx, det_ctx, p);
/* the rule cannot be inspected on this packet: the default policy decides now */
FAIL_IF(!(f.flags & FLOW_ACTION_DROP));
FAIL_IF(f.flags & FLOW_ACTION_ACCEPT);
/* and it is not the rule that dropped: no alert of its own is queued */
for (uint16_t i = 0; i < det_ctx->alert_queue_size; i++) {
FAIL_IF(det_ctx->alert_queue[i].s == s_blk);
}
UTHFreePackets(&p, 1);
DetectEngineThreadCtxDeinit(&th_v, (void *)det_ctx);
DetectEngineCtxFree(de_ctx);
AppLayerParserThreadCtxFree(alp_tctx);
StreamTcpFreeConfig(true);
FLOW_DESTROY(&f);
StatsThreadCleanup(&th_v.stats);
EngineModeSetIDS();
PASS;
}
void SigRegisterTests(void)
{
SigParseRegisterTests();
@ -4813,6 +5099,9 @@ void SigRegisterTests(void)
UtRegisterTest("DetectFwLteCoverageTest01", DetectFwLteCoverageTest01);
UtRegisterTest("DetectFwLteCoverageTest02", DetectFwLteCoverageTest02);
UtRegisterTest("DetectFwLteCoverageTest03", DetectFwLteCoverageTest03);
UtRegisterTest("DetectFwLteMpmWalkTest01", DetectFwLteMpmWalkTest01);
UtRegisterTest("DetectFwLtePendingLiveTest01", DetectFwLtePendingLiveTest01);
UtRegisterTest("DetectFwLtePendingLivePolicyTest01", DetectFwLtePendingLivePolicyTest01);
UtRegisterTest("SigTest01", SigTest01);
UtRegisterTest("SigTest02 -- Offset/Depth match", SigTest02);

Write Preview
Loading…
Cancel
Save