A transform that derives its key material from a runtime byte_extract or
byte_math variable cannot have its buffer precomputed at load time, so the
content of that buffer is unusable as a fast_pattern. Add the engine
plumbing to recognise such transforms and keep them out of prefilter/MPM.
SignatureInitData gains a var_transform flag. RetrieveFPForSig bails out
for any signature that sets it, so the signature falls back to full
inspection rather than contributing an MPM pattern that can't be built
ahead of time.
New helpers support a transform consuming a variable:
- SCSignatureSetVarTransform() raises the disqualification flag.
- SCDetectByteVarResolve() resolves a byte_extract or byte_math variable
by name to its runtime slot, and for a byte_extract reports its natural
key width (a byte_math result has none).
- SCDetectEngineThreadCtxGetByteVar() reads a byte_* variable's runtime
value from the thread context.
The helpers are declared in detect-engine-buffer.h so the bindgen run
picks them up; the Rust bindings are regenerated to match.
Issue: 7847
Make sure that every buffer in a signature is actually used in the
setup of the engines. Reject sigs that are locked to a certain substate
and use buffers that require another.
DOH2 depends on HTTP/2, so it makes no sense to enable it separately.
It would also put the sub state handling in a weird state, as the DOH2
side reuses the registered HTTP/2 callbacks.
In firewall mode the alproto logic when building prefilter and rule
engines is strict, but was out of sync between prefilter and rule app
inspect engines.
For SIGNATURE_HOOK_TYPE_APP rules the logic is strict, with an exception
for HTTP/2 and DOH2. DOH2 is not a full protocol implementation, but
rather HTTP/2 with a different alproto.
Fixes: d64954a873 ("detect: don't register unrelated inspect engines")
With substate support the fixed table approach for policies was no
longer a good fit, so convert to a hash table. The policies are stored
per alproto, sub_state, progress and direction.
Split into 2 sub-states:
- stream, which has the "HTTP" requests and responses, including DOH2
- global, which has the settings and other global or control handling
Introduce a simpler progress tracking for the global sub state:
- HTTP2ProgGlobalStart and HTTP2ProgGlobalComplete.
The stream sub state uses the same state machine as before.
Ticket: #8386.
Support for per transaction sub states: different state machines per
transaction type.
Skip engines not belonging to our substate.
Store tx_type in DetectTransaction.
Each protocol supporting sub states will register states from 1 and up.
To support:
Ticket: #8386.
Allow registration of per substate progress name mappings.
Implement logic for getting sub-state names, id's.
Add transaction type and end of progress values to AppLayerTxData.
Introduce helpers to keep logic clean.
Track the transaction id for each queued SMTP command so replies can update the
transaction that created the command instead of always using the current
transaction.
Ticket: #8393