Make sure the transaction still exists before completing it in the
to-client direction. A pipelined RSET reply may already have completed and
freed it while a later end-of-DATA marker still refers to it.
Found by OSS-Fuzz testcase 5498180758994944.
Bug #8739.
Make sure that every buffer in a signature is actually used in the
setup of the engines. Reject sigs that are locked to a certain substate
and use buffers that require another.
DOH2 depends on HTTP/2, so it makes no sense to enable it separately.
It would also put the sub state handling in a weird state, as the DOH2
side reuses the registered HTTP/2 callbacks.
In firewall mode the alproto logic when building prefilter and rule
engines is strict, but was out of sync between prefilter and rule app
inspect engines.
For SIGNATURE_HOOK_TYPE_APP rules the logic is strict, with an exception
for HTTP/2 and DOH2. DOH2 is not a full protocol implementation, but
rather HTTP/2 with a different alproto.
Fixes: d64954a873 ("detect: don't register unrelated inspect engines")
With substate support the fixed table approach for policies was no
longer a good fit, so convert to a hash table. The policies are stored
per alproto, sub_state, progress and direction.
Split into 2 sub-states:
- stream, which has the "HTTP" requests and responses, including DOH2
- global, which has the settings and other global or control handling
Introduce a simpler progress tracking for the global sub state:
- HTTP2ProgGlobalStart and HTTP2ProgGlobalComplete.
The stream sub state uses the same state machine as before.
Ticket: #8386.
Support for per transaction sub states: different state machines per
transaction type.
Skip engines not belonging to our substate.
Store tx_type in DetectTransaction.
Each protocol supporting sub states will register states from 1 and up.
To support:
Ticket: #8386.
Allow registration of per substate progress name mappings.
Implement logic for getting sub-state names, id's.
Add transaction type and end of progress values to AppLayerTxData.
Introduce helpers to keep logic clean.