instead of a single progress.
Will help for keywords such as http.header which can act on
headers and trailers progress
Tx engines are inspected between min_progress and max_progress
So, we do not give up and says a signature does not match
when it will match on later max_progress
And we can match as early as possible, especially in IPS mode.
Alerts and drops for firewall rules hooked at a sub-state (http2
stream/global) had no firewall.hook field: the logger resolved the state
name without the sub-state, so the lookup found nothing.
Keep the rule's sub-state on the Signature next to app_progress_hook,
set it from the hook and for the default policy signatures, resolve the
name in the sub-state table, and fall back to the plain state name when
that lookup fails.
Ticket: #8944.
DetectEngineInspectGenericList returned NO_MATCH for every keyword miss,
also when the tx had moved past the phase the engine is registered at.
Unlike the buffer engines it never produced a terminal no match, so a
firewall LTE rule using such a keyword (tls.version on a hook) stayed
provisional and was accepted for the rest of the session.
Make a miss terminal at P + 1. This engine also backs the lua
on-request/on-response hooks and the tls_cert_*, dnp3/dnp3_ind and
ftpdata/ftp mode keywords, so a plain IDS rule using them keeps its
verdict once the tx has moved past the registered phase.
Export the parser's end state computation as AppLayerParserGetTxEndState()
here, so the version keyword can use it; the remaining call sites are
converted later.
Ticket: #8944.
A non-matching rule followed by an LTE rule (<hook) with an overlapping
hook can be decided by the wrong default policy. E.g.:
drop:flow tls:client_hello .. tls.sni; content:"match" \
content:"no-match";
accept:flow tls:<client_hello .. tls.sni; content:"match"
Here the `tls:client_started` state needs to be accepted by the second
rule, which acts as `accept:hook` on that state.
A failing `<` rule defers the per-state defaults of the states it
promised to cover; the last pending covering rule decides them, applying
its own hook default and sweeping forward from its own hook. There is no
backward sweep, so the states below that hook get no default of their
own.
The pending LTE coverage accounting was the problem: it counted
candidates before their header predicates were checked, compared against
a max that included the rule being evaluated, and kept rules that
resolved during the pass. Count only rules that pass the header checks,
keep per-hook coverage counts in per-thread scratch, drop a rule from
the counts once it resolves at or past its hook, and let the hook checks
consult the counts instead of the raw LTE flag.
Also fixes two latent issues in the same walk: the candidate list is now
sorted whenever the tx state store adds entries - the old condition
skipped the sort when the prefilter contributed nothing, so the
last-for-progress peek could see an unsorted list; and rule profiling
now receives `r == 1` instead of the raw return code, which used to
decrement the match counter on a no-match.
Ticket: #8944.
IPOnly Engine does not support parsing and recording IPv6 ranges. So,
passing a range of ipv6 there just results in non-fatal errors in
parsing the IP addresses which means they fail to be added to the radix
tree struct.
Re-classify the ipv6 ranges as like_ip_only so they fall out of the
IPOnly engine and are processed per packet.
Ticket: 8847
With substate support the fixed table approach for policies was no
longer a good fit, so convert to a hash table. The policies are stored
per alproto, sub_state, progress and direction.
Support for per transaction sub states: different state machines per
transaction type.
Skip engines not belonging to our substate.
Store tx_type in DetectTransaction.
Each protocol supporting sub states will register states from 1 and up.
To support:
Ticket: #8386.
SCDetectEngineRegisterRateFilterCallback() dereferences the result of
DetectEngineGetCurrent() unconditionally. Add a NULL guard with
SCLogError and return false so callers can detect registration failure.
Flagged by Svace and confirmed by gcc -fanalyzer.
Ticket: 8560
More clearly define the relationship between PacketAlerts for firewall
and threat detection events.
Also no longer count firewall_discarded if drop rule came before a
another rule, causing the later rule to not be evaluated/alerted.
When packet:filter and app:filter alert appear in a single alert queue,
handle accept:hook by keeping track of the detect_table.
Support `alert` as a secondary action in packet firewall policies.
To implement this a Signature object is created per policy that uses
alert, and this is stored in a array table. When the policy is applied
the signature is looked up and used in the PacketAlert.
Ticket: #8566.
Support `alert` as a secondary action in app-layer firewall policies.
To implement this a Signature object is created per policy that uses
alert, and this is stored in a hash table. When the policy is applied
the signature is looked up and used in the PacketAlert.
Ticket: #8566.
Allow a single rule to accept a hook and the hooks prior to it.
Example:
accept:flow tls:<client_hello_done ... \
tls.sni; content:"suricata.io"; endswith;
This will evaluate the SNI at the client_hello_done hook, but will
act as if there is a `accept:hook tls:client_in_progress ...` as well.
Implementation is currently specific to this `<` operator. During
parsing the sig gets flagged for this case. During setup this has 3 main
effects:
1. prefilter is disabled as we need to eval this right at the first
state (0)
2. for state 0 a non-PF "prefilter" engine is setup to make sure the
rule is flagged for evaluation
3. In the Signature::app_inspect list a dummy inspect engine is
registered per state before Signature::app_progress_hook
The matching logic is building on the stateful rule handling. The
stateful rule handling can now tell the inspection loop that a partial
match occured. For this rule type the partial match will act as a match
with action accept:hook.
Next app updates will then use the continue detection logic to continue
the stateful match. When that fully matches, the final actions are
applied, like accept:flow or accept:tx.
Ticket: #8472.
Add a `firewall` stats counter aggregator for all firewall-related
stats.
De-overload "detect.alert_queue_overflow", by adding
"firewall.discarded_alerts" to account for discarded drops in
Firewall mode.
Add Debug statements for tracking corner cases where it can be
difficult to know where a drop is coming from.
Added counters:
- stats.firewall.blocked
- stats.firewall.accepted
- stats.firewall.rejected
- stats.firewall.drop_reason.default_app_policy
- stats.firewall.drop_reason.default_packet_policy
- stats.firewall.drop_reason.flow_drop
- stats.firewall.drop_reason.pre_flow_hook
- stats.firewall.drop_reason.pre_stream_hook
- stats.firewall.drop_reason.rules
- stats.firewall.discarded_alerts
Ticket #7699
Allow configurable policies, including accept. For app-layer this
requires looping all available hooks to apply the policies.
Support configurable policies for packet-filter, pre-stream, pre-flow.
If there are no rules there is also no rule group (sgh). Make sure
the app hooks policies are correctly handled in this case by allowing
a NULL sgh to be handled as well.
For tx rule match actually apply drop directly. Previously this was
always handled by the default drop:flow policy.
Ticket: #7701.
Make `Signature::proto` an optional member, meaning that if it is
NULL we can skip the check. This can be done for `alert ip`, as no check
is needed, and for `alert tcp` and `alert udp` as having a rule in a sgh
for those means that the protocol matches.
Some exceptions are rules that require:
- ipv4/ipv6 specific matching
- frames, due to sharing prefilter between tcp and udp
- ip-only rules, due to those not being per sgh
Hyperscan MPM can cache the compiled contexts to files.
This however grows as rulesets change and leads to bloating
the system. This addition prunes the stale cache files based
on their modified file timestamp.
Part of this work incorporates new model for MPM cache stats
to split it out from the cache save function and aggregate
cache-related stats in one place (newly added pruning).
Ticket: 7830
Remove unused rule prefilter-related stats counters that aren't in use.
94644ac960 (detect: move non-pf rules into special prefilter engines)
removed the logic that made use of and incremented the stats counters:
- det_ctx->counter_fnonmpm_list
- det_ctx->counter_nonmpm_list
Some code was left, registering them, and mentioning them in the
json schema.
Ticket #7834
The det_ctx structure was inflated by the additoin of the array to
handle JSON context. This commit updates the code to use a growing
buffer and limit the impact.
This patch introduces new option to dataset keyword.
Where regular dataset allows match from sets, dataset with json
format allows the same but also adds JSON data to the alert
event. This data is coming from the set definition it self.
For example, an ipv4 set will look like:
[{"ip": "10.16.1.11", "test": "success","context":3}]
The syntax is a JSON array but it can also be a JSON object
with an array inside. The idea is to directly used data coming
from the API of a threat intel management software.
The syntax of the keyword is the following:
dataset:isset,src_ip,type ip,load src.lst,format json, \
enrichment_key src_ip, value_key ip;
Compare to dataset, it just have a supplementary option key
that is used to indicate in which subobject the JSON value
should be added.
The information is added in the even under the alert.extra
subobject:
"alert": {
"extra": {
"src_ip": {
"ip": "10.6.1.11",
"test": "success",
"context": 3
},
The main interest of the feature is to be able to contextualize
a match. For example, if you have an IOC source, you can do
[
{"buffer": "value1", "actor":"APT28","Country":"FR"},
{"buffer": "value2", "actor":"APT32","Country":"NL"}
]
This way, a single dataset is able to produce context to the
event where it was not possible before and multiple signatures
had to be used.
The format introduced in datajson is an evolution of the
historical datarep format. This has some limitations. For example,
if a user fetch IOCs from a threat intel server there is a large
change that the format will be JSON or XML. Suricata has no support
for the second but can support the first one.
Keeping the key value may seem redundant but it is useful to have it
directly accessible in the extra data to be able to query it
independantly of the signature (where it can be multiple metadata
or even be a transformed metadata).
In some case, when interacting with data (mostly coming from
threat intel servers), the JSON array containing the data
to use is not at the root of the object and it is ncessary
to access a subobject.
This patch implements this with support of key in level1.level2.
This is done via the `array_key` option that contains the path
to the data.
Ticket: #7372
This also exposed a difference between the handling of TD alerts in
firewall vs non-firewall mode. In firewall mode the table/hook is also
part of the alert ordering to make sure actions from packet:td are
applied before app:td. Handle that explicitly for now.
Allow keywords to specify in which detect table they can function.
E.g. the pre_flow table will not support flow keywords, as no flow is
availble at this time.
Catena cyber
Philippe Antoine
Victor Julien
Shivani Bhardwaj
Dmitry Uryvchikov
Denis Balashov
Juliana Fajardini
Lukas Sismis
Eric Leblond