Remove unused rule prefilter-related stats counters that aren't in use.
94644ac960 (detect: move non-pf rules into special prefilter engines)
removed the logic that made use of and incremented the stats counters:
- det_ctx->counter_fnonmpm_list
- det_ctx->counter_nonmpm_list
Some code was left, registering them, and mentioning them in the
json schema.
Ticket #7834
We used `copy_column_count`, while just `columns` is more accurate with
what PostgreSQL describes, and what Wireshark shows.
Related to
Task #7644
Task #7645
This sub-protocol inspects messages sent mainly from the frontend to
the backend after a 'COPY FROM STDIN' has been processed by the
backend.
Parses new messages:
- CopyInResponse -- initiates copy-in mode/sub-protocol
- CopyData (In) -- data transfer message, from frontend to backend
- CopyDone -- signals that no more CopyData messages will be seen from
the frontend, for the current transaction
- CopyFail -- used by the frontend to signal some failure to proceed
with sending CopyData messages
Task #7645
The mDNS support is based heavily on the DNS support, reusing the
existing DNS parser where possible. This meant adding variations on
DNS, as mDNS is a little different. Mainly being that *all* mDNS
traffic is to_server, yet there is still the concept of request and
responses.
Keywords added are:
- mdns.queries.rrname
- mdns.answers.rrname
- mdns.additionals.rrname
- mdns.authorities.rrname
- mdns.response.rrname
They are mostly in-line with the DNS keywords, except
mdns.answers.rdata which is a better than that mdns.response.rrname,
as its actually looking at the rdata, and not rrnames.
mDNS has its own logger that differs from the DNS logger:
- No grouped logging
- In answers/additionals/authorities, the rdata is logged in a field
that is named after the rdata type. For example, "txt" data is no
longer logged in the "rdata" field, but instead a "txt" field. We
currently already did this in DNS for fields that were not a single
buffer, like SOA, SRV, etc. So this makes things more consistent. And
gives query like semantics that the "grouped" object was trying to
provide.
- Types are logged in lower case ("txt" instead of "TXT")
- Flags are logged as an array: "flags": ["aa", "z"]
Ticket: #3952
Also:
- Place "additionalProperties" before "properties"
- Place "required" after "additionalProperties"
- Remove "additionalProperties where true, as that is the default
The order should help us spot duplicate keys, and make it easier to
add new keys in their proper place.
Ticket: 6695
"server_handshake" which logs the following:
1. TLS version used during handshake
2. The chosen cipher suite, excluding GREASE
3. TLS extensions, excluding GREASE
Ticket: 6695
Add new custom log fields:
"client_handshake" which logs the following:
1. TLS version used during handshake
2. TLS extensions, excluding GREASE, SNI and ALPN
3. All cipher suites, excluding GREASE
4. All signature algorithms, excluding GREASE
The use-case is for logging TLS handshake parameters in order to survey
them, and so that JA4 hashes can be computed offline (in the case that
they're not already computed for the purposes of rule matching).
If a password message was seen while logging passwords was disabled
for pgsql, this would lead to an empty request being logged.
Instead of simply not logging anything when there is a password message
and this is disabled, however, log instead that said password is
redacted.
Bug #7647
This module uses the sawp-pop3 crate to parse POP3 requests and responses
Features:
- eve logging
- events for parsable but non-RFC-compliant messages
Ticket: 3243
Config:
Firewall rules are like normal rule, with some key differences.
They are loaded separate, and first, from:
```yaml
firewall-rule-path: /etc/suricata/firewall/
firewall-rule-files:
- fw.rules
```
Can also be loaded with --firewall-rules-exclusive: Mostly for QA purposes.
Allow -S with --firewall-rules-exclusive, so that firewall and threat detection
rules can be tested together.
Rules:
Differences with regular "threat detection" rules:
1. these rules are evaluated before threat detection rules
2. these rules are evaluated in the order as they appear in the rule file
3. currently only rules specifying an explicit hook at supported
a. as a consequence, no rules will be treated as (like) IP-only, PD-only or
DE-only
Require explicit action scope for firewall rules. Default policy is
drop for the firewall tables.
Actions:
New action "accept" is added to allow traffic in the filter tables.
New scope "accept:tx" is added to allow accepting a transaction.
Tables:
Rulesets are per table.
Table processing order: `packet:filter` -> `packet:td` -> `app:*:*` -> `app:td`.
Each of the tables has some unique properties:
`packet:filter`:
- default policy is `drop:packet`
- rules are process in order
- action scopes are explicit
- `drop` or `accept` is immediate
- `accept:hook` continues to `packet:td`
`packet:td`:
- default policy is `accept:hook`
- rules are ordered by IDS/IPS ordering logic
- action scopes are implicit
- actions are queued
- continues to `app:*:*` or `alert/action finalize`
`app:*:*`:
- default policy is `drop:flow`
- rules are process in order
- action scopes are explicit
- `drop` is immediate
- `accept` is conditional on possible `drop` from `packet:td`
- `accept:hook` continues to `app:td`, `accept:packet` or `accept:flow`
continues to `alert/action finalize`
`app:td`:
- default policy is `accept:hook`
- rules are ordered by IDS/IPS ordering logic
- action scopes are implicit
- actions are queued
- continues to `alert/action finalize`
Implementation:
During sigorder, split into packet:filter, app:*:* and general td.
Allow fw rules to work when in pass:flow mode. When firewall mode is enabled,
`pass:flow` will not skip the detection engine anymore, but instead
process the firewall rules and then apply the pass before inspecting threat
detect rules.
This sub-protocol inspects messages exchanged between postgresql backend
and frontend after a 'COPY TO STDOUT' has been processed.
Parses new messages:
- CopyOutResponse -- initiates copy-out mode/sub-protocol
- CopyData -- data transfer messages
- CopyDone -- signals that no more CopyData messages will be seen from
the sender for the current transaction
Task #4854
In order to track flow rate and thus determine a course of action or
categorize it as elephant flow, track a flow's byte count per direction
in a ring buffer for a given time interval.
The implementation is simple and keeps overwriting the buffer and
updating the final sum. The sum of all the elements in the ring buffer
at any point in time should reflect the number of bytes for the
respective flow in the last of a given configured interval.
e.g. if the definition says that the flows must be tracked by a rate of
100k bytes in 10 seconds, the ring buffer at any point in time should
carry the total number of bytes seen by the respective flow in the last
10 seconds.
So far, the implementation only supports reading the flow rate
definition from suricata.yaml and using it to track the flows.
This solution adds up a space complexity to the existing Flow struct.
However, the added space complexity should only take effect if the
feature is in use. Since this buffer extends the Flow struct, it does
not impact the usual business logic or complexity of the code.
This implementation is currently limited to defining the time interval
of flow rate in seconds only. However, the number of seconds defined are
directly proportional to the aforementioned added space complexity as
that's the size of the ring buffer.
Feature 5647
To accompany the Exception Policy stats, also add information about any
Exception Policy triggered and for which target to the flow log event.
Task #6215
To some EVE fields and a "suricata" object that contains an array of
keywords. These are the keywords that map directly to this field, or
somehow cover this field.
This is an attempt at tooling to help with EVE and keyword parity.
Related to tickets: #5642, #6463, #4772
Issue 7129
When configured with the existing "ethernet" switch, include the ether
type in the output.
This is most useful with anomaly records indicating unknown ethertypes.
If TCP urgent handling is set to the OOB processing, the number of OOB
bytes is tracked for SEQ offset calculations. If this offset is
non-zero, add the field to the flow record.
Ticket: #7411.
TCP urgent handling is a complex topic due to conflicting RFCs and
implementations.
Until now the URG flag and urgent pointer values were simply ignored,
leading to an effective "inline" processing of urgent data. Many
implementations however, do not default to this behavior.
Many actual implementations use the urgent mechanism to send 1 byte of
data out of band to the application.
Complicating the matter is that the way the urgent logic is handled is
generally configurable both of the OS and the app level. So from the
network it is impossible to know with confidence what the settings are.
This patch adds the following policies:
`stream.reassembly.urgent.policy`:
- drop: drop URG packets before they affect the stream engine
- inline: ignore the urgent pointer and process all data inline
- oob (out of band): treat the last byte as out of band
- gap: skip the last byte, but do no adjust sequence offsets, leading to
gaps in the data
For the `oob` option, tracking of a sequence number offset is required,
as the OOB data does "consume" sequence number space. This is limited to
64k. For this reason, there is a second policy:
`stream.reassembly.urgent.oob-limit-policy`:
- drop: drop URG packets before they affect the stream engine
- inline: ignore the urgent pointer and process all data inline
- gap: skip the last byte, but do no adjust sequence offsets, leading to
gaps in the data
Bug: #7411.
Ticket: 7199
Uses a config parameter detect.guess-applayer-tx to enable
this behavior (off by default)
This feature is requested for use cases with signatures not
using app-layer keywords but still targetting application
layer transactions, such as pass/drop rule combination,
or lua usage.
This overrides the previous behavior of checking if the signature
has a content match, by checking if there is only one live
transaction, in addition to the config parameter being set.
Ticket: 7311
If response_status_number is not a valid poisitive integer,
we should not try to parse it again, and fail again,
but just log the raw string.
Add EVE documentation for QUIC and Pgsql to their respective sections of
the userguide.
Also add a complete EVE reference as an appendix.
Other protocols can be done, but its a manual process to document in the
schema, then add the glue to pull them into the documentation.
The documentation is generated during "make dist", or if it doesn't
exist, "conf.py" will attempt to generate the eve documentation for
building on Readthedocs.
Issue: 4974
1. Use https instead of http everywhere
2. Organize and annotate references by
- Referenced by ET/Open and ET/Pro
- URL resolves and works as intended (to provide supplemental
information regarding a reference value, e.g., bug id, cve
value)
- URL no longer resolves
- URL resolves but doesn't work as intended (to provide
supplemental information)
- Not referenced by ET/Open nor ET/Pro
- URL resolves and works as intended (to provide supplemental
information regarding a reference value, e.g., bug id, cve
value)
- URL no longer resolves
- URL resolves but doesn't work as intended (to provide
supplemental information)
This introduces a new parser registration function for LDAP/UDP, and update
ldap configuration in order to be able to enable/disable a single parser
independently (such as dns).
Also, GAPs are accepted only for TCP parser and not for UDP.
Ticket #7203
V3 style DNS logging fixes the discrepancies between request and
response logging better dns records and alert records.
The main change is that queries and answers are always logged as
arrays, and header fields are not logged in array items.
For alerts this means that answers are now logged as arrays, queries
already were.
DNS records will get this new format as well, but with a configuration
parameter.
Bug: #6281
Feature: 7017
Add DNSRDataOPT struct and DNSRData enum type OPT.
Add OPT parsing function and test function.
Add DNSRData OPT type to lua.rs match.
Log OPT rdata.
Ticket: 3958
- transactions are now bidirectional
- there is a logger
- gap support is improved with probing for resync
- frames support
- app-layer events
- enip_command keyword accepts now string enumeration as values.
- add enip.status keyword
- add keywords :
enip.product_name, enip.protocol_version, enip.revision,
enip.identity_status, enip.state, enip.serial, enip.product_code,
enip.device_type, enip.vendor_id, enip.capabilities,
enip.cip_attribute, enip.cip_class, enip.cip_instance,
enip.cip_status, enip.cip_extendedstatus
Update the Lua allocated to set a code on memory allocation limit
exceeded errors so an appropriate error message can be logged and a
state incremented.
Fixes the tracking of the allocated size by using the difference
between original size, and new size and toss in some debug
validations.
Distinguish between a generic Lua script error and an error created by a
function being blocked, so each is logged once respective of each other.
Also add a stat that is incremented when a script fails due to a
blocked function.
NOTE: This does not catch calls to functions that are blocked by not
having the library loaded, such as "io.open", as they are blocked by
not even loading the "io" library.
If a rule script crashed, the return value was treated as a no
match. This would make a negation of the rule match and alert.
Instead cleanup and exit early if the rule script crashed and don't
run negation logic.
A stat, detect.lua.errors has been added to count how many times a
script crashes.
Also consolidates the running of the Lua script and return value
handling to a common function.
Bug: #6940
Juliana Fajardini
Philippe Antoine
Jeff Lucovsky
Jason Ish
Eric Leblond
Alice Akaki
Victor Julien
Jeff Lucovsky
Richard McConnell
Shivani Bhardwaj
Alex Savage
Giuseppe Longo
Juliana Fajardini
Nathan Scrivens
Philippe Antoine