Ticket: 6695
"server_handshake" which logs the following:
1. TLS version used during handshake
2. The chosen cipher suite, excluding GREASE
3. TLS extensions, excluding GREASE
Ticket: 6695
Add new custom log fields:
"client_handshake" which logs the following:
1. TLS version used during handshake
2. TLS extensions, excluding GREASE, SNI and ALPN
3. All cipher suites, excluding GREASE
4. All signature algorithms, excluding GREASE
The use-case is for logging TLS handshake parameters in order to survey
them, and so that JA4 hashes can be computed offline (in the case that
they're not already computed for the purposes of rule matching).
If a password message was seen while logging passwords was disabled
for pgsql, this would lead to an empty request being logged.
Instead of simply not logging anything when there is a password message
and this is disabled, however, log instead that said password is
redacted.
Bug #7647
Ticket: 7285
As this is the default for websocket, which is bigger than the
defaut for zlib usage
Also limit the decompressed content to the max-payload-size
configuration parameter also used for non-compressed content.
And also use a stateful decoder to store/remember the compression
state to be able to decompress later messages.
This module uses the sawp-pop3 crate to parse POP3 requests and responses
Features:
- eve logging
- events for parsable but non-RFC-compliant messages
Ticket: 3243
Config:
Firewall rules are like normal rule, with some key differences.
They are loaded separate, and first, from:
```yaml
firewall-rule-path: /etc/suricata/firewall/
firewall-rule-files:
- fw.rules
```
Can also be loaded with --firewall-rules-exclusive: Mostly for QA purposes.
Allow -S with --firewall-rules-exclusive, so that firewall and threat detection
rules can be tested together.
Rules:
Differences with regular "threat detection" rules:
1. these rules are evaluated before threat detection rules
2. these rules are evaluated in the order as they appear in the rule file
3. currently only rules specifying an explicit hook at supported
a. as a consequence, no rules will be treated as (like) IP-only, PD-only or
DE-only
Require explicit action scope for firewall rules. Default policy is
drop for the firewall tables.
Actions:
New action "accept" is added to allow traffic in the filter tables.
New scope "accept:tx" is added to allow accepting a transaction.
Tables:
Rulesets are per table.
Table processing order: `packet:filter` -> `packet:td` -> `app:*:*` -> `app:td`.
Each of the tables has some unique properties:
`packet:filter`:
- default policy is `drop:packet`
- rules are process in order
- action scopes are explicit
- `drop` or `accept` is immediate
- `accept:hook` continues to `packet:td`
`packet:td`:
- default policy is `accept:hook`
- rules are ordered by IDS/IPS ordering logic
- action scopes are implicit
- actions are queued
- continues to `app:*:*` or `alert/action finalize`
`app:*:*`:
- default policy is `drop:flow`
- rules are process in order
- action scopes are explicit
- `drop` is immediate
- `accept` is conditional on possible `drop` from `packet:td`
- `accept:hook` continues to `app:td`, `accept:packet` or `accept:flow`
continues to `alert/action finalize`
`app:td`:
- default policy is `accept:hook`
- rules are ordered by IDS/IPS ordering logic
- action scopes are implicit
- actions are queued
- continues to `alert/action finalize`
Implementation:
During sigorder, split into packet:filter, app:*:* and general td.
Allow fw rules to work when in pass:flow mode. When firewall mode is enabled,
`pass:flow` will not skip the detection engine anymore, but instead
process the firewall rules and then apply the pass before inspecting threat
detect rules.
Add an optional bpf filter to pcap-log. If set, packets must match the
filter to be logged, otherwise they will be ignored.
This allows a user to limit what is logged to disk if they have pcap-log
enabled, but still inspect all data captured.
Ticket: #6832
The default value for TLS encryption is generic and not precise.
The new keyword, "track-only" substitues the old keyword, being
more on point what action is actually taken when the TLS flow
reaches an encrypted state.
suricata.yaml output section for smb now parses a types list
and will restrict logging of transactions to these types.
By default, everything still gets logged
Remove unused rs_smb_log_json_request on the way
Ticket: 7620
In order to track flow rate and thus determine a course of action or
categorize it as elephant flow, track a flow's byte count per direction
in a ring buffer for a given time interval.
The implementation is simple and keeps overwriting the buffer and
updating the final sum. The sum of all the elements in the ring buffer
at any point in time should reflect the number of bytes for the
respective flow in the last of a given configured interval.
e.g. if the definition says that the flows must be tracked by a rate of
100k bytes in 10 seconds, the ring buffer at any point in time should
carry the total number of bytes seen by the respective flow in the last
10 seconds.
So far, the implementation only supports reading the flow rate
definition from suricata.yaml and using it to track the flows.
This solution adds up a space complexity to the existing Flow struct.
However, the added space complexity should only take effect if the
feature is in use. Since this buffer extends the Flow struct, it does
not impact the usual business logic or complexity of the code.
This implementation is currently limited to defining the time interval
of flow rate in seconds only. However, the number of seconds defined are
directly proportional to the aforementioned added space complexity as
that's the size of the ring buffer.
Feature 5647
ICE card (Intel E810) was not receiving packets immediatelly
after startup, Suricata workers would act as processing while
it was not. This eliminates the problem by only continuing
in the initialization if the link is already up.
The setting can be turned off manually from the configuraiton
file.
Ticket: 7381
It turned out that having global (interface-specific) mempool
that is shared by the threads of the interface is slower than
having individual mempools per queue for each interface.
The commit brings this change and should be user-invisible,
the config setting remains still as a number of objects of
all mempools summed (of that interface).
Ticket: 7382
The buffer-size value that controls file output buffering defaults to
8k. To be consistent with previous logic, the default is being changed
to 0 (e.g., needed if there are old config files that don't specifically
enable the new value).
If "tpacket-v3" is not present in the configuration file, and we are
in IDS mode, default to "tpacket-v3".
Required moving the check for tpacket-v3 after the copy-mode
configuration, so the warning about tpacket-v3 in active modes has
been moved as well.
Ticket: #4798
Ticket: 7615
Avoids signatures setting extreme hash sizes, which would lead to very
high memory use.
Default to allowing:
- 65536 per dataset
- 16777216 total
To override these built-in defaults:
```yaml
datasets:
# Limits for per rule dataset instances to avoid rules using too many
# resources.
limits:
# Max value for per dataset `hashsize` setting
#single-hashsize: 65536
# Max combined hashsize values for all datasets.
#total-hashsizes: 16777216
```
With the change of the default tpacket-v2 block size from 32k to 128k,
allow it to be configurable for users who may want to make it larger,
or revert it back to the pre 7.0.9 default of 32k.
Ticket: #7458
This commit adds 2 EVE output buffering settings
- buffer-size value which specifies the amount of buffering, if any,
for regular/file output types.
- flush-interval Specifies the cadence at which Suricata will direct
detect threads to flush EVE output.
Issue: 3449
By default, use an empty search path. This gives us a predictable
default. If a user needs access to external modules, the search path
must be set in the configuration file.
Ticket: #7169
If a Suricata inline IPS device is routing traffic over a
non-encrypted tunnel, like IPv6 tunnels, packets in a flow
will be dropped and not be matched. e.g.
The following example is a Suricata inline IPS with an IPv6 tunnel:
request: IPv4]ICMP] -> |IPS| -> IPv6]IPv4]ICMP]
reply: <- |IPS| <- IPv6]IPv4]ICMP]
Both the IPv4 request and IPv6 reply will be seen by Suricata on
ingress. The flows will not be matched due to flow recursion level.
Optionally use pkt recursion level in flow hash. Excluding recursion
level in flow hash allows matching of packet flows and defrag on an
inline IPS Suricata scenario where the IPS device is a tunnel
terminator.
Feature: 6260
So we get:
1. request arrives - buffered due to not ackd
2. response arrives, acks request - request is now parsed, response isn't
3. ack for response, response parsed. Then detect runs for request,
generates alert. We now have 2 txs. txid will be 0 from AppLayerParserGetTransactionInspectId
But txid 1 is unidirectional in the other way, so we can use txid 0
metadata for logging
Ticket: 7449
TCP urgent handling is a complex topic due to conflicting RFCs and
implementations.
Until now the URG flag and urgent pointer values were simply ignored,
leading to an effective "inline" processing of urgent data. Many
implementations however, do not default to this behavior.
Many actual implementations use the urgent mechanism to send 1 byte of
data out of band to the application.
Complicating the matter is that the way the urgent logic is handled is
generally configurable both of the OS and the app level. So from the
network it is impossible to know with confidence what the settings are.
This patch adds the following policies:
`stream.reassembly.urgent.policy`:
- drop: drop URG packets before they affect the stream engine
- inline: ignore the urgent pointer and process all data inline
- oob (out of band): treat the last byte as out of band
- gap: skip the last byte, but do no adjust sequence offsets, leading to
gaps in the data
For the `oob` option, tracking of a sequence number offset is required,
as the OOB data does "consume" sequence number space. This is limited to
64k. For this reason, there is a second policy:
`stream.reassembly.urgent.oob-limit-policy`:
- drop: drop URG packets before they affect the stream engine
- inline: ignore the urgent pointer and process all data inline
- gap: skip the last byte, but do no adjust sequence offsets, leading to
gaps in the data
Bug: #7411.
Ticket: 7199
Uses a config parameter detect.guess-applayer-tx to enable
this behavior (off by default)
This feature is requested for use cases with signatures not
using app-layer keywords but still targetting application
layer transactions, such as pass/drop rule combination,
or lua usage.
This overrides the previous behavior of checking if the signature
has a content match, by checking if there is only one live
transaction, in addition to the config parameter being set.
Since enabling custom logging will replace the extended logging, thus
possibly leading to certain fields disappearing from the logs, mention
this aspect.
Related to
Bug #7333
This was done following the fact that this setting was historically
named incorrectly. The purpose of the setting was always to define the
ports that will be prioritized and have rule groups associated w them on
priority. Rename all occurences of this to correctly reflect the purpose
of the setting.
Ticket: 7191
So as to avoid quadratic complexity in libhtp.
Make the limit configurable from suricata.yaml,
and have an event when network traffic goes over the limit.
If a stream-only rule matches, and we find a tx where we
want to log the app-layer data, store into the tx data that
we already logged, so that we do not log again the app-layer metadata
Ticket: 7085
This introduces a new parser registration function for LDAP/UDP, and update
ldap configuration in order to be able to enable/disable a single parser
independently (such as dns).
Also, GAPs are accepted only for TCP parser and not for UDP.
Ticket #7203
When replaying a pcap file, it is not possible to get rules
profiling because it has to be activated from the unix socket.
This patch adds a new option to be able to activate profiling
collection at start so a pcap run can get rules profiling
information.
Ticket: 3958
- transactions are now bidirectional
- there is a logger
- gap support is improved with probing for resync
- frames support
- app-layer events
- enip_command keyword accepts now string enumeration as values.
- add enip.status keyword
- add keywords :
enip.product_name, enip.protocol_version, enip.revision,
enip.identity_status, enip.state, enip.serial, enip.product_code,
enip.device_type, enip.vendor_id, enip.capabilities,
enip.cip_attribute, enip.cip_class, enip.cip_instance,
enip.cip_status, enip.cip_extendedstatus
When we added feature #5976 (72146b969), we overlook that we also have
a config stats option for the human-readable stats logs to output
0 counters.
Due to not seeing this before, we now have two different setting names
for basically the same thing, but in different logs:
- zero-valued-counters for EVE
- null-values for stats.log
This ensures we use the same terminology, and change the recently added
one to `null-values`, as this one has been around for longer.
Task #6962
The stats config for EVE logs have a comment about exception policy
stats counters. This went in with 72146b969c, but shouldn't have,
as there are no options there.
Some stats can be quite verbose if logging all zero valued-counters.
This allows users to disable logging such counters. Default is still
true, as that's the expected behavior for the engine.
Task #5976
While our documentation indicated what were the possible configuration
settings for exception policies, our yaml only explicitly mentioned
exception policy for the master switch. Clearly indicate which config
settings are about exception policies.
Related to
Task #5816
Ticket: 5926
HTTP2 continuation frames are defined in RFC 9113.
They allow header blocks to be split over multiple HTTP2 frames.
For Suricata to process correctly these header blocks, it
must do the reassembly of the payload of these HTTP2 frames.
Otherwise, we get incomplete decoding for headers names and/or
values while decoding a single frame.
Design is to add a field to the HTTP2 state, as the RFC states that
these continuation frames form a discrete unit :
> Field blocks MUST be transmitted as a contiguous sequence of frames,
> with no interleaved frames of any other type or from any other stream.
So, we do not have to duplicate this reassembly field per stream id.
Another design choice is to wait for the reassembly to be complete
before doing any decoding, to avoid quadratic complexity on partially
decoding of the data.
When the packet load is low, Suricata can run in interrupt
mode. This more resembles the classic approach of processing
packets - CPU cores run low and only fetch packets
on interrupt.
Ticket: #5839
Add a new configuration flag, "datasets.rules.allow-write" to control
if rules can contain "save" or "state" rules which allow write access
to the file system.
Ticket: #6123
For dataset filenames coming from rules, do not allow filenames that
are absolute or contain a directory traversal with "..". This prevents
datasets from escaping the define data-directory which may allow a bad
rule to overwrite any file that Suricata has permission to write to.
Add a new configuration option,
"datasets.rules.allow-absolute-filenames" to allow absolute filenames
in dataset rules. This will be a way to revert back to the pre 6.0.13
behavior where save/state rules could use any filename.
Ticket: #6118
To protect against possible supply chain attacks, disable Lua rules by
default. They can be enabled under the "security" section of
suricata.yaml.
Ticket: #6122
For the rules profiling, we really want to limit the performance
impact to the maximum. So let's use an hash size that is a power
of 2. This will allow to not use the modulo operation that is
costly and simply use a single binary operator.
This code is only active for rules profiling so we are backward
compatible.
As flow.memcap-policy and defrag.memcap-policy do not support flow
actions, clarify that in the documentation. Also fix some typos, and
add missing values in some places where the exception policies were
explained.
Related to
Bug #5940
Adds a new field, "suricata-version" to the configuration file with
the major and minor version of the Suricata that generated the
configuration file.
This may be useful in the future for presenting warnings about
important changes, or even providing different defaults based on what
the user might expect.
Ticket: 5822
Linux is slightly more permissive wrt timestamps than many
other OS'. To avoid many events/issues with linux hosts, add an
option to allow for this slightly more permissive behavior.
Ideally the host-os config would be used, but in practice this
setting is rarely set up correctly, if at all.
This option is enabled by default.
Debug facility to get a per packet view into the stream engine's state.
Logs after a packet has been processed in the stream engine, so the view
into the state includes the updates based on the current packet.
Marked as experimental so it can be changed w/o notice.
Bug: #5876.
Lukas Sismis
Richard McConnell
Juliana Fajardini
Philippe Antoine
Victor Julien
Alex Savage
Jason Ish
Philippe Antoine
Shivani Bhardwaj
Alfredo Cardigliano
Jeff Lucovsky
Cole Dishington
Pierre Chifflier
Adam Kiripolsky
Sascha Steinbiss
Juliana Fajardini
Jeff Lucovsky
Giuseppe Longo
Eric Leblond
Jo Johnson