Refactor pcap file deletion to use a single delete-when-done option
with three values instead of separate boolean options:
- false (default): No deletion
- true: Always delete files
- "non-alerts": Delete only files with no alerts
Also account for alerts produced by pseudo packets (flow timeout / shutdown flush):
- Introduce small capture hooks and invoke on pseudo-packet creation so the
capture layer can retain references and observe alerts emitted after the last
live packet
- Call the hook from both TmThreadDisableReceiveThreads and TmThreadDrainPacketThreads
Key changes:
- Replace should_delete/delete_non_alerts_only bools with enum
- Move alert counter from global to per-file PcapFileFileVars
- Relocate alert counting from PacketAlertFinalize to pcap module
- Ensure thread safety for both single and continuous pcap modes
- Add unit tests for configuration parsing and pseudo-packet alert path
The --pcap-file-delete command line option overrides YAML config
and forces "always delete" mode for backward compatibility.
Documentation updated to reflect the new three-value configuration.
Fixes OISF#7786
Wrapper around `memmem`.
The case sensitive search is implemented by directly calling `memmem`.
As there is no case insensitieve variant available, a wrapper around
memmem is created, that takes a sliding window approach:
1. take a slice of the haystack
2. convert it to lowercase
3. search it using memmem
4. move window forward
Expose elephant flow flags on a flow via rule language. The rule keyword
is defined as "flow.elephant: <dir>;" where the <dir> can expand to:
- toserver
- toclient
- either
- both
Feature 8117
This patch introduces new option to dataset keyword.
Where regular dataset allows match from sets, dataset with json
format allows the same but also adds JSON data to the alert
event. This data is coming from the set definition it self.
For example, an ipv4 set will look like:
[{"ip": "10.16.1.11", "test": "success","context":3}]
The syntax is a JSON array but it can also be a JSON object
with an array inside. The idea is to directly used data coming
from the API of a threat intel management software.
The syntax of the keyword is the following:
dataset:isset,src_ip,type ip,load src.lst,format json, \
enrichment_key src_ip, value_key ip;
Compare to dataset, it just have a supplementary option key
that is used to indicate in which subobject the JSON value
should be added.
The information is added in the even under the alert.extra
subobject:
"alert": {
"extra": {
"src_ip": {
"ip": "10.6.1.11",
"test": "success",
"context": 3
},
The main interest of the feature is to be able to contextualize
a match. For example, if you have an IOC source, you can do
[
{"buffer": "value1", "actor":"APT28","Country":"FR"},
{"buffer": "value2", "actor":"APT32","Country":"NL"}
]
This way, a single dataset is able to produce context to the
event where it was not possible before and multiple signatures
had to be used.
The format introduced in datajson is an evolution of the
historical datarep format. This has some limitations. For example,
if a user fetch IOCs from a threat intel server there is a large
change that the format will be JSON or XML. Suricata has no support
for the second but can support the first one.
Keeping the key value may seem redundant but it is useful to have it
directly accessible in the extra data to be able to query it
independantly of the signature (where it can be multiple metadata
or even be a transformed metadata).
In some case, when interacting with data (mostly coming from
threat intel servers), the JSON array containing the data
to use is not at the root of the object and it is ncessary
to access a subobject.
This patch implements this with support of key in level1.level2.
This is done via the `array_key` option that contains the path
to the data.
Ticket: #7372
Move the SCThreadInfo global function into the suricata.util library as
thread_info().
This is the last global function to be registered, so remove the
supporting functions.
Similar to flowvars and flowints, but a byte var cannot be registered
from a Lua script, but it still needs to be setup. Instead provide an
"map" function that sets it up, or errors out if the byte var is
unknown.
This also required passing the signature into the Lua init method, as
the state of the Signature object and the time of loading the Lua
keyword is required.
The mDNS support is based heavily on the DNS support, reusing the
existing DNS parser where possible. This meant adding variations on
DNS, as mDNS is a little different. Mainly being that *all* mDNS
traffic is to_server, yet there is still the concept of request and
responses.
Keywords added are:
- mdns.queries.rrname
- mdns.answers.rrname
- mdns.additionals.rrname
- mdns.authorities.rrname
- mdns.response.rrname
They are mostly in-line with the DNS keywords, except
mdns.answers.rdata which is a better than that mdns.response.rrname,
as its actually looking at the rdata, and not rrnames.
mDNS has its own logger that differs from the DNS logger:
- No grouped logging
- In answers/additionals/authorities, the rdata is logged in a field
that is named after the rdata type. For example, "txt" data is no
longer logged in the "rdata" field, but instead a "txt" field. We
currently already did this in DNS for fields that were not a single
buffer, like SOA, SRV, etc. So this makes things more consistent. And
gives query like semantics that the "grouped" object was trying to
provide.
- Types are logged in lower case ("txt" instead of "TXT")
- Flags are logged as an array: "flags": ["aa", "z"]
Ticket: #3952
Issue: 7507
Implement the ftp.completion_code sticky buffer. Multi-buffer as an FTP
command can produce multiple responses.
E.g., with the FTP command RETR
RETR temp.txt
150 Opening BINARY mode data connection for temp.txt (1164 bytes).
226 Transfer complete.
Adds a new lua script capability to use a script as a buffer transform
keyword.
It uses a `transform` lua function that returns the input buffer after
modifying it.
Issue: 2290
Issue: 7506
Add a (non-sticky buffer) keyword for ftp.reply_received. This is not a
sticky buffer as the keyword relates to protocol state and not bytes
from the actual protocol exchange.
ftp.reply_received: yes|on|no|off
Ticket: 6695
This introduction splits the use of the handshake parameters into their
own object, HandshakeParams, which is populated by the TLS decoder. The
JA4 object is now very simple. It's a simple String object (the JA4
Hash) which is generated during new().
This introduction is part of a larger idea, which is to enable
outputting these raw parameters without JA3/JA4. These handshake
parameters are the components used to generate the JA4 hash, thus it
makes sense for it to be a user of HandshakeParams.
This also breaks out the fileinfo function into a method per file info
item. And likewise for state, just return the state and add a new method
for checking if the file is stored.
Ticket: #7491
Victor Julien
Philippe Antoine
Ofer Dagan
Shivani Bhardwaj
Fupeng Zhao
Eric Leblond
Jason Ish
Jeff Lucovsky
Jeff Lucovsky
Richard McConnell