Add user-facing documentation for the xor transform's 'extract <nbytes>
<offset>' syntax and offset parameter in rules/transforms.rst.
Add a devguide section (devguide/extending/detect/transforms.rst)
covering the detection pipeline execution order, how transforms run
before detection-time keywords, and how xor reads key bytes directly
from the raw buffer at a specified position and width. Explains how
the engine deduplicates inspection buffers for rules with variable
keys at different buffer positions.
Issue: 8671
Extend the xor transform to accept a variable-position key using
the 'extract <nbytes> <offset>' syntax. The engine reads <nbytes> bytes
at buffer position <offset> at transform time.
The optional 'offset' parameter specifies where XOR decoding
starts in the buffer; bytes before that position are left unchanged.
Syntax:
xor:"<hex_key>"
xor:extract <nbytes> <offset>
xor:offset <N>,"<hex_key>"
xor:offset <N>,extract <nbytes> <offset>
Variable-key helpers (VariableKeyLocation, variable_key_bytes,
parse_key_location, strip_keyword_prefix) are imported from the new
varkey module.
The variable key bytes are copied out of the inspection buffer before
decoding, so an in-place transform cannot corrupt the key when the key
region overlaps the decoded range.
Issue: 8671
The Quickstart guide installs Suricata from the Ubuntu PPA, where the
config lives at /etc/suricata/suricata.yaml. A source build installs it
under the prefix passed to configure, which defaults to /usr/local, so
the file ends up at /usr/local/etc/suricata/suricata.yaml instead. Add a
note next to the edit step so source-build users are not confused when
the package path does not exist on their system.
Ticket: #8358
Signed-off-by: Samaresh Kumar Singh <ssam3003@gmail.com>
Reading a pcap from /dev/stdin or a named pipe currently fails with "failed to get first packet timestamp. pcap_next_ex(): -1" because InitPcapFile calls setvbuf on the FILE* underlying the pcap handle after libpcap has already consumed the pcap header. On a non-seekable fd glibc cannot recover from that and the next read returns -1.
Detect non-regular files via fstat and skip setvbuf in that case so the read keeps working on pipes, fifos and stdin.
Accept pcap-file.buffer-size values of 0, which disables setvbuf buffering as an explicit opt-out, or PCAP_FILE_BUFFER_SIZE_MIN (4 KiB) to PCAP_FILE_BUFFER_SIZE_MAX (64 MiB). Treat any non-zero setvbuf return value as an error, not just negative values.
When pcap-file.buffer-size fails to parse, retain the default buffer size instead of falling through and setting it to 0. The branches are now mutually exclusive so only one of the parse-error, accepted, or out-of-range messages is logged.
Update the user guide: --pcap-file-buffer-size now documents valid values of 0 (disables setvbuf buffering) or 4 KiB to 64 MiB, and pcap-file.rst notes that 0 is the opt-out for non-seekable sources such as stdin and named pipes.
Bug: #8464.
toggle command is not used by any major rulesets and increases the state
complexity of flowbits management. Also, all operations can be carried
out with the combination of other available commands. So, remove it.
Task 8595
Document that DHCP options carried in the overloaded BOOTP sname or
file fields are now merged into the EVE log option set alongside the
main options area.
Bug: #8538.
Ticket: 8518
Keywords that work for HTTP2 headers match now as soon as possible
A push promise is now considered like a headers frame with regards
to the progress (no dedicated "reserved" progress/state)
http.protocol and http.stat_msg keywords are now registered at
earliest progress, since these are synthetic like "HTTP/2" and
not really seen on the wire.
http.request_line and http.response_line match only on data,
and not on headers, since we must wait the end of headers
to be sure to have the full line
http2.size_update now matches at headers progress as it should
http2.frametype, http2.errorcode, http2.priority now match like
http2.window, when the tx is complete from both sides, as a
half-closed client may still send priority, rst_stream
or window_update frames
When an alert is generated from firewall context, add an engine value of
"fw", otherwise "td" (for threat detect).
The engine field is only added when firewall mode is enabled.
Ticket: #8456
Ticket: 7929
Builds a vector of the reason code in a tx to do so,
except if we use the default "any", where we do not append
to the vector, but just run detection while iterating
Store the NTP reference ID as raw network-order bytes so it can be
exposed as a sticky buffer and matched with payload keywords. The
reference ID is often a 4 character string, or an IP address and not
just an integer identifier.
Updates the log reference ID to be a string of colon separated hex
digits as this matches what tshark does.
Ticket: #8488
Wrap the EVE callback handler with a Rust friendly variant that allows
the user to register a callback as a closure which is provided an
already wrapped JsonBuilder object.
Ticket: #8477
Add --plugin <PATH> to load an additional plugin from the command
line. This is more convenient than "--set plugins.X" especially when
you may already have a plugins loaded and you want to load an
additional one.
Ticket: 8463
Sticky buffer to inspect the ethernet header.
Example rule:
alert ether any any -> any any ( \
ether.hdr; content:"|08 06|"; offset:12; depth:2; \
sid:1;)
Ticket: #8327.
This run-mode does not allow for capture ThreadVars to be created at
the appropriate place in the application life cycle. Instead,
applications should register their own custom run-mode.
Ticket: #8259
If the key `app-layer.protocols.PROTO.enabled` is present, the protocol
is enabled for all carrier protocols. This is not ideal.
Only if the key `app-layer.protocols.PROTO.enabled` is missing, an
attempt is made to look for a setting specific to the ipproto passed
at the time of registration e.g.
`app-layer.protocols.PROTO.udp.enabled`.
By default, check for carrier proto specific setting. If it is not
found, then fall back to the generic setting.
Issue a warning in case an inconsistent combination of global and
ipproto specific setting is found.
Bug 8205
Docs were claiming greater than, less than, etc. support even though this was never supported.
Fixes: 51be8f0238 ("doc/dcerpc: add proto keywords")
Ticket: #8330.
Refactor pcap file deletion to use a single delete-when-done option
with three values instead of separate boolean options:
- false (default): No deletion
- true: Always delete files
- "non-alerts": Delete only files with no alerts
Also account for alerts produced by pseudo packets (flow timeout / shutdown flush):
- Introduce small capture hooks and invoke on pseudo-packet creation so the
capture layer can retain references and observe alerts emitted after the last
live packet
- Call the hook from both TmThreadDisableReceiveThreads and TmThreadDrainPacketThreads
Key changes:
- Replace should_delete/delete_non_alerts_only bools with enum
- Move alert counter from global to per-file PcapFileFileVars
- Relocate alert counting from PacketAlertFinalize to pcap module
- Ensure thread safety for both single and continuous pcap modes
- Add unit tests for configuration parsing and pseudo-packet alert path
The --pcap-file-delete command line option overrides YAML config
and forces "always delete" mode for backward compatibility.
Documentation updated to reflect the new three-value configuration.
Fixes OISF#7786
There is an unfortunate side-affect that one has to read
output-eve-bindgen.h for the documentation on this type, however, I
think we can resolve that in time.
Adding the directory "install" to EXTRA_DIST, actually triggers make
to run "make install", which is not what we want. Instead, avoid this
magic keyword and list the files in the install directory
individually.
If the user doesn't have permission to install files to the prefix,
like "/usr", then "make dist" can fail. Worse, even they do have
permission to write into the prefix, a "make dist" will install files
there when it shouldn't.
Ticket: #8279
Redmine ticket: #8261
According to [1], the within pointer (if combined with distance)
includes the distance pointer, which is not clearly visible in the
graphic.
Fixed this in a new graphic by some GIMP arts.
PS: Special thanks to one of our team members Annika C. for initially
spotting this!
[1] https://forum.suricata.io/t/is-within-affected-by-distance/1688
Issue: 6693
Clarify how the bitmask value is used for byte_jump
Snort compatibility says:
- The bitmask value is applied to the extracted value before the
multiplier is applied.
- The result of the bitmask operation is to be right shifted by the
number of trailing 0's in the bitmask value.
Add authentication support to the Redis logging output.
It introduces `username` and `password` configuration options for Redis,
allowing Suricata to authenticate with Redis servers that require it.
Ticket: 7062
Add optional unique_on {src_port|dst_port} to detection_filter for
exact distinct port counting within the seconds window.
Features:
- Runtime uses a single 64k-bit (8192 bytes) union bitmap per
threshold entry with O(1) updates.
- Follows detection_filter semantics: alerting starts after the
threshold (> count), not at it.
- On window expiry, the window is reset and the current packet's
port is recorded as the first distinct of the new window.
Validation:
- unique_on requires a ported transport protocol; reject rules
that are not tcp/udp/sctp or that use ip (protocol any).
Memory management:
- Bitmap memory is bounded by detect.thresholds.memcap.
- New counters: bitmap_memuse and bitmap_alloc_fail.
Tests:
- C unit tests for parsing, distinct counting, window reset, and
allocation failure fallback.
- suricata-verify tests for distinct src/dst port counting.
Task #7928
Adds a chapter indicating what are the main steps when adding exception
policies, how is it possible to extend them, as well as main aspects and
files to consider when doing so.
Task #5612
Hyperscan MPM can cache the compiled contexts to files.
This however grows as rulesets change and leads to bloating
the system. This addition prunes the stale cache files based
on their modified file timestamp.
Part of this work incorporates new model for MPM cache stats
to split it out from the cache save function and aggregate
cache-related stats in one place (newly added pruning).
Ticket: 7830
Jeff Lucovsky
Samaresh Kumar Singh
Giuseppe Longo
Antoine Abou Faysal
Shivani Bhardwaj
Juliana Fajardini
Philippe Antoine
Jason Ish
Giuseppe Longo
Victor Julien
jason taylor
Andreas Dolp
Sven Cuyt
Ofer Dagan
Mingyu Jeon
Fupeng Zhao
Lukas Sismis