Add a `firewall` stats counter aggregator for all firewall-related
stats.
De-overload "detect.alert_queue_overflow", by adding
"firewall.discarded_alerts" to account for discarded drops in
Firewall mode.
Add Debug statements for tracking corner cases where it can be
difficult to know where a drop is coming from.
Added counters:
- stats.firewall.blocked
- stats.firewall.accepted
- stats.firewall.rejected
- stats.firewall.drop_reason.default_app_policy
- stats.firewall.drop_reason.default_packet_policy
- stats.firewall.drop_reason.flow_drop
- stats.firewall.drop_reason.pre_flow_hook
- stats.firewall.drop_reason.pre_stream_hook
- stats.firewall.drop_reason.rules
- stats.firewall.discarded_alerts
Ticket #7699
Always make sure, that smd is not NULL, as we dereference it a few
lines below and base64_data for instance is forbidden as a fast_pattern
Ticket: 8504
If a runmode is missing but other params are passed on the commandline,
issue an error.
Ticket: #5711
Signed-off-by: Promise Charles <descencybobby@gmail.com>
Support scan-build's taint filtering logic. Adds a yaml config to mark
a dedicated debug validation function as filtering a taint.
Use to suppress warnings in reference, threshold and classification file
handling.
Ticket: #3153.
detect-engine.c:2350:69: warning: Out of bound access to memory after the end of 'new_det_ctx' [security.ArrayBound]
2350 | FlowWorkerReplaceDetectCtx(SC_ATOMIC_GET(s->slot_data), new_det_ctx[i]);
| ^~~~~~~~~~~~~~
1 warning generated.
suricata.c:574:5: warning: Potential out of bound access to the heap area with tainted index [security.ArrayBound]
574 | bpf_filter[nm] = '\0';
| ^~~~~~~~~~~~~~
1 warning generated.
Ticket: 8529
When Suricata handles a HTTP1 response body, it does so with a
file, and tries to get the filename from the Content-Disposition
header if any, then from the uri.
If it failed to find a file name, it tried again every time
there was new data from the response body, even if there was
no new data to find a file name in either the header nor the uri.
This causes a slowdown in the case the Content-Disposition header
is big.
Fix is to set the flag on the first call of the callback, to be
sure that we will parse the Content-Disposition header for a
filename header only once per http1 response.
Decoding the reassembled packet inside DefragInsertFrag while holding
the tracker mutex creates a deadlock situation if the reassembled packet
contains tunneled fragments: two threads could each hold one tracker
mutex and deadlock on the other. Move the decode in Defrag() after
DefragTrackerRelease() so no lock is held during decode (and possibly
re-entering the defrag system).
Make sure to fetch tracker->ip_hdr_offset after DefragInsertFrag since
it may be written during that call if the first fragment arrives last.
Issue: 8550
Ticket: 8526
As some C protocols accepted case-insensitive frame names,
a rule using the same frame twice with different cases,
ended up using 2 different buffers,
which ended up in triggering a buffer overflow
Ticket: 8510
Without this check, in case of a hash collision, and the other
fields are equal, we could end up consider adding an IPv6 packet
to a IPv4 tracker (or vice versa).
Doing so, we end up interpreting an IPv6 packet as an IPv4 one,
and we do not benefit from the bounds checks from ipv4/ipv6 decoder.
Allow configurable policies, including accept. For app-layer this
requires looping all available hooks to apply the policies.
Support configurable policies for packet-filter, pre-stream, pre-flow.
If there are no rules there is also no rule group (sgh). Make sure
the app hooks policies are correctly handled in this case by allowing
a NULL sgh to be handled as well.
For tx rule match actually apply drop directly. Previously this was
always handled by the default drop:flow policy.
Ticket: #7701.
Previously a `accept:flow` action would act as both a firewall "accept" and
a threat detection "pass" for the rest of the flow.
This patch changes that. The `accept:flow` action now only accepts the
rest of the packets for the firewall ruleset, but does still continue
threat detection rule evaluation.
Ticket: #8444.
For firewall rules, allow multiple actions to be specified in a list
accept:flow,pass:flow,alert
accept:flow,alert
accept:flow,pass:flow
It is mandatory to make the first action the primary firewall policy
action: accept, drop, reject.
Ticket: #8480.
If a ruleset would use `dns:request_complete` but not have a rule for
`dns:request_started`, the `request_started` hook default policy would
not get invoked.
Add a check to make sure it is invoked.
Ticket: #8495.
Clean up host mode tracking, which is used by reject to control how
rejects are sent. Before this patch there were 2 modes: sniffer only
and router. This patch introduces a bridge mode that is automatically
set by the bridge modes. In bridge mode the `Packet::livedev` is used.
Ticket: #8390.
Ticket: 8336
At the end of a TLS handshake, in IDS mode, the client acks,
and we parse the server hello and use tls.encryption-handling
to know what to do next (for example bypass)
Everything is parsed, but we have not run detection yet on neither
side.
So, in IDS mode, we need to first flush the client side, as the
comment on the function already stated.
Ticket: 8336
When a packet has flag PKT_PSEUDO_DETECTLOG_FLUSH, we do not
expect to rerun detection on the same tx and direction again
So, do not set mpm_in_progress whose purpose is to not store
the state as we will run again.
Allows transactional bidirectional signatures to work on
thse log+flush pair of packets
When an alert is generated from firewall context, add an engine value of
"fw", otherwise "td" (for threat detect).
The engine field is only added when firewall mode is enabled.
Ticket: #8456
The first version of the event set done=true on state->curr_tx and
returned NULL when the limit hit. curr_tx is usually the active
command just received — marking it done mid-request throws off
response matching, and subsequent commands in the flow stop getting
logged.
SMB behavior is mirrored here:
Walk the tx list, find the oldest tx that isn't done, mark
it done and tag it with the event, then fall through and
create the new tx so the flow parsing continues. One stale tx gets
reaped per overflow so memory stays bounded.
Issue: 8489
Other parsers (SMB, POP3, ENIP) already raise a too_many_transactions
event when a flow exceeds its per-flow transaction limit. FTP had
the limit wired up through app-layer.protocols.ftp.max-tx but just
dropped the excess transaction on the floor with a "FTP does not set
events yet..." TODO, so no event was raised when the limit was
reached.
Add the FtpEventTooManyTransactions variant, a stock rule at
sid 2232002, and raise the event from FTPTransactionCreate when the
live-tx count goes past ftp_config_maxtx.
Issue: 8489
when engine-analysis is turned off in suricata.yaml
and we run suricata with --engine-analysis
suricata will try to use data from de_ctx->ea wich is
free'd in SetupEngineAnalysis() because fp_analysis and
rule_analysis is turned off and will receive SIGSEGV
with something like:
0x00007ffff7b4ac7b in _IO_new_fclose (fp=0xad96ab1cf5b38042) at ./libio/iofclose.c:48
Ticket: 8505
Issue: 7672
The subslice transform creates a slice of the input buffer.
Specify the subslice desired -- nbytes and truncate are optional:
subslice: offset <,nbytes> <,truncate>
offset: Specifies the starting offset for the new subslice. When
negative, expresses how far from the end of the input buffer to begin.
nbytes: Specifies the size of the subslice. When negative, specifies
that the subslice will end that many bytes from the end of the input
buffer. Nbytes must be non-zero.
When nbytes is not specified, the size of the subslice will be the size
of the input buffer - offset.
truncate: Specify behavior when offset + nbytes exceeds buffer length.
When present, trims nbytes such that offset + nbytes equals buffer
length. When not present, an empty buffer is produced.
Examples:
subslice: 1; - The subslice will be a copy of the input
buffer but omits the input buffer's first byte
"This is Suricata" -> "his is Suricata"
subslice: 0, 13; - The slice is created from the first 13 bytes
of the input buffer
"This is Suricata" -> "This is Suric"
subslice: 10, -5; - The subslice is created starting at offset 10
and continues to 5 bytes before the end of the input buffer
"This is Suricata" -> "r"
subslice: -3; - The subslice will be the last 3 bytes of the
input buffer.
"This is Suricata" -> "ata"
Ticket: 8501
(m)dns.*.rrname keywords required the rules to start with
alert (m)dns
and refused to load if the rule started with generic alert ip or
alert udp, with error message
Error: detect-parse: invalid alproto 0
When using ASAN, flto=auto and compiling on a platform that supports
AVX-512, this memcpy is optimized as its exactly 64 bytes. However, when
using ASAN, there is a fake stack that may not be 64 bytes aligned, and
this AVX write lands on an unaligned byte, causing a crash.
Use strlcpy avoids this optimization as it needs to find the NUL byte.
Ticket: 6813
So, in the case we have a ftp-data flow with data to client as
expected, but after to_client side completes (with timeout for
example), the client sends some data to server,
In this case, we want to check first the direction, and bail out,
then do the assertion check about the finished state.
As a legacy from Suricata 7, mempool size was kept even internally
as a global size. Since in Suricata 8, mempool allocation moved to
per queue model, it makes sense to calculate the correct size right
when loading the configuration.
The previous implementation misscalculated in some scenarios the size
of the mempool which prevented Suricata starting successfully.
This happened, e.g. when there were 32k RX descriptors and 64 TX descs.
The resulting mempool was incorrectly deduced as 32767 (2^n-1), which
didn't even hold all RX descriptors. Instead, it should result to 65535.
Ticket: 8370
If the last signature in the candidates list is also the last for the
current progress value, it needs to be considered as the last for that
progress, so that if it doesn't match the default policy is applied.
Ticket: 8395
For protocols where tx are complete at start, that is
the `tx_get_progress` function returns 1 inconditionnally,
we can use the progress 1, not losing time on detection in ips
mode, and getting ready for firewall mode
For the other protocols (where a tx needs both a request and response
for instance), keep progress 0 as was the case before
Fixes warning:
conf-yaml-loader.c: In function ‘ConfYamlSetConfDirname’:
conf-yaml-loader.c:87:8: warning: assignment discards ‘const’ qualifier from pointer target type [-Wdiscarded-qualifiers]
87 | ep = strrchr(filename, '\\');
| ^
Add --plugin <PATH> to load an additional plugin from the command
line. This is more convenient than "--set plugins.X" especially when
you may already have a plugins loaded and you want to load an
additional one.
Ticket: 8463
CID 1645613
Ticket: 7664
We assign and dereference next_s like
next_s = *match_array++;
next_sflags = next_s->flags;
So, there is no need to check if next_s is NULL, as it can never
be (otherwise, we could crash before)
Internals
---------
Suricata's stream engine returns data for inspection to the detection
engine from the stream when the chunk size is reached.
Bug
---
Inspection triggered only in the specified chunk sizes may be too late
when it comes to inspection of smaller protocol specific data which
could result in delayed inspection, incorrect data logged with a transaction
and logs misindicating the pkt that triggered an alert.
Fix
---
Fix this by making an explicit call from all respective applayer parsers to
trigger raw stream inspection which shall make the data available for inspection
in the following call of the stream engine. This needs to happen per direction
on the completion of an entity like a request or a response.
Important notes
---------------
1. The above mentioned behavior with and without this patch is
affected internally by the following conditions.
- inspection depth
- stream depth
In these special cases, the inspection window will be affected and
Suricata may not consider all the data that could be expected to be
inspected.
2. This only applies to applayer protocols running over TCP.
3. The inspection window is only considered up to the ACK'd data.
4. This entire issue is about IDS mode only.
FTP parser can handle multiple command lines per direction. Appropriate calls
to trigger raw stream inspection have been added on succesful parsing of each
request line and response line.
Bug 7742
`alert pkthdr` was initially just an alias for `alert ip`, as that was
really just a way of stating that "any" should be matched. However with
the Ethernet matching in place, it no long makes sense to treat `alert
ip` as "any". Since `pkthdr` is used to match on decoder events, also
for packets that completely failed to parse, it should no longer be
treated as `alert ip` but rather as it's own distinct logic.
Sticky buffer to inspect the ethernet header.
Example rule:
alert ether any any -> any any ( \
ether.hdr; content:"|08 06|"; offset:12; depth:2; \
sid:1;)
Ticket: #8327.
Make `Signature::proto` an optional member, meaning that if it is
NULL we can skip the check. This can be done for `alert ip`, as no check
is needed, and for `alert tcp` and `alert udp` as having a rule in a sgh
for those means that the protocol matches.
Some exceptions are rules that require:
- ipv4/ipv6 specific matching
- frames, due to sharing prefilter between tcp and udp
- ip-only rules, due to those not being per sgh
Add flushing logic driven off of the file contexts. This is a simpler
solution that removes the need for logger registration changes.
Overview:
Use the heartbeat-driven thread to periodically flush all registered EVE
contexts via a global flush list.
The global flush list is a mutex-protected TAILQ of LogFileFlushEntry
nodes; each node points to a LogFileCtx. Mutex = log_file_flush_mutex
Periodic flushing performed by a thread according to the
heartbeat.output-flush-interval [1,60]. LogFileFlushAll() is invoked to
initiate flushing of registered LogFileCtx structs; each struct's
fp_mutex is obtained while the flush occurs to synchronize with
LogFileWrite activity.
Interacts with file-rotation via the fp_mutex.
Deadlock prevention: the log_file_flush_mutex must be obtained before
the fp_mutex.
Issue: 8286
Ticket: 8333
Allows to pass on pseudo packets
So, that a ruleset like
pass ssh any any -> any any (geoip:any,US,IN; sid: 1;)
drop ip any any -> any any (flow:established, to_server; sid:2;)
Does not lead to a drop on the pseudo-packet when we flush
because ssh traffic gets encrypted
Simplify statements where there's a conditional statement followed by
return true or return false.
Instead of
if (conditional)
return true;
else
return false;
Use:
return conditional
In some timezones, LogCustomFormatTest01 was failing as we were
specifying the wrong setting for is_dst. Instead set it to -1 so mktime
can make the decision based on the environment. Appears to fix this test
for "Brazil/East" in the summer.
Add GitHub CI tests to test this specific test with a few different
timezones.
When during raw reassembly it is detected that last ack is moved beyond
the progress and also beyond the data retrieved, update progress to the
last ack value.
Bug: #8272.
When mempool size was configured really low (<32), Suricata exhausted
the mempool with the rx_burst call, which led to undefined behavior.
The current fix ensures the burst size is at most the size of the mempool,
if the mempool is smaller than BURST_SIZE macro.
Based on review it seems p->proto should always be set to the same
value as what `PacketGetIPProto` returns. Add a check to validate
in a decoder that the fuzzer executes.
Move the locking mechanism outside of the getter function and hold the
lock until the reference string is no longer reused.
** CID 1682023: Concurrent data access violations (MISSING_LOCK)
/src/util-mpm-hs-cache.c: 139 in HSGetReferenceDbInfo()
This run-mode does not allow for capture ThreadVars to be created at
the appropriate place in the application life cycle. Instead,
applications should register their own custom run-mode.
Ticket: #8259
If the key `app-layer.protocols.PROTO.enabled` is present, the protocol
is enabled for all carrier protocols. This is not ideal.
Only if the key `app-layer.protocols.PROTO.enabled` is missing, an
attempt is made to look for a setting specific to the ipproto passed
at the time of registration e.g.
`app-layer.protocols.PROTO.udp.enabled`.
By default, check for carrier proto specific setting. If it is not
found, then fall back to the generic setting.
Issue a warning in case an inconsistent combination of global and
ipproto specific setting is found.
Bug 8205
DPDK interface which were added to bond and had auto-calculated
mempool size didn't have enough space as the configured
descriptors and the derived mempool only considered one interface
per YAML node entry.
DPDK Bonding PMD allows for multiple interface to cooperate and
therefore, needs more space.
Ticket: 8125
Following DPDK's recommendation to size mempool to 2^n - 1,
the current implementation was flawed for certain combos,
e.g. mp_size 1023 and 1 queue, which resulted to 1022.
Hash Hyperscan installation info (version, CPU features, mode)
into the cache filename. A Hyperscan upgrade or platform change
would now produce a different filename, so stale caches from an
older installation are never opened.
Ticket: 8325
After deserializing a cached Hyperscan database, verify that its
version, CPU features, and mode match the current Hyperscan
installation by comparing hs_database_info output against a
reference database. Reject loading incompatible caches.
Ticket: 8325
HSSaveCache wrote serialized Hyperscan databases using text mode ("w")
while HSReadStream already read them with binary mode ("rb").
Matched file reading modes to the binary format and simplified
write-size check.
Ticket: 8325
HSLoadCache opened the cache file but never used the resulting handle
for reading. The actual read was done by HSReadStream which opened
the same file independently.
Removed the unused fopen/fclose pair and flattened the control flow.
Ticket: 8325
Reduce logging level of a minor informational message.
Split tracking of pruning by age and by version and log those
separately, where the logging only appears if something has been
removed.
Ticket: 8322
Ticket: 7717
Allows for instance to process/log ARP packets over VXLAN.
That means we need to decode the ethernet layer above vxlan
instead of skipping it as part of the vxlan, even if the vxlan
decoder still checks the ethernet layer to avoid FPs.
Instead of directly accessing the field
Will allow PacketTunnelType to hold the precise tunnel type like
DECODE_TUNNEL_ERSPANII with a modification of PacketIsTunnelChild
Refactor pcap file deletion to use a single delete-when-done option
with three values instead of separate boolean options:
- false (default): No deletion
- true: Always delete files
- "non-alerts": Delete only files with no alerts
Also account for alerts produced by pseudo packets (flow timeout / shutdown flush):
- Introduce small capture hooks and invoke on pseudo-packet creation so the
capture layer can retain references and observe alerts emitted after the last
live packet
- Call the hook from both TmThreadDisableReceiveThreads and TmThreadDrainPacketThreads
Key changes:
- Replace should_delete/delete_non_alerts_only bools with enum
- Move alert counter from global to per-file PcapFileFileVars
- Relocate alert counting from PacketAlertFinalize to pcap module
- Ensure thread safety for both single and continuous pcap modes
- Add unit tests for configuration parsing and pseudo-packet alert path
The --pcap-file-delete command line option overrides YAML config
and forces "always delete" mode for backward compatibility.
Documentation updated to reflect the new three-value configuration.
Fixes OISF#7786
Bindgen the Rust bindings to the C JsonBuilder API along with the rest
of the Rust bindings to C. Breaking it out was probably the wrong
idea.
This should make it easier, and more correct to bindgen C functions
that use SCJsonBuilder types.
There is an unfortunate side-affect that one has to read
output-eve-bindgen.h for the documentation on this type, however, I
think we can resolve that in time.
As output-eve-bindgen.h exists to support bindgen, its odd to see
other Suricata C files using it. Instead Suricata C code should import
"output-eve.h", which itself includes "output-eve-bindgen.h", only
broken out to support the external tool bindgen.
Replace duplicated SCFree() calls in error paths with a single
cleanup label using goto pattern. This reduces code duplication
and ensures consistent resource cleanup.
Additional improvements:
- Fix misleading error message when xstats table size changes
between calls (was passing positive value to rte_strerror)
- Use unsigned int for length/index to match DPDK API semantics
- Initialize xstats_names to NULL at declaration for safe cleanup
- Added "not supported" case when the first call to xstats_get returns 0
Ticket: 8273
... to account for midstream sessions.
Commit 497394e removed inspection of app-proto txs for packets
without an established TCP connection. But this meant that the
first packet seen in a session pick mid-stream could go without
inspection (previous bug 5510 seemed to point towards this behavior,
too).
If a flow has more packets, the stream will be inspected as part of
the upcoming packets and this would go unnoticed. In a single-packet
flow, however, the inspection for the packed would be skipped. Although
this might not affect alerts -- as they could be processed as part of
the flow timeout logic, the actual traffic could be evaded in IPS, in
case of a drop rule.
From the above, the most visible scenario is when there is only one packet on the flow,
as then the engine doesn't have "more time" to pick-up real-packets to
inspect for that given flow. But certain tests show that this can also
happen for more than one packet scenarios: there will be one less drop
event, or traffic from a packet that should have been already dropped
will be logged.
This led to the possibility of a real packet not being blocked, in IPS,
or matched against rules, as the corresponding portion of the stream
was only inspected later, as part of the stream/flow-timeout logic.
To ensure that we correctly flag the first packet seen for a given mid-stream
session, we must check for the session state and existance *after* we
have dealt with TCP flags and state.
Related to
Bug #5510
As part of
Bug #5180
During initialization, the engine reports how many rules were loaded, as
well as which types. Pkt-only or stream-pkt rules would cause a "hole"
in such stats, as they're not counted.
and bindgen it to rust
Will make easier the bindgen of RustParser structure which uses
a callback which uses AppLayerTxData
Move also the free function to C SCAppLayerTxDataCleanup
As suricata-sys crate defines AppLayerTxData for rust,
It must itself implement the Drop trait, and thus,
We need to define a feature surest
and bindgen it to rust
Will make easier the bindgen of RustParser structure which uses
a callback which uses AppLayerResult
Keep From<> impl in sys crate that defines it
and bindgen it to rust, and use default trait instead of new
Will make easier the bindgen of RustParser structure which uses
a callback which uses AppLayerStateData
Makes leak sanitizer work without adding LSAN_OPTIONS=use_unaligned=1
Otherwise, leak sanitizer may report the read and black tree pointers
as leaked when it is still owned by some global variable.
Follow-up on d046e82db6
Add authentication support to the Redis logging output.
It introduces `username` and `password` configuration options for Redis,
allowing Suricata to authenticate with Redis servers that require it.
Ticket: 7062
Some drivers (e.g. BNXT) fail to report stats after the device is
stopped. Move stats collection (DPDKDumpCounters and PrintDPDKPortXstats)
to run before rte_eth_dev_stop() in HandleShutdown.
Also change PrintDPDKPortXstats error handling from FatalError to
graceful return since stats collection failures during shutdown
should not crash the application.
The commit removes ThreadExitPrintStats callback as the function had no
useful features after the stats were moved.
Ticket: 8251
To explain a bit more the TOCTOU issue found, we can consider
a case where Suricata starts to prune, yet externally somebody also
starts erasing cache files.
Right after Suricata checks the file age with the stat function,
somebody may delete or update the file of our interest.
Suricata aging decision doesn't reflect the actual state of the file.
This commit additionally adds a check for noent failure of the unlink operation
(considered as a success). The code can still delete a file that is recently
updated but was considered stale.
In the documentation-following deployments this should not happen anyway as
one cache folder should only be used by a single Suricata instance (and within
Suricata instance only one thread handles cache eviction).
Additionally, the `stat` and `unlink` command are immediatelly followed, making
this scenario extra unlikely.
Additional comment in the code explains problems of using fstat and potential
issues on Windows.
Ticket: 8243
Add optional unique_on {src_port|dst_port} to detection_filter for
exact distinct port counting within the seconds window.
Features:
- Runtime uses a single 64k-bit (8192 bytes) union bitmap per
threshold entry with O(1) updates.
- Follows detection_filter semantics: alerting starts after the
threshold (> count), not at it.
- On window expiry, the window is reset and the current packet's
port is recorded as the first distinct of the new window.
Validation:
- unique_on requires a ported transport protocol; reject rules
that are not tcp/udp/sctp or that use ip (protocol any).
Memory management:
- Bitmap memory is bounded by detect.thresholds.memcap.
- New counters: bitmap_memuse and bitmap_alloc_fail.
Tests:
- C unit tests for parsing, distinct counting, window reset, and
allocation failure fallback.
- suricata-verify tests for distinct src/dst port counting.
Task #7928
Makes leak sanitizer work without adding LSAN_OPTIONS=use_unaligned=1
Otherwise, leak sanitizer may report rule_id as leaked
when it is still owned by some global variable
When not sandboxed, a script can get access to the metatable and call
`.__gc` with an invalid value like nil, causing a NULL pointer dereference
in Suricata.
Ticket: #8248
This should make it possible to catch invalid combinations in the same
signature early. This patch covers checking and erroring on the following
invalid cmd combinations:
- set + isset
- unset + isnotset
- set + toggle
- set + unset
- isset + isnotset
- unset + toggle
the same flowbit in the same signature which is basically an unnecessary
operation at runtime.
This also helps bring down the difficulty of handling of actual complex
flowbit chains.
Bug 7772
Bug 7773
Bug 7774
Bug 7817
Bug 7818
Bug 8166
Code refactor to gather all PCAP-related structure members
under one structure.
New pcap_v structure guards protect the union variables from
other capture modes trying to access the packet number incorrectly.
Ticket: 7835
For an easier review process, this is a two-step change process,
in which pcap_cnt is first accessed by functions-to-be, implemented
as simple macros.
In the follow-up commit, the actual refactor is implemented with the new
function. The old macros are deleted.
Ticket: 7835
Use of t_pcapcnt is only relevant when compiled in debug mode only.
This patch adds additional debug guard to also shield the declaration
and assignment.
It does not return anything.
The only known implementation is in dns where it does not return.
And the C code does not bother to check a return value anyways.
Add basic derive counter support, where a counters value is derived from
2 other counters.
Add DERIVE_DIV that divides the first counter value by the second
counters value.
Convert `decoder.avg_pkt_size` to be derived from `decoder.bytes` and
`decoder.pkts`.
Ticket: #5615.
When in a `base64_decode`-`base64_data` pair the decode was depending
on another match through the relative option, the `buffer_offset` would
be updated to the relative position of the previous match. During the
`base64_data` phase, a relative match would use that offset even though
the match happened in a new buffer.
Example::
http.request_body; content:"|27|"; \
base64_decode:relative; \
base64_data; content:"|ff ff ff ff|"; within:16;
This use of the `buffer_offset` is incorrect as that value is relative
to a buffer and the `base64_data` points to a new buffer.
This patch addresses this by resetting DetectEngineThreadCtx::buffer_offset
before inspecting `base64_data`.
Bug: #7842.
Adds a chapter indicating what are the main steps when adding exception
policies, how is it possible to extend them, as well as main aspects and
files to consider when doing so.
Task #5612
Hyperscan MPM can cache the compiled contexts to files.
This however grows as rulesets change and leads to bloating
the system. This addition prunes the stale cache files based
on their modified file timestamp.
Part of this work incorporates new model for MPM cache stats
to split it out from the cache save function and aggregate
cache-related stats in one place (newly added pruning).
Ticket: 7830
As a intermediary step for Hyperscan (MPM) caching,
the MPM config initialization should be part of the default
detect engine context for later dynamic retrieval.
Ticket: 7830
To have a system-level overview of when was the last time the file was
used, update the file modification timestamp to to the current time.
This is needed to remove stale cache files of the system.
Access time is not used as it may be, on the system level, disabled.
Ticket: 7830
If recursive call to DetectPortInsert (in case DetectPortCut returned
a c != NULL) failed, we had successfullt inserted `new` but
we return error, leading to freeing DetectPort `new` in
DetectPortParseInsertString but it
will be freed again by DetectPortCleanupList from DetectPortParse
Ticket: 6187
Wrapper around `memmem`.
The case sensitive search is implemented by directly calling `memmem`.
As there is no case insensitieve variant available, a wrapper around
memmem is created, that takes a sliding window approach:
1. take a slice of the haystack
2. convert it to lowercase
3. search it using memmem
4. move window forward
Firewall rules were supposed to also load when using a config like
##
## Suricata as a Firewall options (experimental)
##
firewall:
# toggle to enable firewall mode
enabled: yes
# Firewall rule file are in their own path and are not managed
# by Suricata-Update.
rule-path: /etc/suricata/firewall/
# List of files with firewall rules. Order matters, files are loaded
# in order and rules are applied in that order (per state, see docs)
rule-files:
- firewall.rules
But they were not because of a missing check.
Bug: #8206.
16384 is used as the max, but a configuration parameter has been
provided. The reason for setting an upper bound is that bit flags can
create a memory amplification as we parse them into individual data
structures.
Ticket: #8181
Lower the number of unreplied requests from 500 to 32 to consider a
flood. At the very least this is an anomaly given the DNP3 spec mentions
that DNP3 should only have one outstanding request at a time, with an
exception for unsolicited responses, so in practice no more than 2
should be seen.
Additionally make this value configurable by introducing the max-tx
parameter.
Ticket: #8181
Complete is a flag used to tell if the message was completely parsed,
as not all messages may be completely parsed if we don't know all
their objects. However, they are still "done".
In the alstate-progress callback, check the done flag, not the
complete flag.
Ticket: #8181
So far, the alert queue was expanded by doubling in size w/o any
boundary checks in place. This led to situations where doubling
the alert_queue_capacity meant overflow of the very same value
stored in det_ctx.
This led to heap-use-after-free in some conditions where
det_ctx->alert_queue_capacity overflowed.
Fix this by capping the max of alert_queue_capacity by checking if its
expansion could result in an overflow.
Security 8190
Ticket: 8156
In case of non-tx alerts, we try to loop over all the txs to find
the xff header. Do not start from tx_id 0, but from min_id
as AppLayerParserTransactionsCleanup to skip txs that were freed
TLS parsers use x509-parser crate which parses X.509 certificates that
use ASN.1 DER encoding that can allow arbitrary byte sequences. An
attacker could inject null byte in a certificate anywhere to stump the
common language parsers terminating the string at a null byte leading to
a bypass of a possibly malicious certificate.
So far, the rust TLS parser for "subjectaltname" used a pattern that involved:
-> Get ASN.1 DER encoded raw data from the x509-parser crate
-> Convert this raw data to a decoded string (Rust)
-> Convert the Rust string to CString
-- The problem lies here. CString only accepts proper strings/byte
buffers and converts it into an owned C-compatible, null-terminated
string. However, if any null byte occurs in the string passed to the
CString then it panics.
In the rust TLS parser, this panic is handled by returning NULL.
This means that the parser will error out during the decoding of the
certificate. However, Suricata must be able to detect the null byte
injection attack being an IDS/IPS.
Hence, replace all such string patterns w.r.t. TLS SAN with a byte
array.
Bug 7887
TLS parsers use x509-parser crate which parses X.509 certificates that
use ASN.1 DER encoding that can allow arbitrary byte sequences. An
attacker could inject null byte in a certificate anywhere to stump the
common language parsers terminating the string at a null byte leading to
a bypass of a possibly malicious certificate.
So far, the rust TLS parser for "issuerdn" used a pattern that involved:
-> Get ASN.1 DER encoded raw data from the x509-parser crate
-> Convert this raw data to a decoded string (Rust)
-> Convert the Rust string to CString
-- The problem lies here. CString only accepts proper strings/byte
buffers and converts it into an owned C-compatible, null-terminated
string. However, if any null byte occurs in the string passed to the
CString then it panics.
In the rust TLS parser, this panic is handled by returning NULL.
This means that the parser will error out during the decoding of the
certificate. However, Suricata must be able to detect the null byte
injection attack being an IDS/IPS.
Hence, replace all such string patterns w.r.t. TLS IssuerDN with a byte
array.
Bug 7887
TLS parsers use x509-parser crate which parses X.509 certificates that
use ASN.1 DER encoding that can allow arbitrary byte sequences. An
attacker could inject null byte in a certificate anywhere to stump the
common language parsers terminating the string at a null byte leading to
a bypass of a possibly malicious certificate.
So far, the rust TLS parser for "Subject" used a pattern that involved:
-> Get ASN.1 DER encoded raw data from the x509-parser crate
-> Convert this raw data to a decoded string (Rust)
-> Convert the Rust string to CString
-- The problem lies here. CString only accepts proper strings/byte
buffers and converts it into an owned C-compatible, null-terminated
string. However, if any null byte occurs in the string passed to the
CString then it panics.
In the rust TLS parser, this panic is handled by returning NULL.
This means that the parser will error out during the decoding of the
certificate. However, Suricata must be able to detect the null byte
injection attack being an IDS/IPS.
Hence, replace all such string patterns w.r.t. TLS Subject with a byte
array.
Bug 7887
When FlowSwap() reverses the direction of a flow, the MAC address sets
stored in the flow also need to be swapped to maintain consistency with
the new direction. Previously, MAC addresses were not swapped along with
other flow properties like packet/byte counters.
Ticket #8172
Make sure the script can use all bytes configured. So exclude setup like
input buffers that are put on the lua state before script is executed.
Bug #8173.
Expose elephant flow flags on a flow via rule language. The rule keyword
is defined as "flow.elephant: <dir>;" where the <dir> can expand to:
- toserver
- toclient
- either
- both
Feature 8117
The tracking for elephant flow is done per direction however, the flag
was set on the flow whenever either of the directions crossed the
rate-limit defined in the settings. Given that the tracking was already
split, it makes sense to split the detection tracking per direction as
well and allow user to have a better control via rule language.
The functionality of "both" can already be easily achieved by using both
"toclient" and "toserver" in a rule. This just adds the ease of
expression in rule. As it is added now, check the direction for the
pre-existing users of direction as well.
Detection and logging skip a lot of work if PKT_STREAM_EST is not set. When
a TFO packet with data comes in the TCP state is not yet established, but
the data still needs to be considered.
So for this case set the PKT_STREAM_EST flag.
Bug #6744.
Issue: 7389
Follow-up change for verbose handling. Instead of displaying the numeric
verbose level, display the level name.
I.e., instead of 4, display Debug
Ticket: 3220
DetectSslVersionMatch did not handle properly negation.
It could never match on a signatrue with ssl_version: !tls1.3
That is because, if we had such a signature and network traffic
with tls1.1, we were looking into DetectSslVersionData field
for tls1.1, which was not set, instead of looking at field
for tls1.3 which was set with negated flag.
Previous DetectSslVersionData was holding redundant information.
It did not need to have it for each ssl version, but just globally.
Also, it did not need to hold the version as a value in the array,
as it was redundant with the index of the array.
Issue: 7389
Continue to recognize multiple 'v' specifications up to the maximum
recognized. The maximum recognized is the number of slots between the
NOTICE and DEBUG levels.
When debug is configured, `-vvvv` is required to set the log level to
debug. Specifying 4 or more `v` will maintain debug level.
When debug is not configured, `-vvvv` and more `v`'s will cause
config messages to be printed.
If we have
- stream->last_ack 0x40021
Then, we call StreamTcpUpdateLastAck with 0x8000fc21
Then we satisfy SEQ_GT((ack), (stream)->last_ack)
But we do not satisfy SEQ_GT(ack, (stream)->base_seq))
and the new last_ack will be compared to base_seq
So, refuse to make such a big update
Ticket: 6865
To avoid overhead of copies during syncs, compress the local stats table.
Previously all counters had 2 64bit ints: one for the value, one to
track the number of times the counter was updated. The updates field was
only used for counters of type "average" however. Since only a few
average counters were in use, this effectively meant that almost half of
the space used for the table was wasted.
This patch reduces `StatsLocalCounter` to a single i64. Regular counters
simply use this directly by using their id as an index into the table.
For avg counters the logic is similar, however they effectively use 2
indexes, idx and idx+1. The value is at `idx`, the updates are at `idx +
1`.
Ticket: #5613.
Minimize the work done under lock when synchronizing thread stats.
Instead of looping all stats to copy them from the threads private store
to the public store, this patch simply memcpy's the whole array.
When the stats thread then takes the stats from the threads public
store, it now also copies the whole array to a local store before
processing the stats one by one.
This keeps the locking time minimal.
Packet Thread(s)
[ private thread stats ]
| lock
| memcpy
V unlock
[ copy of thread stats ]
Stats Thread
[ copy of thread stats ]
| lock
| memcpy
V unlock
[ private copy of thread stats ]
In `add-hostbit`, `remove-hostbit` and `list-hostbit` commands, the IPv6
address parsing was not using the correct variable:
from /usr/include/dirent.h:25,
from suricata-common.h:73,
from runmode-unix-socket.c:18:
In function ‘inet_pton’,
inlined from ‘UnixSocketHostbitAdd’ at runmode-unix-socket.c:1316:13:
/usr/include/x86_64-linux-gnu/bits/inet-fortified.h:56:10: warning: call to ‘__inet_pton_chk_warn’ declared with attribute warning: inet_pton called with a destination buffer size too small [-Wattribute-warning]
56 | return __glibc_fortify (inet_pton, __sz, sizeof (char),
| ^~~~~~~~~~~~~~~
In function ‘inet_pton’,
inlined from ‘UnixSocketHostbitRemove’ at runmode-unix-socket.c:1403:13:
/usr/include/x86_64-linux-gnu/bits/inet-fortified.h:56:10: warning: call to ‘__inet_pton_chk_warn’ declared with attribute warning: inet_pton called with a destination buffer size too small [-Wattribute-warning]
56 | return __glibc_fortify (inet_pton, __sz, sizeof (char),
| ^~~~~~~~~~~~~~~
In function ‘inet_pton’,
inlined from ‘UnixSocketHostbitList’ at runmode-unix-socket.c:1476:13:
/usr/include/x86_64-linux-gnu/bits/inet-fortified.h:56:10: warning: call to ‘__inet_pton_chk_warn’ declared with attribute warning: inet_pton called with a destination buffer size too small [-Wattribute-warning]
56 | return __glibc_fortify (inet_pton, __sz, sizeof (char),
| ^~~~~~~~~~~~~~~
Bug: #8102.
Juliana Fajardini
Jason Ish
Victor Julien
Sergey Pinaev
Promise Charles
Philippe Antoine
Jeff Lucovsky
Urval Kheni
Jeff Lucovsky
Lukas Sismis
Adam Kiripolsky
Shivani Bhardwaj
Philippe Antoine
Ofer Dagan
Mingyu Jeon
Fupeng Zhao
Abhijeet Singh
Sergey Zhidkih
Shivani Bhardwaj
Giuseppe Longo
Ali Saidi
Ambre Iooss
James Kaddu
Jhonny Sousa
Olusegun Fajobi
Cheryl Owala