Implement a new design for handling var name id's. The old logic
was aware of detection engine versions and generally didn't work
well for multi-tenancy cases. Other than memory leaks and crashes,
logging of var names worked or failed based on which tenant was
loaded last.
This patch implements a new approach, where there is a global store
of vars and their id's for the lifetime of the program.
Overall Design:
Base Store: "base"
Used during keyword registration. Operates under lock. Base is shared
between all detect engines, detect engine versions and tenants.
Each variable name is ref counted.
During the freeing of a detect engine / tenant, unregistration decreases
the ref cnt.
Base has both a string to id and a id to string hash table. String to
id is used during parsing/registration. id to string during unregistration.
Active Store Pointer (atomic)
The "active" store atomic pointer points to the active lookup store. The call
to `VarNameStoreActivate` will build a new lookup store and hot swap
the pointer.
Ensuring memory safety. During the hot swap, the pointer is replaced, so
any new call to the lookup functions will automatically use the new store.
This leaves the case of any lookup happening concurrently with the pointer
swap. For this case we add the old store to a free list. It gets a timestamp
before which it cannot be freed.
Free List
The free list contains old stores that are waiting to get removed. They
contain a timestamp that is checked before they are freed.
Bug: #6044.
Bug: #6201.
Move pcre2 data structures used for parsing into the detect engine
context, so that multiple tenant loading threads don't use the same
data structures.
Bug: #4797.
Performance measurement of rules is important on live Suricata
as bad rules can cause severe performance regression. This patch
introduces the --enable-profiling-rules that activate profiling
for the rules. This reduces the performance impact of full
profiling and provide visiblity on the rules performance at
the same time.
Use a fixed type of max_pending_packets instead of intmax_t which can
differ based on the platform/standard library.
Should also prevent lints about possible arithmetic overflow.
Querying an engine mode with an unknown value signals a bug when
the engine mode has not been determined but is already queried by
other functions.
Ticket: #5959
Master exception policy queried engine mode earlier than it was
determined from the configuration file/command line. As a result it
used the default (IDS) mode. However, the engine mode could have been
reconfigured later on to the IPS mode. This lead into an undefined behavior
as master exception policy behaves according to the configured engine mode.
Ticket: #5960
BPF codebase queried engine mode earlier than it was determined from
the configuration file/command line. As a result it used the default (IDS)
mode where it could've been configured later on to the IPS mode.
This could lead into an undefined behavior as some Suricata modules behave
according to the engine mode.
PF-Ring, Netmap and AF-Packet all shared almost identical code for
determining the engine mode. It was put into one common function.
Omitted the usage of SCStrdup function in PF-Ring module as it is
uppercased during thread initialization phase.
Ticket: #5957
suricata.c:691:17: warning: Value stored to 'bits' during its initialization is never read [deadcode.DeadStores]
const char *bits = "<unknown>-bits";
^~~~ ~~~~~~~~~~~~~~~~
suricata.c:692:17: warning: Value stored to 'endian' during its initialization is never read [deadcode.DeadStores]
const char *endian = "<unknown>-endian";
^~~~~~ ~~~~~~~~~~~~~~~~~~
2 warnings generated.
Add a new command line option, --include. This will merge additional
configuration files into the configuration specified in the main
suricata.yaml. It can be provided multiple times and the files will be
included in the order they appear on the command line.
Ticket: 3912
This allows all traffic Exception Policies to be set from one
configuration point. All exception policy options are available in IPS
mode. Bypass, pass and auto (disabled) are also available in iDS mode
Exception Policies set up individually will overwrite this setup for the
given traffic exception.
Task #5219
Configuration and behavior of HTP app layer depends on the copy
mode of Suricata engine. Copy mode was set after the app layer setup.
Decision of engine's copy mode operation is now made earlier.
Ticket: #5706
Commit contains prototype changes of RunModeSetLiveCaptureAutoFp and
RunModeSetLiveCaptureWorkers functions to move the IPS enable logic
out of suricata.c file.
AF_XDP support is a recent technology introduced that aims at improving
capture performance. With this update, Suricata now provides a new
capture source 'af-xdp' that attaches an eBPF program to the network
interface card. Packets received in the NIC queue are forwarded to
a RX ring in user-space, bypassing the Linux network stack.
Note, there is a configuration option (force-xdp-mode) that forces the
packet through the normal Linux network stack.
libxdp and libbpf is required for this feature and is compile time
configured.
This capture source operates on single and multi-queue NIC's via
suricata.yaml. Here, various features can be enabled, disabled
or edited as required by the use case.
This feature currently only supports receiving packets via AF_XDP,
no TX support has been developed.
Ticket: https://redmine.openinfosecfoundation.org/issues/3306
Additional reading:
https://www.kernel.org/doc/html/latest/networking/af_xdp.html
Various parts of ASAN will require creation of threads. At least
LSAN reporting and ASAN stack traces require this. Detect if we
run with ASAN and bypass the noproc setting with a warning.
Bug: #5661.
with setrlimit NPROC.
So that, if Suricata wants to execve or such to create a new process
the OS will forbid it so that RCE exploits are more painful to write.
Ticket: #5373
Each module (thread) updates its status to indicate running.
Main thread awaits for all threads to be in a running state
before continuing the initialisation process
Implements feature 5384
(https://redmine.openinfosecfoundation.org/issues/5384)
Work towards making `suricata-common.h` only introduce system headers
and other things that are independent of complex internal Suricata
data structures.
Update files to compile after this.
Remove special DPDK handling for strlcpy and strlcat, as this caused
many compilation failures w/o including DPDK headers for all files.
Remove packet macros from decode.h and move them into their own file,
turn them into functions and rename them to match our function naming
policy.
If landlock ABI is inferior to 2 (before Linux 5.19) then the
renaming of files is impossible if the protection is enabled. This
patch disables landlock if ABI < 2 and file-store is enabled.
As file store is initialized in output the call to landlock had to
done after the output initialization.
This patch is adding support for Landlock, a Linux
Security Module available since Linux 5.13.
The concept is to prevent any file operation on directories where
Suricata is not supposed to access.
Landlock support is built by default if the header is present. The
feature is disabled by default and need to be activated in the YAML
to be active.
Landlock documentation: https://docs.kernel.org/userspace-api/landlock.html
Feature: #5479
For testing purposes. Meant to simulate a reallocation failure when
dynamically growing the alert queue in DetectEngineThreadCtx, so we can
check that Suri's behavior doesn't break under such circumstances.
Task #5319
Adds a framework for setting exception policies. These would be called
when the engine reaches some kind of exception condition, like hitting
a memcap or some traffic processing error.
The policy gives control over what should happen next: drop the packet,
drop the packet and flow, bypass, etc.
Implements the policy for:
stream: If stream session or reassembly memcaps are hit call the
memcap policy on the packet and flow.
flow: Apply policy when memcap is reached and no flow could be
freed up.
defrag: Apply policy when no tracker could be picked up.
app-layer: Apply ppolicy if a parser reaches an error state.
All options default to 'ignore', which means the default behavior
is unchanged.
Adds commandline options: add simulation options for exceptions. These
are only exposed if compiled with `--enable-debug`.
Ticket: #5214.
Ticket: #5215.
Ticket: #5216.
Ticket: #5218.
Ticket: #5194.
kill(0, ...) re-raises the signal to every processes in the process
group which may impact unrelated processes.
Concretely, in our CI pipeline, a segfaulting Suricata process killed
the test driver.
Victor Julien
Philippe Antoine
Eric Leblond
Jason Ish
Lukas Sismis
Philippe Antoine
Juliana Fajardini
Jeff Lucovsky
Richard McConnell
Arne Welzel