Each thread had its own version of the `machine_start_time`, which
lead to slight time differences. This became apparent mostly in IPS,
where 2 threads each process a side of the flow.
This patch makes the `machine_start_time` global.
Update DROP action handling in tunnel packets. DROP/REJECT action is set
to outer (root) and inner packet.
Check action flags both against outer (root) and inner packet.
Remove PACKET_SET_ACTION macro. Replace with RESET for the one reset usecase.
The reason to remove is to make the logic easier to understand.
Reduce scope of RESET macros.
Rename PacketTestAction to PacketCheckAction except in unittests. Keep
PacketTestAction as a wrapper around PacketCheckAction. This makes it
easier to trace the action handling in the real code.
Fix rate_filter setting actions directly.
General code cleanups.
Bug: #5571.
As per RFC 4648,
Implementations MUST reject the encoded data if it contains characters
outside the base alphabet when interpreting base-encoded data, unless
the specification referring to this document explicitly states
otherwise.
Add a new mode BASE64_MODE_RFC4648, and handle input strictly as per the
specification.
Bug 5223
When looking at an alert event, it was impossible to determine which
side from src or dest IP in the alert was the client and wich side
was the server with regards to the underlying flow. This was a problem
when you try to known who belongs a metadata property such as a HTTP
hostname or a TLS JA3.
This patch updates the code to add src and dest IP in the flow
subobject as well as src and dst port. This way, we can now which
side is the client and which side is the server.
The result is looking like:
{
"event_type": "alert",
"src_ip": "22.47.184.196",
"src_port": 81,
"dest_ip": "192.168.1.47",
"dest_port": 1063,
"proto": "TCP",
"tx_id": 0,
"alert": {
"signature_id": 2018959,
"rev": 3,
},
"app_proto": "http",
"flow": {
"pkts_toserver": 22,
"pkts_toclient": 35,
"bytes_toserver": 1370,
"bytes_toclient": 48852,
"start": "2009-10-28T10:01:46.755232+0100",
"src_ip": "192.168.1.47",
"dest_ip": "22.47.184.196",
"src_port": 1063,
"dest_port": 81
}
}
Add a command to extract the accounting data from a live
flow using the unix socket. It takes the flow_id as param
and return the volume of data seen on the flow as well as
its age.
Previous method was truncating the flow hash value when building
the flow_id. It is interesting not to loose the flow hash value
as it can be used in other tools or to interact with a flow that
is still active.
Worker threads are responsible for final processing of timed out flows.
These are selected by the Flow Manager and inserted into a per thread
queue. The Flow Worker then checks this queue after each packet. Due to
the burstiness of this process, the packet threads would sometimes process
a lot of these flows in the context of a single packet, leading to spike
in latency which might cause packet loss.
This patch changes the behavior to only process at max 2 flows per packet.
This way added processing cost is amortized over many packets.
Add thread local cache to avoid locking overhead for ssns and segments.
A thread will return segments/ssns to a local cache first, and if that
is full, to a return queue where the actual return to the pool returns
a batch, to amortize locking overhead.
Adds segment and session pool/cache counters to see where how effective
the cache is.
Track packets that updated the app-layer, and for those run
the transaction housekeeping and output-tx logging loops.
Do the same of end of flow packets.
This skips needless iterations over the transaction stores.
Work towards making `suricata-common.h` only introduce system headers
and other things that are independent of complex internal Suricata
data structures.
Update files to compile after this.
Remove special DPDK handling for strlcpy and strlcat, as this caused
many compilation failures w/o including DPDK headers for all files.
Remove packet macros from decode.h and move them into their own file,
turn them into functions and rename them to match our function naming
policy.
Introduce AppLayerTxData::file_tx as direction(s) indicator for transactions.
When set to 0, its not a file tx and it will not be considered for file
inspection, logging and housekeeping tasks.
Various tx loop optimizations in housekeeping and output.
Update the "file capable" app-layers to set the fields based on their
directional file support as well as on the traffic.
Update APIs to store files in transactions instead of the per flow state.
Goal is to avoid the overhead of matching up files and transactions in
cases where there are many of both.
Update all protocol implementations to support this.
Update file logging logic to account for having files in transactions. Instead
of it acting separately on file containers, it is now tied into the
transaction logging.
Update the filestore keyword to consider a match if filestore output not
enabled.
Issue: 5077
This commit
- Converts the PCRE based parser to Rust.
- Adds unit tests to the new Rust modules
- Removes the PCRE parser from detect-bytemath.c
- Adjusts the C source modules to refer to the Rust definitions
- Includes the multiply operator (missing from the C parser)
Make sure to exit the parser early on incomplete header data.
Additionally, make sure to not create duplicated tls frames in this
case.
Add a debug validation check for the header parser parsing too much
data, which should never happen.
Improve unknown record handling. Inspired by Wireshark 'unknown record'
handling, we take a best effort approach for records with unknown content
types in TLS versions 1.0, 1.1 and 1.2.
Improve record length check and set 'invalid_record_length' event instead
of 'invalid_tls_header'.
Parse client cerificates and store them in the state similar to how
this is done for server certificates.
Update "progress" handling to not consider the TLS handshake complete
if the server indicated a client cert was needed.
TCP Buffering is now done in the app-layer using the incomplete API, on
the SSL/TLS record level. TLS level fragmentation will be implemented
separately.
To avoid overhead of stream buffering for records we don't do
much with anyway, pass through application records instead of
buffering the entire record in the stream engine.
The TLS record header is parsed in streaming mode still, but once the
record size is known we tell the app-layer API to give us the full
record.
Ticket: #5481
My review of the FreeBSD kernel code reveals that this setting
a) is ignored by the kernel b) is not required. The sending
side of divert(4) never checks so->so_options, but always gives
IP_ALLOWBROADCAST to ip_output().
If landlock ABI is inferior to 2 (before Linux 5.19) then the
renaming of files is impossible if the protection is enabled. This
patch disables landlock if ABI < 2 and file-store is enabled.
As file store is initialized in output the call to landlock had to
done after the output initialization.
This patch is adding support for Landlock, a Linux
Security Module available since Linux 5.13.
The concept is to prevent any file operation on directories where
Suricata is not supposed to access.
Landlock support is built by default if the header is present. The
feature is disabled by default and need to be activated in the YAML
to be active.
Landlock documentation: https://docs.kernel.org/userspace-api/landlock.html
Feature: #5479
This enables the usage of 'reject' as an exception policy. As for both
IPS and IDS modes the intended result of sending a reject packet is to
reject the related flow, this will effectively mean setting the reject
action to the packet that triggered the exception condition, and then
dropping the associated flow.
Task #5503
This allows to set a midstream-policy that can:
- fail closed (stream.midstream-policy=drop-flow)
- fail open (stream.midstream-policy=pass-flow)
- bypass stream (stream.midstream-policy=bypass)
- do nothing (default behavior)
Usage and behavior:
If stream.midstream-policy is set then if Suricata identifies a midstream flow
it will apply the corresponding action associated with the policy.
No setting means Suricata will not apply such policies, either inspecting the
flow (if stream.midstream=true) or ignoring it stream.midstream=false.
Task #5468
For protocols with multi buffer inspection there could be multiple times
the same sid would be queued into the candidates queue. This triggered
a debug validation check.
W/o debug validation this would lead to duplicate work and possibly multiple
alerts where a single one would be appropriate.
Bug: 5419.
Add tls.random keyword that matches on the 32 bytes of the TLS
random field for client as well as server.
Add tls.random_time keyword that matches on the first 4 bytes of the TLS
random field for client as well as server.
Add tls.random_bytes keyword that matches on the last 28 bytes of the TLS
random field for client as well as server.
All these are sticky buffers.
Feature 5190
Previous adaptive model would have a large time range when scanning the
hash when not so busy. In the default case it would take up to 4 minutes
for a full hash scan. In case of sudden increase in business, where the
hash would fill up rapidily during a few seconds, the flow manager would
be forced to suddenly consider a much larger slice of the hash leading
to a burst of work. This burst would increase pressure on the rest of the
system leading to packet loss as the worker threads would be overloaded
with flow housekeeping tasks.
This patch reduces the max scan time to 10 seconds, and ramps up quickly
to increase the slice of the hash scanned.
Before, if an invalid value was passed as exception policy, Suricata
would log a warning and set the exception policy to "ignore". This is a
very different result, than, say, dropping or bypassing a midstream flow.
Task #5504
Bug 5458 states that the reject action is no longer working. While SV
tests that use the reject action still pass, it indeed seems that a
regression has happened with commit aa93984, because while the
function that applies rule actions to the flow (RuleActionToFlow) does
check for the reject action, the newly added function PacketApply
SignatureActions only checks for ACTION_DROP or ACTION_PASS when
deciding to call RuleActionToFlow.
Bug #5458
A Packet may be dropped due to several different reasons. This change
adds action as a parameter, so we can update the packet action when we
drop it, instead of setting it to drop.
Related to
Bug #5458
When building with the following options:
./configure CC=clang --enable-luajit --enable-geoip --enable-unittests
There is a build failure:
runmode-unittests.c:234:9: error: implicit declaration of function 'LuajitSetupStatesPool' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
if (LuajitSetupStatesPool() != 0) {