This changeset makes changes to the TX logging path. Since the txn
is passed to the TX logger, the TX can be used directly instead of
through the TX id.
This changeset adds a mechanism to track when individual events
are logged. Transactions can be provided more than once; track
events to prevent event re-logging.
Declare _POSIX_C_SOURCE before sys/time.h to avoid:
util-time.c: In function 'SCUtcTime':
util-time.c:222:12: error: implicit declaration of function 'gmtime_r'; did you mean 'gmtime_s'? [-Werror=implicit-function-declaration]
222 | return gmtime_r(&timep, result);
| ^~~~~~~~
| gmtime_s
util-time.c:222:12: warning: returning 'int' from a function with return type 'struct tm *' makes pointer from integer without a cast [-Wint-conversion]
222 | return gmtime_r(&timep, result);
| ^~~~~~~~~~~~~~~~~~~~~~~~
util-time.c: In function 'SCLocalTime':
util-time.c:305:9: error: implicit declaration of function 'localtime_r'; did you mean 'localtime_s'? [-Werror=implicit-function-declaration]
305 | localtime_r(&timep, &cached_local_tm[lru]);
| ^~~~~~~~~~~
| localtime_s
util-time.c:321:56: warning: comparison between pointer and integer
321 | if (localtime_r(&timep, &cached_local_tm[lru]) == NULL)
| ^~
cc1.exe: some warnings being treated as errors
Tickets: #2994#3051
Packets from captured bypassed flows that are received by Suricata
before the capture method start to bypass them can represent an
important part due to various buffer and insertion delay.
This patch adds a two counters to know the number of packets and
bytes in this case.
This patch improves the bypass error handling add adds more counters
to the interface so it is possible to get a view on success and
failure of insertion in the eBPF maps via the `iface-bypassed-stat`
command.
This patch introduces and uses a new bypass strategy
based on a callback. EBPF bypass implementation is
updated to use this new strategy.
Once the flow manager detect that a flow should be timeouted,
it asks the capture method if it has seen packets in the interval.
If it is the case the lastts of the flow is updated and the timeout
is postponed.
The flow bypass stats is computed at every pass so the accounting
needs to be done at each pass. This patch fixes the accounting
in the flow_bypassed counters.
At the time of the writing, libbpf output useful error message
on strdout only and errno is not really interesting. So let's
tell user to look at stdout.
The loop on bypassed flow maps can take a few seconds on heavily
loaded system causing Suricata to not honor a stop before a few
seconds.
This patch adds the code needed to detect the need to exit from
the check loop.
For capture method that have their own flow structure (not maintained
by Suricata), it can make sense to bypass a packet even if there is
no Flow in Suricata.
For AF_PACKET it does not make sense as the eBPF map entry will
be destroyed as soon as it will be checked by the flow bypass
manager. Thus we shortcut the bypass function if ever no Flow is
attached to the packet.
This path also removes reference to Flow in the bypass functions
for AF_PACKET. It was not necessary and we possibly could benefit
of it if ever we change the bypass algorithm.
There is a synchronization issue occuring when a flow is
added to the eBPF bypass maps. The flow can have packets
in the ring buffer that have already passed the eBPF stage.
By consequences, they are not accounted in the eBPF counter
but are accounted by Suricata flow engine.
This was causing counters to be completely wrong. This code
fixes the issue by avoiding the counter change in invalid
case.
To avoid adding 4 64bits integers to the Flow structure for the
bypass accounting, we use instead a FlowStorage. This limits the
memory usage to the size of a pointer.
Only reason clock_gettime could fail is a permission so let's
error and leave the flow bypass manager if it is the case.
Also let's suppress the error message if ever the error appear in
the middle of a run (which is unlikely).
An alignement issue was preventing the code to work properly.
We introduce macros taken from Linux source code sample to get
something that should work on the long term.
By pinning the maps we are creating a file in /sys/fs/bpf that can
be used by external program to access the map. This has multiple
benefits such as handling list from an external program.
The pinned maps could be persistent accross Suricata reload but
this can be complicated in term of handling everything in the life
of Suricata.
Kernel time is not available (and/or costly) on NIC such as
Netronome so we update the logic to detect dead flows based on a
lack of update of packets counters. This way, the XDP filter will
be usable by network card.
This patch also updates the ebpf code to support per CPU and
regular mapping. Netronome is not supporting it and the structure
is using atomic for counter so the cost of simultaneous update
is really low.
This patch also updates the xdp_filter to be able to select if the
flow table is per CPU on shared. Second option will be used for
hardward offload. To deactivate the per cpu hash, you need to set
USE_PERCPU_HASH to 0.
This patch also adds an new option to af-packet named no-percpu-hash
If this option is set to yes then the Flow bypassed manager thread
will use one CPU instead of the number of cores. By doing that
we are able to handle the case where USE_PERCPU_HASH is unset (so
hardware offload for Netronome).
This patch also remove aligment indications in the eBPF filter. This
was not really needed and it seems it is causing problem with
some recent version of LLVM toolchain.
So far, if the rule loaded had a backslash character ("\") in its
content field, the rule will fail to load but without giving a
descriptive error message. This patch tells the user to escape the
troubling character.
Before
```
9409] 7/6/2019 -- 16:12:22 - (detect-engine-loader.c:184) <Error> (DetectLoadSigFile) -- [ERRCODE: SC_ERR_INVALID_SIGNATURE(39)] - error parsing signature "alert tcp $HOME_NET any -> $EXTERNAL_NET any (msg:"Backslash needs escaping msg"; flow:established,to_server; content:"MyBackslash\here"; sid:86; rev:1; )" from file /var/lib/suricata/rules/myrule.rules at line 1
```
After
```
[9409] 7/6/2019 -- 16:12:22 - (detect-content.c:155) <Error> (DetectContentDataParse) -- [ERRCODE: SC_ERR_INVALID_SIGNATURE(39)] - '\' has to be escaped
[9409] 7/6/2019 -- 16:12:22 - (detect-engine-loader.c:184) <Error> (DetectLoadSigFile) -- [ERRCODE: SC_ERR_INVALID_SIGNATURE(39)] - error parsing signature "alert tcp $HOME_NET any -> $EXTERNAL_NET any (msg:"Backslash needs escaping msg"; flow:established,to_server; content:"MyBackslash\here"; sid:86; rev:1; )" from file /var/lib/suricata/rules/myrule.rules at line 1
```
Closes redmine ticket #2626
Stream reassembly memcap is regulated by the Init and Cleanup
callbacks. If Init fails due to memcap reached, Cleanup had no
way of knowing and it would decrease the memcap even if it hadn't
been increased by Init. This could lead to too much memory use and
memcap counter underflow.
This patch fixes the issue by not calling Cleanup in this case. It's
fair to see a failed Init the responsibility of Init.
The capture threads can receive packets from the flow manager in their
Threadvars::stream_pq packet queue. This mechanism makes sure the packets
the flow manager injects into the engine are processed by the correct
worker thread.
If the capture thread(s) would not receive packets for a long time, the
Threadvars::stream_pq would not be checked and processed. This could
lead to packet pool depletion in the flow manager. It would also lead
to flows not being timed out/logged until either packets started flowing
again or until the engine was shut down.
The scenario is more likely to happen in a test (e.g. replay) but could
also delay logging on low traffic sensors.
If waiting for N packets move the return stack to the main
stack every time we take the return stack lock.
Make sure we consider enough packets when setting the pending pool
flush logic. This should at least make sure to have the 9 packets
the flow manager requires per run.
Match on JA3S hash using ja3s.hash keyword, e.g:
alert tls any any -> any any (msg:"ja3s.hash test";
ja3s.hash; content:"b26c652e0a402a24b5ca2a660e84f9d5"; sid:1;)
Add Ja3SGetString() to return the content of the JA3S string buffer from
the TLS session.
Example:
function init (args)
local needs = {}
needs["protocol"] = "tls"
return needs
end
function setup (args)
filename = SCLogPath() .. "/ja3s_string.log"
file = assert(io.open(filename, "a"))
end
function log (args)
ja3s_string = Ja3SGetString()
if ja3s_string == nil then
return
end
file:write(ja3s_string .. "\n")
file:flush()
end
function deinit (args)
file:close()
end
Add Ja3SGetHash() to return the content of the JA3S hash buffer from
the TLS session.
Example:
function init (args)
local needs = {}
needs["protocol"] = "tls"
return needs
end
function setup (args)
filename = SCLogPath() .. "/ja3s_hash.log"
file = assert(io.open(filename, "a"))
end
function log (args)
ja3s_hash = Ja3SGetHash()
if ja3s_hash == nil then
return
end
file:write(ja3s_hash .. "\n")
file:flush()
end
function deinit (args)
file:close()
end
In the example above, each JA3S hash is logged to a log file.
Add keyword to do "raw" matching on each of the certificates in the
TLS certificate sticky buffer.
Example:
alert tls any any -> any any (msg:"tls.certs test"; tls.certs; \
content:"|01 02 03 04|"; sid:1;)
This change ensures that each anomaly is tagged with an
event type to support querying.
Each anomalous event will include `"event_type": "anomaly"`
in the log record.
This changeset adds anomaly logging to suricata for issue 2282.
Anomaly logging is controlled via the `anomaly` section within eve-log.
There is a single option -- `packethdr` -- for including the packet header
in the anomaly.
When killed with SIGHUP, suricata reopens the log files. If filemode
is set in the config, it needs pledge promise "fattr" to allow the
chmod(2) on OpenBSD.
Make sure all file sig mismatches indicate this in their return
code, not just the ones with filestore enabled. This is needed
to tell the stateful detect engine that it is dealing with a file
sig, so it can make sure these are inspected correctly even if
there are possibly multiple files per tx.
If the filename inspection function is returning nomatch this will
trigger iterative inspections with same content (aka filename) being
inspected. To avoid this we change the return as the buffer inspection
has not to be inspected anymore.
This patch implement an option named 'buffer' that can be used in the
init function of a lua signature:
function init (args)
local needs = {}
needs["buffer"] = tostring(true)
return needs
end
With this, the lua script will get access to the sticky buffer
content.
This fixes redmine bug #2057 by setting pseudopacket iface and vlan from
flow values, solving the problem of missing vlan/iface when psuedopacket
gets logged/alerted on.
Adds SSH_FLAG_VERSION_PARSED to flags before each return
This way, we are sure SSHParseBanner does not get called again
And proto_version does not get leaked
This changeset addresses 2 issues:
- 2459
- 2527
and improves handling for FTP active mode over IPv4 and IPv6.
Active mode is triggered when the FTP client conveys the port
that should be used for a data connection (PORT, EPRT).
When this occurs, the FTP state is marked as "active".
Instead of casting the packet buffer to a uint32, memcpy it to
avoid misaligned read error, as caught by the undefined behavior
detector (ubsan).
Redmine issue:
https://redmine.openinfosecfoundation.org/issues/2903