By pinning the maps we are creating a file in /sys/fs/bpf that can
be used by external program to access the map. This has multiple
benefits such as handling list from an external program.
The pinned maps could be persistent accross Suricata reload but
this can be complicated in term of handling everything in the life
of Suricata.
Kernel time is not available (and/or costly) on NIC such as
Netronome so we update the logic to detect dead flows based on a
lack of update of packets counters. This way, the XDP filter will
be usable by network card.
This patch also updates the ebpf code to support per CPU and
regular mapping. Netronome is not supporting it and the structure
is using atomic for counter so the cost of simultaneous update
is really low.
This patch also updates the xdp_filter to be able to select if the
flow table is per CPU on shared. Second option will be used for
hardward offload. To deactivate the per cpu hash, you need to set
USE_PERCPU_HASH to 0.
This patch also adds an new option to af-packet named no-percpu-hash
If this option is set to yes then the Flow bypassed manager thread
will use one CPU instead of the number of cores. By doing that
we are able to handle the case where USE_PERCPU_HASH is unset (so
hardware offload for Netronome).
This patch also remove aligment indications in the eBPF filter. This
was not really needed and it seems it is causing problem with
some recent version of LLVM toolchain.
So far, if the rule loaded had a backslash character ("\") in its
content field, the rule will fail to load but without giving a
descriptive error message. This patch tells the user to escape the
troubling character.
Before
```
9409] 7/6/2019 -- 16:12:22 - (detect-engine-loader.c:184) <Error> (DetectLoadSigFile) -- [ERRCODE: SC_ERR_INVALID_SIGNATURE(39)] - error parsing signature "alert tcp $HOME_NET any -> $EXTERNAL_NET any (msg:"Backslash needs escaping msg"; flow:established,to_server; content:"MyBackslash\here"; sid:86; rev:1; )" from file /var/lib/suricata/rules/myrule.rules at line 1
```
After
```
[9409] 7/6/2019 -- 16:12:22 - (detect-content.c:155) <Error> (DetectContentDataParse) -- [ERRCODE: SC_ERR_INVALID_SIGNATURE(39)] - '\' has to be escaped
[9409] 7/6/2019 -- 16:12:22 - (detect-engine-loader.c:184) <Error> (DetectLoadSigFile) -- [ERRCODE: SC_ERR_INVALID_SIGNATURE(39)] - error parsing signature "alert tcp $HOME_NET any -> $EXTERNAL_NET any (msg:"Backslash needs escaping msg"; flow:established,to_server; content:"MyBackslash\here"; sid:86; rev:1; )" from file /var/lib/suricata/rules/myrule.rules at line 1
```
Closes redmine ticket #2626
Stream reassembly memcap is regulated by the Init and Cleanup
callbacks. If Init fails due to memcap reached, Cleanup had no
way of knowing and it would decrease the memcap even if it hadn't
been increased by Init. This could lead to too much memory use and
memcap counter underflow.
This patch fixes the issue by not calling Cleanup in this case. It's
fair to see a failed Init the responsibility of Init.
The capture threads can receive packets from the flow manager in their
Threadvars::stream_pq packet queue. This mechanism makes sure the packets
the flow manager injects into the engine are processed by the correct
worker thread.
If the capture thread(s) would not receive packets for a long time, the
Threadvars::stream_pq would not be checked and processed. This could
lead to packet pool depletion in the flow manager. It would also lead
to flows not being timed out/logged until either packets started flowing
again or until the engine was shut down.
The scenario is more likely to happen in a test (e.g. replay) but could
also delay logging on low traffic sensors.
If waiting for N packets move the return stack to the main
stack every time we take the return stack lock.
Make sure we consider enough packets when setting the pending pool
flush logic. This should at least make sure to have the 9 packets
the flow manager requires per run.
Match on JA3S hash using ja3s.hash keyword, e.g:
alert tls any any -> any any (msg:"ja3s.hash test";
ja3s.hash; content:"b26c652e0a402a24b5ca2a660e84f9d5"; sid:1;)
Add Ja3SGetString() to return the content of the JA3S string buffer from
the TLS session.
Example:
function init (args)
local needs = {}
needs["protocol"] = "tls"
return needs
end
function setup (args)
filename = SCLogPath() .. "/ja3s_string.log"
file = assert(io.open(filename, "a"))
end
function log (args)
ja3s_string = Ja3SGetString()
if ja3s_string == nil then
return
end
file:write(ja3s_string .. "\n")
file:flush()
end
function deinit (args)
file:close()
end
Add Ja3SGetHash() to return the content of the JA3S hash buffer from
the TLS session.
Example:
function init (args)
local needs = {}
needs["protocol"] = "tls"
return needs
end
function setup (args)
filename = SCLogPath() .. "/ja3s_hash.log"
file = assert(io.open(filename, "a"))
end
function log (args)
ja3s_hash = Ja3SGetHash()
if ja3s_hash == nil then
return
end
file:write(ja3s_hash .. "\n")
file:flush()
end
function deinit (args)
file:close()
end
In the example above, each JA3S hash is logged to a log file.
Add keyword to do "raw" matching on each of the certificates in the
TLS certificate sticky buffer.
Example:
alert tls any any -> any any (msg:"tls.certs test"; tls.certs; \
content:"|01 02 03 04|"; sid:1;)
This change ensures that each anomaly is tagged with an
event type to support querying.
Each anomalous event will include `"event_type": "anomaly"`
in the log record.
This changeset adds anomaly logging to suricata for issue 2282.
Anomaly logging is controlled via the `anomaly` section within eve-log.
There is a single option -- `packethdr` -- for including the packet header
in the anomaly.
When killed with SIGHUP, suricata reopens the log files. If filemode
is set in the config, it needs pledge promise "fattr" to allow the
chmod(2) on OpenBSD.
Make sure all file sig mismatches indicate this in their return
code, not just the ones with filestore enabled. This is needed
to tell the stateful detect engine that it is dealing with a file
sig, so it can make sure these are inspected correctly even if
there are possibly multiple files per tx.
If the filename inspection function is returning nomatch this will
trigger iterative inspections with same content (aka filename) being
inspected. To avoid this we change the return as the buffer inspection
has not to be inspected anymore.
This patch implement an option named 'buffer' that can be used in the
init function of a lua signature:
function init (args)
local needs = {}
needs["buffer"] = tostring(true)
return needs
end
With this, the lua script will get access to the sticky buffer
content.
This fixes redmine bug #2057 by setting pseudopacket iface and vlan from
flow values, solving the problem of missing vlan/iface when psuedopacket
gets logged/alerted on.
Adds SSH_FLAG_VERSION_PARSED to flags before each return
This way, we are sure SSHParseBanner does not get called again
And proto_version does not get leaked
This changeset addresses 2 issues:
- 2459
- 2527
and improves handling for FTP active mode over IPv4 and IPv6.
Active mode is triggered when the FTP client conveys the port
that should be used for a data connection (PORT, EPRT).
When this occurs, the FTP state is marked as "active".
Instead of casting the packet buffer to a uint32, memcpy it to
avoid misaligned read error, as caught by the undefined behavior
detector (ubsan).
Redmine issue:
https://redmine.openinfosecfoundation.org/issues/2903
- There is now an option to automatically create streams on the
correct NUMA node when using cpu affinity.
- When not using cpu affinity the user can specify streams to be
created in the suricata.yaml file. It is no longer required to
use NTPL to create streams before running suricata.
- The legacy usage model of running NTPL to create streams is still
available. This can be used for legacy configurations and complex
configurations that cannot be satisfied by the auto-config option.
pledge(2) can be used on OpenBSD to restrict suricata possible
operation on the system once initialization is completed.
The process promises to only make use of:
- "stdio" to allow read(2) on IPS rules and write(2) on log file
- "rpath wpath cpath" to allow log rotation
- "unix" to operate the control unix socket and log unix sockets
- "dns" to retrieve DNS from recvfrom(2)/sento(2) in IPFW mode
- "bpf" as suricata uses libpcap, which uses the BIOCGSTATS operation
Signed-off-by: Emmanuel Roullit <emmanuel.roullit@cognitix.de>
The mode input in chmod is an octal integer. However when the warning is logged,
the file mode is printed in decimal which is confusing.
Signed-off-by: Emmanuel Roullit <emmanuel.roullit@cognitix.de>
The return value from the options decoder in TCP and IPv4 is ignored.
This commit changes the return type of the function to `void` and
modifies existing return points to return without a value.
When an error occurs, the packet state is being set to indicate whether
it's valid or not and the existing return value is never used.
This fixes two memleaks found with ASAN.
Direct leak of 96 byte(s) in 1 object(s) allocated from:
#0 0x7f59cf4a4d28 in malloc (/usr/lib/x86_64-linux-gnu/libasan.so.3+0xc1d28)
#1 0xd7f92f in ReceiveNFLOGThreadInit /home/glongo/suricata/src/source-nflog.c:221
#2 0xe9c8eb in TmThreadsSlotPktAcqLoop /home/glongo/suricata/src/tm-threads.c:293
#3 0x7f59cd7aa4a3 in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x74a3)
Indirect leak of 70000 byte(s) in 1 object(s) allocated from:
#0 0x7f59cf4a4d28 in malloc (/usr/lib/x86_64-linux-gnu/libasan.so.3+0xc1d28)
#1 0xd814ea in ReceiveNFLOGThreadInit /home/glongo/suricata/src/source-nflog.c:324
#2 0xe9c8eb in TmThreadsSlotPktAcqLoop /home/glongo/suricata/src/tm-threads.c:293
#3 0x7f59cd7aa4a3 in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x74a3)
SUMMARY: AddressSanitizer: 70096 byte(s) leaked in 2 allocation(s).
When Suricata picks up a flow it assumes the first packet is
toserver. In a perfect world without packet loss and where all
sessions neatly start after Suricata itself started, this would be
true. However, in reality we have to account for packet loss and
Suricata starting to get packets for flows already active be for
Suricata is (re)started.
The protocol records on the wire would often be able to tell us more
though. For example in SMB1 and SMB2 records there is a flag that
indicates whether the record is a request or a response. This patch
is enabling the procotol detection engine to utilize this information
to 'reverse' the flow.
There are three ways in which this is supported in this patch:
1. patterns for detection are registered per direction. If the proto
was not recognized in the traffic direction, and midstream is
enabled, the pattern set for the opposing direction is also
evaluated. If that matches, the flow is considered to be in the
wrong direction and is reversed.
2. probing parsers now have a way to feed back their understanding
of the flow direction. They are now passed the direction as
Suricata sees the traffic when calling the probing parsers. The
parser can then see if its own observation matches that, and
pass back it's own view to the caller.
3. a new pattern + probing parser set up: probing parsers can now
be registered with a pattern, so that when the pattern matches
the probing parser is called as well. The probing parser can
then provide the protocol detection engine with the direction
of the traffic.
The process of reversing takes a multi step approach as well:
a. reverse the current packets direction
b. reverse most of the flows direction sensitive flags
c. tag the flow as 'reversed'. This is because the 5 tuple is
*not* reversed, since it is immutable after the flows creation.
Most of the currently registered parsers benefit already:
- HTTP/SMTP/FTP/TLS patterns are registered per direction already
so they will benefit from the pattern midstream logic in (1)
above.
- the Rust based SMB parser uses a mix of pattern + probing parser
as described in (3) above.
- the NFS detection is purely done by probing parser and is updated
to consider the direction in that parser.
Other protocols, such as DNS, are still to do.
Ticket: #2572
Better handle case where pcap file receive thread fails to initialize. Allow
initialize to complete, but terminate the thread quickly. Delay exiting
unix socket runmode as late as possible.
Global NFQ contexts were not freed properly causing
'use-after-free' error. Moving contexts cleanup to a
separate NFQContextsCleanup() and calling it from
GlobalsDestroy(), like it's done for AFPacket, solves
the problem.
In case the pid file is given in the config file, the file name is
stored in volatile memory. Removal of the pid file happens after
cleanup of config memory. Create a copy of the name which will be
freed after the pid file has been removed.
Flow direction doesn't need explicit checking as the rule groups (sgh)
are already per direction. So if a rule sets only flow:to_server or
flow:to_client, we can avoid adding a sigmatch to the signature.