Support either the __thread GNUism or the C11 _Thread_local.
Use 'thread_local' to point to the one that is used. Convert existing
__thread user to 'thread_local'.
Remove non-thread-local code from the packet pool code.
Update the default packet size computation to use LiveDeviceName
instead of LiveDevice as the LiveDevice list is not built when
the default packet size is built.
Create a single function to return the version string, to avoid lots
of ifdefs in multiple places.
Make the version determine the 'release' status. If the version from
autoconf has '-dev' in the name, it is not a release. If it hasn't
it is considered a release version.
Add --strict-rule-keywords commandline option to enable strict rule
parsing.
It can be used without options or with a comma separated list:
--strict-rule-keywords
--strict-rule-keywords=all
--strict-rule-keywords=classtype,reference
Parsing implementations can use SigMatchStrictEnabled to check
if strict parsing is enabled for them and act accordingly.
At the startup, if the default log dir provided either by command line
options or suricat.yaml is not writable, the error comes quite later.
This patch makes suricata exit if there is such an error in the
beginning itself.
Closes redmine ticket #2386.
Datasets are sets/lists of data that can be accessed or added from
the rule language.
This patch implements 3 data types:
1. string (or buffer)
2. md5
3. sha256
The patch also implements 2 new rule keywords:
1. dataset
2. datarep
The dataset keyword allows matching against a list of values to see if
it exists or not. It can also add the value to the set. The set can
optionally be stored to disk on exit.
The datarep support matching/lookups only. With each item in the set a
reputation value is stored and this value can be matched against. The
reputation value is unsigned 16 bit, so values can be between 0 and 65535.
Datasets can be registered in 2 ways:
1. through the yaml
2. through the rules
The goal of this rules based approach is that rule writers can start using
this without the need for config changes.
A dataset is implemented using a thash hash table. Each dataset is its own
separate thash.
Date makes it even clearer that when was the last commit for the build
that one is running. Add this info alongwith rev. Change inspired by
rustc.
Before
```
$ suricata -V
This is Suricata version 5.0.0-dev (rev 2d217e666)
```
After
```
This is Suricata version 5.0.0-dev (2d217e666 2019-07-12)
```
Closes redmine ticket #3092
Fill in the vlan_id fields unconditionally. We can now remove the check
for the vlan.use-for-tracking setting in decode.c. The debug log message
is moved to suricata.c.
If vlan.use-for-tracking is disabled, set the vlan_id fields to 0 when
hashing or comparing flows. This is done using a bitmask as suggested by
Victor Julien in IRC, in order to avoid adding more branches to this
code.
Currently, suricata does not fill in vlan_id fields if
vlan.use-for-tracking is disabled and instead leaves them at the default
0 value, so this commit makes no functional change. This change is in
preparation for future commits where the vlan_ids will be always filled
in.
Related to https://redmine.openinfosecfoundation.org/issues/3076
This patch introduces and uses a new bypass strategy
based on a callback. EBPF bypass implementation is
updated to use this new strategy.
Once the flow manager detect that a flow should be timeouted,
it asks the capture method if it has seen packets in the interval.
If it is the case the lastts of the flow is updated and the timeout
is postponed.
There is a synchronization issue occuring when a flow is
added to the eBPF bypass maps. The flow can have packets
in the ring buffer that have already passed the eBPF stage.
By consequences, they are not accounted in the eBPF counter
but are accounted by Suricata flow engine.
This was causing counters to be completely wrong. This code
fixes the issue by avoiding the counter change in invalid
case.
To avoid adding 4 64bits integers to the Flow structure for the
bypass accounting, we use instead a FlowStorage. This limits the
memory usage to the size of a pointer.
pledge(2) can be used on OpenBSD to restrict suricata possible
operation on the system once initialization is completed.
The process promises to only make use of:
- "stdio" to allow read(2) on IPS rules and write(2) on log file
- "rpath wpath cpath" to allow log rotation
- "unix" to operate the control unix socket and log unix sockets
- "dns" to retrieve DNS from recvfrom(2)/sento(2) in IPFW mode
- "bpf" as suricata uses libpcap, which uses the BIOCGSTATS operation
Signed-off-by: Emmanuel Roullit <emmanuel.roullit@cognitix.de>
Global NFQ contexts were not freed properly causing
'use-after-free' error. Moving contexts cleanup to a
separate NFQContextsCleanup() and calling it from
GlobalsDestroy(), like it's done for AFPacket, solves
the problem.
In case the pid file is given in the config file, the file name is
stored in volatile memory. Removal of the pid file happens after
cleanup of config memory. Create a copy of the name which will be
freed after the pid file has been removed.
Process multiple packets at nm_dispatch. Use zero copy for workers
recv mode.
Add configure check netmap check for API 11+ and find netmap api version.
Add netmap guide to the userguide.
If one needs to use multiple sequential Netfilter queues,
it can be done with a new '-q' option's syntax: "start:end"
(just like it's done with iptables '--queue-balance' option).
In the eve log the decoder events are added as optional counters. This
behaviour is enabled by default. However, lots of the counters are
missing, as the names colide with other counters.
E.g.
decoder.ipv6 counts ipv6 packets
decoder.ipv6.unknown_next_header counts how often an unknown next
header is encountered.
In this example 'ipv6' would be both a json integer and a json object.
It appears that jansson favours the first that is generated, so the
event counters are mostly missing.
This patch registers them as 'decoder.events.<event>' instead. As
these names are generated on the fly, a hash table to contain the
allocated strings was added as well.
Files and directories passed via command line option -r should be checked for
existence during command line parsing and not start additional suricata
functionality.
4474889667 did break the logic of IPS
mode detection for layer 2 IPS mode like Netmap and AF_PACKET. This
pach fixes the issue by calling the IPS detection mode after
complete registration of the interfaces.
Reported-by: Alexander Gozman <a.gozman@securitycode.ru>
https://redmine.openinfosecfoundation.org/issues/2417
Add option to have pcap files deleted after they have been processed.
This option combines well with pcap file continuous and streaming
files to a directory being processed.
Last multi-detect changes broken delayed-detect by refusing to reload
a 'stub' detect engine. This patch distinguishes between a stub for
multi-tenancy and for delayed detect.
There are 3 types of detect engine objects:
1. normal
The normal detection engine if no multi-tenancy is in use
2. tenant
A per tenant detection engine
3. stub
A stub (or minimal as it was called before) detect engine
that is needed to have something in place when there are
only tenants.
A stub is also used in case of 'delayed detect', where we
need a minimal detect engine to start up which is replaced
by a full (normal type) detect engine after startup.
This patch adds a new field 'type' to the DetectEngineCtx object
to distinguish between the types. This replaces the boolean 'minimal'.
Enables IPS functionality on Windows using the open-source
(LGPLv3/GPLv2) WinDivert driver and API.
From https://www.reqrypt.org/windivert-doc.html : "WinDivert is a
user-mode capture/sniffing/modification/blocking/re-injection package
for Windows Vista, Windows Server 2008, Windows 7, and Windows 8.
WinDivert can be used to implement user-mode packet filters, packet
sniffers, firewalls, NAT, VPNs, tunneling applications, etc., without
the need to write kernel-mode code."
- adds `--windivert [filter string]` and `--windivert-forward [filter
string]` command-line options to enable WinDivert IPS mode.
`--windivert[-forward] true` will open a filter for all traffic. See
https://www.reqrypt.org/windivert-doc.html#filter_language for more
information.
Limitation: currently limited to `autofp` runmode.
Additionally:
- `tmm_modules` now zeroed during `RegisterAllModules`
- fixed Windows Vista+ `inet_ntop` call in `PrintInet`
- fixed `GetRandom` bug (nonexistent keys) on fresh Windows installs
- fixed `RandomGetClock` building on Windows builds
- Added WMI queries for MTU
https://redmine.openinfosecfoundation.org/issues/2394
Certain parameters of delay and poll interval could cause newly added
files in a directory to be missed. Cleaned up how time is handled for
files in a directory and fix which time is used for future directory
traversals. Add a mutex to make sure processing time is not optimized
away.
Device storage requires the devices to be created after storage
is finalized so we need to first get the list of devices then
create them when the storage is finalized.
This patch introduces the LiveDeviceName structure that is a list
of device name used during registration.
Code uses LiveRegisterDeviceName for pre registration and keep
using the LiveRegisterDevice function for part of the code that
create the interface during the runmode creation.
This patch implements bypass capability for af-packet.
The filter only bypass TCP and UDP in IPv4 and IPv6. It don't
don't bypass IPv6 with extended headers.
This patch also introduces a bypassed flow manager that takes
care of timeouting the bypassed flows. It uses a 60 sec
timeout on flow. As they are supposed to be active we can
try that. If they are not active then we don't care to get them
back in Suricata.
according to its man page, sigprocmask has undefined behavior in
multithreaded environments. Instead of explictly blocking the handling
of SIGUSR2 in every thread, direct block handling SIGUSR2 before
creating the threads and enable again the handling of this signal
afterwards. In this way, only the main thread will be able to manage
this signal properly.
Free txs that are done out of order if we can. Some protocol
implementations have transactions running in parallel, where it is
possible that a tx that started later finishes earlier than other
transactions. Support freeing those.
Also improve handling on asynchronious transactions. If transactions
are unreplied, e.g. in the dns flood case, the parser may at some
point free transactions on it's own. Handle this case in
the app-layer engine so that the various tracking id's (inspect, log,
and 'min') are updated accordingly.
Next, free txs much more aggressively. Instead of freeing old txs
at the app-layer parsing stage, free all complete txs at the end
of the flow-worker. This frees txs much sooner in many cases.
This doesn't need to be registered from suricata.c. And moving
it to the init function makes sure its only registered if
the logger is actually enabled.
Multiple NULL-pointer dereferences after ConfGet in PostConfLoadedSetup can cause suricata to terminate with segfaults. The ASAN-output:
ASAN:DEADLYSIGNAL =================================================================
5734ERROR: AddressSanitizer: SEGV on unknown address 0x000000000000 (pc 0x7f1a9a3967cc bp 0x7ffdff033ad0 sp 0x7ffdff033250 T0)
0 0x7f1a9a3967cb (/usr/lib/x86_64-linux-gnu/libasan.so.3+0x447cb)
1 0x55ba65f66f27 in PostConfLoadedSetup /root/suricata-1/src/suricata.c:2652
2 0x55ba65f6870e in main /root/suricata-1/src/suricata.c:2898
3 0x7f1a96aeb2b0 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x202b0)
4 0x55ba65af9039 in _start (/usr/local/bin/suricata+0xc8039)
AddressSanitizer can not provide additional info.
SUMMARY: AddressSanitizer: SEGV (/usr/lib/x86_64-linux-gnu/libasan.so.3+0x447cb)
This commit fixes Bug #2370 by replacing ConfGet by ConfGetValue
In some cases the rule reload could hang. The pending USR2 signals will
be recognized even with the <2 check. Also the SCLogWarning shouldn't be
used in the handler (see Warning about SCLog* API above in the code).
https://redmine.openinfosecfoundation.org/issues/2353
Command line option pcap-file-continuous was ignoring command line options
passed after its usage. Fixed bug, fixed formatting of help command
regarding pcap-file-continuous.
Currently, when live reload is executed through
unix-socket, suri prints in the console the following
error message:
"Live rule reload not possible if -s or -S option used at runtime."
Instead, prints "done" in unix socket,
when the live reload is not executed.
Remove the DONE state to fix a problem with state not being
changed correctly when multiple reload were done. As DONE was
not really useful, we can remove it.
Don't use pcre for the high level rule parsing, instead
using a tokenizing parser for breaking out the rule
into keywords and options.
Much faster, especially on older CPUs. Should also allow
us to provide better context where a rule parse error
occurs.
https://redmine.openinfosecfoundation.org/issues/2215
Fixes issue with events being dropped since socket was non-blocking for
offline run modes.
Add a method for determining offline from run mode. Make sure SCInstance
offline is set correctly. Use current run mode to set socket flags.
In normal records it will try to continue parsing.
GAP 'data' will be passed to file api as '0's. New call is used
so that the file API does know it is dealing with a GAP. Such
files are flagged as truncated at the end of the file and no
checksums are calculated.
Current file storing approach is using a open file, write data,
close file logic. If this technic is fixing the problem of getting
too much open files in Suricata it is not optimal.
Test on a loop shows that open, write, close on a single file is
two time slower than a single open, loop of write, close.
This patch updates the logic by storing the fd in the File structure.
This is done for a certain number of files. If this amount is exceeded
then the previous logic is used.
This patch also adds two counters. First is the number of
currently open files. The second one is the number of time
the open, write, close sequence has been used due to too much
open files.
In EVE, the entries are:
stats {file_store: {"open_files_max_hit":0,"open_files":5}}
Wrapper around Suricata's File and FileContainer API. Built around
assumption that a rust owned structure will have a
'SuricataFileContainer' member that is managed by the C-side of
things.
Where the context is a struct passed from C with pointers
to all the functions that may be called.
Instead of referencing C functions directly, wrap them
in function pointers so pure Rust unit tests can still run.
Improve output of unix mode in --list-runmodes
Honor the runmode commandline setting. Supported are 'single'
and 'autofp', with the latter still the default.
Set flags by default:
-Wmissing-prototypes
-Wmissing-declarations
-Wstrict-prototypes
-Wwrite-strings
-Wcast-align
-Wbad-function-cast
-Wformat-security
-Wno-format-nonliteral
-Wmissing-format-attribute
-funsigned-char
Fix minor compiler warnings for these new flags on gcc and clang.
If running against a pcap there is no reason to drop events,
a blocking socket is fine here. So only do non-blocking writes
when running off a live device.
Due to the use of AFL_LOOP and initialization/deinit outside of it,
part of the fuzzing relied on the global 'state' in flow and defrag.
Because of this crashes that were found could not be reproduced. The
saved crash input was only the last in the series.
This patch addresses that. It requires a new output directory 'dump'
where the packet fuzzers will store all their input. If the AFL_LOOP
fails the files will not be removed and this 'serie' can be read
again for reproducing the issue.
e.g.: AFL would work with:
--afl-decoder-ppp=@@
and after a crash is found the produced serie can be read with:
--afl-decoder-ppp-serie=1486656919-514163
The series have a timestamp as name and a suffix that controls the
order in which the files will be 'replayed' in Suricata.
Luajit has a strange memory requirement, it's 'states' need to be in the
first 2G of the process' memory.
This patch improves the pool approach by moving it to the front of the
start up.
A new config option 'luajit.states' is added to control how many states
are preallocated. It defaults to 128.
Add a warning when more states are used then preallocated. This may fail
if flow/stream/detect engines use a lot of memory. Add hint at exit that
gives the max states in use if it's higher than the default.
Add support for the ENIP/CIP Industrial protocol
This is an app layer implementation which uses the "enip" protocol
and "cip_service" and "enip_command" keywords
Implements AFL entry points
Register keywords globally at start up.
Create a map of the registery per detection engine. This we need because
the sgh_mpm_context value is set per detect engine.
Remove APP_MPMS_MAX.
Add a generic 'capture' section to the YAML:
# general settings affecting packet capture
capture:
# disable NIC offloading. It's restored when Suricata exists.
# Enabled by default
#disable-offloading: false
#
# disable checksum validation. Same as setting '-k none' on the
# commandline
#checksum-validation: none
When running in live mode, the new default 'auto' value of
unix-command.enabled causes unix-command to be activated. This
will allow users of live capture to benefit from the feature and
result in no side effect for user running in offline capture.
We did ignore additional USR2 signals while a rule-reload was running.
This changes the counter to be incremented with every additional USR2
signal so we don't ignore them anymore but it's still limited to prevent
huge overload or even overflow.
As the logging modules are no longer threading modules, rename
them so they don't look like they are being registered as
threading modules.
Also, move the registration to the output.c which will handle
registration of the loggers.
Introduces a new thread module, TMM_LOGGER, which is the
root most logger.
Only handles loggers in the packet path, stats and flow
logging are not included.
The loggers are made up of a hierarchy of loggers. At the top we
have the root logger which is the main entry point to
logging. Under the root there exists parent loggers that are the
entry point for specific types of loggers such as packet logger,
transaction loggers, etc. Each parent logger may have 0 or more
loggers that actual handle the job of producing output to something
like a file.
Fix improper fread string handling. Improve error handling.
Skip trailing spaces for slightly more pretty printing.
Coverity CID 400763.
Thanks to Steve Grubb for helping address this issue.
The structure for create the alert preceding each tagged packet
was not being initialized, preventing tagged packets from being
logged.
Note: Snort unified2 does not precede tagged packets with an
alert like is done here, so this just fixes what the code
intended to do, it does not make it Snort unified2
compatible.
Address issue:
https://redmine.openinfosecfoundation.org/issues/1854
Joshua Lumb
Eric Leblond
Philippe Antoine
Victor Julien
Shivani Bhardwaj
Pierre Chifflier
Jason Ish
Jeff Lucovsky
Danny Browning
Yujie Zhao
Jason Ish
Andreas Herz
Max Fillinger
Emmanuel Roullit
Alexander Gozman
Alexander Bluhm
Jacob Masen-Smith
Mats Klepsland
Maurizio Abba
Wolfgang Hotwagner
Dana Helwig
Giuseppe Longo
kwong
Duarte Silva