The maximum of possible alerts triggered by a unique packet was
hardcoded to 15. With usage of 'noalert' rules, that limit could be
reached somewhat easily. Make that configurable via suricata.yaml.
Conf Bug#4941
Task #4207
Register a new runmode - DPDK. This enables a new flag on Suricata start
(--dpdk).
With the flag given, DPDK runmode is enabled.
Runmode loads the configuration and then initializes EAL.
If successful, it configures the physical NICs according to the configuration
file. After that, worker threads are initialized and then are in continuous
receive loop.
Unify handling of signature matches between various rule types and
between noalert and regular rules.
"noalert" sigs are added to the alert queue initially, but removed
from it after handling their actions. This way all actions are applied
from a single place.
Make sure flow drop and pass are mutually exclusive.
The above addresses issue with pass and drops not getting applied
correctly in various cases.
Bug: #4663
Bug: #4670
Allow a plugin to register itself as a capture source. This isn't that
much different than how current sources register, it just happens
a little later on during startup.
One "slot" is reserved for capture plugins, but multiple plugins
implementing a capture can be loaded. The --capture-plugin command
line option must be used to tell Suricata which plugin
to use.
This is still very much a work in progress, but can load
PF_RING as a capture plugin.
Goals:
- reduce locking
- take advantage of 'hot' caches
- better locality
Locking reduction
New flow spare pool. The global pool is implmented as a list of blocks,
where each block has a 100 spare flows. Worker threads fetch a block at
a time, storing the block in the local thread storage.
Flow Recycler now returns flows to the pool is blocks as well.
Flow Recycler fetches all flows to be processed in one step instead of
one at a time.
Cache 'hot'ness
Worker threads now check the timeout of flows they evaluate during lookup.
The worker will have to read the flow into cache anyway, so the added
overhead of checking the timeout value is minimal. When a flow is considered
timed out, one of 2 things happens:
- if the flow is 'owned' by the thread it is handled locally. Handling means
checking if the flow needs 'timeout' work.
- otherwise, the flow is added to a special 'evicted' list in the flow
bucket where it will be picked up by the flow manager.
Flow Manager timing
By default the flow manager now tries to do passes of the flow hash in
smaller steps, where the goal is to do full pass in 8 x the lowest timeout
value it has to enforce. So if the lowest timeout value is 30s, a full pass
will take 4 minutes. The goal here is to reduce locking overhead and not
get in the way of the workers.
In emergency mode each pass is full, and lower timeouts are used.
Timing of the flow manager is also no longer relying on pthread condition
variables, as these generally cause waking up much quicker than the desired
timout. Instead a simple (u)sleep loop is used.
Both changes reduce the number of hash passes a lot.
Emergency behavior
In emergency mode there a number of changes to the workers. In this scenario
the flow memcap is fully used up and it is unavoidable that some flows won't
be tracked.
1. flow spare pool fetches are reduced to once a second. This avoids locking
overhead, while the chance of success was very low.
2. getting an active flow directly from the hash skips flows that had very
recent activity to avoid the scenario where all flows get only into the
NEW state before getting reused. Rather allow some to have a chance of
completing.
3. TCP packets that are not SYN packets will not get a used flow, unless
stream.midstream is enabled. The goal here is again to avoid evicting
active flows unnecessarily.
Better Localily
Flow Manager injects flows into the worker threads now, instead of one or
two packets. Advantage of this is that the worker threads can get packets
from their local packet pools, avoiding constant overhead of packets returning
to 'foreign' pools.
Counters
A lot of flow counters have been added and some have been renamed.
Overall the worker threads increment 'flow.wrk.*' counters, while the flow
manager increments 'flow.mgr.*'.
Additionally, none of the counters are snapshots anymore, they all increment
over time. The flow.memuse and flow.spare counters are exceptions.
Misc
FlowQueue has been split into a FlowQueuePrivate (unlocked) and FlowQueue.
Flow no longer has 'prev' pointers and used a unified 'next' pointer for
both hash and queue use.
This commit adds MAC address output to the EVE-JSON format. We follow the
remarks made in Redmine ticket #962: for packets, log MAC src/dst as a
scalar field in EVE; for flows, log MAC src/dst as lists in EVE. Field names
are different between flow and packet context to avoid type confusion
(src_mac vs. src_macs). Configuration approach and JSON representation is
taken from previous GitHub PR #2700.
Previously each 'TmSlot' had it's own packet queue that was passed
to the registered SlotFunc as an argument. This was used mostly for
tunnel packets by the decoders and by defrag.
This patch removes that in favor of a single queue in the ThreadVars:
decode_pq. This is the non-locked version of the queue as this is
only a temporary store for handling packets within a thread.
This patch removes the PacketQueue pointer argument from the API.
The new queue can be accessed directly through the ThreadVars
pointer.
Fill in the vlan_id fields unconditionally. We can now remove the check
for the vlan.use-for-tracking setting in decode.c. The debug log message
is moved to suricata.c.
Since the vlan.use-for-tracking setting is now handled in flow-hash.c,
we can fill in the vlan_id fields unconditionally. This makes the vlanh
fields unnecessary.
Related to https://redmine.openinfosecfoundation.org/issues/3076
Implement port config handling. Also check both src port and dest
port for tunnels that only set the destination port to the VXLAN
port. At the point of the check we don't know the packet direction
yet.
Implement as Suricata tunnel similar to Teredo.
Cleanups.
No longer set stream events after a gap or wrong thread. We know
we lost sync and are now in 'lets make the best of it'-mode. No
point in flooding the system with stream events.
Ticket #2484
In the eve log the decoder events are added as optional counters. This
behaviour is enabled by default. However, lots of the counters are
missing, as the names colide with other counters.
E.g.
decoder.ipv6 counts ipv6 packets
decoder.ipv6.unknown_next_header counts how often an unknown next
header is encountered.
In this example 'ipv6' would be both a json integer and a json object.
It appears that jansson favours the first that is generated, so the
event counters are mostly missing.
This patch registers them as 'decoder.events.<event>' instead. As
these names are generated on the fly, a hash table to contain the
allocated strings was added as well.
Invalid Teredo can lead to valid DNS traffic (or other UDP traffic)
being misdetected as Teredo. This leads to false negatives in the
UDP payload inspection.
Make the teredo code only consider a packet teredo if the encapsulated
data was decoded without any 'invalid' events being set.
Bug #2736.
Change the decode handler signature to increase the size of its decode
handler, from uint16 to uint32. This is necessary to let suricata use
interfaces with mtu > 65535 (ex: lo interface has default size 65536).
It's necessary to change several primitive for Packet manipulation, to
unify the parameter "packet length" whenever we are before IP decoding.
Add tests before calling DecodeIPVX function to avoid a possible
integer overflow over the len parameter.
Enables IPS functionality on Windows using the open-source
(LGPLv3/GPLv2) WinDivert driver and API.
From https://www.reqrypt.org/windivert-doc.html : "WinDivert is a
user-mode capture/sniffing/modification/blocking/re-injection package
for Windows Vista, Windows Server 2008, Windows 7, and Windows 8.
WinDivert can be used to implement user-mode packet filters, packet
sniffers, firewalls, NAT, VPNs, tunneling applications, etc., without
the need to write kernel-mode code."
- adds `--windivert [filter string]` and `--windivert-forward [filter
string]` command-line options to enable WinDivert IPS mode.
`--windivert[-forward] true` will open a filter for all traffic. See
https://www.reqrypt.org/windivert-doc.html#filter_language for more
information.
Limitation: currently limited to `autofp` runmode.
Additionally:
- `tmm_modules` now zeroed during `RegisterAllModules`
- fixed Windows Vista+ `inet_ntop` call in `PrintInet`
- fixed `GetRandom` bug (nonexistent keys) on fresh Windows installs
- fixed `RandomGetClock` building on Windows builds
- Added WMI queries for MTU
Juliana Fajardini
Victor Julien
Lukas Sismis
Victor Julien
Jeff Lucovsky
Philippe Antoine
Emmanuel Thompson
Carl Smith
Ali Jad Khalil
Jason Ish
Sascha Steinbiss
Max Fillinger
Henrik Lund Kramshoej
Maurizio Abba
Jacob Masen-Smith
fooinha