Kernel time is not available (and/or costly) on NIC such as
Netronome so we update the logic to detect dead flows based on a
lack of update of packets counters. This way, the XDP filter will
be usable by network card.
This patch also updates the ebpf code to support per CPU and
regular mapping. Netronome is not supporting it and the structure
is using atomic for counter so the cost of simultaneous update
is really low.
This patch also updates the xdp_filter to be able to select if the
flow table is per CPU on shared. Second option will be used for
hardward offload. To deactivate the per cpu hash, you need to set
USE_PERCPU_HASH to 0.
This patch also adds an new option to af-packet named no-percpu-hash
If this option is set to yes then the Flow bypassed manager thread
will use one CPU instead of the number of cores. By doing that
we are able to handle the case where USE_PERCPU_HASH is unset (so
hardware offload for Netronome).
This patch also remove aligment indications in the eBPF filter. This
was not really needed and it seems it is causing problem with
some recent version of LLVM toolchain.
This patch adds a boolean option "xdp-cpu-redirect" to af-packet
interface configuration. If set, then the XDP filter will load
balance the skb creation on specified CPUs instead of doing the
creation on the CPU handling the packet. In the case of a card
with asymetric hashing this will allow to avoid saturating the
single CPU handling the trafic.
The XDP filter must contains a set of map allowing load balancing.
This is the case of xdp_filter.bpf.
Fixed-by: Jesper Dangaard Brouer <netoptimizer@brouer.com>
This patch implements bypass capability for af-packet.
The filter only bypass TCP and UDP in IPv4 and IPv6. It don't
don't bypass IPv6 with extended headers.
This patch also introduces a bypassed flow manager that takes
care of timeouting the bypassed flows. It uses a 60 sec
timeout on flow. As they are supposed to be active we can
try that. If they are not active then we don't care to get them
back in Suricata.
This patch introduces the ebpf cluster mode. This mode is using
an extended BPF function that is loaded into the kernel and
provide the load balancing.
An example of cluster function is provided in the ebpf
subdirectory and provide ippair load balancing function.
This is a function which uses the same method as
the one used in autofp ippair to provide a symetrical
load balancing based on IP addresses.
A simple filter example allowing to drop IPv6 is added to the
source.
This patch also prepares the infrastructure to be able to load
and use map inside eBPF files. This will be used later for flow
bypass.
The new OutputInitResult is a struct return type that allows
logger init functions to return a NULL context without
raising error.
Instead of returning NULL to signal error, the "ok" field will
be set to false. If ok, but the ctx is NULL, then silently
move on to the next logger.
Use case: multiple versions of a specific logger, and one
implementation decides the configuration is not for that
implemenation. It can return NULL, ok.
Set flags by default:
-Wmissing-prototypes
-Wmissing-declarations
-Wstrict-prototypes
-Wwrite-strings
-Wcast-align
-Wbad-function-cast
-Wformat-security
-Wno-format-nonliteral
-Wmissing-format-attribute
-funsigned-char
Fix minor compiler warnings for these new flags on gcc and clang.
Only use RSS queue count when cluster_qm is used. Only use core count
when cluster_flow is used.
Use a local variable to simplify the check so that we don't have to deal
with the extra flags.
Older system may pretend they can support FANOUT but then fail to
work at runtime. CentOS6 is an example of this. It would fail to
start up with the default configuration with errors like:
[15770] 21/6/2016 -- 16:00:13 - (tm-threads.c:2168) <Notice> (TmThreadWaitOnThreadInit) -- all 4 packet processing threads, 4 management threads initialized, engine started.
[15785] 21/6/2016 -- 16:00:13 - (source-af-packet.c:1907) <Error> (AFPCreateSocket) -- [ERRCODE: SC_ERR_AFP_CREATE(190)] - Coudn't set fanout mode, error Protocol not available
[15785] 21/6/2016 -- 16:00:13 - (source-af-packet.c:1337) <Error> (ReceiveAFPLoop) -- [ERRCODE: SC_ERR_AFP_CREATE(190)] - Couldn't init AF_PACKET socket, fatal error
[15770] 21/6/2016 -- 16:00:13 - (suricata.c:2664) <Notice> (main) -- Signal Received. Stopping engine.
[15787] 21/6/2016 -- 16:00:13 - (source-af-packet.c:1907) <Error> (AFPCreateSocket) -- [ERRCODE: SC_ERR_AFP_CREATE(190)] - Coudn't set fanout mode, error Protocol not available
[15788] 21/6/2016 -- 16:00:13 - (source-af-packet.c:1907) <Error> (AFPCreateSocket) -- [ERRCODE: SC_ERR_AFP_CREATE(190)] - Coudn't set fanout mode, error Protocol not available
[15786] 21/6/2016 -- 16:00:13 - (source-af-packet.c:1907) <Error> (AFPCreateSocket) -- [ERRCODE: SC_ERR_AFP_CREATE(190)] - Coudn't set fanout mode, error Protocol not available
[15789] 21/6/2016 -- 16:00:13 - (flow-manager.c:693) <Perf> (FlowManager) -- 0 new flows, 0 established flows were timed out, 0 flows in closed state
[15787] 21/6/2016 -- 16:00:13 - (source-af-packet.c:1337) <Error> (ReceiveAFPLoop) -- [ERRCODE: SC_ERR_AFP_CREATE(190)] - Couldn't init AF_PACKET socket, fatal error
[15788] 21/6/2016 -- 16:00:13 - (source-af-packet.c:1337) <Error> (ReceiveAFPLoop) -- [ERRCODE: SC_ERR_AFP_CREATE(190)] - Couldn't init AF_PACKET socket, fatal error
[15786] 21/6/2016 -- 16:00:13 - (source-af-packet.c:1337) <Error> (ReceiveAFPLoop) -- [ERRCODE: SC_ERR_AFP_CREATE(190)] - Couldn't init AF_PACKET socket, fatal error
This patch adds a test that if run before the number of threads
is determined. If the test fails, only 1 thread is created.
As AF_PACKET handles csum offloading don't check for this type of
offloading. Other methods like pcap and netmap do require it to be
turned off.
Improve disable command suggestion wording.
Update the code to use mmap capture by default even in unset in
configuration file. mmap capture is now be turned off by using
explicitely 'use-mmap: no' in configuration.
If TPACKET_V3 is not defined then it is not available and we should
not build anything related to tpacket_v3. This will allow us to
activate it dy default and fallback to v2 if not available.
It is used to set the block size in tpacket_v3. It will allow user
to tune the capture depending on his bandwidth.
Default block size value has been updated to a bigger value to
allow more efficient wlak on block.
This patch adds a basic implementation of AF_PACKET tpacket v3. It
is basic in the way it is only working for 'workers' runnning mode.
If not in 'workers' mode there is a fallback to tpacket_v2. Feature
is activated via tpacket-v3 option in the af-packet section of
Suricata YAML.
For af-packet, pf-ring, netmap, and pcap use a generic
lookup function to find the configuration node for an
interface.
The new lookup function does not depend on the ordering
of the items inside the device configuration.
Update FlowManager/Recycler to use global name.
Also add # into thread number.
Update af-packet to use global threadnames.
Update pcap to use global threadnames.
Update pfring to use global threadnames.
Update erf-dag to use global threadnames.
Update nflog to use global threadnames.
Update netmap to use global threadnames.
Update napatech to use global threadnames.
This patch implements the rollover option in af_packet capture.
This should heavily minimize the packet drops as well as the
maximum bandwidth treated for a single flow.
The option has been deactivated by default but it is activated in
the af_packet default section. This ensure there is no change for
old users using an existing YAML. And new users will benefit from
the change.
This option is available since Linux 3.10. An analysis of af_packet
kernel code shows that setting the flag in all cases should not
cause any trouble for older kernel.
This patch implements the fanout load balancing modes available
in kernel 4.0. The more interesting is cluster_qm that does the
load balancing based on the RSS queues. So if the network card
is doing a flow based load balancing then a given socket will
receive all packets of a flow indepently of the CPU affinity.
If we manage to read the number of RSS queues from an interface,
this means that the optimal number of capture threads is equal
to the minimum of this number and of the number of cores on the
system.
This patch implements this logic thanks to the newly introduced
function GetIfaceRSSQueuesNum.
Add a new default value for the 'threads:' setting in af-packet: "auto".
This will create as many capture threads as there are cores.
Default runmode of af-packet to workers.
Eric Leblond
Jason Ish
Victor Julien
maxtors
Zachary Rasmor