The global variable suricata_ctl_flags needs to volatile, otherwise the
compiler might not cause the variable to be read every time because it
doesn't know other threads might write the variable.
This was causing Suricata to not exit under some conditions.
Using a stack for free Packet storage causes recently freed Packets to be
reused quickly, while there is more likelihood of the data still being in
cache.
The new structure has a per-thread private stack for allocating Packets
which does not need any locking. Since Packets can be freed by any thread,
there is a second stack (return stack) for freeing packets by other threads.
The return stack is protected by a mutex. Packets are moved from the return
stack to the private stack when the private stack is empty.
Returning packets back to their "home" stack keeps the stacks from getting out
of balance.
The PacketPoolInit() function is now called by each thread that will be
allocating packets. Each thread allocates max_pending_packets, which is a
change from before, where that was the total number of packets across all
threads.
During socket creation all error cases were leading to suricata to
retry the opening of capture. This patch updates this behavior to
have fatal and recoverable error case. In case of a fatal error,
suricata is leaving cleanly.
Previously the sync code would depend on traffic to complete. This
patch adds poll support and can complete the setup if the poll timeout
is reached as well.
Part of bug #1130.
This patch is updating af-packet to discard packets that have been
sent to a socket before all socket in a fanout group have been setup.
Without this, there is no way to assure that all packets for a single
flow will be treated by the same thread.
Tests have been done on a system with an ixgbe network card. When using
'cluster_flow' load balancing and disactivating receive hash on the iface:
ethtool -K IFACE rxhash off
then suricata is behaving as expected and all packets for a single flow
are treated by the same thread.
For some unknown reason, this is not the case when using cluster_cpu. It
seems that in that case the load balancing is not perfect on the card side.
The rxhash offloading has a direct impact on the cluster_flow load balancing
because load balancing is done by using a generic hash key attached to
each skb. This hash can be computed by the network card or can be
computed by the kernel. In the xase of a ixgbe network card, it seems there
is some issue with the hash key for TCP. This explains why it is necessary to
remove the rxhash offloading to have a correct behavior. This could also
explain why cluster_cpu is currently failing because the card is using the
same hash key computation to do the RSS queues load balancing.
Some of the packets counters were using a 32bit integer. Given the
bandwidth that is often seen, this is not a good idea. This patch
switches to 64bit counter.
Packets counter is incremented in AFPDumpCounters and it was
also incremented during packet reading. The result was a value
that is twice the expected result.
Spotted-by: Victor Julien <victor@inliniac.net>
In really short Suricata runtimes the capture counters would not
be updated. This patch does a force update at the end of the
capture loops in pcap and af-packet.
Some old distribution don't ship recent enough linux header. This
result in TP_STATUS_VLAN_VALID being undefined. This patch defines
the constant and use it as it is used in backward compatible method
in the code: the flag is not set by kernel and a test on vci value
will be made.
This should fix https://redmine.openinfosecfoundation.org/issues/1106
Flow-timeout code injects pseudo packets into the decoders, leading
to various issues. For a full explanation, see:
https://redmine.openinfosecfoundation.org/issues/1107
This patch works around the issues with a hack. It adds a check to
each of the decoder entry points to bail out as soon as a pseudo
packet from the flow timeout is encountered.
Ticket #1107.
This patch uses the new function SCKernelVersionIsAtLeast to know
that we've got a old kernel that do not strip the VLAN header from
the message before sending it to userspace.
Since commit in kernel
commit a3bcc23e890a6d49d6763d9eb073d711de2e0469
Author: Ben Greear <greearb@candelatech.com>
Date: Wed Jun 1 06:49:10 2011 +0000
af-packet: Add flag to distinguish VID 0 from no-vlan.
a flag is set to indicate VLAN has been set in packet header.
As suggested in commit message, using a test of the flag followed
by a check on vci value ensure backward compatibility of the test.
To be able to register counters from AppLayerGetCtxThread, the
ThreadVars pointer needs to be available in it and thus in it's
callers:
- AppLayerGetCtxThread
- DecodeThreadVarsAlloc
- StreamTcpReassembleInitThreadCtx
Logic of patch 98e4a14f6d was correct
but implementation is wrong because TP_STATUS_KERNEL is equal to
zero and thus can not be evaluated in a binary operation. This patch
updates the logic by doing two tests.
Reported-by: Alessandro Guido
This patch updates the logic of the packet acquisition loop. When
the reader loop function is called and when the data to read
at offset is a without data (kernel) or still used by suricata. We
try to iter for a loop on the ring to try to find kernel put by
data.
As we are entering the function because the poll said there was some
data. This allow us to jump to the data added to the ring by the
kernel.
When using suricata in autofp mode, with multiple detect threads and
packet acquisition threads attached to a dedicated CPU, the reader
loop function was looping really fast because poll call was returning
immediatly because we did read the data available.
Live device counter was in fact the number of packets seen by suricata
and not the total number of packet reported by kernel. This patch fixes
this by using counter provided by kernel instead.
The counter is Clear On Read, so by adding the value fetch at each call
and earch sockets we get the number of packets and drops for the
interface.
This patch adds and increments a invalid packet counter. It
does this by introducing PacketDecodeFinalize function
This function is incrementing the invalid counter and is also
signalling the packet to CUDA.
This commit allows handling Packets allocated by different methods.
The ReleaseData function pointer in the Packet structure is replaced
with ReleasePacket function pointer, which is then always called to
release the memory associated with a Packet.
Currently, the only usage of ReleaseData is in AF Packet. Previously
ReleaseData was only called when it was not NULL. To implement the
same functionality as before in AF Packet, a new function is defined
in AF Packet to first call the AFP specific ReleaseData function and
then releases the Packet structure.
Three new general functions are defined for releasing packets in the
default case:
1) PacketFree() - To release a packet alloced with SCMalloc()
2) PacketPoolReturnPacket() - For packets allocated from the Packet Pool.
Calls RECYCLE_PACKET(p)
3) PacketFreeOrRelease() - Calls PacketFree() or PacketPoolReturnPacket()
based on the PKT_ALLOC flag.
Having these functions removes the need to check the PKT_ALLOC flag
when releasing a packet in most cases, since the ReleasePacket
function encodes how the Packet was allocated. The PKT_ALLOC flag is
still set and is needed when AF Packet releases a packet, since it
replaces the ReleasePacket function pointer with its own function and
then calls PacketFreeOfRelease(), which uses the PKT_ALLOC flag.
Use test macro instead of direct access to action field.
This patch has been obtained by using the following
spatch file:
@@
Packet *p;
expression E;
@@
- p->action & E
+ TEST_PACKET_ACTION(p, E)
Victor Julien
Eric Leblond
Ken Steele
Anoop Saldanha