Now that the flow lookup is done in the worker threads the flow
queue handlers running after the capture thread(s) no longer have
access to the flow. This limits the options of how flow balancing
can be done.
This patch removes all code that is now useless. The only 2 methods
that still make sense are 'hash' and 'ippair'.
Instead of handling the packet update during flow lookup, handle
it in the stream/detect threads. This lowers the load of the
capture thread(s) in autofp mode.
The decoders now set a flag in the packet if the packet needs a
flow lookup. Then the workers will take care of this. The decoders
also already calculate the raw flow hash value. This is so that
this value can be used in flow balancing in autofp.
Because the flow lookup/creation is now done in the worker threads,
the flow balancing can no longer use the flow. It's not yet
available. Autofp load balancing uses raw hash values instead.
In the same line, move UDP AppLayer out of the DecodeUDP module,
and also into the stream/detect threads.
Handle TCP session reuse inside the flow engine itself. If a looked up
flow matches the packet, but is a TCP stream starter, check if the
ssn needs to be reused. If that is the case handle it within the
lookup function. Simplies the locking and removes potential race
conditions.
Add "ippair" autofp scheduler to split traffic based on source and
destination IP only (not ports).
- This is useful when using the "xbits" feature to track events
that occur between the same hosts but not necessarily the same
flow (such as exploit kit landings/expoits/payloads)
- The disadvantage is that traffic may be balanced very unevenly
between threads if some host pairs are much more frequently seen
than others, so it may be only practicable for sandbox or pcap
analysis
- not tested for IPv6
See https://redmine.openinfosecfoundation.org/issues/1661
This fixes:
96 bytes in 4 blocks are definitely lost in loss record 178 of 320
at 0x4C29C0F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
by 0x8FF88C: TmqhOutputFlowSetupCtx (tmqh-flow.c:163)
by 0x908D5F: TmThreadCreate (tm-threads.c:1097)
by 0x909A62: TmThreadCreatePacketHandler (tm-threads.c:1156)
by 0x8790CF: RunModeFilePcapAutoFp (runmode-pcap-file.c:188)
by 0x8837CB: RunModeDispatch (runmodes.c:372)
by 0x87F2A5: UnixSocketPcapFilesCheck (runmode-unix-socket.c:393)
by 0x910460: UnixCommandBackgroundTasks (unix-manager.c:430)
by 0x91420D: UnixManager (unix-manager.c:980)
by 0x907853: TmThreadsManagement (tm-threads.c:602)
by 0x68DE283: start_thread (pthread_create.c:333)
by 0x80A6A4C: clone (in /lib/x86_64-linux-gnu/libc-2.21.so)
Queue name was sometimes allocated and sometimes not. This
patch updates the behavior of creation function so it is
always allocated. This way we can free it at exit and fix
memory leak.
This fixes:
900 bytes in 110 blocks are definitely lost in loss record 322 of 329
at 0x4C29C0F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
by 0x803E0A9: strdup (in /lib/x86_64-linux-gnu/libc-2.21.so)
by 0x90090E: StoreQueueId (tmqh-flow.c:112)
by 0x8FFEA8: TmqhOutputFlowSetupCtx (tmqh-flow.c:180)
by 0x908C7F: TmThreadCreate (tm-threads.c:1095)
by 0x909982: TmThreadCreatePacketHandler (tm-threads.c:1154)
by 0x87906F: RunModeFilePcapAutoFp (runmode-pcap-file.c:188)
by 0x88376B: RunModeDispatch (runmodes.c:372)
by 0x87F245: UnixSocketPcapFilesCheck (runmode-unix-socket.c:393)
by 0x9102B0: UnixCommandBackgroundTasks (unix-manager.c:430)
by 0x91405D: UnixManager (unix-manager.c:980)
by 0x907773: TmThreadsManagement (tm-threads.c:600)
tm-queue: fix logic WIP
On Tile, replace pthread_mutex_locks with queued spin locks (ticket
locks) for dataplane processing code. This is safe when running on
dataplane cores with one thread per core. The condition variables are
no-ops when the thread is spinning anyway.
For control plane threads, unix-manager, stats-logs, thread startup,
use pthread_mutex_locks. For these locks replaced SCMutex with SCCtrlMutex
and SCCond with SCCtrlCond.
Bug #939: thread name buffers are sized inconsistently
These buffers are now all fixed at 16 bytes.
Bug #914: Having a high number of pickup queues (216+) makes suricata crash
Fixed so that we can now have 256 pickup queues, which is the current built-in
maximum. Improved the error reporting.
Bug #928: Max number of threads
Error reporting improved. Issue was the same as #914.
When handling error case on SCMallog, SCCalloc or SCStrdup
we are in an unlikely case. This patch adds the unlikely()
expression to indicate this to gcc.
This patch has been obtained via coccinelle. The transformation
is the following:
@istested@
identifier x;
statement S1;
identifier func =~ "(SCMalloc|SCStrdup|SCCalloc)";
@@
x = func(...)
... when != x
- if (x == NULL) S1
+ if (unlikely(x == NULL)) S1
clang was issuing some warnings related to unused return in function.
This patch adds some needed error treatment and ignore the rest of the
warnings by adding a cast to void.
- Implement "closing" state in flow.
- Add protocol specific timeouts.
- Lots of stream tracking updates, fixing a lot of out of window issues.
- Stream reassembly fixes.
- Implement a new IDS runmode with 4 stream and detect threads.
- Added a BUG_ON macro that aborts the engine if the expression is true.
- Better balance the flow queue handler for traffic that doesn't have flow (like icmp currently).
- Simplify application level protocol in the Tcp Session.
- Add some debugging memory counters.
Victor Julien
Jason Ish
cdwakelin
Eric Leblond
Ken Steele
Anoop Saldanha
Gerardo Iglesias Galvan
William Metcalf
Pablo Rincon