At shutdown, all flows that still need work are handled by the flow
force reassembly logic. This means one or more flow end pseudo packets
are generated and pushed through the engine for final detection and
logging.
In some cases this would not work correctly. This was caused by the
flow timeout logic kicking in before all the 'live' packets were
processed. Before the flow timeout handling runs the receive threads
are disabled, however the engine did not wait for the in-flight
packets to be fully processed. In autofp mode, packets could still
be in the queue between receive thread(s) and flow worker(s).
This patch adds a new function that 'drains' all the packet threads
of any in-progress packets before moving on the flow timeout logic.
Bug #1946.
Thread restarts never worked well and the rest of the engine was
never really expecting errors to lead to thread restarts. Either
and error is recoverable in the thread, or not at all.
So this patch removes the functionality completely.
Initial version of the 'FlowWorker' thread module. This module
combines Flow handling, TCP handling, App layer handling and
Detection in a single module. It does all flow related processing
under a single flow lock.
When we run on live traffic, time handling is simple. Packets have a
timestamp set by the capture method. Management threads can simply
use 'gettimeofday' to know the current time. There should never be
any serious gap between the two or major differnces between the
threads.
In offline mode, things are dramatically different. Here we try to keep
the time from the pcap, which means that if the packets are recorded in
2011 the log output should also reflect this. Multiple issues:
1. merged pcaps might have huge time jumps or time going backward
2. slowly recorded pcaps may be processed much faster than their
'realtime'
3. management threads need a concept of what the 'current' time is for
enforcing timeouts
4. due to (1) individual threads may have very different views on what
the current time is. E.g. T1 processed packet 1 with TS X, while T2
at the very same time processes packet 2 with TS X+100000s.
The changes in flow handling make the problems worse. The capture thread
no longer handles the flow lookup, while it did set the global 'time'.
This meant that a thread may be working on Packet 1 with TS 1, while the
capture thread already saw packet 2 with TS 10000. Management threads
would take TS 10000 as the 'current time', considering a flow created by
the first thread as timed out immediately.
This was less of a problem before the flow changes as the capture thread
would also create a flow reference for a packet, meaning the flow
couldn't time out as easily. Packets in the queues between capture
thread and workers would all hold such references.
The patch updates the time handling to be as follows.
In offline mode we keep the timestamp per thread. If a management thread
needs current time, it will get the minimum of the threads' values. This
is to avoid the problem that T2s time value might already trigger a flow
timeout as the flow lastts + 100000s is almost certainly meaning the
flow would be considered timed out.
- Changed FlowManagerThread to FM-
- Changed FlowRecyclerThread to FR-
- Changed use of strcasecmp to strncasecmp. This was used in the
killing and disabling of FM/FR Threads.
TmThreadCreate copy string provided as name for threads to
avoid any issue is a non allocated string is used.
This patch also introduce TmThreadSetGroupName function. This
function is used to be sure we have an allocation when
assigning the thread group name. This way we can free allocated
memory at exit.
Both code changes have required some fixes in different parts of
the code to be in sync with the new API.
Good point about these changes is that it fixes an inconsistency
were some names were not allocated and some were.
Fix cleaning of threads where mutex and condition where not freed.
This fixes:
352 (192 direct, 160 indirect) bytes in 4 blocks are definitely lost in loss record 301 of 327
at 0x4C29C0F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
by 0x909404: TmThreadInitMC (tm-threads.c:1764)
by 0x908DE7: TmThreadCreate (tm-threads.c:1120)
by 0x90A326: TmThreadCreateMgmtThread (tm-threads.c:1183)
by 0x4CA0AD: StatsSpawnThreads (counters.c:856)
by 0x87F254: UnixSocketPcapFilesCheck (runmode-unix-socket.c:396)
by 0x910330: UnixCommandBackgroundTasks (unix-manager.c:430)
by 0x9140DD: UnixManager (unix-manager.c:980)
by 0x9077F3: TmThreadsManagement (tm-threads.c:600)
by 0x68DE283: start_thread (pthread_create.c:333)
by 0x80A6A4C: clone (in /lib/x86_64-linux-gnu/libc-2.21.so)
This patch adds a new callback PktAcqBreakLoop() in TmModule to let
packet acquisition modules define "break-loop" functions to terminate
the capture loop. This is useful in case of blocking functions that
need special actions to take place in order to stop the execution.
Implement this for PF_RING
If TmThreadsUnregisterThread was called with out of range 'id', a lock
would not be cleared after returning from the function.
** CID 1264421: Missing unlock (LOCK)
/src/tm-threads.c: 2186 in TmThreadsUnregisterThread()
Don't kill flow manager and recyclers before the rest of the threads. The
packet threads may still have packets from their pools. As the flow threads
would destroy their pools the packets would be lost.
This patch doesn't kill the threads, it just pulls them out of their run
loop and into a wait loop. The packet pools won't be cleared until all
threads are killed.
Wait for flow management threads to close before moving on to the
next steps in the shutdown process.
Don't destroy flow force reassembly packet pool too early. Worker
threads may still want to return packets to it.
Update pktacq loop to process flow timeouts in a running engine.
Add a new step to the shutdown phase of packet acquisition loop
threads (pktacqloop).
The shutdown code lets the pktacqloop break out of it's packet
acquisition loop. The thread then enters a flow timeout loop, where
it processes packets from it's tv->stream_pq queue until it's
empty _and_ the KILL flag is set.
Make sure receive threads are done before moving on to flow hash
cleanup (recycle all). Without this the flow recycler could start
it's unconditional hash clean up while detect threads are still
running on the flows.
Update unix socket to match live modes.
Add function TmThreadsInjectPacketById that is to be used to inject flow
timeout packets into the threads stream_pq queue.
TmThreadsInjectPacketById will also wake up listening threads if
applicable.
Packets are passed all packets together in an NULL terminated array
to reduce locking overhead.
Create thread registration and unregistration API for assigning unique
thread id's.
Threadvars is static even if a thread restarts, so we can do the
registration before the threads start.
A thread is unregistered when the ThreadVars are freed.
Currently management threads do their own thread setup and handling. This
patch introduces a new way of handling management threads.
Functionality that needs to run as a management thread can now register
itself as a regular 'thread module' (TmModule), where the 'Management'
callback is registered.
Victor Julien
Zachary Rasmor
maxtors
Eric Leblond
cardigliano
Ken Steele