Example:
util-runmodes.c: In function ‘RunModeSetIPSAutoFp’:
util-runmodes.c:496:40: error: ‘snprintf’ output may be truncated before the last format character [-Werror=format-truncation=]
snprintf(qname, sizeof(qname), "pickup%d", thread+1);
^~~~~~~~~~
util-runmodes.c:496:9: note: ‘snprintf’ output between 8 and 17 bytes into a destination of size16
snprintf(qname, sizeof(qname), "pickup%d", thread+1);
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Solved by reducing 'thread' to a uint16_t and limiting the max
thread count to 1024.
Introduces a new thread module, TMM_LOGGER, which is the
root most logger.
Only handles loggers in the packet path, stats and flow
logging are not included.
The loggers are made up of a hierarchy of loggers. At the top we
have the root logger which is the main entry point to
logging. Under the root there exists parent loggers that are the
entry point for specific types of loggers such as packet logger,
transaction loggers, etc. Each parent logger may have 0 or more
loggers that actual handle the job of producing output to something
like a file.
Initial version of the 'FlowWorker' thread module. This module
combines Flow handling, TCP handling, App layer handling and
Detection in a single module. It does all flow related processing
under a single flow lock.
Instead of handling the packet update during flow lookup, handle
it in the stream/detect threads. This lowers the load of the
capture thread(s) in autofp mode.
The decoders now set a flag in the packet if the packet needs a
flow lookup. Then the workers will take care of this. The decoders
also already calculate the raw flow hash value. This is so that
this value can be used in flow balancing in autofp.
Because the flow lookup/creation is now done in the worker threads,
the flow balancing can no longer use the flow. It's not yet
available. Autofp load balancing uses raw hash values instead.
In the same line, move UDP AppLayer out of the DecodeUDP module,
and also into the stream/detect threads.
Handle TCP session reuse inside the flow engine itself. If a looked up
flow matches the packet, but is a TCP stream starter, check if the
ssn needs to be reused. If that is the case handle it within the
lookup function. Simplies the locking and removes potential race
conditions.
The shortening of the interfacenames is now dependent on the
size of the destination buffer, so that this can be easily
changed in the future. The process uses snprintf and strlcat.
Also changed the buffer sizes in the util-runmodes to 12
so that they can hold 11 chars + null terminator.
Changed out strcpy, strncpy to strlcat and strlcpy. Also added
checks to see if the shortening did work or if it would fail in
advance. Fixed code in util-device and util-runmodes.
Added function LiveSafeDeviceName in util-device that shortens an
NIC device name if the name is over a given length and turns
it in to Ex: longi...eeth1
TmThreadCreate copy string provided as name for threads to
avoid any issue is a non allocated string is used.
This patch also introduce TmThreadSetGroupName function. This
function is used to be sure we have an allocation when
assigning the thread group name. This way we can free allocated
memory at exit.
Both code changes have required some fixes in different parts of
the code to be in sync with the new API.
Good point about these changes is that it fixes an inconsistency
were some names were not allocated and some were.
In case of autofp (or more general, when flow and stream engine
run in different threads) the flow engine should not trigger a flow
reuse as this can lead to race conditions between the flow and the
stream engine.
In such cases, the flow engine can be far ahead of the stream engine
as packets are in a queue between the threads.
Observed:
Flow engine tags packet 10 as start of new flow. Flow is tagged as
'reused'.
Stream engine evaluates packet 5 which belongs to the old flow. It
rejects the flow as it's tagged 'reused'. Attaches packet 5 to the
new flow which is wrong.
Solution:
This patch connects the flow engines handling of reuse cases to
the runmode. It hooks into the RunmodeSetFlowStreamAsync() call to
notify the flow engine that it shouldn't handle the reuse.
This patch is a result of applying the following coccinelle
transformation to suricata sources:
@istested@
identifier x;
statement S1;
identifier func =~ "(SCMalloc|SCStrdup|SCCalloc|SCMallocAligned|SCRealloc)";
@@
x = func(...)
... when != x
- if (x == NULL) S1
+ if (unlikely(x == NULL)) S1
Bug #939: thread name buffers are sized inconsistently
These buffers are now all fixed at 16 bytes.
Bug #914: Having a high number of pickup queues (216+) makes suricata crash
Fixed so that we can now have 256 pickup queues, which is the current built-in
maximum. Improved the error reporting.
Bug #928: Max number of threads
Error reporting improved. Issue was the same as #914.
When handling error case on SCMallog, SCCalloc or SCStrdup
we are in an unlikely case. This patch adds the unlikely()
expression to indicate this to gcc.
This patch has been obtained via coccinelle. The transformation
is the following:
@istested@
identifier x;
statement S1;
identifier func =~ "(SCMalloc|SCStrdup|SCCalloc)";
@@
x = func(...)
... when != x
- if (x == NULL) S1
+ if (unlikely(x == NULL)) S1
This patch modifies the init of Detect threads. They are now started
with a dummy function and their initialisation is done after the
signatures are loaded. Just after this, the dummy function is switched
to normal one.
In IPS mode, this permit to route packets without waiting for the
signature to start and should fix#488.
Offline mode such as pcap file don't use this mode to be sure to
analyse all packets in the file.
The patch introduces a "delayed-detect" configuration variable
under detect-engine. It can be used to activate the feature
(set to "yes" to have signature loaded after capture is started).
This patch adds the 'auto', 'autofp' and 'worker' runmode for
IPS. It provides a set of ready-to-use functions that can be
used by NFQ and IPFW to implement this running mode.
This patch modifies the RunModeSetLiveCaptureAuto() prototype to
be able to detect that a 'threads' variable (telling how much
threads must listen to one socket in IDS mode) has been used
in the configuration file. It then print a warning message
if this is the case.
Sascha Steinbiss
Victor Julien
Ray Ruvinskiy
Jason Ish
Andreas Herz
Zachary Rasmor
maxtors
Eric Leblond
cardigliano
Ken Steele
Anoop Saldanha
pi-rho