When a rules match and fired filestore we may want
to increase the stream reassembly depth for this specific.
This add the 'depth' setting in file-store config,
which permits to specify how much data we want to reassemble
into a stream.
As the logging modules are no longer threading modules, rename
them so they don't look like they are being registered as
threading modules.
Also, move the registration to the output.c which will handle
registration of the loggers.
Make the file storage use the streaming buffer API.
As the individual file chunks were not needed by themselves, this
approach uses a chunkless implementation.
Code was unused and was leaking memory.
This fixes:
Direct leak of 614240 byte(s) in 3839 object(s) allocated from:
#0 0x4c396b in malloc (/opt/suricata-asan/bin/suricata+0x4c396b)
#1 0x11bc12e in LogFileNewCtx /home/pmanev/sandnet-qa/stage/oisf/src/util-logopenfile.c:474:27
#2 0xcf7ef2 in LogFilestoreLogInitCtx /home/pmanev/sandnet-qa/stage/oisf/src/log-filestore.c:430:31
#3 0xec3275 in RunModeInitializeOutputs /home/pmanev/sandnet-qa/stage/oisf/src/runmodes.c:763:26
#4 0xeae17f in UnixSocketPcapFilesCheck /home/pmanev/sandnet-qa/stage/oisf/src/runmode-unix-socket.c:391:9
#5 0x109bc37 in UnixCommandBackgroundTasks /home/pmanev/sandnet-qa/stage/oisf/src/unix-manager.c:430:20
#6 0x10a9be2 in UnixManager /home/pmanev/sandnet-qa/stage/oisf/src/unix-manager.c:977:9
#7 0x1075643 in TmThreadsManagement /home/pmanev/sandnet-qa/stage/oisf/src/tm-threads.c:600:9
#8 0x7fbc9fcb3181 in start_thread /build/eglibc-3GlaMS/eglibc-2.19/nptl/pthread_create.c:312
With the previous code the order of the logging modules in the
YAML were determining which module was run first. This was not
wished and a consequences was that the EVE fileinfo module was
not correctly displaying the key 'stored' because it was
depending on a flag set alter by the filestore module.
This patch adds a priority file to the TmModule structure. The
higher the priority is set, the sooner the module is run in the
logging process. The RunModeOutput structure has also been
updated to contain the name of the original TmModule. Thus allowing
to define a priority for a RunModeOutput.
Currently only the filestore has a priority set. The rest of them is
set to the default value of zero.
The TM_FLAG_LOGAPI_TM flag indicates that a module is run by the log
api, not by the 'regular' thread module call functions.
Set flag in all all Log API users' registration code.
Purpose of this flag is in profiling. In profiling output it will be
used to list log api thread modules separately.
app-layer.[ch], app-layer-detect-proto.[ch] and app-layer-parser.[ch].
Things addressed in this commit:
- Brings out a proper separation between protocol detection phase and the
parser phase.
- The dns app layer now is registered such that we don't use "dnstcp" and
"dnsudp" in the rules. A user who previously wrote a rule like this -
"alert dnstcp....." or
"alert dnsudp....."
would now have to use,
alert dns (ipproto:tcp;) or
alert udp (app-layer-protocol:dns;) or
alert ip (ipproto:udp; app-layer-protocol:dns;)
The same rules extend to other another such protocol, dcerpc.
- The app layer parser api now takes in the ipproto while registering
callbacks.
- The app inspection/detection engine also takes an ipproto.
- All app layer parser functions now take direction as STREAM_TOSERVER or
STREAM_TOCLIENT, as opposed to 0 or 1, which was taken by some of the
functions.
- FlowInitialize() and FlowRecycle() now resets proto to 0. This is
needed by unittests, which would try to clean the flow, and that would
call the api, AppLayerParserCleanupParserState(), which would try to
clean the app state, but the app layer now needs an ipproto to figure
out which api to internally call to clean the state, and if the ipproto
is 0, it would return without trying to clean the state.
- A lot of unittests are now updated where if they are using a flow and
they need to use the app layer, we would set a flow ipproto.
- The "app-layer" section in the yaml conf has also been updated as well.
There were 8 identical copies of CreateTimeString() in 8 files.
Most used SCLocalTime, to replace localtime_r(), but some did not.
Created one copy in util-time.c.
Improved accuracy, improved performance. Performance improvement
noticeable with http heavy traffic and ruleset.
A lot of other cosmetic changes carried out as well. Wrappers introduced
for a lot of app layer functions.
Failing dce unittests disabled. Will be reintroduced in the updated dce
engine.
Cross transaction matching taken care of. FPs emanating from these
matches have now disappeared. Double inspection of transactions taken
care of as well.
This patch modifies the file store system to have it create the
file store directory if needed. It dos not create the full
directory tree as the parent directory must have already been
created.
When handling error case on SCMallog, SCCalloc or SCStrdup
we are in an unlikely case. This patch adds the unlikely()
expression to indicate this to gcc.
This patch has been obtained via coccinelle. The transformation
is the following:
@istested@
identifier x;
statement S1;
identifier func =~ "(SCMalloc|SCStrdup|SCCalloc)";
@@
x = func(...)
... when != x
- if (x == NULL) S1
+ if (unlikely(x == NULL)) S1
clang was issuing some warnings related to unused return in function.
This patch adds some needed error treatment and ignore the rest of the
warnings by adding a cast to void.
This function is a wrapper to localtime_r. It is needed to avoid
a compilation warning on OpenBSD. I'm forced to type the function
to a non pointer first parameter. If not we will have to use two
differents functions in OpenBSD where tv->tv_sec is a long
(different from time_t).
Giuseppe Longo
Duarte Silva
Jason Ish
Victor Julien
maxtors
Eric Leblond
Tom DeCanio
David Abarbanel
Ken Steele
Anoop Saldanha
Ludovico Cavedon