Work towards making `suricata-common.h` only introduce system headers
and other things that are independent of complex internal Suricata
data structures.
Update files to compile after this.
Remove special DPDK handling for strlcpy and strlcat, as this caused
many compilation failures w/o including DPDK headers for all files.
Remove packet macros from decode.h and move them into their own file,
turn them into functions and rename them to match our function naming
policy.
Update APIs to store files in transactions instead of the per flow state.
Goal is to avoid the overhead of matching up files and transactions in
cases where there are many of both.
Update all protocol implementations to support this.
Update file logging logic to account for having files in transactions. Instead
of it acting separately on file containers, it is now tied into the
transaction logging.
Update the filestore keyword to consider a match if filestore output not
enabled.
Required including NSS header in places that depended on
util-file.h including it.
All filestore suricata-verify tests now pass without libnss.
Also enabled detect-file{md5,sha1,sha256} without NSS support.
Improve flow file flags and file flags updates. Introduce a mask
that is set at start up to avoid lots of runtime checks.
Disable cocci flags check as it doesn't support the more dynamic
nature of the flag updates.
Now that we call the file prune loop very regularly, we can move the
SMTP specific inspection pruning logic into this loop. Helps with
cases there we don't (often) update a files inspection trackers.
If a file transfer stops on flow timeout, it won't be closed or
truncated. This patch makes sure that in such cases the files
are indeed truncated. This fixes the filestore-v2 output module,
as that requires a sha256 for storing the partial file correctly.
In normal records it will try to continue parsing.
GAP 'data' will be passed to file api as '0's. New call is used
so that the file API does know it is dealing with a GAP. Such
files are flagged as truncated at the end of the file and no
checksums are calculated.
Current file storing approach is using a open file, write data,
close file logic. If this technic is fixing the problem of getting
too much open files in Suricata it is not optimal.
Test on a loop shows that open, write, close on a single file is
two time slower than a single open, loop of write, close.
This patch updates the logic by storing the fd in the File structure.
This is done for a certain number of files. If this amount is exceeded
then the previous logic is used.
This patch also adds two counters. First is the number of
currently open files. The second one is the number of time
the open, write, close sequence has been used due to too much
open files.
In EVE, the entries are:
stats {file_store: {"open_files_max_hit":0,"open_files":5}}
Some protocols transfer multiple files in parallel. To support this add
a 'track id' to the API. This track id is set by the protocol parser. It
will use this id to indicate what file in the FileContainer it wants to
act on.
Set flags by default:
-Wmissing-prototypes
-Wmissing-declarations
-Wstrict-prototypes
-Wwrite-strings
-Wcast-align
-Wbad-function-cast
-Wformat-security
-Wno-format-nonliteral
-Wmissing-format-attribute
-funsigned-char
Fix minor compiler warnings for these new flags on gcc and clang.
This patch introduces the FileDataSize and FileTrackedSize functions.
The first one is just a renaming of the initial FilSize function
whereas the other one is using the newly introduced size field as
value.
The file size returned by FileSize is invalid if file store is not
used so we introduce a new size field in File structure that is used
to store the size.
When a rules match and fired filestore we may want
to increase the stream reassembly depth for this specific.
This add the 'depth' setting in file-store config,
which permits to specify how much data we want to reassemble
into a stream.
Moved and adapted code from detect-filemd5 to util-detect-file-hash,
generalised code to work with SHA-1 and SHA-256 and added necessary
flags and other constants.
Make the file storage use the streaming buffer API.
As the individual file chunks were not needed by themselves, this
approach uses a chunkless implementation.
When the file API is used to do content inspection (currently only
smtp does this), the detection should be considered while pruning
the file chunks.
This patch introduces a new flag for the file API: FILE_USE_DETECT
When it is used, 'FilePrune' will not remove chunks that are (partly)
beyond the File::content_inspected tracker.
When using this flag, it's important to realize that when the detect
engine is disabled or rules are not matching, content_inspected
might not get updated.
Improved accuracy, improved performance. Performance improvement
noticeable with http heavy traffic and ruleset.
A lot of other cosmetic changes carried out as well. Wrappers introduced
for a lot of app layer functions.
Failing dce unittests disabled. Will be reintroduced in the updated dce
engine.
Cross transaction matching taken care of. FPs emanating from these
matches have now disappeared. Double inspection of transactions taken
care of as well.
Victor Julien
Victor Julien
Philippe Antoine
Jason Ish
magenbluten
Eric Leblond
Giuseppe Longo
Duarte Silva
Tom DeCanio
Giuseppe Longo
Anoop Saldanha