This commit fixes a leak of mmap'ed ring buffer that was not
unmaped when a socket was closed. In addition, the leak could
break an inline channel on certain configurations.
Also slightly changed AFPCreateSocket():
1. If an interface is not up, it does not try to apply any
settings to a socket. This reduces a number of error messages
while an interface is down.
2. Interface is considered active if both IFF_UP and IFF_RUNNING
are present.
So far, the suricata socket suricata-command.socket has the rights
rw-r----- suricata:user.
When suricata is used with restricted access, an other application
(suricatasc like) that needs to access to the command socket also
with restricted access can not write to the socket since it is not
the owner (e.g suricata within container, with an hardened value
for umask and hardened rights for users).
The socket should be set as rw-rw----. Use chmod instead of fchmod
and set it after the socket creation.
https://redmine.openinfosecfoundation.org/issues/2412
Suricatasc is not supporting pcap-file processing in continuous mode.
Register a new command pcap-file-continuous in the unix manager to work
with suricatasc. Add defaulted arguments for pcap-file to support
backwards compatibility.
RAND_MAX is not guaranteed to be a divisor of ULONG_MAX, so take the
necessary precautions to get unbiased random numbers. Although the
bias might be negligible, it's not advisable to rely on it.
according to its man page, sigprocmask has undefined behavior in
multithreaded environments. Instead of explictly blocking the handling
of SIGUSR2 in every thread, direct block handling SIGUSR2 before
creating the threads and enable again the handling of this signal
afterwards. In this way, only the main thread will be able to manage
this signal properly.
Add startswith modifier to simplify matching patterns at the start
of a buffer.
Instead of:
content:"abc"; depth:3;
This enables:
content:"abc"; startswith;
Especially with longer patterns this makes the intention of the rule
more clear and eases writing the rules.
Internally it's simply a shorthand for 'depth:<pattern len>;'.
Ticket https://redmine.openinfosecfoundation.org/issues/742
Fix the inspection of multiple files in a single TX, where new files
may be added to the TX after inspection started.
Assign the hard coded id DE_STATE_FLAG_FILE_INSPECT to the file
inspect engine.
Make sure that sigs that do file inspection and don't match on the
current file always store a detailed state. This state will include
the DE_STATE_FLAG_FILE_INSPECT flag.
When the app-layer indicates a new file is available, for each sig
that has the DE_STATE_FLAG_FILE_INSPECT flag set, reset part of the
state so that the sig is evaluated again.
Free txs that are done out of order if we can. Some protocol
implementations have transactions running in parallel, where it is
possible that a tx that started later finishes earlier than other
transactions. Support freeing those.
Also improve handling on asynchronious transactions. If transactions
are unreplied, e.g. in the dns flood case, the parser may at some
point free transactions on it's own. Handle this case in
the app-layer engine so that the various tracking id's (inspect, log,
and 'min') are updated accordingly.
Next, free txs much more aggressively. Instead of freeing old txs
at the app-layer parsing stage, free all complete txs at the end
of the flow-worker. This frees txs much sooner in many cases.
When stateless rules are depending on a flowbit being set by a stateful
rule, the inspection order is almost certainly wrong.
Switch stateless rules depending on stateful rules to being stateful.
This is used to turn 'TCP stream' inspecting rules (which are stateless
unless mixed with stateful keywords) into stateful rules.
Use per tx detect_flags to track prefilter. Detect flags are used for 2
things:
1. marking tx as fully inspected
2. tracking already run prefilter (incl mpm) engines
This supercedes the MpmIDs API for directionless tracking
of the prefilter engines.
When we have no SGH we have to flag the txs that are 'complete'
as inspected as well.
Special handling for the stream engine:
If a rule mixes TX inspection and STREAM inspection, we can encounter
the case where the rule is evaluated against multiple transactions
during a single inspection run. As the stream data is exactly the same
for each of those runs, it's wasteful to rerun inspection of the stream
portion of the rule.
This patch enables caching of the stream 'inspect engine' result in
the local 'RuleMatchCandidateTx' array. This is valid only during the
live of a single inspection run.
Remove stateful inspection from 'mask' (SignatureMask). The mask wasn't
used in most cases for those rules anyway, as there we rely on the
prefilter. Add a alproto check to catch the remaining cases.
When building the active non-mpm/non-prefilter list check not just
the mask, but also the alproto. This especially helps stateful rules
with negated mpm.
Simplify AppLayerParserHasDecoderEvents usage in detection to only
return true if protocol detection events are set. Other detection is done
in inspect engines.
Move rule group lookup and handling into it's own function. Handle
'post lookup' tasks immediately, instead of after the first detect
run. The tasks were independent of the initial detection.
Many cleanups and much refactoring.
Add API meant to replace the MpmIDs API. It uses a u64 for each direction
in a tx to keep track of 2 things:
1. is inspection done?
2. which prefilter engines (like mpm) are already completed
Analyze flowbits to find which bits are only checked.
Track whether they are set and checked on the same level of 'statefulness'
for later used.
Dump flowbits to json including the sids that set/check etc the bit.
There is probably not too much bad about enabling both, but
open file counts can get messy with both enabled. And v1
should be schedule for deprecation soon enough.
This doesn't need to be registered from suricata.c. And moving
it to the init function makes sure its only registered if
the logger is actually enabled.
Track each type of error warning and only log it once. Also create
a new stat, file_store.fs_errors to count each file system type
error (open, rename, unlink).
Also remove exit stats, they are of limited value.
Use a new directory, Python to host the Suricata python modules.
One entry point is suricatactl, a control script for
miscalleneous tasks. Currently onl filestore pruning
is implemented.
As fileinfo records are logged to the main eve log, disable
metadata by default. But when enabled, just use the fileinfo
record.
Metadata is stored in a file named:
<sha256>.<seconds>.<file_id>.json
where the sha256 is the same as the file logged, the seconds
is the unix timestamp in seconds for the fileinfo record,
and the file_id is an atomically incremented integer per
Suricata instance.
This should allow for each occurrence of the same file to have
its own metadata file. But a collision is expected when running
Suricata repeatedly over the same pcap, as that would be the
exact same occurrence of a file.
Even if a file is truncated, force the SHA256 if force sha256
is set to yes.
The new file store requires the sha256 regardless of the file
state if it is to be logged, as the filename is based on the
sha256.
Filestore v2 is starts as a copy of log-filestore with the
following changes.
- NSS is required as file names as based on the SHA256.
- Work/tmp files are stored in a temp. directory, then
moved into a directory tree where the directory names
are the first 2 characters of the hex SHA256.
- Removes the need for a waldo file or pid in the filenames.
Split the building of the fileinfo record from the writing
of the record so the building can be called from other code.
Specifically the new filestore output which uses fileinfo
records as the metadata.
Alexander Gozman
Danny Browning
Pascal Delalande
Martin Natano
Victor Julien
Maurizio Abba
Ruslan Usmanov
Jason Ish