Coccinelle test was doing a false positive on the function
AppLayerParserStateSetFlag and AppLayerParserStateIssetFlag.
To address that, this patch adds a new coccinelle markup:
/* coccinelle: AppLayerParserStateSetFlag():2,2:APP_LAYER_PARSER_ */
It indicates that AppLayerParserStateSetFlag is a setter and getter
and that the checks should be disabled inside the function.
Currently this markup is only used for that but following patch will
add some checks on option value.
Add method to check if a parser for an app-layer protocol
supports tx detect flags.
This is a bit of a hack for now as where we need to run
this check from we do not have the IP protocol.
If a protocol does not support TxDetectFlags, don't try to use them.
The consequence of trying to use them was that a TX would never be
considered done, and it would never be freed. This would lead to excessive
memory use and performance problems due to walking an ever increasing
list.
Extend the Rust parsing infrastructure with the "get event info by id"
calls. This changeset extends the parser structure, the C-based
registration handlers and the template parser.
This changeset makes changes to the TX logging path. Since the txn
is passed to the TX logger, the TX can be used directly instead of
through the TX id.
The app layers with a custom iterator would skip a tx if during
the ..Cleanup() pass a transaction was removed.
Address this by storing the current index instead of the next
index. Also pass in the next "min_tx_id" to be incremented from
the last TX. Update loops to do this increment.
Also make sure that the min_id is properly updated if the last
TX is removed when out of order.
Finally add a SMB unittest to test this.
Reported by: Ilya Bakhtin
When an app-layer parser is enabled, it could set its
own stream_depth value calling the API AppLayerParserSetStreamDepth.
Then, the function AppLayerParserPostStreamSetup will replace
the stream_depth value already set with stream_config.reassembly_depth.
To avoid overwriting, in AppLayerParserSetStreamDepth API a flag
will be set internally to specify that a value is already set.
This is a DHCP decoder and logger written in Rust. Unlike most
parsers, this one is stateless so responses are not matched
up to requests by Suricata. However, the output does contain
enough fields to match them up in post-processing.
Rules are included to alert of malformed or truncated options.
Add a new parser for Internet Key Exchange version (IKEv2), defined in
RFC 7296.
The IKEv2 parser itself is external. The embedded code includes the
parser state and associated variables, the state machine, and the
detection code.
The parser looks the first two messages of a connection, and analyzes
the client and server proposals to check the cryptographic parameters.
Also remove the now useless 'state' argument from the SetTxDetectState
calls. For those app-layer parsers that use a state == tx approach,
the state pointer is passed as tx.
Update app-layer parsers to remove the unused call and update the
modified call.
Until now, the transaction space is assumed to be terse. Transactions
are handled sequentially so the difference between the lowest and highest
active tx id's is small. For this reason the logic of walking every id
between the 'minimum' and max id made sense. The space might look like:
[..........TTTT]
Here the looping starts at the first T and loops 4 times.
This assumption isn't a great fit though. A protocol like NFS has 2 types
of transactions. Long running file transfer transactions and short lived
request/reply pairs are causing the id space to be sparse. This leads to
a lot of unnecessary looping in various parts of the engine, but most
prominently: detection, tx house keeping and tx logging.
[.T..T...TTTT.T]
Here the looping starts at the first T and loops for every spot, even
those where no tx exists anymore.
Cases have been observed where the lowest tx id was 2 and the highest
was 50k. This lead to a lot of unnecessary looping.
This patch add an alternative approach. It allows a protocol to register
an iterator function, that simply returns the next transaction until
all transactions are returned. To do this it uses a bit of state the
caller must keep.
The registration is optional. If no iterator is registered the old
behaviour will be used.
TFTP parsing and logging written in Rust.
Log on eve.json the type of request (read or write), the name of the file and
the mode.
Example of output:
"tftp":{"packet":"read","file":"rfc1350.txt","mode":"octet"}
Free txs that are done out of order if we can. Some protocol
implementations have transactions running in parallel, where it is
possible that a tx that started later finishes earlier than other
transactions. Support freeing those.
Also improve handling on asynchronious transactions. If transactions
are unreplied, e.g. in the dns flood case, the parser may at some
point free transactions on it's own. Handle this case in
the app-layer engine so that the various tracking id's (inspect, log,
and 'min') are updated accordingly.
Next, free txs much more aggressively. Instead of freeing old txs
at the app-layer parsing stage, free all complete txs at the end
of the flow-worker. This frees txs much sooner in many cases.
Use per tx detect_flags to track prefilter. Detect flags are used for 2
things:
1. marking tx as fully inspected
2. tracking already run prefilter (incl mpm) engines
This supercedes the MpmIDs API for directionless tracking
of the prefilter engines.
When we have no SGH we have to flag the txs that are 'complete'
as inspected as well.
Special handling for the stream engine:
If a rule mixes TX inspection and STREAM inspection, we can encounter
the case where the rule is evaluated against multiple transactions
during a single inspection run. As the stream data is exactly the same
for each of those runs, it's wasteful to rerun inspection of the stream
portion of the rule.
This patch enables caching of the stream 'inspect engine' result in
the local 'RuleMatchCandidateTx' array. This is valid only during the
live of a single inspection run.
Remove stateful inspection from 'mask' (SignatureMask). The mask wasn't
used in most cases for those rules anyway, as there we rely on the
prefilter. Add a alproto check to catch the remaining cases.
When building the active non-mpm/non-prefilter list check not just
the mask, but also the alproto. This especially helps stateful rules
with negated mpm.
Simplify AppLayerParserHasDecoderEvents usage in detection to only
return true if protocol detection events are set. Other detection is done
in inspect engines.
Move rule group lookup and handling into it's own function. Handle
'post lookup' tasks immediately, instead of after the first detect
run. The tasks were independent of the initial detection.
Many cleanups and much refactoring.
Add API meant to replace the MpmIDs API. It uses a u64 for each direction
in a tx to keep track of 2 things:
1. is inspection done?
2. which prefilter engines (like mpm) are already completed
Avoid looping in transaction output.
Update app-layer API to store the bits in one step
and retrieve the bits in a single step as well.
Update users of the API.
Create a bitmap of the loggers per protocol. This is done at runtime
based on the loggers that are enabled. Take the logger_id for each
logger and store it as a bitmap in the app-layer protcol storage.
Goal is to be able to use it as an expectation later.
Victor Julien
Eric Leblond
Jason Ish
Konstantin Klinger
Giuseppe Longo
Zach Kelly
Jeff Lucovsky
jason taylor
Pierre Chifflier
Jason Ish
Giuseppe Longo
Clement Galland