and bindgen it to rust
Will make easier the bindgen of RustParser structure which uses
a callback which uses AppLayerTxData
Move also the free function to C SCAppLayerTxDataCleanup
As suricata-sys crate defines AppLayerTxData for rust,
It must itself implement the Drop trait, and thus,
We need to define a feature surest
While debug_validate_bug_on is still used, it does not need to be
imported directly, as that macro is marked with `macro_export`, making
it globally available to the crate.
The functions around TriggerRawStreamReassembly are misnomers in the
current layout of the code. The functions were named appropriately when
they were created as per the structural and logical layout of the code
at the time.
These functions in today's code are being used to track, update and
trigger progress of inspection in the raw stream only. Hence, rename them
to TriggerRawStreamInspection.
Internals
---------
Suricata's stream engine returns data for inspection to the detection
engine from the stream when the chunk size is reached.
Bug
---
Inspection triggered only in the specified chunk sizes may be too late
when it comes to inspection of smaller protocol specific data which
could result in delayed inspection, incorrect data logged with a transaction
and logs misindicating the pkt that triggered an alert.
Fix
---
Fix this by making an explicit call from all respective applayer parsers to
trigger raw stream reassembly which shall make the data available for inspection
in the following call of the stream engine. This needs to happen per direction
on the completion of an entity like a request or a response.
Important notes
---------------
1. The above mentioned behavior with and without this patch is
affected internally by the following conditions.
- inspection depth
- stream depth
In these special cases, the inspection window will be affected and
Suricata may not consider all the data that could be expected to be
inspected.
2. This only applies to applayer protocols running over TCP.
3. The inspection window is only considered up to the ACK'd data.
4. This entire issue is about IDS mode only.
Modbus has a classic request response model, so, a call to trigger raw
stream reassembly is added on completion of each request and response.
Optimization 7026
Bug 7004
Have bindgen generate bindings for app-layer-protos.h, then use the
generated definitions of AppProto/AppProtoEnum instead if defining
them ourselves.
This header was chosen as its used by Rust, and its a simple header
with no circular dependencies.
Ticket: #7341
Because some alprotos will remain static and defined as a constant,
such as ALPROTO_UNKNOWN=0, or ALPROTO_FAILED.
The regular already used protocols keep for now their static
identifier such as ALPROTO_SNMP, but this could be made more
dynamic in a later commit.
ALPROTO_FAILED was used in comparison and these needed to change to use
either ALPROTO_MAX or use standard function AppProtoIsValid
Allow `set_uint` to accept any number value that can be converted to a
u64. Prevents callers from having to do `as u64`.
This required fixing up any callers that used `.into()` to just pass in
their value without the into conversion.
Most calls using `as u64` can have that cast removed, with the exception
of `usize` values which must still be cast is conversion can't be
guaranteed to be non-fallible.
To optimize detection, and logging, to avoid going through
all the live transactions when only a few were modified.
Two boolean fields are added to the tx data: updated_tc and ts
The app-layer parsers are now responsible to set these when
needed, and the logging and detection uses them to skip
transactions that were not updated.
There may some more optimization remaining by when we set
both updated_tc and updated_ts in functions returning
a mutable transaction, by checking if all the callers
are called in one direction only (request or response)
Ticket: 7087
truncate fn is only active and used by dcerpc and smb parsers. In case
stream depth is reached for any side, truncate fn is supposed to set the
tx entity (request/response) in the same direction as complete so the
other side is not forever waiting for data.
However, whether the stream depth is reached is already checked by
AppLayerParserGetStateProgress fn which is called by:
- DetectTx
- DetectEngineInspectBufferGeneric
- AppLayerParserSetTransactionInspectId
- OutputTxLog
- AppLayerParserTransactionsCleanup
and, in such a case, StateGetProgressCompletionStatus is returned for
the respective direction. This fn following efc9a7a, always returns 1
as long as the direction is valid meaning that the progress for the
current direction is marked complete. So, there is no need for the additional
callback to mark the entities as done in case of depth or a gap.
Remove all such glue code and callbacks for truncate fns.
Bug 7044
Ticket: 7013
Done consistently for all protocols
This may change some protocols behaviors which failed early
if they found there was not enough data...
sawp 0.12 is available and addresses future compilation failures in
dependent crates.
Updated modbus test case to expect 12 bytes needed instead of 15. This
aligns with expectations as the test case slices 3 bytes off the end of
a 12 byte message so needing 12 bytes is correct.
Ticket #5989
Update APIs to store files in transactions instead of the per flow state.
Goal is to avoid the overhead of matching up files and transactions in
cases where there are many of both.
Update all protocol implementations to support this.
Update file logging logic to account for having files in transactions. Instead
of it acting separately on file containers, it is now tied into the
transaction logging.
Update the filestore keyword to consider a match if filestore output not
enabled.
Instead of a method that is required to return a slice of transactions,
use 2 methods, one to return the number of transactions in the
collection, and another to get a transaction by its index in the
collection.
This allows for the transaction collection to not be a contiguous array
and instead can be a VecDeque, or possibly another collection type that
supports retrieval by index.
Ticket #5278
Every transaction has an existing mandatory field, tx_data. As
DetectEngineState is also mandatory, include it in tx_data.
This allows us to remove the boilerplate every app-layer has
for managing detect engine state.
When there are a lot of open transactions, as is possible with
modbus, the default tx_iterator will loop for the whole
transacations vector to find each transaction, that means
quadratic complexity.
Reusing the tx_iterator from the template, and keeping as a state
the last index where to start looking avoids this quadratic
complexity.
DNP3, ENIP, HTTP2 and Modbus are supposed to be disabled
by default. That means the default configuration does it,
but that also means that, if they are not in suricata.yaml,
the protocol should stay disabled.
Based on the Rust clippy lint that recommends that any public
function that dereferences a raw pointer, mark all FFI functions
that reference raw pointers with build_slice and cast_pointer
as unsafe.
This commits starts by removing the unsafe wrapper inside
the build_slice and cast_pointer macros then marks all
functions that use these macros as unsafe.
Then fix all not_unsafe_ptr_arg_deref warnings from clippy.
Fixes clippy lint:
https://rust-lang.github.io/rust-clippy/master/index.html#not_unsafe_ptr_arg_deref
Philippe Antoine
Jason Ish
Shivani Bhardwaj
Victor Julien
Philippe Antoine
Daniel Olatunji
William Correia
Victor Julien