Every transaction has an existing mandatory field, tx_data. As
DetectEngineState is also mandatory, include it in tx_data.
This allows us to remove the boilerplate every app-layer has
for managing detect engine state.
The stream depth setting was broken since it was moved to Rust because
of a missing parser for memory values in configuration.
Use get_memval fn from conf.rs to calculate and fetch the correct
values.
Calling a function in unwrap_or causes that function to always
be called even when not needed. Instead use unwrap_or_else with
a closure which will only be called when needed.
Based on the Rust clippy lint that recommends that any public
function that dereferences a raw pointer, mark all FFI functions
that reference raw pointers with build_slice and cast_pointer
as unsafe.
This commits starts by removing the unsafe wrapper inside
the build_slice and cast_pointer macros then marks all
functions that use these macros as unsafe.
Then fix all not_unsafe_ptr_arg_deref warnings from clippy.
Fixes clippy lint:
https://rust-lang.github.io/rust-clippy/master/index.html#not_unsafe_ptr_arg_deref
All cases of our transmute can be replaced with more idiomatic
solutions and do no require the power of transmute.
When returning an object to C for life-time management, use
Box::into_raw to convert the boxed object to pointer and use
Box::from_raw to convert back.
For cases where we're just returning a pointer to Rust managed
data, use a cast.
Before, even if there were no outputs, all the arguments
were evaluated, which could turn expensive
All variables which are used only in certain build configurations
are now prefixed by underscore to avoid warnings
Since the completion status was a constant for all parsers, remove the
callback logic and instead register the values themselves. This should
avoid a lot of unnecessary callback calls.
Update all parsers to take advantage of this.
Scenario is use of dummy padding in write AndX request
or other similar commands using a data offset.
Parsing skips now these dummy bytes, and generates one event
Evasion scenario is
- a first dummy write of one byte at offset 0 is done
- the second full write of EICAR at offset 0 is then done
and does not trigger detection
The last write had the final value, and as we cannot "cancel"
the previous write, we set an event which is then transformed into
an app-layer decoder alert
This parameter is NULL or the pointer to the previous state
for the previous protocol in the case of a protocol change,
for instance from HTTP1 to HTTP2
This way, the new protocol can use the old protocol context.
For instance, HTTP2 mimicks the HTTP1 request, to have a HTTP2
transaction with both request and response
In case of lossy connections the SMB state would properly clean up
transactions, including file transactions. However for files the
state was never set to 'truncated', leading to files to stay 'active'.
This would lead these files staying in the SMB's state. In long running
sessions with lots of files this would lead to performance and memory
use issues.
This patch cleans truncates the file that was being transmitted when
a file transaction is being closed.
All the dead code in C after the Rust implementation is hereby removed.
Invalid/migrated tests have also been deleted.
All the function calls in C have been replaced with appropriate calls to
Rust functions. Same has been done for smb/detect.rs as a part of this
migration.
This patch simplifies the return codes app-layer parsers use,
in preparation of a patch set for overhauling the return type.
Introduce two macros:
APP_LAYER_OK (value 0)
APP_LAYER_ERROR (value -1)
Update all parsers to use this.
Unfortunately, the transition to nom 5 (and functions instead of macros)
has side-effects, one of them being requiring lots of types annotations
when using a parsing, for ex in a match instruction.
After a GAP all normal transactions are closed. File transactions
are left open as they can handle GAPs in principle. However, the
GAP might have contained the closing of a file and therefore it
may remain active until the end of the flow.
This patch introduces a time based heuristic for these transactions.
After the GAP all file transactions are stamped with the current
timestamp. If 60 seconds later a file has seen no update, its marked
as closed.
This is meant to fix resource starvation issues observed in long
running SMB sessions where packet loss was causing GAPs.
Rust 1.40 in strict mode will now fail the build on the
presence of unnecessary parentheses.
warning: unnecessary parentheses around type
--> src/smb/smb2_ioctl.rs:41:12
|
41 | -> (&mut SMBTransaction)
| ^^^^^^^^^^^^^^^^^^^^^ help: remove these parentheses
|
= note: `#[warn(unused_parens)]` on by default
This changeset makes changes to the TX logging path. Since the txn
is passed to the TX logger, the TX can be used directly instead of
through the TX id.
When Suricata picks up a flow it assumes the first packet is
toserver. In a perfect world without packet loss and where all
sessions neatly start after Suricata itself started, this would be
true. However, in reality we have to account for packet loss and
Suricata starting to get packets for flows already active be for
Suricata is (re)started.
The protocol records on the wire would often be able to tell us more
though. For example in SMB1 and SMB2 records there is a flag that
indicates whether the record is a request or a response. This patch
is enabling the procotol detection engine to utilize this information
to 'reverse' the flow.
There are three ways in which this is supported in this patch:
1. patterns for detection are registered per direction. If the proto
was not recognized in the traffic direction, and midstream is
enabled, the pattern set for the opposing direction is also
evaluated. If that matches, the flow is considered to be in the
wrong direction and is reversed.
2. probing parsers now have a way to feed back their understanding
of the flow direction. They are now passed the direction as
Suricata sees the traffic when calling the probing parsers. The
parser can then see if its own observation matches that, and
pass back it's own view to the caller.
3. a new pattern + probing parser set up: probing parsers can now
be registered with a pattern, so that when the pattern matches
the probing parser is called as well. The probing parser can
then provide the protocol detection engine with the direction
of the traffic.
The process of reversing takes a multi step approach as well:
a. reverse the current packets direction
b. reverse most of the flows direction sensitive flags
c. tag the flow as 'reversed'. This is because the 5 tuple is
*not* reversed, since it is immutable after the flows creation.
Most of the currently registered parsers benefit already:
- HTTP/SMTP/FTP/TLS patterns are registered per direction already
so they will benefit from the pattern midstream logic in (1)
above.
- the Rust based SMB parser uses a mix of pattern + probing parser
as described in (3) above.
- the NFS detection is purely done by probing parser and is updated
to consider the direction in that parser.
Other protocols, such as DNS, are still to do.
Ticket: #2572
Only use ssn_id and msg_id for mapping a response to a request.
By not using the tree_id it can always be included in the tx.hdr which
means it can be logged properly in case of IOCTL and DCERPC.
The app layers with a custom iterator would skip a tx if during
the ..Cleanup() pass a transaction was removed.
Address this by storing the current index instead of the next
index. Also pass in the next "min_tx_id" to be incremented from
the last TX. Update loops to do this increment.
Also make sure that the min_id is properly updated if the last
TX is removed when out of order.
Finally add a SMB unittest to test this.
Reported by: Ilya Bakhtin
If we missed the tree connect we can't know for sure if we're
reading from a (DCERPC) PIPE or not. In this case probe the data
to see if it looks like DCERPC.
If the detection succeeds, use a special 'suricata::dcerpc' service
in the TX.
Simplify handling of DCERPC records that cross records
Update logging for the response only TXs.
Calls with both START and MIDSTREAM mean the record might be cut and the
start of it could be missing. For this case, enable the same logic as is
used when catching up after a GAP. Search for the start of the record
instead of assuming it sits exactly at the start of the input data.
When skipping records the skip tracker could underflow if the record
parsing had more data than expected.
Enforce the calculation by moving it into a method and make the actual
fields private.
parse_smb2_response_read()/parse_smb2_response_write() can be called on
incomplete data, so they didn't use the read/write length field to grab
the data field. Instead it just used rest(). However in some cases
SMB2 records have trailing data, which would be included in the
READ/WRITE data.
This patch addresses this by using the length field if enough data is
available.