This commit adds keyword/build support for the entropy keyword. The
entropy keyword compares an entropy value with a value calculated
according to the Shannon entropy on the available content.
Issue: 4162
This commit adds
- Parser for the entropy keyword
- Calculation of content the Shannon entropy value
Issue: 4162
The entropy keyword syntax is the keyword entropy followed by options
and the entropy value for comparison.
The minimum entropy keyword specification is:
entropy: value <entropy-spec>
This results in the calculated entropy value being compared with
<entropy-spec> with the equality operator.
Calculated entropy values are between 0.0 and 8.0, inclusive.
A match occurs when the values and operator agree. This example matches
if the calculated and entropy value are the same.
When entropy keyword options are specified, all options and "value" must
be comma-separated. Options and value may be specified in any order.
Options have default values:
- bytes is equal to the current content length
- offset is 0
- comparison with value is equality
entropy: [bytes <byteval>] [offset <offsetval>] value <entropy-spec>
Using default values:
entropy: bytes 0, offset 0, value =<entropy-spec>
<entropy-spec> is: <operator> (see below) and a value, e.g., "< 4.1"
The following operators are available from the float crate:
- = (default): Match when calculated entropy value equals specified entropy value
- < Match when calculated entropy value is strictly less than specified entropy value
- <= Match when calculated entropy value is less than or equal to specified entropy value
- > Match when calculated entropy value is strictly greater than specified entropy value
- >= Match when calculated entropy value is greater than or equal to specified entropy value
- != Match when calculated entropy value is not equal to specified entropy value
- x-y Match when calculated entropy value is in the range, exclusive
- !x-y Match when calculated entropy value is not in the range, exclusive
Reduce per tx space for tracking detection/prefilter progress. Instead
of a per direction u64 of flags, where each bit reflected a progress
value, use a simple u8 to track the linear progression through the
progress values. Use an offset to allow 0 to mean no value.
Add flags field as well to track "skip detect" and "inspect complete".
Deranged v0.4.1 (a dependency of the time crate) has implemented
PartialOrd for some integer types that conflict with the
implementation in the standard library creating an ambiguity as such
implementation are global. For more info see
https://github.com/jhpratt/deranged/issues/18.
To fix, use "::from" directly, instead of using .into() which is where
we run into amgibuity.
Notable changes from the previous API:
- rcode will return the rcode as an integer
- rcode_string will return the string representation
Also fixes an issue where an rcode of 0 was returned as nil.
Ticket: #7602
Feature: 7012
Add dns.response sticky buffer to match on dns response fields.
Add rust functions to return dns response packet data.
Unit tests verifying signature matching.
ldap.responses.message matches on LDAPResult error message
This keyword maps the following eve fields:
ldap.responses[].bind_response.message
ldap.responses[].search_result_done.message
ldap.responses[].modify_response.message
ldap.responses[].add_response.message
ldap.responses[].del_response.message
ldap.responses[].mod_dn_response.message
ldap.responses[].compare_response.message
ldap.responses[].extended_response.message
It is a sticky buffer
Supports prefiltering
Ticket: #7532
ldap.responses.result_code matches on LDAP result code
This keyword maps the following eve fields:
ldap.responses[].bind_response.result_code
ldap.responses[].search_result_done.result_code
ldap.responses[].modify_response.result_code
ldap.responses[].add_response.result_code
ldap.responses[].del_response.result_code
ldap.responses[].mod_dn_response.result_code
ldap.responses[].compare_response.result_code
ldap.responses[].extended_response.result_code
It is an unsigned 32-bit integer
Doesn't support prefiltering
Ticket: #7532
Funciton ldap_tx_get_responses_dn returns empty buffer in case
the response doesn't contain the distinguished name field
Fixes: 73ae6e997f ("detect: add ldap.responses.dn")
Change variable name 'req' to 'resp' in function ldap_tx_get_responses_dn and documentation nits
Fixes:
73ae6e997f ("detect: add ldap.responses.dn")
16dcee46fc ("detect: add ldap.request.dn")
- remove rs_prefix, replace with SC if needed
- remove pub and no_mangle where not needed
- remove some unused functions and fields
Related to ticket: #7498
This uses the cbindgen found during ./configure, and not the one
found on the path during "make", which while often the same, aren't
always the same.
Ticket: #6384
As references to static mutables are highly discouraged, remove the
global suppressing of the compiler warning. Each use case can be
suppressed as needed.
Ticket: #7417
It doesn't appear to be needed. The vec being cleared is only set once
per run, so never needs to be cleared.
Removes one point where we have to supress the static_mut_refs compiler
warning.
Ticket: #7417
Issue: 4082
Move the configuration file handling to Rust.
These changes will no longer terminate Suricata when there's an invalid
value for ftp.memcap. Like earlier Suricata releases, an error message
is logged "Invalid value <value> for ftp.memcap" but Suricata will no
longer terminate execution. It will use a default value of "0" instead.
Ticket: 7567
After a gap, we search a new record that may start later than
the beginning of current stream slice.
If so, consume the first bytes before the start of the record,
so that AppLayerResult::incomplete can be consistent and not
trigger assertion !((res.needed + res.consumed < input_len))
Ticket: 7556
See RFC 9000 section 17.2.5.2 :
After the client has received and processed an Initial
or Retry packet from the server,
it MUST discard any subsequent Retry packets that it receives.
ldap.responses.dn matches on LDAPDN from responses operations
This keyword maps the following eve fields:
ldap.responses[].search_result_entry.base_object
ldap.responses[].bind_response.matched_dn
ldap.responses[].search_result_done.matched_dn
ldap.responses[].modify_response.matched_dn
ldap.responses[].add_response.matched_dn
ldap.responses[].del_response.matched_dn
ldap.responses[].mod_dn_response.matched_dn
ldap.responses[].compare_response.matched_dn
ldap.responses[].extended_response.matched_dn
It is a sticky buffer
Supports prefiltering
Ticket: #7471
ldap.request.dn matches on LDAPDN from request operations
This keyword maps the following eve fields:
ldap.request.bind_request.name
ldap.request.add_request.entry
ldap.request.search_request.base_object
ldap.request.modify_request.object
ldap.request.del_request.dn
ldap.request.mod_dn_request.entry
ldap.request.compare_request.entry
It is a sticky buffer
Supports prefiltering
Ticket: #7471
Ticket: 7556
To do so, we need to add 2 buffers (one for each direction)
to the QuicState structure, so that on parsing the second packet
with hello/crypto fragment, we still have the data of the first
hello/crypto fragment.
Use a hardcoded limit so that these buffers cannot grow indefinitely
and set an event when reaching the limit
This may happen in some situations if the app-layer parser only sees
unknown messages and sets an event: there will be an empty transaction,
but nothing to log.
Related to
Task #5566
No state change, but since we added Unknown responses, we should handle
that case -- should we have a specific state for such cases?
Related to
Bug #5524
Task #5566
Some inner parsers were using it, some weren't. Better to standardize
this. Also take the time to avoid magic numbers for representing the
expected lengths for pgsql PDUs.
Also throwing PgsqlParseError and allowing for incomplete results.
Related to
Task #5566
Bug #5524
Some backend messages can be the shortest pgsql length possible,
4 bytes, but the parser expectd all messages to be longer than that.
Related to
Bug #5524
Also disable bindgen's generated layout tests. They are valid for the
platform generating the tests, but may not be valid for other
platforms. For example, if the tests are generated on a 64 bit
platform the tests will not be valid when run on a 32 bit platform as
pointers are a different size.
However, the generating bindings are valid for both platform.
Ticket: #7341
We don't keep bindgen's autogenerated do not edit line as it contains
the bindgen version which could break the CI check for out of date
bindings. So add our own do not edit line.
Ticket: #7341
Have bindgen generate bindings for app-layer-protos.h, then use the
generated definitions of AppProto/AppProtoEnum instead if defining
them ourselves.
This header was chosen as its used by Rust, and its a simple header
with no circular dependencies.
Ticket: #7341
Bindgen works by processing a header file which includes all other
header files it should generate bindings for. For this I've created
bindgen.h which just includes app-layer-protos.h for now as an
example.
These bindings are then generated and saved in the "suricata-sys"
crate and become availale as "suricata_sys::sys".
Ticket: #7341
Follow Rust convention of using a "sys" crate for bindings to C
functions. The bindings don't exist yet, but will be generated by
bindgen and put into this crate.
Ticket: #7341
In a recent warning reported by scan-build, datasets were found to be
using a blocking call in a critical section.
datasets.c:187:12: warning: Call to blocking function 'fgets' inside of critical section [unix.BlockInCriticalSection]
187 | while (fgets(line, (int)sizeof(line), fp) != NULL) {
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
datasets.c:292:12: warning: Call to blocking function 'fgets' inside of critical section [unix.BlockInCriticalSection]
292 | while (fgets(line, (int)sizeof(line), fp) != NULL) {
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
datasets.c:368:12: warning: Call to blocking function 'fgets' inside of critical section [unix.BlockInCriticalSection]
368 | while (fgets(line, (int)sizeof(line), fp) != NULL) {
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
datasets.c:442:12: warning: Call to blocking function 'fgets' inside of critical section [unix.BlockInCriticalSection]
442 | while (fgets(line, (int)sizeof(line), fp) != NULL) {
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
datasets.c:512:12: warning: Call to blocking function 'fgets' inside of critical section [unix.BlockInCriticalSection]
512 | while (fgets(line, (int)sizeof(line), fp) != NULL) {
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
5 warnings generated.
These calls are blocking in the multi tenant mode where several tenants
may be trying to load the same dataset in parallel.
In a single tenant mode, this operation is performed as a part of a
single thread before the engine startup.
In order to evade the warning and simplify the code, the initial file
reading is moved to Rust with this commit with a much simpler handling
of dataset and datarep.
Bug 7398
A no padding option is provided as a mode, as its a variant suitable
for encoding and decoding.
A padding optional function is added that is indifferent to padding
when decoding. This can be useful when you're not sure if padding
exists, and don't really care.
Frames of the following types have been added for toserver direction:
1. Pdu: The entire Protocol Data Unit
2. Hdr: Header of the request
3. Data: PDU data
Feature 4904
With the introduction of AppLayerResult::incomplete API, fragmented data
is no longer handled fully in the dcerpc code. Given that these code
paths are already covered by the following s-v tests, these tests can now be
safely removed.
- dce-gap-handling
- dcerpc-dce-iface-*
Ticket 5699
Instead of own internal mechanism of buffering in case of fragmented
data, use AppLayerResult::incomplete API to let the AppLayer Parser take
care of it. This makes the memory use more efficient.
Remove any unneeded variables and code with the introduction of this
API.
Ticket 5699
TCP data can be presented to the protocol parser in any way e.g. one
byte at a time, single complete PDU, fragmented PDU, multiple PDUs at
once. A limit of 1MB can be easily reached in some of such scenarios.
Remove the check that rejects data that is more than 1MB.
Jason Ish
Jeff Lucovsky
Alice Akaki
Victor Julien
Philippe Antoine
Juliana Fajardini
Bryan Benson
Nathan Scrivens
Shivani Bhardwaj
Juliana Fajardini
Ilya Bakhtin