Commit Graph

17462 Commits (02bdea2bce46b7833133f5e2c2dfa34630fe047c)
 

Author SHA1 Message Date
Lukas Sismis e780a20f82 doc: update available options in the example config 2 months ago
Lukas Sismis 58df970391 flow-manager: move time check after RUNNNING state change
When running in pcap-file mode and with a continous directory
reading, the provided directory can be empty.
By having no packets and being in offline mode, the initial packet timestamp
is never set. However, Flow Manager waited until the timestamp was set
to proceed with transferring its state to "RUNNING".
Other pcap-related threads (RX / workers) are set in RUNNING state while
waiting for the PCAP to appear in the directory.

As a result, the main Suricata-Main thread timed out after the default
60 seconds budget for threads to turn from INIT_DONE to RUNNING state.

Ticket: 7568
2 months ago
Philippe Antoine b8e4ba6e3e files: remove legacy option force-md5
The modern option is force-hash: md5

Ticket: 7353
2 months ago
Philippe Antoine a2b5225612 rust: bindgen SCSigMatchAppendSMToList
Ticket: 7667
2 months ago
Philippe Antoine 78034b218d rust: bindgen SCDetectSignatureSetAppProto
Ticket: 7667
2 months ago
Philippe Antoine b1ef498a81 rust: bindgen detect-parse.h
Ticket: 7667

Currently no functions are exported.

DetectFile* struct are moved to detect-file-data.h where
they make more sense.

ifndef SURICATA_BINDGEN_H is used for bindgen to exclude
pcre2 related code
2 months ago
Philippe Antoine 465f94fafb detect/tls: fix -Wshorten-64-to-32 warnings
Ticket: #6186
2 months ago
Philippe Antoine 3c743529b7 detect/frames: fix -Wshorten-64-to-32 warnings
Ticket: #6186
2 months ago
Philippe Antoine 6eed2f1d7a detect/transforms: fix -Wshorten-64-to-32 warnings
Ticket: #6186
2 months ago
Philippe Antoine 148f0ea775 detect/ftp: fix -Wshorten-64-to-32 warnings
Ticket: #6186
2 months ago
Jeff Lucovsky 1a13244b4b doc/tls: Update Lua TLS functions
Issue: 7608

Update the documentation to reflect the new and expanded functions
available form the Lua TLS library

There are now "server" and "client" versions of most functions. The TLS
object getter is now "get_tx"
2 months ago
Jeff Lucovsky 35bda27bd4 lua/tls: convert LUA functions to lib: suricata.tls
Ticket: 7608

Convert the Lua functions to a library.

Modified the existing Lua functions to include client and server
versions as appropriate.
2 months ago
Jeff Lucovsky 77139e0cb1 doc/ftp: Document ftp.completion_code sticky buffer
This commit adds documentation for the ftp.completion_code sticky
buffer. This is a multi-buffer match.

Issue: 7507
2 months ago
Jeff Lucovsky f8575dab50 detect/ftp: Add ftp.completion_code keyword
Issue: 7507

Implement the ftp.completion_code sticky buffer. Multi-buffer as an FTP
command can produce multiple responses.

E.g., with the FTP command RETR
    RETR temp.txt
    150 Opening BINARY mode data connection for temp.txt (1164 bytes).
    226 Transfer complete.
2 months ago
Shivani Bhardwaj 31a395c734 pop3: fix incorrect direction matching
sawp crate has its own Direction enum as follows.

pub enum Direction {
    ToClient = 0,
    ToServer = 1,
    Unknown = 2,
}

While it is correct to send this Direction enum as argument to the
sawp_pop3 parser as it expects, it is not correct to use it where the
direction param is obtained from the internal API of Suricata.
The reason is that Suricata's definition of its Direction enum is as
follows.

pub enum Direction {
    ToServer = 0x04,
    ToClient = 0x08,
}

This can lead to issues like incorrect progress tracking of a transaction in
a direction which could cause inspection on incorrect data and buggy behavior.
2 months ago
Jason Ish dbc2f9d1db github-ci: add check for schema property ordering 2 months ago
Jason Ish 371d7cba06 schema: add script to check or sort the schema 2 months ago
Jason Ish 5a4eba2f57 etc/schema: sort properties in alphabetic order
Also:
- Place "additionalProperties" before "properties"
- Place "required" after "additionalProperties"
- Remove "additionalProperties where true, as that is the default

The order should help us spot duplicate keys, and make it easier to
add new keys in their proper place.
2 months ago
Jeff Lucovsky 53c8a0f8f1 doc: Document luaxform transform
Issue: 2290
2 months ago
Jeff Lucovsky 6ed386082e detect/luaxform: initial lua transform support
Adds a new lua script capability to use a script as a buffer transform
keyword.

It uses a `transform` lua function that returns the input buffer after
modifying it.

Issue: 2290
2 months ago
Victor Julien c02f56877e detect: make packet and tx_id available earlier
So transforms can access them through DetectEngineThreadCtx
2 months ago
Jeff Lucovsky aec2513799 doc/ftp: Document ftp.reply_received
Add documentation for the ftp.reply_received keyword.
2 months ago
Jeff Lucovsky 263d56fd11 detect/ftp: Add ftp.received_reply
Issue: 7506

Add a (non-sticky buffer) keyword for ftp.reply_received. This is not a
sticky buffer as the keyword relates to protocol state and not bytes
from the actual protocol exchange.

ftp.reply_received: yes|on|no|off
2 months ago
Philippe Antoine dc20129195 rust: bindgen SCDetectHelperBufferMpmRegister
Ticket: 7667
2 months ago
Philippe Antoine e2603fa820 detect/single-buf: new simple wrapper
Introduce DetectGetSingleData which does the generic wrapping,
including the transforms, using a new callback prototype
DetectTxGetBufferPtr

The goal is to replace most InspectionBufferGetDataPtr.
For this commit, we do not change every callback to keep the
change relatively small.

Focus here is to remove DetectHelperGetData as its functionality is
provided more directly by the new DetectTxGetBufferPtr.
2 months ago
Philippe Antoine 6bf2b3c47e rust/derive: fix clippy collapsible_match warning
And now deny warnings
2 months ago
Philippe Antoine c6dfadd4b4 ja4: check more ifdef HAVE_JA4
Avoids a unused macro warning
2 months ago
Richard McConnell d81b76d852 output/tls: Allow logging of sv-handshake params
Ticket: 6695

"server_handshake" which logs the following:
1. TLS version used during handshake
2. The chosen cipher suite, excluding GREASE
3. TLS extensions, excluding GREASE
2 months ago
Richard McConnell 94c8be22d4 output/tls: Allow logging of cl-handshake params
Ticket: 6695

Add new custom log fields:

"client_handshake" which logs the following:
1. TLS version used during handshake
2. TLS extensions, excluding GREASE, SNI and ALPN
3. All cipher suites, excluding GREASE
4. All signature algorithms, excluding GREASE

The use-case is for logging TLS handshake parameters in order to survey
them, and so that JA4 hashes can be computed offline (in the case that
they're not already computed for the purposes of rule matching).
2 months ago
Richard McConnell 912030cbf4 tls: Move tls-versions to rust
This commit is designed in preparation of enabling the handshake object
to log it's own contents rather than being done on the C side.
Moving the tls versions functionality to rust has a couple of uses:
1. Allows both rust and C side to use these fields
2. Moves more of the tls related logic to rust
3. C side can still use these values because of cbindgen
2 months ago
Richard McConnell 6c1238b7bd tls: Integrate ALPNs into HandshakeParams object
Ticket: 6695

With the introduction of the HandshakeParams object we're able to
utilise the theory further by using it as the object to track the ALPNs.

The HandshakeParams object is now responsible for holding all ALPNS. The
user of this HandshakeParams object i.e. JA4, can use whichever fields
are needed. So only when we generate a JA4 hash do we use the first ALPN
and require to format it. Other users of HandshakeParams may opt to use
all ALPN's i.e. during TlsAlpnGetData().
2 months ago
Richard McConnell 468a037daa tls: Introduce HandshakeParams object for tracking
Ticket: 6695

This introduction splits the use of the handshake parameters into their
own object, HandshakeParams, which is populated by the TLS decoder. The
JA4 object is now very simple. It's a simple String object (the JA4
Hash) which is generated during new().

This introduction is part of a larger idea, which is to enable
outputting these raw parameters without JA3/JA4. These handshake
parameters are the components used to generate the JA4 hash, thus it
makes sense for it to be a user of HandshakeParams.
2 months ago
Philippe Antoine 4085cf44db detect: fix some -Wshorten-64-to-32 warnings
Ticket: #6186
2 months ago
Philippe Antoine 256804b7b1 detect/threshold: timeout handling precision improvement
As found by -Wshorten-64-to-32 warnings

Ticket: #6186

Use SCTime_t instead of u32, which increases memory usage for
the structures changed here, while making it more correct.
2 months ago
Philippe Antoine ee386ac6eb detect/tag: timeout handling precision improvement
As found by -Wshorten-64-to-32 warnings

Ticket: #6186

Use SCTime_t instead of u32, which increases memory usage for
the structures changed here, while making it more correct.
2 months ago
Philippe Antoine bad7d2f16d detect/xbits: timeout handling precision improvement
As found by -Wshorten-64-to-32 warnings

Ticket: #6186

Use SCTime_t instead of u32, which increases memory usage for
the structures changed here, while making it more correct.
2 months ago
Jason Ish 259a304f3e rust/applayer: collapse nested if let to remove clippy warning
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#collapsible_match
   = note: `#[warn(clippy::collapsible_match)]` on by default
2 months ago
Jason Ish 90297788f6 rust/htp: suppress io_other_error lint
The fix for this lint requires Rust 1.74.
2 months ago
Jason Ish 061ed2d6a4 github-ci: use rust 1.85.0 for clippy check on templates
Currently our CI clippy test is based on 1.85.0, but the clippy test for
the templates was using latest stable.  Unify the clippy checks to
1.85.0.
2 months ago
Shivani Bhardwaj 91353fdb61 mqtt: trigger raw stream reassembly
Internals
---------
Suricata's stream engine returns data for inspection to the detection
engine from the stream when the chunk size is reached.

Bug
---
Inspection triggered only in the specified chunk sizes may be too late
when it comes to inspection of smaller protocol specific data which
could result in delayed inspection, incorrect data logged with a transaction
and logs misindicating the pkt that triggered an alert.

Fix
---
Fix this by making an explicit call from all respective applayer parsers to
trigger raw stream reassembly which shall make the data available for inspection
in the following call of the stream engine. This needs to happen per direction
on the completion of an entity like a request or a response.

Important notes
---------------
1. The above mentioned behavior with and without this patch is
affected internally by the following conditions.
- inspection depth
- stream depth
In these special cases, the inspection window will be affected and
Suricata may not consider all the data that could be expected to be
inspected.
2. This only applies to applayer protocols running over TCP.
3. The inspection window is only considered up to the ACK'd data.
4. This entire issue is about IDS mode only.

MQTT creates a transaction per message per direction, so, a call to
trigger raw stream reassembly has been made on completion of each
transaction in the respective direction.

Optimization 7026
Bug 7004
2 months ago
Shivani Bhardwaj 42978ca9a7 modbus: trigger raw stream reassembly
Internals
---------
Suricata's stream engine returns data for inspection to the detection
engine from the stream when the chunk size is reached.

Bug
---
Inspection triggered only in the specified chunk sizes may be too late
when it comes to inspection of smaller protocol specific data which
could result in delayed inspection, incorrect data logged with a transaction
and logs misindicating the pkt that triggered an alert.

Fix
---
Fix this by making an explicit call from all respective applayer parsers to
trigger raw stream reassembly which shall make the data available for inspection
in the following call of the stream engine. This needs to happen per direction
on the completion of an entity like a request or a response.

Important notes
---------------
1. The above mentioned behavior with and without this patch is
affected internally by the following conditions.
- inspection depth
- stream depth
In these special cases, the inspection window will be affected and
Suricata may not consider all the data that could be expected to be
inspected.
2. This only applies to applayer protocols running over TCP.
3. The inspection window is only considered up to the ACK'd data.
4. This entire issue is about IDS mode only.

Modbus has a classic request response model, so, a call to trigger raw
stream reassembly is added on completion of each request and response.

Optimization 7026
Bug 7004
2 months ago
Shivani Bhardwaj ca7e9f8daf ldap: trigger raw stream reassembly
Internals
---------
Suricata's stream engine returns data for inspection to the detection
engine from the stream when the chunk size is reached.

Bug
---
Inspection triggered only in the specified chunk sizes may be too late
when it comes to inspection of smaller protocol specific data which
could result in delayed inspection, incorrect data logged with a transaction
and logs misindicating the pkt that triggered an alert.

Fix
---
Fix this by making an explicit call from all respective applayer parsers to
trigger raw stream reassembly which shall make the data available for inspection
in the following call of the stream engine. This needs to happen per direction
on the completion of an entity like a request or a response.

Important notes
---------------
1. The above mentioned behavior with and without this patch is
affected internally by the following conditions.
- inspection depth
- stream depth
In these special cases, the inspection window will be affected and
Suricata may not consider all the data that could be expected to be
inspected.
2. This only applies to applayer protocols running over TCP.
3. The inspection window is only considered up to the ACK'd data.
4. This entire issue is about IDS mode only.

LDAP can have multiple responses corresponding to a request. The call to
trigger raw stream reassembly has been added on common call sites that
see the completion of a request or any of the responses.

Optimization 7026
Bug 7004
2 months ago
Shivani Bhardwaj fd683ed151 krb: trigger raw stream reassembly
Internals
---------
Suricata's stream engine returns data for inspection to the detection
engine from the stream when the chunk size is reached.

Bug
---
Inspection triggered only in the specified chunk sizes may be too late
when it comes to inspection of smaller protocol specific data which
could result in delayed inspection, incorrect data logged with a transaction
and logs misindicating the pkt that triggered an alert.

Fix
---
Fix this by making an explicit call from all respective applayer parsers to
trigger raw stream reassembly which shall make the data available for inspection
in the following call of the stream engine. This needs to happen per direction
on the completion of an entity like a request or a response.

Important notes
---------------
1. The above mentioned behavior with and without this patch is
affected internally by the following conditions.
- inspection depth
- stream depth
In these special cases, the inspection window will be affected and
Suricata may not consider all the data that could be expected to be
inspected.
2. This only applies to applayer protocols running over TCP.
3. The inspection window is only considered up to the ACK'd data.
4. This entire issue is about IDS mode only.

KRB5 creates a transaction based on how each input is parsed. It could
be parsed as a request or response but that is the concern of the
parser. The call to trigger raw stream reassembly has been added after
successful parsing of the respective request/response.

Optimization 7026
Bug 7004
2 months ago
Shivani Bhardwaj d0655ed30e http2: trigger raw stream reassembly
Internals
---------
Suricata's stream engine returns data for inspection to the detection
engine from the stream when the chunk size is reached.

Bug
---
Inspection triggered only in the specified chunk sizes may be too late
when it comes to inspection of smaller protocol specific data which
could result in delayed inspection, incorrect data logged with a transaction
and logs misindicating the pkt that triggered an alert.

Fix
---
Fix this by making an explicit call from all respective applayer parsers to
trigger raw stream reassembly which shall make the data available for inspection
in the following call of the stream engine. This needs to happen per direction
on the completion of an entity like a request or a response.

Important notes
---------------
1. The above mentioned behavior with and without this patch is
affected internally by the following conditions.
- inspection depth
- stream depth
In these special cases, the inspection window will be affected and
Suricata may not consider all the data that could be expected to be
inspected.
2. This only applies to applayer protocols running over TCP.
3. The inspection window is only considered up to the ACK'd data.
4. This entire issue is about IDS mode only.

HTTP2 has a classic request response model, so, a call to trigger raw
stream reassembly is added on completion of each request and response.
HTTP2 parser has its own maximum reassembly setting. The call has been
added irrespective of this setting as it is prudent to make all data so
far available for inspection if maximum was reached until the maximum.

Optimization 7026
Bug 7004
2 months ago
Shivani Bhardwaj eca13b8dd8 enip: trigger raw stream reassembly
Internals
---------
Suricata's stream engine returns data for inspection to the detection
engine from the stream when the chunk size is reached.

Bug
---
Inspection triggered only in the specified chunk sizes may be too late
when it comes to inspection of smaller protocol specific data which
could result in delayed inspection, incorrect data logged with a transaction
and logs misindicating the pkt that triggered an alert.

Fix
---
Fix this by making an explicit call from all respective applayer parsers to
trigger raw stream reassembly which shall make the data available for inspection
in the following call of the stream engine. This needs to happen per direction
on the completion of an entity like a request or a response.

Important notes
---------------
1. The above mentioned behavior with and without this patch is
affected internally by the following conditions.
- inspection depth
- stream depth
In these special cases, the inspection window will be affected and
Suricata may not consider all the data that could be expected to be
inspected.
2. This only applies to applayer protocols running over TCP.
3. The inspection window is only considered up to the ACK'd data.
4. This entire issue is about IDS mode only.

ENIP has a classic request response model, so, a call to trigger raw
stream reassembly is added on completion of each request and response.

Optimization 7026
Bug 7004
2 months ago
Philippe Antoine 622476c47d http1: use a tx iterator
Better performance than the defaut iterator as we do not need to
read all first elements every time
2 months ago
Philippe Antoine c2756dec75 htp: move transactions list from BTree to VecDeque
As it is more efficient in our case of pipelining requests
2 months ago
Philippe Antoine 756f28d086 http: simplify tx removal interface 2 months ago
Philippe Antoine ba186f5d25 htp: remove unused code 2 months ago
Jeff Lucovsky 07205ab057 detect/xform: Support transform identity data
Transforms that support optional strings, like from_base64 and
pcrexform, should also support identity-strings to treat transforms with
like transform options as the same.

This commit adds transform identity data handling:
- When computing a hash, include identity data from the transform
- When comparing, include the identity data from the transforms
- Omitting the "options" ptr from the transform hash/compare
- Modify xor, pcrexform and from_base64 to supply identification data for
  disambiguation in the compare/hash logic.
2 months ago