When running in pcap-file mode and with a continous directory
reading, the provided directory can be empty.
By having no packets and being in offline mode, the initial packet timestamp
is never set. However, Flow Manager waited until the timestamp was set
to proceed with transferring its state to "RUNNING".
Other pcap-related threads (RX / workers) are set in RUNNING state while
waiting for the PCAP to appear in the directory.
As a result, the main Suricata-Main thread timed out after the default
60 seconds budget for threads to turn from INIT_DONE to RUNNING state.
Ticket: 7568
Ticket: 7667
Currently no functions are exported.
DetectFile* struct are moved to detect-file-data.h where
they make more sense.
ifndef SURICATA_BINDGEN_H is used for bindgen to exclude
pcre2 related code
Issue: 7608
Update the documentation to reflect the new and expanded functions
available form the Lua TLS library
There are now "server" and "client" versions of most functions. The TLS
object getter is now "get_tx"
Issue: 7507
Implement the ftp.completion_code sticky buffer. Multi-buffer as an FTP
command can produce multiple responses.
E.g., with the FTP command RETR
RETR temp.txt
150 Opening BINARY mode data connection for temp.txt (1164 bytes).
226 Transfer complete.
sawp crate has its own Direction enum as follows.
pub enum Direction {
ToClient = 0,
ToServer = 1,
Unknown = 2,
}
While it is correct to send this Direction enum as argument to the
sawp_pop3 parser as it expects, it is not correct to use it where the
direction param is obtained from the internal API of Suricata.
The reason is that Suricata's definition of its Direction enum is as
follows.
pub enum Direction {
ToServer = 0x04,
ToClient = 0x08,
}
This can lead to issues like incorrect progress tracking of a transaction in
a direction which could cause inspection on incorrect data and buggy behavior.
Also:
- Place "additionalProperties" before "properties"
- Place "required" after "additionalProperties"
- Remove "additionalProperties where true, as that is the default
The order should help us spot duplicate keys, and make it easier to
add new keys in their proper place.
Adds a new lua script capability to use a script as a buffer transform
keyword.
It uses a `transform` lua function that returns the input buffer after
modifying it.
Issue: 2290
Issue: 7506
Add a (non-sticky buffer) keyword for ftp.reply_received. This is not a
sticky buffer as the keyword relates to protocol state and not bytes
from the actual protocol exchange.
ftp.reply_received: yes|on|no|off
Introduce DetectGetSingleData which does the generic wrapping,
including the transforms, using a new callback prototype
DetectTxGetBufferPtr
The goal is to replace most InspectionBufferGetDataPtr.
For this commit, we do not change every callback to keep the
change relatively small.
Focus here is to remove DetectHelperGetData as its functionality is
provided more directly by the new DetectTxGetBufferPtr.
Ticket: 6695
"server_handshake" which logs the following:
1. TLS version used during handshake
2. The chosen cipher suite, excluding GREASE
3. TLS extensions, excluding GREASE
Ticket: 6695
Add new custom log fields:
"client_handshake" which logs the following:
1. TLS version used during handshake
2. TLS extensions, excluding GREASE, SNI and ALPN
3. All cipher suites, excluding GREASE
4. All signature algorithms, excluding GREASE
The use-case is for logging TLS handshake parameters in order to survey
them, and so that JA4 hashes can be computed offline (in the case that
they're not already computed for the purposes of rule matching).
This commit is designed in preparation of enabling the handshake object
to log it's own contents rather than being done on the C side.
Moving the tls versions functionality to rust has a couple of uses:
1. Allows both rust and C side to use these fields
2. Moves more of the tls related logic to rust
3. C side can still use these values because of cbindgen
Ticket: 6695
With the introduction of the HandshakeParams object we're able to
utilise the theory further by using it as the object to track the ALPNs.
The HandshakeParams object is now responsible for holding all ALPNS. The
user of this HandshakeParams object i.e. JA4, can use whichever fields
are needed. So only when we generate a JA4 hash do we use the first ALPN
and require to format it. Other users of HandshakeParams may opt to use
all ALPN's i.e. during TlsAlpnGetData().
Ticket: 6695
This introduction splits the use of the handshake parameters into their
own object, HandshakeParams, which is populated by the TLS decoder. The
JA4 object is now very simple. It's a simple String object (the JA4
Hash) which is generated during new().
This introduction is part of a larger idea, which is to enable
outputting these raw parameters without JA3/JA4. These handshake
parameters are the components used to generate the JA4 hash, thus it
makes sense for it to be a user of HandshakeParams.
As found by -Wshorten-64-to-32 warnings
Ticket: #6186
Use SCTime_t instead of u32, which increases memory usage for
the structures changed here, while making it more correct.
As found by -Wshorten-64-to-32 warnings
Ticket: #6186
Use SCTime_t instead of u32, which increases memory usage for
the structures changed here, while making it more correct.
As found by -Wshorten-64-to-32 warnings
Ticket: #6186
Use SCTime_t instead of u32, which increases memory usage for
the structures changed here, while making it more correct.
Internals
---------
Suricata's stream engine returns data for inspection to the detection
engine from the stream when the chunk size is reached.
Bug
---
Inspection triggered only in the specified chunk sizes may be too late
when it comes to inspection of smaller protocol specific data which
could result in delayed inspection, incorrect data logged with a transaction
and logs misindicating the pkt that triggered an alert.
Fix
---
Fix this by making an explicit call from all respective applayer parsers to
trigger raw stream reassembly which shall make the data available for inspection
in the following call of the stream engine. This needs to happen per direction
on the completion of an entity like a request or a response.
Important notes
---------------
1. The above mentioned behavior with and without this patch is
affected internally by the following conditions.
- inspection depth
- stream depth
In these special cases, the inspection window will be affected and
Suricata may not consider all the data that could be expected to be
inspected.
2. This only applies to applayer protocols running over TCP.
3. The inspection window is only considered up to the ACK'd data.
4. This entire issue is about IDS mode only.
MQTT creates a transaction per message per direction, so, a call to
trigger raw stream reassembly has been made on completion of each
transaction in the respective direction.
Optimization 7026
Bug 7004
Internals
---------
Suricata's stream engine returns data for inspection to the detection
engine from the stream when the chunk size is reached.
Bug
---
Inspection triggered only in the specified chunk sizes may be too late
when it comes to inspection of smaller protocol specific data which
could result in delayed inspection, incorrect data logged with a transaction
and logs misindicating the pkt that triggered an alert.
Fix
---
Fix this by making an explicit call from all respective applayer parsers to
trigger raw stream reassembly which shall make the data available for inspection
in the following call of the stream engine. This needs to happen per direction
on the completion of an entity like a request or a response.
Important notes
---------------
1. The above mentioned behavior with and without this patch is
affected internally by the following conditions.
- inspection depth
- stream depth
In these special cases, the inspection window will be affected and
Suricata may not consider all the data that could be expected to be
inspected.
2. This only applies to applayer protocols running over TCP.
3. The inspection window is only considered up to the ACK'd data.
4. This entire issue is about IDS mode only.
Modbus has a classic request response model, so, a call to trigger raw
stream reassembly is added on completion of each request and response.
Optimization 7026
Bug 7004
Internals
---------
Suricata's stream engine returns data for inspection to the detection
engine from the stream when the chunk size is reached.
Bug
---
Inspection triggered only in the specified chunk sizes may be too late
when it comes to inspection of smaller protocol specific data which
could result in delayed inspection, incorrect data logged with a transaction
and logs misindicating the pkt that triggered an alert.
Fix
---
Fix this by making an explicit call from all respective applayer parsers to
trigger raw stream reassembly which shall make the data available for inspection
in the following call of the stream engine. This needs to happen per direction
on the completion of an entity like a request or a response.
Important notes
---------------
1. The above mentioned behavior with and without this patch is
affected internally by the following conditions.
- inspection depth
- stream depth
In these special cases, the inspection window will be affected and
Suricata may not consider all the data that could be expected to be
inspected.
2. This only applies to applayer protocols running over TCP.
3. The inspection window is only considered up to the ACK'd data.
4. This entire issue is about IDS mode only.
LDAP can have multiple responses corresponding to a request. The call to
trigger raw stream reassembly has been added on common call sites that
see the completion of a request or any of the responses.
Optimization 7026
Bug 7004
Internals
---------
Suricata's stream engine returns data for inspection to the detection
engine from the stream when the chunk size is reached.
Bug
---
Inspection triggered only in the specified chunk sizes may be too late
when it comes to inspection of smaller protocol specific data which
could result in delayed inspection, incorrect data logged with a transaction
and logs misindicating the pkt that triggered an alert.
Fix
---
Fix this by making an explicit call from all respective applayer parsers to
trigger raw stream reassembly which shall make the data available for inspection
in the following call of the stream engine. This needs to happen per direction
on the completion of an entity like a request or a response.
Important notes
---------------
1. The above mentioned behavior with and without this patch is
affected internally by the following conditions.
- inspection depth
- stream depth
In these special cases, the inspection window will be affected and
Suricata may not consider all the data that could be expected to be
inspected.
2. This only applies to applayer protocols running over TCP.
3. The inspection window is only considered up to the ACK'd data.
4. This entire issue is about IDS mode only.
KRB5 creates a transaction based on how each input is parsed. It could
be parsed as a request or response but that is the concern of the
parser. The call to trigger raw stream reassembly has been added after
successful parsing of the respective request/response.
Optimization 7026
Bug 7004
Internals
---------
Suricata's stream engine returns data for inspection to the detection
engine from the stream when the chunk size is reached.
Bug
---
Inspection triggered only in the specified chunk sizes may be too late
when it comes to inspection of smaller protocol specific data which
could result in delayed inspection, incorrect data logged with a transaction
and logs misindicating the pkt that triggered an alert.
Fix
---
Fix this by making an explicit call from all respective applayer parsers to
trigger raw stream reassembly which shall make the data available for inspection
in the following call of the stream engine. This needs to happen per direction
on the completion of an entity like a request or a response.
Important notes
---------------
1. The above mentioned behavior with and without this patch is
affected internally by the following conditions.
- inspection depth
- stream depth
In these special cases, the inspection window will be affected and
Suricata may not consider all the data that could be expected to be
inspected.
2. This only applies to applayer protocols running over TCP.
3. The inspection window is only considered up to the ACK'd data.
4. This entire issue is about IDS mode only.
HTTP2 has a classic request response model, so, a call to trigger raw
stream reassembly is added on completion of each request and response.
HTTP2 parser has its own maximum reassembly setting. The call has been
added irrespective of this setting as it is prudent to make all data so
far available for inspection if maximum was reached until the maximum.
Optimization 7026
Bug 7004
Internals
---------
Suricata's stream engine returns data for inspection to the detection
engine from the stream when the chunk size is reached.
Bug
---
Inspection triggered only in the specified chunk sizes may be too late
when it comes to inspection of smaller protocol specific data which
could result in delayed inspection, incorrect data logged with a transaction
and logs misindicating the pkt that triggered an alert.
Fix
---
Fix this by making an explicit call from all respective applayer parsers to
trigger raw stream reassembly which shall make the data available for inspection
in the following call of the stream engine. This needs to happen per direction
on the completion of an entity like a request or a response.
Important notes
---------------
1. The above mentioned behavior with and without this patch is
affected internally by the following conditions.
- inspection depth
- stream depth
In these special cases, the inspection window will be affected and
Suricata may not consider all the data that could be expected to be
inspected.
2. This only applies to applayer protocols running over TCP.
3. The inspection window is only considered up to the ACK'd data.
4. This entire issue is about IDS mode only.
ENIP has a classic request response model, so, a call to trigger raw
stream reassembly is added on completion of each request and response.
Optimization 7026
Bug 7004
Transforms that support optional strings, like from_base64 and
pcrexform, should also support identity-strings to treat transforms with
like transform options as the same.
This commit adds transform identity data handling:
- When computing a hash, include identity data from the transform
- When comparing, include the identity data from the transforms
- Omitting the "options" ptr from the transform hash/compare
- Modify xor, pcrexform and from_base64 to supply identification data for
disambiguation in the compare/hash logic.
Lukas Sismis
Philippe Antoine
Jeff Lucovsky
Shivani Bhardwaj
Jason Ish
Victor Julien
Richard McConnell