This commit is designed in preparation of enabling the handshake object
to log it's own contents rather than being done on the C side.
Moving the tls versions functionality to rust has a couple of uses:
1. Allows both rust and C side to use these fields
2. Moves more of the tls related logic to rust
3. C side can still use these values because of cbindgen
Ticket: 6695
With the introduction of the HandshakeParams object we're able to
utilise the theory further by using it as the object to track the ALPNs.
The HandshakeParams object is now responsible for holding all ALPNS. The
user of this HandshakeParams object i.e. JA4, can use whichever fields
are needed. So only when we generate a JA4 hash do we use the first ALPN
and require to format it. Other users of HandshakeParams may opt to use
all ALPN's i.e. during TlsAlpnGetData().
Ticket: 6695
This introduction splits the use of the handshake parameters into their
own object, HandshakeParams, which is populated by the TLS decoder. The
JA4 object is now very simple. It's a simple String object (the JA4
Hash) which is generated during new().
This introduction is part of a larger idea, which is to enable
outputting these raw parameters without JA3/JA4. These handshake
parameters are the components used to generate the JA4 hash, thus it
makes sense for it to be a user of HandshakeParams.
As found by -Wshorten-64-to-32 warnings
Ticket: #6186
Use SCTime_t instead of u32, which increases memory usage for
the structures changed here, while making it more correct.
As found by -Wshorten-64-to-32 warnings
Ticket: #6186
Use SCTime_t instead of u32, which increases memory usage for
the structures changed here, while making it more correct.
As found by -Wshorten-64-to-32 warnings
Ticket: #6186
Use SCTime_t instead of u32, which increases memory usage for
the structures changed here, while making it more correct.
Internals
---------
Suricata's stream engine returns data for inspection to the detection
engine from the stream when the chunk size is reached.
Bug
---
Inspection triggered only in the specified chunk sizes may be too late
when it comes to inspection of smaller protocol specific data which
could result in delayed inspection, incorrect data logged with a transaction
and logs misindicating the pkt that triggered an alert.
Fix
---
Fix this by making an explicit call from all respective applayer parsers to
trigger raw stream reassembly which shall make the data available for inspection
in the following call of the stream engine. This needs to happen per direction
on the completion of an entity like a request or a response.
Important notes
---------------
1. The above mentioned behavior with and without this patch is
affected internally by the following conditions.
- inspection depth
- stream depth
In these special cases, the inspection window will be affected and
Suricata may not consider all the data that could be expected to be
inspected.
2. This only applies to applayer protocols running over TCP.
3. The inspection window is only considered up to the ACK'd data.
4. This entire issue is about IDS mode only.
MQTT creates a transaction per message per direction, so, a call to
trigger raw stream reassembly has been made on completion of each
transaction in the respective direction.
Optimization 7026
Bug 7004
Internals
---------
Suricata's stream engine returns data for inspection to the detection
engine from the stream when the chunk size is reached.
Bug
---
Inspection triggered only in the specified chunk sizes may be too late
when it comes to inspection of smaller protocol specific data which
could result in delayed inspection, incorrect data logged with a transaction
and logs misindicating the pkt that triggered an alert.
Fix
---
Fix this by making an explicit call from all respective applayer parsers to
trigger raw stream reassembly which shall make the data available for inspection
in the following call of the stream engine. This needs to happen per direction
on the completion of an entity like a request or a response.
Important notes
---------------
1. The above mentioned behavior with and without this patch is
affected internally by the following conditions.
- inspection depth
- stream depth
In these special cases, the inspection window will be affected and
Suricata may not consider all the data that could be expected to be
inspected.
2. This only applies to applayer protocols running over TCP.
3. The inspection window is only considered up to the ACK'd data.
4. This entire issue is about IDS mode only.
Modbus has a classic request response model, so, a call to trigger raw
stream reassembly is added on completion of each request and response.
Optimization 7026
Bug 7004
Internals
---------
Suricata's stream engine returns data for inspection to the detection
engine from the stream when the chunk size is reached.
Bug
---
Inspection triggered only in the specified chunk sizes may be too late
when it comes to inspection of smaller protocol specific data which
could result in delayed inspection, incorrect data logged with a transaction
and logs misindicating the pkt that triggered an alert.
Fix
---
Fix this by making an explicit call from all respective applayer parsers to
trigger raw stream reassembly which shall make the data available for inspection
in the following call of the stream engine. This needs to happen per direction
on the completion of an entity like a request or a response.
Important notes
---------------
1. The above mentioned behavior with and without this patch is
affected internally by the following conditions.
- inspection depth
- stream depth
In these special cases, the inspection window will be affected and
Suricata may not consider all the data that could be expected to be
inspected.
2. This only applies to applayer protocols running over TCP.
3. The inspection window is only considered up to the ACK'd data.
4. This entire issue is about IDS mode only.
LDAP can have multiple responses corresponding to a request. The call to
trigger raw stream reassembly has been added on common call sites that
see the completion of a request or any of the responses.
Optimization 7026
Bug 7004
Internals
---------
Suricata's stream engine returns data for inspection to the detection
engine from the stream when the chunk size is reached.
Bug
---
Inspection triggered only in the specified chunk sizes may be too late
when it comes to inspection of smaller protocol specific data which
could result in delayed inspection, incorrect data logged with a transaction
and logs misindicating the pkt that triggered an alert.
Fix
---
Fix this by making an explicit call from all respective applayer parsers to
trigger raw stream reassembly which shall make the data available for inspection
in the following call of the stream engine. This needs to happen per direction
on the completion of an entity like a request or a response.
Important notes
---------------
1. The above mentioned behavior with and without this patch is
affected internally by the following conditions.
- inspection depth
- stream depth
In these special cases, the inspection window will be affected and
Suricata may not consider all the data that could be expected to be
inspected.
2. This only applies to applayer protocols running over TCP.
3. The inspection window is only considered up to the ACK'd data.
4. This entire issue is about IDS mode only.
KRB5 creates a transaction based on how each input is parsed. It could
be parsed as a request or response but that is the concern of the
parser. The call to trigger raw stream reassembly has been added after
successful parsing of the respective request/response.
Optimization 7026
Bug 7004
Internals
---------
Suricata's stream engine returns data for inspection to the detection
engine from the stream when the chunk size is reached.
Bug
---
Inspection triggered only in the specified chunk sizes may be too late
when it comes to inspection of smaller protocol specific data which
could result in delayed inspection, incorrect data logged with a transaction
and logs misindicating the pkt that triggered an alert.
Fix
---
Fix this by making an explicit call from all respective applayer parsers to
trigger raw stream reassembly which shall make the data available for inspection
in the following call of the stream engine. This needs to happen per direction
on the completion of an entity like a request or a response.
Important notes
---------------
1. The above mentioned behavior with and without this patch is
affected internally by the following conditions.
- inspection depth
- stream depth
In these special cases, the inspection window will be affected and
Suricata may not consider all the data that could be expected to be
inspected.
2. This only applies to applayer protocols running over TCP.
3. The inspection window is only considered up to the ACK'd data.
4. This entire issue is about IDS mode only.
HTTP2 has a classic request response model, so, a call to trigger raw
stream reassembly is added on completion of each request and response.
HTTP2 parser has its own maximum reassembly setting. The call has been
added irrespective of this setting as it is prudent to make all data so
far available for inspection if maximum was reached until the maximum.
Optimization 7026
Bug 7004
Internals
---------
Suricata's stream engine returns data for inspection to the detection
engine from the stream when the chunk size is reached.
Bug
---
Inspection triggered only in the specified chunk sizes may be too late
when it comes to inspection of smaller protocol specific data which
could result in delayed inspection, incorrect data logged with a transaction
and logs misindicating the pkt that triggered an alert.
Fix
---
Fix this by making an explicit call from all respective applayer parsers to
trigger raw stream reassembly which shall make the data available for inspection
in the following call of the stream engine. This needs to happen per direction
on the completion of an entity like a request or a response.
Important notes
---------------
1. The above mentioned behavior with and without this patch is
affected internally by the following conditions.
- inspection depth
- stream depth
In these special cases, the inspection window will be affected and
Suricata may not consider all the data that could be expected to be
inspected.
2. This only applies to applayer protocols running over TCP.
3. The inspection window is only considered up to the ACK'd data.
4. This entire issue is about IDS mode only.
ENIP has a classic request response model, so, a call to trigger raw
stream reassembly is added on completion of each request and response.
Optimization 7026
Bug 7004
Transforms that support optional strings, like from_base64 and
pcrexform, should also support identity-strings to treat transforms with
like transform options as the same.
This commit adds transform identity data handling:
- When computing a hash, include identity data from the transform
- When comparing, include the identity data from the transforms
- Omitting the "options" ptr from the transform hash/compare
- Modify xor, pcrexform and from_base64 to supply identification data for
disambiguation in the compare/hash logic.
This also breaks out the fileinfo function into a method per file info
item. And likewise for state, just return the state and add a new method
for checking if the file is stored.
Ticket: #7491
If a password message was seen while logging passwords was disabled
for pgsql, this would lead to an empty request being logged.
Instead of simply not logging anything when there is a password message
and this is disabled, however, log instead that said password is
redacted.
Bug #7647
PKT_STREAM_EOF flag is set only when a pseudo packet is created. In all
the users of this flag, it suffices to use PKT_PSEUDO_STREAM_END
instead. PKT_PSEUDO_STREAM_END is the more widely used flag as well so
keep it and remove this unneeded flag creating a vacancy.
Whether minimum inspection depth should be respected is an important
parameter but it is only used by one caller of StreamReassembleRawDo fn
to calculate progress value. This progress value is already passed as a
parameter to the said fn.
Richard McConnell
Philippe Antoine
Jason Ish
Shivani Bhardwaj
Jeff Lucovsky
Victor Julien
Juliana Fajardini