The default value for TLS encryption is generic and not precise.
The new keyword, "track-only" substitues the old keyword, being
more on point what action is actually taken when the TLS flow
reaches an encrypted state.
suricata.yaml output section for smb now parses a types list
and will restrict logging of transactions to these types.
By default, everything still gets logged
Remove unused rs_smb_log_json_request on the way
Ticket: 7620
Ticket: #2696
There are a lot of changes here, which are described below.
In general these changes are renaming constants to conform to the
libhtp-rs versions (which are generated by cbindgen); making all htp
types opaque and changing struct->member references to
htp_struct_member() function calls; and a handful of changes to offload
functionality onto libhtp-rs from suricata, such as URI normalization
and transaction cleanup.
Functions introduced to handle opaque htp_tx_t:
- tx->parsed_uri => htp_tx_parsed_uri(tx)
- tx->parsed_uri->path => htp_uri_path(htp_tx_parsed_uri(tx)
- tx->parsed_uri->hostname => htp_uri_hostname(htp_tx_parsed_uri(tx))
- htp_tx_get_user_data() => htp_tx_user_data(tx)
- htp_tx_is_http_2_upgrade(tx) convenience function introduced to detect response status 101
and “Upgrade: h2c" header.
Functions introduced to handle opaque htp_tx_data_t:
- d->len => htp_tx_data_len()
- d->data => htp_tx_data_data()
- htp_tx_data_tx(data) function to get the htp_tx_t from the htp_tx_data_t
- htp_tx_data_is_empty(data) convenience function introduced to test if the data is empty.
Other changes:
Build libhtp-rs as a crate inside rust. Update autoconf to no longer
use libhtp as an external dependency. Remove HAVE_HTP feature defines
since they are no longer needed.
Make function arguments and return values const where possible
htp_tx_destroy(tx) will now free an incomplete transaction
htp_time_t replaced with standard struct timeval
Callbacks from libhtp now provide the htp_connp_t and the htp_tx_data_t
as separate arguments. This means the connection parser is no longer
fetched from the transaction inside callbacks.
SCHTPGenerateNormalizedUri() functionality moved inside libhtp-rs, which
now provides normalized URI values.
The normalized URI is available with accessor function: htp_tx_normalized_uri()
Configuration settings added to control the behaviour of the URI normalization:
- htp_config_set_normalized_uri_include_all()
- htp_config_set_plusspace_decode()
- htp_config_set_convert_lowercase()
- htp_config_set_double_decode_normalized_query()
- htp_config_set_double_decode_normalized_path()
- htp_config_set_backslash_convert_slashes()
- htp_config_set_bestfit_replacement_byte()
- htp_config_set_convert_lowercase()
- htp_config_set_nul_encoded_terminates()
- htp_config_set_nul_raw_terminates()
- htp_config_set_path_separators_compress()
- htp_config_set_path_separators_decode()
- htp_config_set_u_encoding_decode()
- htp_config_set_url_encoding_invalid_handling()
- htp_config_set_utf8_convert_bestfit()
- htp_config_set_normalized_uri_include_all()
- htp_config_set_plusspace_decode()
Constants related to configuring uri normalization:
- HTP_URL_DECODE_PRESERVE_PERCENT => HTP_URL_ENCODING_HANDLING_PRESERVE_PERCENT
- HTP_URL_DECODE_REMOVE_PERCENT => HTP_URL_ENCODING_HANDLING_REMOVE_PERCENT
- HTP_URL_DECODE_PROCESS_INVALID => HTP_URL_ENCODING_HANDLING_PROCESS_INVALID
htp_config_set_field_limits(soft_limit, hard_limit) changed to
htp_config_set_field_limit(limit) because libhtp didn't implement soft
limits.
libhtp logging API updated to provide HTP_LOG_CODE constants along with
the message. This eliminates the need to perform string matching on
message text to map log messages to HTTP_DECODER_EVENT values, and the
HTP_LOG_CODE values can be used directly. In support of this,
HTP_DECODER_EVENT values are mapped to their corresponding HTP_LOG_CODE
values.
New log events to describe additional anomalies:
HTP_LOG_CODE_REQUEST_TOO_MANY_LZMA_LAYERS
HTP_LOG_CODE_RESPONSE_TOO_MANY_LZMA_LAYERS
HTP_LOG_CODE_PROTOCOL_CONTAINS_EXTRA_DATA
HTP_LOG_CODE_CONTENT_LENGTH_EXTRA_DATA_START
HTP_LOG_CODE_CONTENT_LENGTH_EXTRA_DATA_END
HTP_LOG_CODE_SWITCHING_PROTO_WITH_CONTENT_LENGTH
HTP_LOG_CODE_DEFORMED_EOL
HTP_LOG_CODE_PARSER_STATE_ERROR
HTP_LOG_CODE_MISSING_OUTBOUND_TRANSACTION_DATA
HTP_LOG_CODE_MISSING_INBOUND_TRANSACTION_DATA
HTP_LOG_CODE_ZERO_LENGTH_DATA_CHUNKS
HTP_LOG_CODE_REQUEST_LINE_UNKNOWN_METHOD
HTP_LOG_CODE_REQUEST_LINE_UNKNOWN_METHOD_NO_PROTOCOL
HTP_LOG_CODE_REQUEST_LINE_UNKNOWN_METHOD_INVALID_PROTOCOL
HTP_LOG_CODE_REQUEST_LINE_NO_PROTOCOL
HTP_LOG_CODE_RESPONSE_LINE_INVALID_PROTOCOL
HTP_LOG_CODE_RESPONSE_LINE_INVALID_RESPONSE_STATUS
HTP_LOG_CODE_RESPONSE_BODY_INTERNAL_ERROR
HTP_LOG_CODE_REQUEST_BODY_DATA_CALLBACK_ERROR
HTP_LOG_CODE_RESPONSE_INVALID_EMPTY_NAME
HTP_LOG_CODE_REQUEST_INVALID_EMPTY_NAME
HTP_LOG_CODE_RESPONSE_INVALID_LWS_AFTER_NAME
HTP_LOG_CODE_RESPONSE_HEADER_NAME_NOT_TOKEN
HTP_LOG_CODE_REQUEST_INVALID_LWS_AFTER_NAME
HTP_LOG_CODE_LZMA_DECOMPRESSION_DISABLED
HTP_LOG_CODE_CONNECTION_ALREADY_OPEN
HTP_LOG_CODE_COMPRESSION_BOMB_DOUBLE_LZMA
HTP_LOG_CODE_INVALID_CONTENT_ENCODING
HTP_LOG_CODE_INVALID_GAP
HTP_LOG_CODE_ERROR
The new htp_log API supports consuming log messages more easily than
walking a list and tracking the current offset. Internally, libhtp-rs
now provides log messages as a queue of htp_log_t, which means the
application can simply call htp_conn_next_log() to fetch the next log
message until the queue is empty. Once the application is done with a
log message, they can call htp_log_free() to dispose of it.
Functions supporting htp_log_t:
htp_conn_next_log(conn) - Get the next log message
htp_log_message(log) - To get the text of the message
htp_log_code(log) - To get the HTP_LOG_CODE value
htp_log_free(log) - To free the htp_log_t
email.x_mailer matches on MIME EMAIL X-Mailer
This keyword maps to the EVE field email.x_mailer
It is a sticky buffer
Supports prefiltering
Ticket: #7598
email.message_id matches on MIME EMAIL Message-Id
This keyword maps to the EVE field email.message_id
It is a sticky buffer
Supports prefiltering
Ticket: #7593
This commits adds documentation for the entropy keyword.
The entropy keyword calculates the Shannon entropy value for content
with the calculated value used to determine whether an alert occurs.
When running in non-forwarding (IDS) mode, it is not required
to create TX queues for the interface.
This can be acheived by setting tx-descriptors configuration
field to 0.
Ticket: 7633
ICE card (Intel E810) was not receiving packets immediatelly
after startup, Suricata workers would act as processing while
it was not. This eliminates the problem by only continuing
in the initialization if the link is already up.
The setting can be turned off manually from the configuraiton
file.
Ticket: 7381
It turned out that having global (interface-specific) mempool
that is shared by the threads of the interface is slower than
having individual mempools per queue for each interface.
The commit brings this change and should be user-invisible,
the config setting remains still as a number of objects of
all mempools summed (of that interface).
Ticket: 7382
Moves the nDPI documentation to an nDPI page in the plugins
section. Remove the duplication of installation and setup
documentation.
Includes some minor cleanups.
Cache Hyperscan serialized databases to disk to prevent compilation
of the same databases when Suricata is run again with the same
ruleset.
Hyperscan binary files are stored per rulegroup in the designated
folder, by default in the cached library folder.
Since caching is per signature group heads,
some chunk of the ruleset can change and it still can reuse part of
the unchanged signature groups.
Loading *fresh* ET Open ruleset: 19 seconds
Loading *cached* ET Open ruleset: 07 seconds
Ticket: 7170
The stats for exception policies are only logged/ present when any of
the exception policies are enabled (which means any value other than
"auto" or "ignore" in IDS mode, or "ignore" in IPS mode).
This wasn't clearly stated in the docs.
To accompany the Exception Policy stats, also add information about any
Exception Policy triggered and for which target to the flow log event.
Task #6215
Ticket: 7613
Avoids potential large per-thread memory allocation. A buffer with the
size of the largest decode_base64 buffer size setting would be allocated
per thread. As this was a u32, it could mean a per-thread 4GiB memory
allocation.
64KiB was already the built-in default for cases where bytes size wasn't
specified.
We were mentioning "Suricata Support" page, which could be a bit
misleading -- and also used a link that is actually redirected to the
Suricata Community page, anyways.
Add keywords dns.additionals.rrname and dns.authorities.rrname. Along
the way, consolidate dns.query.name and dns.answer.name into a single file
and register them altogether since there is a lot of common code.
ldap.responses.message matches on LDAPResult error message
This keyword maps the following eve fields:
ldap.responses[].bind_response.message
ldap.responses[].search_result_done.message
ldap.responses[].modify_response.message
ldap.responses[].add_response.message
ldap.responses[].del_response.message
ldap.responses[].mod_dn_response.message
ldap.responses[].compare_response.message
ldap.responses[].extended_response.message
It is a sticky buffer
Supports prefiltering
Ticket: #7532
ldap.responses.result_code matches on LDAP result code
This keyword maps the following eve fields:
ldap.responses[].bind_response.result_code
ldap.responses[].search_result_done.result_code
ldap.responses[].modify_response.result_code
ldap.responses[].add_response.result_code
ldap.responses[].del_response.result_code
ldap.responses[].mod_dn_response.result_code
ldap.responses[].compare_response.result_code
ldap.responses[].extended_response.result_code
It is an unsigned 32-bit integer
Doesn't support prefiltering
Ticket: #7532
Funciton ldap_tx_get_responses_dn returns empty buffer in case
the response doesn't contain the distinguished name field
Fixes: 73ae6e997f ("detect: add ldap.responses.dn")
Change variable name 'req' to 'resp' in function ldap_tx_get_responses_dn and documentation nits
Fixes:
73ae6e997f ("detect: add ldap.responses.dn")
16dcee46fc ("detect: add ldap.request.dn")
Issue: 4082
Move the configuration file handling to Rust.
These changes will no longer terminate Suricata when there's an invalid
value for ftp.memcap. Like earlier Suricata releases, an error message
is logged "Invalid value <value> for ftp.memcap" but Suricata will no
longer terminate execution. It will use a default value of "0" instead.
ldap.responses.dn matches on LDAPDN from responses operations
This keyword maps the following eve fields:
ldap.responses[].search_result_entry.base_object
ldap.responses[].bind_response.matched_dn
ldap.responses[].search_result_done.matched_dn
ldap.responses[].modify_response.matched_dn
ldap.responses[].add_response.matched_dn
ldap.responses[].del_response.matched_dn
ldap.responses[].mod_dn_response.matched_dn
ldap.responses[].compare_response.matched_dn
ldap.responses[].extended_response.matched_dn
It is a sticky buffer
Supports prefiltering
Ticket: #7471
ldap.request.dn matches on LDAPDN from request operations
This keyword maps the following eve fields:
ldap.request.bind_request.name
ldap.request.add_request.entry
ldap.request.search_request.base_object
ldap.request.modify_request.object
ldap.request.del_request.dn
ldap.request.mod_dn_request.entry
ldap.request.compare_request.entry
It is a sticky buffer
Supports prefiltering
Ticket: #7471
If a Suricata inline IPS device is routing traffic over a
non-encrypted tunnel, like IPv6 tunnels, packets in a flow
will be dropped and not be matched. e.g.
The following example is a Suricata inline IPS with an IPv6 tunnel:
request: IPv4]ICMP] -> |IPS| -> IPv6]IPv4]ICMP]
reply: <- |IPS| <- IPv6]IPv4]ICMP]
Both the IPv4 request and IPv6 reply will be seen by Suricata on
ingress. The flows will not be matched due to flow recursion level.
Optionally use pkt recursion level in flow hash. Excluding recursion
level in flow hash allows matching of packet flows and defrag on an
inline IPS Suricata scenario where the IPS device is a tunnel
terminator.
Feature: 6260
ldap.responses.count matches on the number of LDAP responses
This keyword maps to the eve field len(ldap.responses[])
It is an unsigned 32-bit integer
Doesn't support prefiltering
Ticket: #7453
ldap.responses.operation matches on Lightweight Directory Access Protocol response operations
This keyword maps to the eve field ldap.responses[].operation
It is an unsigned 8-bit integer
Doesn't support prefiltering
Ticket: #7453
ldap.request.operation matches on Lightweight Directory Access Protocol request operations
This keyword maps to the eve field ldap.request.operation
It is an unsigned 8-bit integer
Doesn't support prefiltering
Ticket: #7453
Describe how to use the git commit template. The template helps ensure
that the information needed for evaluation and context is included in
the commit message.
Ticket: <Redmine ticket number>
Ticket: 7199
Uses a config parameter detect.guess-applayer-tx to enable
this behavior (off by default)
This feature is requested for use cases with signatures not
using app-layer keywords but still targetting application
layer transactions, such as pass/drop rule combination,
or lua usage.
This overrides the previous behavior of checking if the signature
has a content match, by checking if there is only one live
transaction, in addition to the config parameter being set.
Similar keywords use `isnotset`, while `flowint` only accepted `notset`
Opted to change the code, not only the regex, to keep the underlying
code also following the same patterns.
Task #7426
Lukas Sismis
Philippe Antoine
Juliana Fajardini
Jeff Lucovsky
Philippe Antoine
Todd Mortimer
Shivani Bhardwaj
Alice Akaki
Giuseppe Longo
Jason Ish
Alfredo Cardigliano
Victor Julien
Juliana Fajardini
Joyce Yu
Nathan Scrivens
AlirezaPourchali
Cole Dishington