Decouple app.protocols.tls.encryption-handling and stream.bypass.
There's no apparent reason why encrypted TLS bypass traffic should
depend on stream bypass, as these are unrelated features.
Ticket: 6788
The default value for TLS encryption is generic and not precise.
The new keyword, "track-only" substitues the old keyword, being
more on point what action is actually taken when the TLS flow
reaches an encrypted state.
suricata.yaml output section for smb now parses a types list
and will restrict logging of transactions to these types.
By default, everything still gets logged
Remove unused rs_smb_log_json_request on the way
Ticket: 7620
Ticket: #2696
There are a lot of changes here, which are described below.
In general these changes are renaming constants to conform to the
libhtp-rs versions (which are generated by cbindgen); making all htp
types opaque and changing struct->member references to
htp_struct_member() function calls; and a handful of changes to offload
functionality onto libhtp-rs from suricata, such as URI normalization
and transaction cleanup.
Functions introduced to handle opaque htp_tx_t:
- tx->parsed_uri => htp_tx_parsed_uri(tx)
- tx->parsed_uri->path => htp_uri_path(htp_tx_parsed_uri(tx)
- tx->parsed_uri->hostname => htp_uri_hostname(htp_tx_parsed_uri(tx))
- htp_tx_get_user_data() => htp_tx_user_data(tx)
- htp_tx_is_http_2_upgrade(tx) convenience function introduced to detect response status 101
and “Upgrade: h2c" header.
Functions introduced to handle opaque htp_tx_data_t:
- d->len => htp_tx_data_len()
- d->data => htp_tx_data_data()
- htp_tx_data_tx(data) function to get the htp_tx_t from the htp_tx_data_t
- htp_tx_data_is_empty(data) convenience function introduced to test if the data is empty.
Other changes:
Build libhtp-rs as a crate inside rust. Update autoconf to no longer
use libhtp as an external dependency. Remove HAVE_HTP feature defines
since they are no longer needed.
Make function arguments and return values const where possible
htp_tx_destroy(tx) will now free an incomplete transaction
htp_time_t replaced with standard struct timeval
Callbacks from libhtp now provide the htp_connp_t and the htp_tx_data_t
as separate arguments. This means the connection parser is no longer
fetched from the transaction inside callbacks.
SCHTPGenerateNormalizedUri() functionality moved inside libhtp-rs, which
now provides normalized URI values.
The normalized URI is available with accessor function: htp_tx_normalized_uri()
Configuration settings added to control the behaviour of the URI normalization:
- htp_config_set_normalized_uri_include_all()
- htp_config_set_plusspace_decode()
- htp_config_set_convert_lowercase()
- htp_config_set_double_decode_normalized_query()
- htp_config_set_double_decode_normalized_path()
- htp_config_set_backslash_convert_slashes()
- htp_config_set_bestfit_replacement_byte()
- htp_config_set_convert_lowercase()
- htp_config_set_nul_encoded_terminates()
- htp_config_set_nul_raw_terminates()
- htp_config_set_path_separators_compress()
- htp_config_set_path_separators_decode()
- htp_config_set_u_encoding_decode()
- htp_config_set_url_encoding_invalid_handling()
- htp_config_set_utf8_convert_bestfit()
- htp_config_set_normalized_uri_include_all()
- htp_config_set_plusspace_decode()
Constants related to configuring uri normalization:
- HTP_URL_DECODE_PRESERVE_PERCENT => HTP_URL_ENCODING_HANDLING_PRESERVE_PERCENT
- HTP_URL_DECODE_REMOVE_PERCENT => HTP_URL_ENCODING_HANDLING_REMOVE_PERCENT
- HTP_URL_DECODE_PROCESS_INVALID => HTP_URL_ENCODING_HANDLING_PROCESS_INVALID
htp_config_set_field_limits(soft_limit, hard_limit) changed to
htp_config_set_field_limit(limit) because libhtp didn't implement soft
limits.
libhtp logging API updated to provide HTP_LOG_CODE constants along with
the message. This eliminates the need to perform string matching on
message text to map log messages to HTTP_DECODER_EVENT values, and the
HTP_LOG_CODE values can be used directly. In support of this,
HTP_DECODER_EVENT values are mapped to their corresponding HTP_LOG_CODE
values.
New log events to describe additional anomalies:
HTP_LOG_CODE_REQUEST_TOO_MANY_LZMA_LAYERS
HTP_LOG_CODE_RESPONSE_TOO_MANY_LZMA_LAYERS
HTP_LOG_CODE_PROTOCOL_CONTAINS_EXTRA_DATA
HTP_LOG_CODE_CONTENT_LENGTH_EXTRA_DATA_START
HTP_LOG_CODE_CONTENT_LENGTH_EXTRA_DATA_END
HTP_LOG_CODE_SWITCHING_PROTO_WITH_CONTENT_LENGTH
HTP_LOG_CODE_DEFORMED_EOL
HTP_LOG_CODE_PARSER_STATE_ERROR
HTP_LOG_CODE_MISSING_OUTBOUND_TRANSACTION_DATA
HTP_LOG_CODE_MISSING_INBOUND_TRANSACTION_DATA
HTP_LOG_CODE_ZERO_LENGTH_DATA_CHUNKS
HTP_LOG_CODE_REQUEST_LINE_UNKNOWN_METHOD
HTP_LOG_CODE_REQUEST_LINE_UNKNOWN_METHOD_NO_PROTOCOL
HTP_LOG_CODE_REQUEST_LINE_UNKNOWN_METHOD_INVALID_PROTOCOL
HTP_LOG_CODE_REQUEST_LINE_NO_PROTOCOL
HTP_LOG_CODE_RESPONSE_LINE_INVALID_PROTOCOL
HTP_LOG_CODE_RESPONSE_LINE_INVALID_RESPONSE_STATUS
HTP_LOG_CODE_RESPONSE_BODY_INTERNAL_ERROR
HTP_LOG_CODE_REQUEST_BODY_DATA_CALLBACK_ERROR
HTP_LOG_CODE_RESPONSE_INVALID_EMPTY_NAME
HTP_LOG_CODE_REQUEST_INVALID_EMPTY_NAME
HTP_LOG_CODE_RESPONSE_INVALID_LWS_AFTER_NAME
HTP_LOG_CODE_RESPONSE_HEADER_NAME_NOT_TOKEN
HTP_LOG_CODE_REQUEST_INVALID_LWS_AFTER_NAME
HTP_LOG_CODE_LZMA_DECOMPRESSION_DISABLED
HTP_LOG_CODE_CONNECTION_ALREADY_OPEN
HTP_LOG_CODE_COMPRESSION_BOMB_DOUBLE_LZMA
HTP_LOG_CODE_INVALID_CONTENT_ENCODING
HTP_LOG_CODE_INVALID_GAP
HTP_LOG_CODE_ERROR
The new htp_log API supports consuming log messages more easily than
walking a list and tracking the current offset. Internally, libhtp-rs
now provides log messages as a queue of htp_log_t, which means the
application can simply call htp_conn_next_log() to fetch the next log
message until the queue is empty. Once the application is done with a
log message, they can call htp_log_free() to dispose of it.
Functions supporting htp_log_t:
htp_conn_next_log(conn) - Get the next log message
htp_log_message(log) - To get the text of the message
htp_log_code(log) - To get the HTP_LOG_CODE value
htp_log_free(log) - To free the htp_log_t
email.x_mailer matches on MIME EMAIL X-Mailer
This keyword maps to the EVE field email.x_mailer
It is a sticky buffer
Supports prefiltering
Ticket: #7598
email.message_id matches on MIME EMAIL Message-Id
This keyword maps to the EVE field email.message_id
It is a sticky buffer
Supports prefiltering
Ticket: #7593
This commits adds documentation for the entropy keyword.
The entropy keyword calculates the Shannon entropy value for content
with the calculated value used to determine whether an alert occurs.
When running in non-forwarding (IDS) mode, it is not required
to create TX queues for the interface.
This can be acheived by setting tx-descriptors configuration
field to 0.
Ticket: 7633
ICE card (Intel E810) was not receiving packets immediatelly
after startup, Suricata workers would act as processing while
it was not. This eliminates the problem by only continuing
in the initialization if the link is already up.
The setting can be turned off manually from the configuraiton
file.
Ticket: 7381
It turned out that having global (interface-specific) mempool
that is shared by the threads of the interface is slower than
having individual mempools per queue for each interface.
The commit brings this change and should be user-invisible,
the config setting remains still as a number of objects of
all mempools summed (of that interface).
Ticket: 7382
Moves the nDPI documentation to an nDPI page in the plugins
section. Remove the duplication of installation and setup
documentation.
Includes some minor cleanups.
Cache Hyperscan serialized databases to disk to prevent compilation
of the same databases when Suricata is run again with the same
ruleset.
Hyperscan binary files are stored per rulegroup in the designated
folder, by default in the cached library folder.
Since caching is per signature group heads,
some chunk of the ruleset can change and it still can reuse part of
the unchanged signature groups.
Loading *fresh* ET Open ruleset: 19 seconds
Loading *cached* ET Open ruleset: 07 seconds
Ticket: 7170
The stats for exception policies are only logged/ present when any of
the exception policies are enabled (which means any value other than
"auto" or "ignore" in IDS mode, or "ignore" in IPS mode).
This wasn't clearly stated in the docs.
To accompany the Exception Policy stats, also add information about any
Exception Policy triggered and for which target to the flow log event.
Task #6215
Ticket: 7613
Avoids potential large per-thread memory allocation. A buffer with the
size of the largest decode_base64 buffer size setting would be allocated
per thread. As this was a u32, it could mean a per-thread 4GiB memory
allocation.
64KiB was already the built-in default for cases where bytes size wasn't
specified.
We were mentioning "Suricata Support" page, which could be a bit
misleading -- and also used a link that is actually redirected to the
Suricata Community page, anyways.
Add keywords dns.additionals.rrname and dns.authorities.rrname. Along
the way, consolidate dns.query.name and dns.answer.name into a single file
and register them altogether since there is a lot of common code.
ldap.responses.message matches on LDAPResult error message
This keyword maps the following eve fields:
ldap.responses[].bind_response.message
ldap.responses[].search_result_done.message
ldap.responses[].modify_response.message
ldap.responses[].add_response.message
ldap.responses[].del_response.message
ldap.responses[].mod_dn_response.message
ldap.responses[].compare_response.message
ldap.responses[].extended_response.message
It is a sticky buffer
Supports prefiltering
Ticket: #7532
ldap.responses.result_code matches on LDAP result code
This keyword maps the following eve fields:
ldap.responses[].bind_response.result_code
ldap.responses[].search_result_done.result_code
ldap.responses[].modify_response.result_code
ldap.responses[].add_response.result_code
ldap.responses[].del_response.result_code
ldap.responses[].mod_dn_response.result_code
ldap.responses[].compare_response.result_code
ldap.responses[].extended_response.result_code
It is an unsigned 32-bit integer
Doesn't support prefiltering
Ticket: #7532
Funciton ldap_tx_get_responses_dn returns empty buffer in case
the response doesn't contain the distinguished name field
Fixes: 73ae6e997f ("detect: add ldap.responses.dn")
Change variable name 'req' to 'resp' in function ldap_tx_get_responses_dn and documentation nits
Fixes:
73ae6e997f ("detect: add ldap.responses.dn")
16dcee46fc ("detect: add ldap.request.dn")
Issue: 4082
Move the configuration file handling to Rust.
These changes will no longer terminate Suricata when there's an invalid
value for ftp.memcap. Like earlier Suricata releases, an error message
is logged "Invalid value <value> for ftp.memcap" but Suricata will no
longer terminate execution. It will use a default value of "0" instead.
ldap.responses.dn matches on LDAPDN from responses operations
This keyword maps the following eve fields:
ldap.responses[].search_result_entry.base_object
ldap.responses[].bind_response.matched_dn
ldap.responses[].search_result_done.matched_dn
ldap.responses[].modify_response.matched_dn
ldap.responses[].add_response.matched_dn
ldap.responses[].del_response.matched_dn
ldap.responses[].mod_dn_response.matched_dn
ldap.responses[].compare_response.matched_dn
ldap.responses[].extended_response.matched_dn
It is a sticky buffer
Supports prefiltering
Ticket: #7471
ldap.request.dn matches on LDAPDN from request operations
This keyword maps the following eve fields:
ldap.request.bind_request.name
ldap.request.add_request.entry
ldap.request.search_request.base_object
ldap.request.modify_request.object
ldap.request.del_request.dn
ldap.request.mod_dn_request.entry
ldap.request.compare_request.entry
It is a sticky buffer
Supports prefiltering
Ticket: #7471
If a Suricata inline IPS device is routing traffic over a
non-encrypted tunnel, like IPv6 tunnels, packets in a flow
will be dropped and not be matched. e.g.
The following example is a Suricata inline IPS with an IPv6 tunnel:
request: IPv4]ICMP] -> |IPS| -> IPv6]IPv4]ICMP]
reply: <- |IPS| <- IPv6]IPv4]ICMP]
Both the IPv4 request and IPv6 reply will be seen by Suricata on
ingress. The flows will not be matched due to flow recursion level.
Optionally use pkt recursion level in flow hash. Excluding recursion
level in flow hash allows matching of packet flows and defrag on an
inline IPS Suricata scenario where the IPS device is a tunnel
terminator.
Feature: 6260
ldap.responses.count matches on the number of LDAP responses
This keyword maps to the eve field len(ldap.responses[])
It is an unsigned 32-bit integer
Doesn't support prefiltering
Ticket: #7453
ldap.responses.operation matches on Lightweight Directory Access Protocol response operations
This keyword maps to the eve field ldap.responses[].operation
It is an unsigned 8-bit integer
Doesn't support prefiltering
Ticket: #7453
ldap.request.operation matches on Lightweight Directory Access Protocol request operations
This keyword maps to the eve field ldap.request.operation
It is an unsigned 8-bit integer
Doesn't support prefiltering
Ticket: #7453
Describe how to use the git commit template. The template helps ensure
that the information needed for evaluation and context is included in
the commit message.
Ticket: <Redmine ticket number>
Ticket: 7199
Uses a config parameter detect.guess-applayer-tx to enable
this behavior (off by default)
This feature is requested for use cases with signatures not
using app-layer keywords but still targetting application
layer transactions, such as pass/drop rule combination,
or lua usage.
This overrides the previous behavior of checking if the signature
has a content match, by checking if there is only one live
transaction, in addition to the config parameter being set.
Similar keywords use `isnotset`, while `flowint` only accepted `notset`
Opted to change the code, not only the regex, to keep the underlying
code also following the same patterns.
Task #7426
For example, "requires: foo bar" is an unknown requirement, however
its not tracked, nor an error as it follows the syntax. Instead,
record these unknown keywords, and fail the requirements check if any
are present.
A future version of Suricata may have new requires keywords, for
example a check for keywords.
Ticket: #7418
Ticket: 2224
It takes an argument to match only if the buffer is absent,
or it can still match if the buffer is present, but we test
the absence of some content.
For multi buffers, absent matches if there are 0 buffers.
For file keywords, absent matches if there is no file.
Since enabling custom logging will replace the extended logging, thus
possibly leading to certain fields disappearing from the logs, mention
this aspect.
Related to
Bug #7333
Add EVE documentation for QUIC and Pgsql to their respective sections of
the userguide.
Also add a complete EVE reference as an appendix.
Other protocols can be done, but its a manual process to document in the
schema, then add the glue to pull them into the documentation.
The documentation is generated during "make dist", or if it doesn't
exist, "conf.py" will attempt to generate the eve documentation for
building on Readthedocs.
Ticket: 7191
So as to avoid quadratic complexity in libhtp.
Make the limit configurable from suricata.yaml,
and have an event when network traffic goes over the limit.
Use sed + regex to replace all occurrences of suricata.yaml terms that
used underscore for their up-to-date dash version.
Also search for such terms in the eve-log.yaml partials file, as that
is referenced in the configuration section.
commands used:
sed -i 's/\(^ *[a-z]*\)_\([a-z]*:\)/\1-\2/g'
sed -i 's/\(^ *[a-z]*\)_\([a-z]*\)_\([a-z]*:\)/\1-\2-\3/g'
Some other instances were found manually.
Task #7260
If a stream-only rule matches, and we find a tx where we
want to log the app-layer data, store into the tx data that
we already logged, so that we do not log again the app-layer metadata
Ticket: 7085
Although the `prune-flows` option was removed with a5587fec2e,
when documentation for the suricata.yaml config file was added with
b252b0d, this option was also included - as has remained until now.
They have updated their docs domain, leading to the link we had
returning a 404.
Also checked the other links. Although some seem to only contain old
traffic, they all still work.
This ensures that the correct spelling of the `security_result` EVE
field for RFB (as opposed to `security-result`) is also reflected in the
documentation.
Ticket: #7210
While not incorrect, the previous wording made the sentence almost
paradoxical. While at it, also highlight a side effect that might not be
so clear to users.
Related to
Bug #6976
Ticket: 5734
Adds frames for SSH records, that come after banner, and before
the data is encrypted.
These records may contain cipher lists for instance.
Ticket: 4863
On the way, convert some keywords to use the first-class integer
support.
And helpers for pure rust the support for multi-buffer.
Move the C unit tests about keyword mqtt.protocol_version
to unit tests for generic integer parsing, and test version 5
instead of testing twice version 3.
Also iterate all tx's messages for reason code as is done for other
keywords.
And allow detection on empty topics.
Dean Balandin
Lukas Sismis
Philippe Antoine
Juliana Fajardini
Jeff Lucovsky
Philippe Antoine
Todd Mortimer
Shivani Bhardwaj
Alice Akaki
Giuseppe Longo
Jason Ish
Alfredo Cardigliano
Victor Julien
Juliana Fajardini
Joyce Yu
Nathan Scrivens
AlirezaPourchali
Cole Dishington
Sascha Steinbiss
jason taylor
Jeff Lucovsky