Add live tests to verify that mempool, mempool cache sizes, and
potentially other DPDK features are initialized correctly during the
startup.
Follows up on bugs found in mempool settings on real/bonded devices.
The tests are placed in the new folder qa/live/ after a comment
in https://github.com/OISF/suricata/pull/14896 which indicated other
live tests are going to be moved there as well.
Ticket: 8320
The Rust build in rust.yml was just an AlmaLinux 9 build with some extra
checks. Instead, use an existing AlmaLinux 9 build in builds.yml, make
it use Rustup (there are other AlmaLinux 9 builds that use the RPM), and
add the checks.
Saves us one build in CI.
Implementing a bridge test with af-packet IPS and a routing test
with iptables + nfqueue.
Very helpful explanation and guidance on network namespaces can be
found here: https://www.redhat.com/en/blog/net-namespaces
Sets up 3 network namespaces:
1. client - for running client tools like ping, curl, wget
2. server - for running a server, currently only Caddy
3. dut - for running Suricata, this namespace connects the client and
server namespaces
Validate IPS operations in 3 ways:
1. check return codes of the client tools
2. check Suricata's IPS stats
3. use tshark to validate expected drops
Run Suricata in AF_PACKET IPS mode for both autofp and workers mode. Do
the same for NFQUEUE.
Tshark's JSON output is used with JQ to validate that pings are dropped.
All tests are codecov enabled.
To support alternative cargo and rustc programs (such as cargo-1.82),
respect CARGO and RUSTC environment variables during ./configure much
like CC.
RUSTFMT is also respected as that is required for the tests, and Cargo
can't figure this out like it can for rustc (perhaps a bug in the
packaging).
For cbindgen, we have also have to make sure the cargo environment
variable is set for each invocation.
To build with Ubuntu's Rust 1.82 packaging:
CARGO=cargo-1.82 RUSTC=rustc-1.82 RUSTDOC=rustdoc-1.82 \
./configure
Note that setting RUSTDOC is only required for commands like "make
check" to pass.
Ticket: #7877
Based on the current AlmaLinux 9 build, with plugin tests, etc.
Remove cppclean as its not installed and was previously disabled with
commit 2d308c000d.
Ubuntu and Fedora packing system build with -flto=auto by default, so
update one test to use -flto=auto. Also build with -O2 as that
combination can cause issues such as
https://redmine.openinfosecfoundation.org/issues/7824.
Also adds vectorscan to the build.
tzname is a POSIX variable, WIN32 has prefixed many POSIX variables
with "_". While Mingw64 supports both, UCRT64 emits a compiler warning
on the usage of "tzname".
This triggered a rather large clang-format update.
Ticket: #2696
There are a lot of changes here, which are described below.
In general these changes are renaming constants to conform to the
libhtp-rs versions (which are generated by cbindgen); making all htp
types opaque and changing struct->member references to
htp_struct_member() function calls; and a handful of changes to offload
functionality onto libhtp-rs from suricata, such as URI normalization
and transaction cleanup.
Functions introduced to handle opaque htp_tx_t:
- tx->parsed_uri => htp_tx_parsed_uri(tx)
- tx->parsed_uri->path => htp_uri_path(htp_tx_parsed_uri(tx)
- tx->parsed_uri->hostname => htp_uri_hostname(htp_tx_parsed_uri(tx))
- htp_tx_get_user_data() => htp_tx_user_data(tx)
- htp_tx_is_http_2_upgrade(tx) convenience function introduced to detect response status 101
and “Upgrade: h2c" header.
Functions introduced to handle opaque htp_tx_data_t:
- d->len => htp_tx_data_len()
- d->data => htp_tx_data_data()
- htp_tx_data_tx(data) function to get the htp_tx_t from the htp_tx_data_t
- htp_tx_data_is_empty(data) convenience function introduced to test if the data is empty.
Other changes:
Build libhtp-rs as a crate inside rust. Update autoconf to no longer
use libhtp as an external dependency. Remove HAVE_HTP feature defines
since they are no longer needed.
Make function arguments and return values const where possible
htp_tx_destroy(tx) will now free an incomplete transaction
htp_time_t replaced with standard struct timeval
Callbacks from libhtp now provide the htp_connp_t and the htp_tx_data_t
as separate arguments. This means the connection parser is no longer
fetched from the transaction inside callbacks.
SCHTPGenerateNormalizedUri() functionality moved inside libhtp-rs, which
now provides normalized URI values.
The normalized URI is available with accessor function: htp_tx_normalized_uri()
Configuration settings added to control the behaviour of the URI normalization:
- htp_config_set_normalized_uri_include_all()
- htp_config_set_plusspace_decode()
- htp_config_set_convert_lowercase()
- htp_config_set_double_decode_normalized_query()
- htp_config_set_double_decode_normalized_path()
- htp_config_set_backslash_convert_slashes()
- htp_config_set_bestfit_replacement_byte()
- htp_config_set_convert_lowercase()
- htp_config_set_nul_encoded_terminates()
- htp_config_set_nul_raw_terminates()
- htp_config_set_path_separators_compress()
- htp_config_set_path_separators_decode()
- htp_config_set_u_encoding_decode()
- htp_config_set_url_encoding_invalid_handling()
- htp_config_set_utf8_convert_bestfit()
- htp_config_set_normalized_uri_include_all()
- htp_config_set_plusspace_decode()
Constants related to configuring uri normalization:
- HTP_URL_DECODE_PRESERVE_PERCENT => HTP_URL_ENCODING_HANDLING_PRESERVE_PERCENT
- HTP_URL_DECODE_REMOVE_PERCENT => HTP_URL_ENCODING_HANDLING_REMOVE_PERCENT
- HTP_URL_DECODE_PROCESS_INVALID => HTP_URL_ENCODING_HANDLING_PROCESS_INVALID
htp_config_set_field_limits(soft_limit, hard_limit) changed to
htp_config_set_field_limit(limit) because libhtp didn't implement soft
limits.
libhtp logging API updated to provide HTP_LOG_CODE constants along with
the message. This eliminates the need to perform string matching on
message text to map log messages to HTTP_DECODER_EVENT values, and the
HTP_LOG_CODE values can be used directly. In support of this,
HTP_DECODER_EVENT values are mapped to their corresponding HTP_LOG_CODE
values.
New log events to describe additional anomalies:
HTP_LOG_CODE_REQUEST_TOO_MANY_LZMA_LAYERS
HTP_LOG_CODE_RESPONSE_TOO_MANY_LZMA_LAYERS
HTP_LOG_CODE_PROTOCOL_CONTAINS_EXTRA_DATA
HTP_LOG_CODE_CONTENT_LENGTH_EXTRA_DATA_START
HTP_LOG_CODE_CONTENT_LENGTH_EXTRA_DATA_END
HTP_LOG_CODE_SWITCHING_PROTO_WITH_CONTENT_LENGTH
HTP_LOG_CODE_DEFORMED_EOL
HTP_LOG_CODE_PARSER_STATE_ERROR
HTP_LOG_CODE_MISSING_OUTBOUND_TRANSACTION_DATA
HTP_LOG_CODE_MISSING_INBOUND_TRANSACTION_DATA
HTP_LOG_CODE_ZERO_LENGTH_DATA_CHUNKS
HTP_LOG_CODE_REQUEST_LINE_UNKNOWN_METHOD
HTP_LOG_CODE_REQUEST_LINE_UNKNOWN_METHOD_NO_PROTOCOL
HTP_LOG_CODE_REQUEST_LINE_UNKNOWN_METHOD_INVALID_PROTOCOL
HTP_LOG_CODE_REQUEST_LINE_NO_PROTOCOL
HTP_LOG_CODE_RESPONSE_LINE_INVALID_PROTOCOL
HTP_LOG_CODE_RESPONSE_LINE_INVALID_RESPONSE_STATUS
HTP_LOG_CODE_RESPONSE_BODY_INTERNAL_ERROR
HTP_LOG_CODE_REQUEST_BODY_DATA_CALLBACK_ERROR
HTP_LOG_CODE_RESPONSE_INVALID_EMPTY_NAME
HTP_LOG_CODE_REQUEST_INVALID_EMPTY_NAME
HTP_LOG_CODE_RESPONSE_INVALID_LWS_AFTER_NAME
HTP_LOG_CODE_RESPONSE_HEADER_NAME_NOT_TOKEN
HTP_LOG_CODE_REQUEST_INVALID_LWS_AFTER_NAME
HTP_LOG_CODE_LZMA_DECOMPRESSION_DISABLED
HTP_LOG_CODE_CONNECTION_ALREADY_OPEN
HTP_LOG_CODE_COMPRESSION_BOMB_DOUBLE_LZMA
HTP_LOG_CODE_INVALID_CONTENT_ENCODING
HTP_LOG_CODE_INVALID_GAP
HTP_LOG_CODE_ERROR
The new htp_log API supports consuming log messages more easily than
walking a list and tracking the current offset. Internally, libhtp-rs
now provides log messages as a queue of htp_log_t, which means the
application can simply call htp_conn_next_log() to fetch the next log
message until the queue is empty. Once the application is done with a
log message, they can call htp_log_free() to dispose of it.
Functions supporting htp_log_t:
htp_conn_next_log(conn) - Get the next log message
htp_log_message(log) - To get the text of the message
htp_log_code(log) - To get the HTP_LOG_CODE value
htp_log_free(log) - To free the htp_log_t
The RPM has been updated to handle the conversion of suricatasc and
suricatactl to Rust.
Also fixes the "if" expression to prevent one job running both upload
sections.
There will be changes in our development branch that the RPMs need to
adapt to, but that can't be done until the changes have been merged to
master, then the RPM can catchup.
This gives us a single variable to turn off RPM building.
Victor Julien
dependabot[bot]
Lukas Sismis
Jason Ish
Philippe Antoine
Ambre Iooss
Jeff Lucovsky
Shivani Bhardwaj
Todd Mortimer