You cannot select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
suricata/doc/userguide/devguide/codebase/testing.rst

159 lines
8.1 KiB
ReStructuredText

****************
Testing Suricata
****************
.. contents:: Table of Contents
General Concepts
================
There are a few ways of testing Suricata:
- **Unit tests**: for independently checking specific functions or portions of code. This guide has specific sections to
further explain those, for C and Rust;
- `Suricata-Verify <https://github.com/OISF/suricata-verify>`_: those are used to check more complex behavior, like the log output or the alert counts for a given input, where that input is usually comprised of several packets;
- **Static and dynamic analysis tools**: to help in finding bugs, memory leaks and other issues (like `scan-build <https://clang-analyzer.llvm.org/scan-build.html#scanbuild_basicusage>`_, from `clang`, which is also used for our C formatting checks; or ASAN, which checks for memory issues);
- **Fuzz testing**: especially good for uncovering existing, often non-trivial bugs. For more on how to fuzz test Suricata, check :doc:`fuzz-testing`;
- **CI checks**: each PR submitted to the project's public repositories will be run against a suit of Continuous Integration
workflows, as part of our QA process. Those cover: formatting and commit checks; fuzz tests (CI Fuzz), and several builds. See our `github workflows <https://github.com/OISF/suricata/tree/master/.github/workflows>`_ for details and those in
action at `<https://github.com/OISF/suricata/actions>`_.
.. note:: If you can run unit tests or other checks and report failures in our `issue tracker <https://redmine.openinfosecfoundation.org/projects/suricata/issues>`_, that is rather useful and appreciated!
The focus of this document are Unit tests and Suricata-Verify tests, especially on offering some guidance regarding when to use each type of test, and how to prepare input
for them.
Unit tests
==========
Use these to check that specific functions behave as expected, in success and in failure scenarios. Specially useful
during development, for nom parsers in the Rust codebase, for instance, or for checking that messages
or message parts of a protocol/stream are processed as they should.
http: Use libhtp-rs. Ticket: #2696 There are a lot of changes here, which are described below. In general these changes are renaming constants to conform to the libhtp-rs versions (which are generated by cbindgen); making all htp types opaque and changing struct->member references to htp_struct_member() function calls; and a handful of changes to offload functionality onto libhtp-rs from suricata, such as URI normalization and transaction cleanup. Functions introduced to handle opaque htp_tx_t: - tx->parsed_uri => htp_tx_parsed_uri(tx) - tx->parsed_uri->path => htp_uri_path(htp_tx_parsed_uri(tx) - tx->parsed_uri->hostname => htp_uri_hostname(htp_tx_parsed_uri(tx)) - htp_tx_get_user_data() => htp_tx_user_data(tx) - htp_tx_is_http_2_upgrade(tx) convenience function introduced to detect response status 101 and “Upgrade: h2c" header. Functions introduced to handle opaque htp_tx_data_t: - d->len => htp_tx_data_len() - d->data => htp_tx_data_data() - htp_tx_data_tx(data) function to get the htp_tx_t from the htp_tx_data_t - htp_tx_data_is_empty(data) convenience function introduced to test if the data is empty. Other changes: Build libhtp-rs as a crate inside rust. Update autoconf to no longer use libhtp as an external dependency. Remove HAVE_HTP feature defines since they are no longer needed. Make function arguments and return values const where possible htp_tx_destroy(tx) will now free an incomplete transaction htp_time_t replaced with standard struct timeval Callbacks from libhtp now provide the htp_connp_t and the htp_tx_data_t as separate arguments. This means the connection parser is no longer fetched from the transaction inside callbacks. SCHTPGenerateNormalizedUri() functionality moved inside libhtp-rs, which now provides normalized URI values. The normalized URI is available with accessor function: htp_tx_normalized_uri() Configuration settings added to control the behaviour of the URI normalization: - htp_config_set_normalized_uri_include_all() - htp_config_set_plusspace_decode() - htp_config_set_convert_lowercase() - htp_config_set_double_decode_normalized_query() - htp_config_set_double_decode_normalized_path() - htp_config_set_backslash_convert_slashes() - htp_config_set_bestfit_replacement_byte() - htp_config_set_convert_lowercase() - htp_config_set_nul_encoded_terminates() - htp_config_set_nul_raw_terminates() - htp_config_set_path_separators_compress() - htp_config_set_path_separators_decode() - htp_config_set_u_encoding_decode() - htp_config_set_url_encoding_invalid_handling() - htp_config_set_utf8_convert_bestfit() - htp_config_set_normalized_uri_include_all() - htp_config_set_plusspace_decode() Constants related to configuring uri normalization: - HTP_URL_DECODE_PRESERVE_PERCENT => HTP_URL_ENCODING_HANDLING_PRESERVE_PERCENT - HTP_URL_DECODE_REMOVE_PERCENT => HTP_URL_ENCODING_HANDLING_REMOVE_PERCENT - HTP_URL_DECODE_PROCESS_INVALID => HTP_URL_ENCODING_HANDLING_PROCESS_INVALID htp_config_set_field_limits(soft_limit, hard_limit) changed to htp_config_set_field_limit(limit) because libhtp didn't implement soft limits. libhtp logging API updated to provide HTP_LOG_CODE constants along with the message. This eliminates the need to perform string matching on message text to map log messages to HTTP_DECODER_EVENT values, and the HTP_LOG_CODE values can be used directly. In support of this, HTP_DECODER_EVENT values are mapped to their corresponding HTP_LOG_CODE values. New log events to describe additional anomalies: HTP_LOG_CODE_REQUEST_TOO_MANY_LZMA_LAYERS HTP_LOG_CODE_RESPONSE_TOO_MANY_LZMA_LAYERS HTP_LOG_CODE_PROTOCOL_CONTAINS_EXTRA_DATA HTP_LOG_CODE_CONTENT_LENGTH_EXTRA_DATA_START HTP_LOG_CODE_CONTENT_LENGTH_EXTRA_DATA_END HTP_LOG_CODE_SWITCHING_PROTO_WITH_CONTENT_LENGTH HTP_LOG_CODE_DEFORMED_EOL HTP_LOG_CODE_PARSER_STATE_ERROR HTP_LOG_CODE_MISSING_OUTBOUND_TRANSACTION_DATA HTP_LOG_CODE_MISSING_INBOUND_TRANSACTION_DATA HTP_LOG_CODE_ZERO_LENGTH_DATA_CHUNKS HTP_LOG_CODE_REQUEST_LINE_UNKNOWN_METHOD HTP_LOG_CODE_REQUEST_LINE_UNKNOWN_METHOD_NO_PROTOCOL HTP_LOG_CODE_REQUEST_LINE_UNKNOWN_METHOD_INVALID_PROTOCOL HTP_LOG_CODE_REQUEST_LINE_NO_PROTOCOL HTP_LOG_CODE_RESPONSE_LINE_INVALID_PROTOCOL HTP_LOG_CODE_RESPONSE_LINE_INVALID_RESPONSE_STATUS HTP_LOG_CODE_RESPONSE_BODY_INTERNAL_ERROR HTP_LOG_CODE_REQUEST_BODY_DATA_CALLBACK_ERROR HTP_LOG_CODE_RESPONSE_INVALID_EMPTY_NAME HTP_LOG_CODE_REQUEST_INVALID_EMPTY_NAME HTP_LOG_CODE_RESPONSE_INVALID_LWS_AFTER_NAME HTP_LOG_CODE_RESPONSE_HEADER_NAME_NOT_TOKEN HTP_LOG_CODE_REQUEST_INVALID_LWS_AFTER_NAME HTP_LOG_CODE_LZMA_DECOMPRESSION_DISABLED HTP_LOG_CODE_CONNECTION_ALREADY_OPEN HTP_LOG_CODE_COMPRESSION_BOMB_DOUBLE_LZMA HTP_LOG_CODE_INVALID_CONTENT_ENCODING HTP_LOG_CODE_INVALID_GAP HTP_LOG_CODE_ERROR The new htp_log API supports consuming log messages more easily than walking a list and tracking the current offset. Internally, libhtp-rs now provides log messages as a queue of htp_log_t, which means the application can simply call htp_conn_next_log() to fetch the next log message until the queue is empty. Once the application is done with a log message, they can call htp_log_free() to dispose of it. Functions supporting htp_log_t: htp_conn_next_log(conn) - Get the next log message htp_log_message(log) - To get the text of the message htp_log_code(log) - To get the HTP_LOG_CODE value htp_log_free(log) - To free the htp_log_t
2 years ago
To execute all unit tests (both from C and Rust code) from the Suricata main directory, run::
make check
Check the Suricata Devguide on :doc:`unittests-c` or :doc:`unittests-rust` for more on how to write and run unit tests,
given that the way to do so differs, depending on the language.
Code Examples
^^^^^^^^^^^^^
An example from the `DNS parser <https://github.com/OISF/suricata/blob/master/rust/src/dns/parser.rs#L417>`_. This
checks that the given raw input (note the comments indicating what it means), once processed by ``dns_parse_name`` yields
the expected result, including the unparsed portion.
.. code-block:: rust
/// Parse a simple name with no pointers.
#[test]
fn test_dns_parse_name() {
let buf: &[u8] = &[
0x09, 0x63, /* .......c */
0x6c, 0x69, 0x65, 0x6e, 0x74, 0x2d, 0x63, 0x66, /* lient-cf */
0x07, 0x64, 0x72, 0x6f, 0x70, 0x62, 0x6f, 0x78, /* .dropbox */
0x03, 0x63, 0x6f, 0x6d, 0x00, 0x00, 0x01, 0x00, /* .com.... */
];
let expected_remainder: &[u8] = &[0x00, 0x01, 0x00];
let (remainder,name) = dns_parse_name(buf, buf).unwrap();
assert_eq!("client-cf.dropbox.com".as_bytes(), &name[..]);
assert_eq!(remainder, expected_remainder);
}
From the C side, ``decode-ethernet.c`` offers an good example:
.. code-block:: c
/**
* Test a DCE ethernet frame that is too small.
*/
static int DecodeEthernetTestDceTooSmall(void)
{
uint8_t raw_eth[] = {
0x00, 0x10, 0x94, 0x55, 0x00, 0x01, 0x00, 0x10,
0x94, 0x56, 0x00, 0x01, 0x89, 0x03,
};
Packet *p = PacketGetFromAlloc();
FAIL_IF_NULL(p);
ThreadVars tv;
DecodeThreadVars dtv;
memset(&dtv, 0, sizeof(DecodeThreadVars));
memset(&tv, 0, sizeof(ThreadVars));
DecodeEthernet(&tv, &dtv, p, raw_eth, sizeof(raw_eth));
FAIL_IF_NOT(ENGINE_ISSET_EVENT(p, DCE_PKT_TOO_SMALL));
PacketFree(p);
PASS;
}
Suricata-Verify
===============
As mentioned above, these tests are used to check more complex behavior that involve a complete flow, with exchange of requests and responses. This can be done in an easier and more straightforward way,
since one doesn't have to simulate the network traffic and Suricata engine mechanics - one simply runs it, with the desired input packet capture,
configuration and checks.
A Suricata-verify test can help to ensure that code refactoring doesn't affect protocol logs, or signature detection,
for instance, as this could have a major impact to Suricata users and integrators.
For simpler tests, providing the pcap input is enough. But it is also possible to provide Suricata rules to be
inspected, and have Suricata Verify match for alerts and specific events.
Refer to the `Suricata Verify readme <https://github.com/OISF/suricata-verify#readme>`_ for details on how to create
this type of test. It suffices to have a packet capture representative of the behavior one wants to test, and then
follow the steps described there.
The Git repository for the Suricata Verify tests is a great source for examples, like the `app-layer-template <https://github.com/OISF/suricata-verify/tree/master/tests/app-layer-template>`_ one.
Generating Input
================
Using real traffic
^^^^^^^^^^^^^^^^^^
Having a packet capture for the desired protocol you want to test, open it in `Wireshark <https://www.wireshark.org/>`_, and select the specific
packet chosen for the test input, then use the Wireshark option ``Follow [TCP/UDP/HTTP/HTTP2/QUIC] Stream``. This allows for inspecting the whole network traffic stream in a different window.
There, it's possible to choose to ``Show and save data as`` ``C Arrays``, as well as to select if one wants to see the whole conversation or just **client** or **server** packets.
It is also possible to reach the same effect by accessing the **Analyze->Follow->TCP Stream** top menu in Wireshark.
(There are other stream options, the available one will depend on the type of network traffic captured).
This option will show the packet data as hexadecimal compatible with C-array style, and easily adapted for Rust,
as well. As shown in the image:
.. image:: img/InputCaptureExample.png
Wireshark can be also used to `capture sample network traffic <https://gitlab.com/wireshark/wireshark/-/wikis/CaptureSetup>`_ and generate pcap files.
Crafting input samples with Scapy
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
It is also possible to use Scapy to create specific traffic: `Scapy usage
<https://scapy.readthedocs.io/en/latest/usage.html>`_
Suricata-verify tests have several examples of pcaps generated in such a way. Look for Python scripts like the one used
for the `dce-udp-scapy
<https://github.com/OISF/suricata-verify/blob/master/tests/dcerpc/dcerpc-udp-scapy/dcerpc_udp_scapy.py>`_.
Other examples from our Suricata-Verify tests:
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Going through Suricata-Verify tests `readme` files it is also possible to find an assorted collection of pcap generation possibilities, some with explanation on the how-tos. To list a few:
- `http2-range <https://github.com/OISF/suricata-verify/blob/master/tests/http2-range/README.md>`_
- `http-range <https://github.com/OISF/suricata-verify/blob/master/tests/http-range/README.md>`_
- `smb2-delete <https://github.com/OISF/suricata-verify/blob/master/tests/smb2-delete/README.md>`_
- `smtp-rset <https://github.com/OISF/suricata-verify/blob/master/tests/smtp-rset/README.md>`_
- `http-auth-unrecognized <https://github.com/OISF/suricata-verify/blob/master/tests/http-auth-unrecognized/README.md>`_
Finding Capture Samples
^^^^^^^^^^^^^^^^^^^^^^^
If you can't capture traffic for the desired protocol from live traffic, or craft something up, you can try finding the type of traffic you
are interested in in public data sets. There's a thread for `Sharing good sources of sample captures
<https://forum.suricata.io/t/sharing-good-sources-of-sample-captures/1766/4>`_ in our forum.