Commit Graph

1032 Commits (90666c7da1b3c8bd5a611a3e925ac80a48b9e90a)

Author SHA1 Message Date
Philippe Antoine 90666c7da1 doc/sdp: fix doc to match real keywords names
Ticket: 7291
4 months ago
Todd Mortimer 9c324b796e http: Use libhtp-rs.
Ticket: #2696

There are a lot of changes here, which are described below.

In general these changes are renaming constants to conform to the
libhtp-rs versions (which are generated by cbindgen); making all htp
types opaque and changing struct->member references to
htp_struct_member() function calls; and a handful of changes to offload
functionality onto libhtp-rs from suricata, such as URI normalization
and transaction cleanup.

Functions introduced to handle opaque htp_tx_t:
- tx->parsed_uri => htp_tx_parsed_uri(tx)
- tx->parsed_uri->path => htp_uri_path(htp_tx_parsed_uri(tx)
- tx->parsed_uri->hostname => htp_uri_hostname(htp_tx_parsed_uri(tx))
- htp_tx_get_user_data() => htp_tx_user_data(tx)
- htp_tx_is_http_2_upgrade(tx) convenience function introduced to detect response status 101
  and “Upgrade: h2c" header.

Functions introduced to handle opaque htp_tx_data_t:
- d->len => htp_tx_data_len()
- d->data => htp_tx_data_data()
- htp_tx_data_tx(data) function to get the htp_tx_t from the htp_tx_data_t
- htp_tx_data_is_empty(data) convenience function introduced to test if the data is empty.

Other changes:

Build libhtp-rs as a crate inside rust. Update autoconf to no longer
use libhtp as an external dependency. Remove HAVE_HTP feature defines
since they are no longer needed.

Make function arguments and return values const where possible

htp_tx_destroy(tx) will now free an incomplete transaction

htp_time_t replaced with standard struct timeval

Callbacks from libhtp now provide the htp_connp_t and the htp_tx_data_t
as separate arguments. This means the connection parser is no longer
fetched from the transaction inside callbacks.

SCHTPGenerateNormalizedUri() functionality moved inside libhtp-rs, which
now provides normalized URI values.
The normalized URI is available with accessor function: htp_tx_normalized_uri()
Configuration settings added to control the behaviour of the URI normalization:
- htp_config_set_normalized_uri_include_all()
- htp_config_set_plusspace_decode()
- htp_config_set_convert_lowercase()
- htp_config_set_double_decode_normalized_query()
- htp_config_set_double_decode_normalized_path()
- htp_config_set_backslash_convert_slashes()
- htp_config_set_bestfit_replacement_byte()
- htp_config_set_convert_lowercase()
- htp_config_set_nul_encoded_terminates()
- htp_config_set_nul_raw_terminates()
- htp_config_set_path_separators_compress()
- htp_config_set_path_separators_decode()
- htp_config_set_u_encoding_decode()
- htp_config_set_url_encoding_invalid_handling()
- htp_config_set_utf8_convert_bestfit()
- htp_config_set_normalized_uri_include_all()
- htp_config_set_plusspace_decode()
Constants related to configuring uri normalization:
- HTP_URL_DECODE_PRESERVE_PERCENT => HTP_URL_ENCODING_HANDLING_PRESERVE_PERCENT
- HTP_URL_DECODE_REMOVE_PERCENT => HTP_URL_ENCODING_HANDLING_REMOVE_PERCENT
- HTP_URL_DECODE_PROCESS_INVALID => HTP_URL_ENCODING_HANDLING_PROCESS_INVALID

htp_config_set_field_limits(soft_limit, hard_limit) changed to
htp_config_set_field_limit(limit) because libhtp didn't implement soft
limits.

libhtp logging API updated to provide HTP_LOG_CODE constants along with
the message. This eliminates the need to perform string matching on
message text to map log messages to HTTP_DECODER_EVENT values, and the
HTP_LOG_CODE values can be used directly. In support of this,
HTP_DECODER_EVENT values are mapped to their corresponding HTP_LOG_CODE
values.

New log events to describe additional anomalies:
HTP_LOG_CODE_REQUEST_TOO_MANY_LZMA_LAYERS
HTP_LOG_CODE_RESPONSE_TOO_MANY_LZMA_LAYERS
HTP_LOG_CODE_PROTOCOL_CONTAINS_EXTRA_DATA
HTP_LOG_CODE_CONTENT_LENGTH_EXTRA_DATA_START
HTP_LOG_CODE_CONTENT_LENGTH_EXTRA_DATA_END
HTP_LOG_CODE_SWITCHING_PROTO_WITH_CONTENT_LENGTH
HTP_LOG_CODE_DEFORMED_EOL
HTP_LOG_CODE_PARSER_STATE_ERROR
HTP_LOG_CODE_MISSING_OUTBOUND_TRANSACTION_DATA
HTP_LOG_CODE_MISSING_INBOUND_TRANSACTION_DATA
HTP_LOG_CODE_ZERO_LENGTH_DATA_CHUNKS
HTP_LOG_CODE_REQUEST_LINE_UNKNOWN_METHOD
HTP_LOG_CODE_REQUEST_LINE_UNKNOWN_METHOD_NO_PROTOCOL
HTP_LOG_CODE_REQUEST_LINE_UNKNOWN_METHOD_INVALID_PROTOCOL
HTP_LOG_CODE_REQUEST_LINE_NO_PROTOCOL
HTP_LOG_CODE_RESPONSE_LINE_INVALID_PROTOCOL
HTP_LOG_CODE_RESPONSE_LINE_INVALID_RESPONSE_STATUS
HTP_LOG_CODE_RESPONSE_BODY_INTERNAL_ERROR
HTP_LOG_CODE_REQUEST_BODY_DATA_CALLBACK_ERROR
HTP_LOG_CODE_RESPONSE_INVALID_EMPTY_NAME
HTP_LOG_CODE_REQUEST_INVALID_EMPTY_NAME
HTP_LOG_CODE_RESPONSE_INVALID_LWS_AFTER_NAME
HTP_LOG_CODE_RESPONSE_HEADER_NAME_NOT_TOKEN
HTP_LOG_CODE_REQUEST_INVALID_LWS_AFTER_NAME
HTP_LOG_CODE_LZMA_DECOMPRESSION_DISABLED
HTP_LOG_CODE_CONNECTION_ALREADY_OPEN
HTP_LOG_CODE_COMPRESSION_BOMB_DOUBLE_LZMA
HTP_LOG_CODE_INVALID_CONTENT_ENCODING
HTP_LOG_CODE_INVALID_GAP
HTP_LOG_CODE_ERROR

The new htp_log API supports consuming log messages more easily than
walking a list and tracking the current offset. Internally, libhtp-rs
now provides log messages as a queue of htp_log_t, which means the
application can simply call htp_conn_next_log() to fetch the next log
message until the queue is empty. Once the application is done with a
log message, they can call htp_log_free() to dispose of it.

Functions supporting htp_log_t:
htp_conn_next_log(conn) - Get the next log message
htp_log_message(log) - To get the text of the message
htp_log_code(log) - To get the HTP_LOG_CODE value
htp_log_free(log) - To free the htp_log_t
4 months ago
Shivani Bhardwaj 9409a5a49a doc: update configuration with flow rate-tracking
Feature 5647
4 months ago
Alice Akaki 341369f203 detect: add email.x_mailer keyword
email.x_mailer matches on MIME EMAIL X-Mailer
This keyword maps to the EVE field email.x_mailer
It is a sticky buffer
Supports prefiltering

Ticket: #7598
4 months ago
Alice Akaki 52e12410ed detect: add email.message_id keyword
email.message_id matches on MIME EMAIL Message-Id
This keyword maps to the EVE field email.message_id
It is a sticky buffer
Supports prefiltering

Ticket: #7593
4 months ago
Philippe Antoine 1851030010 detect: generic callback for md5-like keywords
Ticket: 5634
4 months ago
Giuseppe Longo 330241b162 doc: add sdp sticky buffers 4 months ago
Jason Ish 22b77b0c56 conf: prefix conf API with SC 4 months ago
Jeff Lucovsky ed2a81dc05 doc/entropy: Add documentation for the entropy keyword
This commits adds documentation for the entropy keyword.
The entropy keyword calculates the Shannon entropy value for content
with the calculated value used to determine whether an alert occurs.
4 months ago
Shivani Bhardwaj 65349442f8 doc: add http library supported fns
Task 7604
4 months ago
Shivani Bhardwaj 5ba0ccaf4b doc: remove http cookie ua from list of buffers
as they are available via library using the tx
- tx:request_header("Cookie")
- tx:response_header("Cookie")
- tx:request_header("User-Agent")
4 months ago
Juliana Fajardini d8c6a56a62 doc/exceptions: change stats counters names
As we've changed them for more search-friendly ones.

Related to
Task #7185
4 months ago
Lukas Sismis 4f2ce17dc5 dpdk: allow zero TX queues when running in IDS mode
When running in non-forwarding (IDS) mode, it is not required
to create TX queues for the interface.
This can be acheived by setting tx-descriptors configuration
field to 0.

Ticket: 7633
4 months ago
Lukas Sismis 1be1c65b6e docs: double quote technical terms in DPDK section 4 months ago
Lukas Sismis fbe5ce7a2b dpdk: document vlan stripping offload
Ticket: 5838
4 months ago
Lukas Sismis 640d0985c2 dpdk: check for link up before full startup
ICE card (Intel E810) was not receiving packets immediatelly
after startup, Suricata workers would act as processing while
it was not. This eliminates the problem by only continuing
in the initialization if the link is already up.

The setting can be turned off manually from the configuraiton
file.

Ticket: 7381
4 months ago
Lukas Sismis cb997a64dc dpdk: replace global with per-thread mempools
It turned out that having global (interface-specific) mempool
that is shared by the threads of the interface is slower than
having individual mempools per queue for each interface.

The commit brings this change and should be user-invisible,
the config setting remains still as a number of objects of
all mempools summed (of that interface).

Ticket: 7382
4 months ago
Lukas Sismis 2ef2a9e26f dpdk: auto configure Rx/Tx descriptors and mempool size
Ticket: 7380
Ticket: 7373
4 months ago
Philippe Antoine 3a092f3027 detect: allow rule which need both directions to match
Ticket: 5665

This is done with `alert ip any any => any any`
The => operator means that we will need both directions
4 months ago
Jason Ish 66eb29affd doc/ndpi: move ndpi docs to new plugins section
Moves the nDPI documentation to an nDPI page in the plugins
section. Remove the duplication of installation and setup
documentation.

Includes some minor cleanups.
4 months ago
Alfredo Cardigliano dfd9ef5784 ndpi: initial implementation of nDPI plugin
Ticket: #7231
4 months ago
Alice Akaki ce2e7aed74 detect: add email.date keyword
email.date matches on MIME EMAIL DATE
This keyword maps to the EVE field email.date
It is a sticky buffer
Supports prefiltering

Ticket: #7591
4 months ago
Victor Julien 8c9dfafc6d doc/tls: add more detail on tls.random 4 months ago
Lukas Sismis 7dc65c2f8a hyperscan: add caching mechanism for hyperscan contexts
Cache Hyperscan serialized databases to disk to prevent compilation
of the same databases when Suricata is run again with the same
ruleset.
Hyperscan binary files are stored per rulegroup in the designated
folder, by default in the cached library folder.
Since caching is per signature group heads,
some chunk of the ruleset can change and it still can reuse part of
the unchanged signature groups.

Loading *fresh* ET Open ruleset:  19 seconds
Loading *cached* ET Open ruleset: 07 seconds

Ticket: 7170
4 months ago
Philippe Antoine 879a733c12 doc/http2: explicit behavior for some http keywords
HTTP/2 does not define a way to carry the version or reason phrase
that is included in an HTTP/1.1 status line.

Ticket: 6548
4 months ago
Juliana Fajardini cd69955d7f doc/userguide: add lua flowlib docs
Task #7489
4 months ago
Juliana Fajardini 9480272509 doc: remove old lua flow methods
Task #7489
4 months ago
Alice Akaki 7ba4ebdc2c detect: add email.cc keyword
email.cc matches on MIME EMAIL Carbon Copy
This keyword maps to the EVE field email.cc[]
It is a sticky buffer
Supports prefiltering

Ticket: #7588
4 months ago
Alice Akaki 9e7d23d73f doc: add keywords to the multi-buffer-matching list 4 months ago
Jason Ish 1a47fdfd46 doc/userguide: group af-packet upgrade notes together
Also fix the rendering of the sip nest list.
4 months ago
Jason Ish 080d48ba29 doc/userguide: upgrade note about defrag now off for inline use
Ticket: #7617
4 months ago
Jason Ish 8fe526006d doc/userguide: upgrade note about tpacket-v3 default for ids
Ticket: #4798
4 months ago
Alice Akaki 5d6a072e35 detect: add email.to keyword
email.to matches on MIME EMAIL TO
This keyword maps to the EVE field email.to[]
It is a sticky buffer
Supports prefiltering

Ticket: #7596
4 months ago
Alice Akaki 09db7c7ac1 detect: add mime email.subject keyword
email.subject matches on MIME EMAIL SUBJECT
This keyword maps to the EVE field email.subject
It is a sticky buffer
Supports prefiltering

Ticket: #7595
4 months ago
Jason Ish 374762d202 af-packet: remove use-mmap option
This option is obsolete and was not used in 7.0 as tpacket-v1 support
was removed (see ticket #4796).
4 months ago
Juliana Fajardini a9b2a62ee4 userguide/exceptions: clarify when stats are logged
The stats for exception policies are only logged/ present when any of
the exception policies are enabled (which means any value other than
"auto" or "ignore" in IDS mode, or "ignore" in IPS mode).

This wasn't clearly stated in the docs.
4 months ago
Juliana Fajardini 08e928988f flow/output: log triggered exception policies
To accompany the Exception Policy stats, also add information about any
Exception Policy triggered and for which target to the flow log event.

Task #6215
4 months ago
Jason Ish a6b116bcbe lua: document new suricata.dns lua library
Ticket: #7602
4 months ago
Alice Akaki 90aab0d62f detect: add email.from
email.from matches on MIME EMAIL FROM
This keyword maps to the EVE field email.from
It is a sticky buffer
Supports prefiltering

Ticket: #7592
4 months ago
Joyce Yu dac0d6371e Doc: update eve-json-output ethernet description
Document getting mac addresses from flow when flow timeout.
4 months ago
Shivani Bhardwaj be372ce39d doc: explain priority port setting
Ticket 7329
4 months ago
Shivani Bhardwaj 040c694256 doc: format and align suricata.yaml section 4 months ago
Juliana Fajardini 3985b24e1b upgrade: list inspection recursion default limit
As the yaml indicated before that if no value was specified there were
no limits, and now there will be one.
4 months ago
Juliana Fajardini e1f9e66af0 doc/upgrade: add datasets hash size limit note 4 months ago
Jason Ish c6d18fc871 doc/userguide: af-packet upgrade notes
Add note about increased block size and how to change it back to old
defaults if needed.

Ticket: #7458
4 months ago
Philippe Antoine 32d0bd2bbb detect: limit base64_decode `bytes` to 64KiB
Ticket: 7613

Avoids potential large per-thread memory allocation. A buffer with the
size of the largest decode_base64 buffer size setting would be allocated
per thread. As this was a u32, it could mean a per-thread 4GiB memory
allocation.

64KiB was already the built-in default for cases where bytes size wasn't
specified.
4 months ago
Juliana Fajardini d8523d9d97 userguide/header-keywords: fix typos, adjust format 5 months ago
Juliana Fajardini 28407b2fb8 doc/rule-types: remove trailing underscore
And other minor fixes that were overseen.
5 months ago
Juliana Fajardini 4a8da8c448 userguide/suricatactl: use suricata community page
We were mentioning "Suricata Support" page, which could be a bit
misleading -- and also used a link that is actually redirected to the
Suricata Community page, anyways.
5 months ago
Jason Ish 814e9ffb7a dns: add keywords for additionals and authorities rrnames
Add keywords dns.additionals.rrname and dns.authorities.rrname. Along
the way, consolidate dns.query.name and dns.answer.name into a single file
and register them altogether since there is a lot of common code.
5 months ago