Commit Graph

1061 Commits (be483dc873b67cbf3de55a0f8f73865596a76b66)

Author SHA1 Message Date
Jason Ish be483dc873 doc/userguide: document that lua dns rules need hooks
And remove the old "keywords" that a lua Rule can register with for
DNS.
3 months ago
Jason Ish b99f254105 lua: add suricata.rule library
Add a "suricata.rule" library for accessing rule information from a
Lua rule, or a Lua output script.

This lib replaces the following global Lua functions:
- SCRuleIds
- SCRuleAction
- SCRuleMsg
- SCRuleClass

Ticket: #7490
3 months ago
Jason Ish a5e662cb8a doc/lua/dns: fix typo 3 months ago
Philippe Antoine 8757ad5fd3 detect/dns: support string for dns.rrtype
Ticket: 6723
3 months ago
Philippe Antoine 44a6f7f8ca detect/dns: support string for dns.rcode
Ticket: 6723
3 months ago
Eric Leblond adfa46ab1c dox/userguide: add tx_cnt documentation 3 months ago
Alice Akaki bda0890834 detect: add email.received keyword
email.received matches on MIME EMAIL Received
This keyword maps to the EVE field email.received[]
It is a sticky buffer
Supports multiple buffer matching
Supports prefiltering

Ticket: #7599
3 months ago
James deb761367d doc: Update bypass docs to use new keyword format
Ticket: #7143

Update documentation to reflect new sticky buffer keyword format
3 months ago
Victor Julien e3c6554ee6 detect/app-layer-protocol: allow matching on 'unknown' 3 months ago
Alice Akaki ca429ef5e3 detect: add email.url keyword
email.url matches on URLs extracted from an email
This keyword maps to the EVE field email.url[]
Supports multiple buffer matching
Supports prefiltering

Ticket: #7597
3 months ago
Jason Ish b23c96198d doc/userguide: type in pfring upgrade notes 4 months ago
Victor Julien 57c73880db lua: enable lua rules by default
Now that sandboxing is in place, lua rule support is enabled by default.
4 months ago
Alice Akaki d4ec5b9765 detect: add ldap.responses.attribute_type
ldap.responses.attribute_type matches on LDAP attribute type/description
This keyword maps the eve field ldap.responses[].search_result_entry.attributes[].type
It is a sticky buffer
Supports multiple buffer matching
Supports prefiltering

Ticket: #7533
4 months ago
Alice Akaki 75fb352bde detect: add ldap.request.attribute_type
ldap.request.attribute_type matches on LDAP attribute type/description
This keyword maps the following eve fields:
ldap.request.search_request.attributes[]
ldap.request.modify_request.changes[].modification.attribute_type
ldap.request.add_request.attributes[].name
ldap.request.compare_request.attribute_value_assertion.description
It is a sticky buffer
Supports multiple buffer matching
Supports prefiltering

Ticket: #7533
4 months ago
Jeff Lucovsky 88c38fc4a0 doc/ftp: Document the ftp.reply keyword
Issue: 7508
4 months ago
Juliana Fajardini 63adfc6b1e userguide/pop3: minor typo and formatting fixing 4 months ago
Alex Savage 778053876b pop3: app-layer parser using sawp-pop3
This module uses the sawp-pop3 crate to parse POP3 requests and responses
Features:
- eve logging
- events for parsable but non-RFC-compliant messages

Ticket: 3243
4 months ago
Eric Leblond ed20e7cfe4 doc/userguide: doc domain and tld transforms 4 months ago
Philippe Antoine 8cb6a4b82b plugin: document app-layer plugins
Ticket: 7149
Ticket: 7150
Ticket: 7153
4 months ago
Victor Julien 5c7173c6df doc/firewall: start of documentation 4 months ago
Victor Julien 6f5fd77cb9 detect/app-layer-state: keyword for protocol state
Allow matching on the app-layer protocol state.
4 months ago
Philippe Antoine 706a0bd99b output: option to add suricata version in eve logs
Ticket: 4853
4 months ago
Jason Ish 2fce106aec pcap-log: add bpf filter for packets that are logged
Add an optional bpf filter to pcap-log. If set, packets must match the
filter to be logged, otherwise they will be ignored.

This allows a user to limit what is logged to disk if they have pcap-log
enabled, but still inspect all data captured.

Ticket: #6832
4 months ago
Lukas Sismis fbed416e26 doc: stylistic and grammar fixes 4 months ago
Dean Balandin 3cdb0ceb7e stream: decouple stream.bypass dependency from tls bypass
Decouple app.protocols.tls.encryption-handling and stream.bypass.
There's no apparent reason why encrypted TLS bypass traffic should
depend on stream bypass, as these are unrelated features.

Ticket: 6788
4 months ago
Lukas Sismis 185123a130 tls: deprecate "default" as a encryption-handling option
The default value for TLS encryption is generic and not precise.
The new keyword, "track-only" substitues the old keyword, being
more on point what action is actually taken when the TLS flow
reaches an encrypted state.
4 months ago
Philippe Antoine 6afad1af51 smb/log: configuration option for types logging
suricata.yaml output section for smb now parses a types list
and will restrict logging of transactions to these types.

By default, everything still gets logged

Remove unused rs_smb_log_json_request on the way

Ticket: 7620
4 months ago
Juliana Fajardini f06ec5336e doc/pgsql: document CopyOut-related response msgs
Related to
Task #4854
4 months ago
Jeff Lucovsky 590c8f6131 doc/ftp: Add ftp.command_data
This commits adds documentation for the ftp.command_data rule keyword
that includes usage examples.
4 months ago
Philippe Antoine 90666c7da1 doc/sdp: fix doc to match real keywords names
Ticket: 7291
4 months ago
Todd Mortimer 9c324b796e http: Use libhtp-rs.
Ticket: #2696

There are a lot of changes here, which are described below.

In general these changes are renaming constants to conform to the
libhtp-rs versions (which are generated by cbindgen); making all htp
types opaque and changing struct->member references to
htp_struct_member() function calls; and a handful of changes to offload
functionality onto libhtp-rs from suricata, such as URI normalization
and transaction cleanup.

Functions introduced to handle opaque htp_tx_t:
- tx->parsed_uri => htp_tx_parsed_uri(tx)
- tx->parsed_uri->path => htp_uri_path(htp_tx_parsed_uri(tx)
- tx->parsed_uri->hostname => htp_uri_hostname(htp_tx_parsed_uri(tx))
- htp_tx_get_user_data() => htp_tx_user_data(tx)
- htp_tx_is_http_2_upgrade(tx) convenience function introduced to detect response status 101
  and “Upgrade: h2c" header.

Functions introduced to handle opaque htp_tx_data_t:
- d->len => htp_tx_data_len()
- d->data => htp_tx_data_data()
- htp_tx_data_tx(data) function to get the htp_tx_t from the htp_tx_data_t
- htp_tx_data_is_empty(data) convenience function introduced to test if the data is empty.

Other changes:

Build libhtp-rs as a crate inside rust. Update autoconf to no longer
use libhtp as an external dependency. Remove HAVE_HTP feature defines
since they are no longer needed.

Make function arguments and return values const where possible

htp_tx_destroy(tx) will now free an incomplete transaction

htp_time_t replaced with standard struct timeval

Callbacks from libhtp now provide the htp_connp_t and the htp_tx_data_t
as separate arguments. This means the connection parser is no longer
fetched from the transaction inside callbacks.

SCHTPGenerateNormalizedUri() functionality moved inside libhtp-rs, which
now provides normalized URI values.
The normalized URI is available with accessor function: htp_tx_normalized_uri()
Configuration settings added to control the behaviour of the URI normalization:
- htp_config_set_normalized_uri_include_all()
- htp_config_set_plusspace_decode()
- htp_config_set_convert_lowercase()
- htp_config_set_double_decode_normalized_query()
- htp_config_set_double_decode_normalized_path()
- htp_config_set_backslash_convert_slashes()
- htp_config_set_bestfit_replacement_byte()
- htp_config_set_convert_lowercase()
- htp_config_set_nul_encoded_terminates()
- htp_config_set_nul_raw_terminates()
- htp_config_set_path_separators_compress()
- htp_config_set_path_separators_decode()
- htp_config_set_u_encoding_decode()
- htp_config_set_url_encoding_invalid_handling()
- htp_config_set_utf8_convert_bestfit()
- htp_config_set_normalized_uri_include_all()
- htp_config_set_plusspace_decode()
Constants related to configuring uri normalization:
- HTP_URL_DECODE_PRESERVE_PERCENT => HTP_URL_ENCODING_HANDLING_PRESERVE_PERCENT
- HTP_URL_DECODE_REMOVE_PERCENT => HTP_URL_ENCODING_HANDLING_REMOVE_PERCENT
- HTP_URL_DECODE_PROCESS_INVALID => HTP_URL_ENCODING_HANDLING_PROCESS_INVALID

htp_config_set_field_limits(soft_limit, hard_limit) changed to
htp_config_set_field_limit(limit) because libhtp didn't implement soft
limits.

libhtp logging API updated to provide HTP_LOG_CODE constants along with
the message. This eliminates the need to perform string matching on
message text to map log messages to HTTP_DECODER_EVENT values, and the
HTP_LOG_CODE values can be used directly. In support of this,
HTP_DECODER_EVENT values are mapped to their corresponding HTP_LOG_CODE
values.

New log events to describe additional anomalies:
HTP_LOG_CODE_REQUEST_TOO_MANY_LZMA_LAYERS
HTP_LOG_CODE_RESPONSE_TOO_MANY_LZMA_LAYERS
HTP_LOG_CODE_PROTOCOL_CONTAINS_EXTRA_DATA
HTP_LOG_CODE_CONTENT_LENGTH_EXTRA_DATA_START
HTP_LOG_CODE_CONTENT_LENGTH_EXTRA_DATA_END
HTP_LOG_CODE_SWITCHING_PROTO_WITH_CONTENT_LENGTH
HTP_LOG_CODE_DEFORMED_EOL
HTP_LOG_CODE_PARSER_STATE_ERROR
HTP_LOG_CODE_MISSING_OUTBOUND_TRANSACTION_DATA
HTP_LOG_CODE_MISSING_INBOUND_TRANSACTION_DATA
HTP_LOG_CODE_ZERO_LENGTH_DATA_CHUNKS
HTP_LOG_CODE_REQUEST_LINE_UNKNOWN_METHOD
HTP_LOG_CODE_REQUEST_LINE_UNKNOWN_METHOD_NO_PROTOCOL
HTP_LOG_CODE_REQUEST_LINE_UNKNOWN_METHOD_INVALID_PROTOCOL
HTP_LOG_CODE_REQUEST_LINE_NO_PROTOCOL
HTP_LOG_CODE_RESPONSE_LINE_INVALID_PROTOCOL
HTP_LOG_CODE_RESPONSE_LINE_INVALID_RESPONSE_STATUS
HTP_LOG_CODE_RESPONSE_BODY_INTERNAL_ERROR
HTP_LOG_CODE_REQUEST_BODY_DATA_CALLBACK_ERROR
HTP_LOG_CODE_RESPONSE_INVALID_EMPTY_NAME
HTP_LOG_CODE_REQUEST_INVALID_EMPTY_NAME
HTP_LOG_CODE_RESPONSE_INVALID_LWS_AFTER_NAME
HTP_LOG_CODE_RESPONSE_HEADER_NAME_NOT_TOKEN
HTP_LOG_CODE_REQUEST_INVALID_LWS_AFTER_NAME
HTP_LOG_CODE_LZMA_DECOMPRESSION_DISABLED
HTP_LOG_CODE_CONNECTION_ALREADY_OPEN
HTP_LOG_CODE_COMPRESSION_BOMB_DOUBLE_LZMA
HTP_LOG_CODE_INVALID_CONTENT_ENCODING
HTP_LOG_CODE_INVALID_GAP
HTP_LOG_CODE_ERROR

The new htp_log API supports consuming log messages more easily than
walking a list and tracking the current offset. Internally, libhtp-rs
now provides log messages as a queue of htp_log_t, which means the
application can simply call htp_conn_next_log() to fetch the next log
message until the queue is empty. Once the application is done with a
log message, they can call htp_log_free() to dispose of it.

Functions supporting htp_log_t:
htp_conn_next_log(conn) - Get the next log message
htp_log_message(log) - To get the text of the message
htp_log_code(log) - To get the HTP_LOG_CODE value
htp_log_free(log) - To free the htp_log_t
4 months ago
Shivani Bhardwaj 9409a5a49a doc: update configuration with flow rate-tracking
Feature 5647
4 months ago
Alice Akaki 341369f203 detect: add email.x_mailer keyword
email.x_mailer matches on MIME EMAIL X-Mailer
This keyword maps to the EVE field email.x_mailer
It is a sticky buffer
Supports prefiltering

Ticket: #7598
4 months ago
Alice Akaki 52e12410ed detect: add email.message_id keyword
email.message_id matches on MIME EMAIL Message-Id
This keyword maps to the EVE field email.message_id
It is a sticky buffer
Supports prefiltering

Ticket: #7593
4 months ago
Philippe Antoine 1851030010 detect: generic callback for md5-like keywords
Ticket: 5634
4 months ago
Giuseppe Longo 330241b162 doc: add sdp sticky buffers 4 months ago
Jason Ish 22b77b0c56 conf: prefix conf API with SC 4 months ago
Jeff Lucovsky ed2a81dc05 doc/entropy: Add documentation for the entropy keyword
This commits adds documentation for the entropy keyword.
The entropy keyword calculates the Shannon entropy value for content
with the calculated value used to determine whether an alert occurs.
4 months ago
Shivani Bhardwaj 65349442f8 doc: add http library supported fns
Task 7604
4 months ago
Shivani Bhardwaj 5ba0ccaf4b doc: remove http cookie ua from list of buffers
as they are available via library using the tx
- tx:request_header("Cookie")
- tx:response_header("Cookie")
- tx:request_header("User-Agent")
4 months ago
Juliana Fajardini d8c6a56a62 doc/exceptions: change stats counters names
As we've changed them for more search-friendly ones.

Related to
Task #7185
4 months ago
Lukas Sismis 4f2ce17dc5 dpdk: allow zero TX queues when running in IDS mode
When running in non-forwarding (IDS) mode, it is not required
to create TX queues for the interface.
This can be acheived by setting tx-descriptors configuration
field to 0.

Ticket: 7633
4 months ago
Lukas Sismis 1be1c65b6e docs: double quote technical terms in DPDK section 4 months ago
Lukas Sismis fbe5ce7a2b dpdk: document vlan stripping offload
Ticket: 5838
4 months ago
Lukas Sismis 640d0985c2 dpdk: check for link up before full startup
ICE card (Intel E810) was not receiving packets immediatelly
after startup, Suricata workers would act as processing while
it was not. This eliminates the problem by only continuing
in the initialization if the link is already up.

The setting can be turned off manually from the configuraiton
file.

Ticket: 7381
4 months ago
Lukas Sismis cb997a64dc dpdk: replace global with per-thread mempools
It turned out that having global (interface-specific) mempool
that is shared by the threads of the interface is slower than
having individual mempools per queue for each interface.

The commit brings this change and should be user-invisible,
the config setting remains still as a number of objects of
all mempools summed (of that interface).

Ticket: 7382
4 months ago
Lukas Sismis 2ef2a9e26f dpdk: auto configure Rx/Tx descriptors and mempool size
Ticket: 7380
Ticket: 7373
4 months ago
Philippe Antoine 3a092f3027 detect: allow rule which need both directions to match
Ticket: 5665

This is done with `alert ip any any => any any`
The => operator means that we will need both directions
4 months ago
Jason Ish 66eb29affd doc/ndpi: move ndpi docs to new plugins section
Moves the nDPI documentation to an nDPI page in the plugins
section. Remove the duplication of installation and setup
documentation.

Includes some minor cleanups.
4 months ago
Alfredo Cardigliano dfd9ef5784 ndpi: initial implementation of nDPI plugin
Ticket: #7231
4 months ago