Create a new list of headers for headers that should never be
installed, such as private headers that should only be available
within the Suricata project.
For now, just the util-device-private.h belongs to this set.
util-device.h exposes some details that are particularly problematic
for C++, even when wrapped in 'extern "C"'. To address this, break the
header into public and private parts. The public part exposes
LiveDevice as an opaque data structure, while the private header has
the actual definition.
The idea is that only Suricata C source files should include the
private header, it should not be re-included in any other header
file. And this is the header library users should use, however we
don't enforce it with tecnical means, a library user could still
include the private header, but the clue there is in the name.
New Lua lib, "suricata.flowvar" for working with flowvars from Lua.
Replaces functions:
- SCFlowvarGet (and ScFlowvarGet)
- SCFlowvarSet (and SCFlowvarSet)
Of note, the DetectLuaData has been made available to the init and
thread_init methods, instead of just the match. This is due to an
issue that if a flow variable is not registered in init, it will not
be logged, registering in thread_init is too late.
Ticket: #7486
Issue: 7504
Add implementation of the ftp.dynamic_port rule keyword. The
implementation uses the U16 integer matching/parsing and thus supports
the comparison operations such as <, >, <=, >=, !, !=, and range (-).
Add a "suricata.rule" library for accessing rule information from a
Lua rule, or a Lua output script.
This lib replaces the following global Lua functions:
- SCRuleIds
- SCRuleAction
- SCRuleMsg
- SCRuleClass
Ticket: #7490
Ticket: 5053
Move enum OutputJsonLogDirection and struct
EveJsonTxLoggerRegistrationData to a public header user by rust
thanks to bindgen
Rename to use SC prefix on the way
And make EveJsonSimpleTxLogFunc use a const pointer to transaction
Ticket: #2696
There are a lot of changes here, which are described below.
In general these changes are renaming constants to conform to the
libhtp-rs versions (which are generated by cbindgen); making all htp
types opaque and changing struct->member references to
htp_struct_member() function calls; and a handful of changes to offload
functionality onto libhtp-rs from suricata, such as URI normalization
and transaction cleanup.
Functions introduced to handle opaque htp_tx_t:
- tx->parsed_uri => htp_tx_parsed_uri(tx)
- tx->parsed_uri->path => htp_uri_path(htp_tx_parsed_uri(tx)
- tx->parsed_uri->hostname => htp_uri_hostname(htp_tx_parsed_uri(tx))
- htp_tx_get_user_data() => htp_tx_user_data(tx)
- htp_tx_is_http_2_upgrade(tx) convenience function introduced to detect response status 101
and “Upgrade: h2c" header.
Functions introduced to handle opaque htp_tx_data_t:
- d->len => htp_tx_data_len()
- d->data => htp_tx_data_data()
- htp_tx_data_tx(data) function to get the htp_tx_t from the htp_tx_data_t
- htp_tx_data_is_empty(data) convenience function introduced to test if the data is empty.
Other changes:
Build libhtp-rs as a crate inside rust. Update autoconf to no longer
use libhtp as an external dependency. Remove HAVE_HTP feature defines
since they are no longer needed.
Make function arguments and return values const where possible
htp_tx_destroy(tx) will now free an incomplete transaction
htp_time_t replaced with standard struct timeval
Callbacks from libhtp now provide the htp_connp_t and the htp_tx_data_t
as separate arguments. This means the connection parser is no longer
fetched from the transaction inside callbacks.
SCHTPGenerateNormalizedUri() functionality moved inside libhtp-rs, which
now provides normalized URI values.
The normalized URI is available with accessor function: htp_tx_normalized_uri()
Configuration settings added to control the behaviour of the URI normalization:
- htp_config_set_normalized_uri_include_all()
- htp_config_set_plusspace_decode()
- htp_config_set_convert_lowercase()
- htp_config_set_double_decode_normalized_query()
- htp_config_set_double_decode_normalized_path()
- htp_config_set_backslash_convert_slashes()
- htp_config_set_bestfit_replacement_byte()
- htp_config_set_convert_lowercase()
- htp_config_set_nul_encoded_terminates()
- htp_config_set_nul_raw_terminates()
- htp_config_set_path_separators_compress()
- htp_config_set_path_separators_decode()
- htp_config_set_u_encoding_decode()
- htp_config_set_url_encoding_invalid_handling()
- htp_config_set_utf8_convert_bestfit()
- htp_config_set_normalized_uri_include_all()
- htp_config_set_plusspace_decode()
Constants related to configuring uri normalization:
- HTP_URL_DECODE_PRESERVE_PERCENT => HTP_URL_ENCODING_HANDLING_PRESERVE_PERCENT
- HTP_URL_DECODE_REMOVE_PERCENT => HTP_URL_ENCODING_HANDLING_REMOVE_PERCENT
- HTP_URL_DECODE_PROCESS_INVALID => HTP_URL_ENCODING_HANDLING_PROCESS_INVALID
htp_config_set_field_limits(soft_limit, hard_limit) changed to
htp_config_set_field_limit(limit) because libhtp didn't implement soft
limits.
libhtp logging API updated to provide HTP_LOG_CODE constants along with
the message. This eliminates the need to perform string matching on
message text to map log messages to HTTP_DECODER_EVENT values, and the
HTP_LOG_CODE values can be used directly. In support of this,
HTP_DECODER_EVENT values are mapped to their corresponding HTP_LOG_CODE
values.
New log events to describe additional anomalies:
HTP_LOG_CODE_REQUEST_TOO_MANY_LZMA_LAYERS
HTP_LOG_CODE_RESPONSE_TOO_MANY_LZMA_LAYERS
HTP_LOG_CODE_PROTOCOL_CONTAINS_EXTRA_DATA
HTP_LOG_CODE_CONTENT_LENGTH_EXTRA_DATA_START
HTP_LOG_CODE_CONTENT_LENGTH_EXTRA_DATA_END
HTP_LOG_CODE_SWITCHING_PROTO_WITH_CONTENT_LENGTH
HTP_LOG_CODE_DEFORMED_EOL
HTP_LOG_CODE_PARSER_STATE_ERROR
HTP_LOG_CODE_MISSING_OUTBOUND_TRANSACTION_DATA
HTP_LOG_CODE_MISSING_INBOUND_TRANSACTION_DATA
HTP_LOG_CODE_ZERO_LENGTH_DATA_CHUNKS
HTP_LOG_CODE_REQUEST_LINE_UNKNOWN_METHOD
HTP_LOG_CODE_REQUEST_LINE_UNKNOWN_METHOD_NO_PROTOCOL
HTP_LOG_CODE_REQUEST_LINE_UNKNOWN_METHOD_INVALID_PROTOCOL
HTP_LOG_CODE_REQUEST_LINE_NO_PROTOCOL
HTP_LOG_CODE_RESPONSE_LINE_INVALID_PROTOCOL
HTP_LOG_CODE_RESPONSE_LINE_INVALID_RESPONSE_STATUS
HTP_LOG_CODE_RESPONSE_BODY_INTERNAL_ERROR
HTP_LOG_CODE_REQUEST_BODY_DATA_CALLBACK_ERROR
HTP_LOG_CODE_RESPONSE_INVALID_EMPTY_NAME
HTP_LOG_CODE_REQUEST_INVALID_EMPTY_NAME
HTP_LOG_CODE_RESPONSE_INVALID_LWS_AFTER_NAME
HTP_LOG_CODE_RESPONSE_HEADER_NAME_NOT_TOKEN
HTP_LOG_CODE_REQUEST_INVALID_LWS_AFTER_NAME
HTP_LOG_CODE_LZMA_DECOMPRESSION_DISABLED
HTP_LOG_CODE_CONNECTION_ALREADY_OPEN
HTP_LOG_CODE_COMPRESSION_BOMB_DOUBLE_LZMA
HTP_LOG_CODE_INVALID_CONTENT_ENCODING
HTP_LOG_CODE_INVALID_GAP
HTP_LOG_CODE_ERROR
The new htp_log API supports consuming log messages more easily than
walking a list and tracking the current offset. Internally, libhtp-rs
now provides log messages as a queue of htp_log_t, which means the
application can simply call htp_conn_next_log() to fetch the next log
message until the queue is empty. Once the application is done with a
log message, they can call htp_log_free() to dispose of it.
Functions supporting htp_log_t:
htp_conn_next_log(conn) - Get the next log message
htp_log_message(log) - To get the text of the message
htp_log_code(log) - To get the HTP_LOG_CODE value
htp_log_free(log) - To free the htp_log_t
In order to track flow rate and thus determine a course of action or
categorize it as elephant flow, track a flow's byte count per direction
in a ring buffer for a given time interval.
The implementation is simple and keeps overwriting the buffer and
updating the final sum. The sum of all the elements in the ring buffer
at any point in time should reflect the number of bytes for the
respective flow in the last of a given configured interval.
e.g. if the definition says that the flows must be tracked by a rate of
100k bytes in 10 seconds, the ring buffer at any point in time should
carry the total number of bytes seen by the respective flow in the last
10 seconds.
So far, the implementation only supports reading the flow rate
definition from suricata.yaml and using it to track the flows.
This solution adds up a space complexity to the existing Flow struct.
However, the added space complexity should only take effect if the
feature is in use. Since this buffer extends the Flow struct, it does
not impact the usual business logic or complexity of the code.
This implementation is currently limited to defining the time interval
of flow rate in seconds only. However, the number of seconds defined are
directly proportional to the aforementioned added space complexity as
that's the size of the ring buffer.
Feature 5647
This commit adds keyword/build support for the entropy keyword. The
entropy keyword compares an entropy value with a value calculated
according to the Shannon entropy on the available content.
Issue: 4162
Add library source and runmode modules. Reorganized
library example to create a worker thread and replay a pcap
file using the library mode.
No API layer is added at this stage.
Edits by Jason Ish:
- fix guard
- add copyright/license headers
Ticket: #7240
It turned out that having global (interface-specific) mempool
that is shared by the threads of the interface is slower than
having individual mempools per queue for each interface.
The commit brings this change and should be user-invisible,
the config setting remains still as a number of objects of
all mempools summed (of that interface).
Ticket: 7382
Cache Hyperscan serialized databases to disk to prevent compilation
of the same databases when Suricata is run again with the same
ruleset.
Hyperscan binary files are stored per rulegroup in the designated
folder, by default in the cached library folder.
Since caching is per signature group heads,
some chunk of the ruleset can change and it still can reuse part of
the unchanged signature groups.
Loading *fresh* ET Open ruleset: 19 seconds
Loading *cached* ET Open ruleset: 07 seconds
Ticket: 7170
Methods:
`get` creates the flow object.
`id` returns the flow id.
`has_alerts` returns a boolean indicating if the flow triggered alerts.
`app_layer_proto` returns various app-layer related fields as 5 strings:
alproto, alproto_ts, alproto_tc, alproto_orig, alproto_expect.
`stats` returns cnts for bytes and packets to sever and to client, as 4
numbers.
`tuple` -- returns various fields: srcip, dstip, proto, sp, dp.
`timestamps` returns time as 4 numbers: seconds and microseconds, for
first and last packet of the flow.
`timestring_legacy` returns the first packet from the flow's timestring
as a string (like fastlog).
`timestring_iso8601` returns the first packet from the flow's
timestring as an iso8601 compat timestring (like eve).
Example:
```
name = "lua-scflowstats.log"
local flow = require("suricata.flow")
function init(args)
local needs = {}
needs["type"] = "flow"
return needs
end
function setup(args)
filename = SCLogPath() .. "/" .. name
file = assert(io.open(filename, "a"))
SCLogInfo("lua SCFlowStats Log Filename " .. filename)
end
function log(args)
local f = flow.get()
timestring = f:timestring_legacy()
tscnt, tsbytes, tccnt, tcbytes = f:stats()
file:write ("[**] " .. timestring .. "\nSCFlowStats is\nPacket count to server: " .. tscnt .. "\nByte count to server: " .. tsbytes .. "\nPacket count to client: " .. tccnt .. "\nByte count to client: " .. tcbytes .. "\n[**]")
file:flush()
end
function deinit(args)
file:close(file)
end
```
Task #7489
Add keywords dns.additionals.rrname and dns.authorities.rrname. Along
the way, consolidate dns.query.name and dns.answer.name into a single file
and register them altogether since there is a lot of common code.
Feature: 7012
Add dns.response sticky buffer to match on dns response fields.
Add rust functions to return dns response packet data.
Unit tests verifying signature matching.
Bindgen works by processing a header file which includes all other
header files it should generate bindings for. For this I've created
bindgen.h which just includes app-layer-protos.h for now as an
example.
These bindings are then generated and saved in the "suricata-sys"
crate and become availale as "suricata_sys::sys".
Ticket: #7341
Example:
```
local packet = require "suricata.packet"
function init (args)
local needs = {}
return needs
end
function match (args)
p = packet.get()
payload = p:payload()
ts = p:timestring()
for line in payload:gmatch("([^\r\n]*)[\r\n]+") do
if line == "GET /index.html HTTP/1.0" then
ipver, srcip, dstip, proto, sp, dp = p:tuple()
SCLogNotice(string.format("%s %s->%s %d->%d (pcap_cnt:%d) match! %s", ts, srcip, dstip, sp, dp, p:pcap_cnt(), line));
return 1
end
end
return 0
end
```
Methods:
`get` creates the packet object.
`payload` returns the packet payload as a buffer
`packet` returns the whole packet (includes headers)
`pcap_cnt` returns the `pcap_cnt` (pcap file mode only)
`tuple` returns various fields: srcip, dstip, proto, sp, dp
`sp` returns source port
`dp` returns destination port
`timestamp` returns time as 2 numbers: seconds and microseconds
`timestring_legacy` returns a timestamp as a string (like fastlog)
`timestring_iso8601` returns a iso8601 compat timestring (like eve)
Ticket: #7488.
The configuration of this rule is the same as for ixgbe driver except
the hash function is not RTE_ETH_HASH_FUNCTION_DEFAULT but
RTE_ETH_HASH_FUNCTION_TOEPLITZ.
The syntax in dpdk-testpmd for this rule with attributes:
port index == 0
used rx queue indices == 0 1 2 3
<hash_key> == 6d5a symmetric hash key
is as follows:
"flow create 0 ingress pattern eth / end actions rss types ipv4 ipv6
end queues 0 1 2 3 end key <hash_key> key_len 40 func toeplitz / end"
Ticket: 7337
Move and adjust the base of RSS configuration from util-dpdk-i40e.c to
a new file that can be later utilized by other cards.
RSS configuration can be configured via rte_flow rules. This is useful
for possible future features such as specific header offload
(vxlan, nvgre) also implemented via rte_flow rules, as rte_flow
rules can be chained via groups and priorities.
i40e uses multiple different rte_flow rules to setup RSS. At first,
function DeviceSetRSSFlowQueues() is used to setup rx queues.
This rule matches all types of traffic, so the equivalent
to dpdk-testpmd pattern would be "pattern end"
This rule can not contain hash types (ipv4, ipv6 etc.) nor hash key.
The hash function used here is RTE_ETH_HASH_FUNCTION_DEFAULT.
The syntax in dpdk-testpmd for this rule with attributes:
port index == 0
used rx queue indices == 0 1 2 3
is as follows:
"flow create 0 ingress pattern end actions rss queues 0 1 2 3 end
func default / end"
The other rules configured by i40eDeviceSetRSSFlowIPv4() and
i40eDeviceSetRSSFlowIPv6() match specific type of traffic by l4 protocol
(none, TCP, UDP, SCTP). For example, pattern to match l3 ipv4 with l4
tcp traffic in dpdk-testpmd syntax would be equivalent of
"pattern eth / ipv4 / tcp / end".
These rules can not have rx queues configured, but have hash types
(l3 src and dst address). This means that the traffic distribution
is affected only by l3 addresses, independent of the l4 specifics.
Also these pattern matching rules have symmetric 6d5a
hash key configured. The length of the key is dependent on DPDK version.
The hash function (either RTE_ETH_HASH_FUNCTION_SYMMETRIC_TOEPLITZ or
RTE_ETH_HASH_FUNCTION_TOEPLITZ, depending on DPKD version) used
in these rules hashes symmetricaly due to the symmetric hash key.
The syntax in dpdk-testpmd for rule to match ipv4-tcp traffic with
attributes:
port index == 0
<hash_key> == 52 bytes long 6d5a symmetric hash key
is as follows:
"flow create 0 ingress pattern eth / ipv4 / tcp / end actions rss types
ipv4-tcp l3-src-only l3-dst-only end queues end key <hash_key>
key_len 52 func toeplitz / end"
(queues need to be set to NULL)
Ticket: 7337
Use a single array of built-ins and provide 2 functions for
registering them:
- SCLuaLoadBuiltIn: for loading built-in modules in sandboxed
environments.
- SCLuaRequirefBuiltIns: registers built-in modules with the standard
package tool, allows built-ins to be loaded by output scripts that are
not restricted
I hope to refactor the sandbox so they can use SCLuaRequirefBuiltIns
as well.
This is mainly for header sanitization to avoid pulling in detect
modules into the Lua sandbox definition.
Plus if we namespace modules with names like "suricata.dataset", it
probably makes sense to keep those modules in their own files.
Implement a more compact set of trees specifically for IPv4
and IPv6 addresses. This allows for more compact data structures
and fewer memory allocations.
Based on the existing radix tree implementation.
rust-bindings.h was not being installed with "make install-headers",
and its now pulled in by a header used for plugin support, so make
sure its installed.
We first attempt to install the "dist" version if exists, otherwise
install the "gen" one. Also install the "gen" even if the "dist" one
exists, as its going to be newer.
Adds user registerable callbacks for flow initialization, flow
update and flow finish.
Some plugins, such as other DPI libraries like nDPI need a way to hook
into these flow lifecycle events.
Ticket: #7319
Ticket: #7320
Jason Ish
Philippe Antoine
Jeff Lucovsky
Victor Julien
Eric Leblond
Todd Mortimer
Shivani Bhardwaj
Angelo Mirabella
Lukas Sismis
Juliana Fajardini
Alice Akaki
Nathan Scrivens
Adam Kiripolsky