Cache Hyperscan serialized databases to disk to prevent compilation
of the same databases when Suricata is run again with the same
ruleset.
Hyperscan binary files are stored per rulegroup in the designated
folder, by default in the cached library folder.
Since caching is per signature group heads,
some chunk of the ruleset can change and it still can reuse part of
the unchanged signature groups.
Loading *fresh* ET Open ruleset: 19 seconds
Loading *cached* ET Open ruleset: 07 seconds
Ticket: 7170
Methods:
`get` creates the flow object.
`id` returns the flow id.
`has_alerts` returns a boolean indicating if the flow triggered alerts.
`app_layer_proto` returns various app-layer related fields as 5 strings:
alproto, alproto_ts, alproto_tc, alproto_orig, alproto_expect.
`stats` returns cnts for bytes and packets to sever and to client, as 4
numbers.
`tuple` -- returns various fields: srcip, dstip, proto, sp, dp.
`timestamps` returns time as 4 numbers: seconds and microseconds, for
first and last packet of the flow.
`timestring_legacy` returns the first packet from the flow's timestring
as a string (like fastlog).
`timestring_iso8601` returns the first packet from the flow's
timestring as an iso8601 compat timestring (like eve).
Example:
```
name = "lua-scflowstats.log"
local flow = require("suricata.flow")
function init(args)
local needs = {}
needs["type"] = "flow"
return needs
end
function setup(args)
filename = SCLogPath() .. "/" .. name
file = assert(io.open(filename, "a"))
SCLogInfo("lua SCFlowStats Log Filename " .. filename)
end
function log(args)
local f = flow.get()
timestring = f:timestring_legacy()
tscnt, tsbytes, tccnt, tcbytes = f:stats()
file:write ("[**] " .. timestring .. "\nSCFlowStats is\nPacket count to server: " .. tscnt .. "\nByte count to server: " .. tsbytes .. "\nPacket count to client: " .. tccnt .. "\nByte count to client: " .. tcbytes .. "\n[**]")
file:flush()
end
function deinit(args)
file:close(file)
end
```
Task #7489
Add keywords dns.additionals.rrname and dns.authorities.rrname. Along
the way, consolidate dns.query.name and dns.answer.name into a single file
and register them altogether since there is a lot of common code.
Feature: 7012
Add dns.response sticky buffer to match on dns response fields.
Add rust functions to return dns response packet data.
Unit tests verifying signature matching.
Bindgen works by processing a header file which includes all other
header files it should generate bindings for. For this I've created
bindgen.h which just includes app-layer-protos.h for now as an
example.
These bindings are then generated and saved in the "suricata-sys"
crate and become availale as "suricata_sys::sys".
Ticket: #7341
Example:
```
local packet = require "suricata.packet"
function init (args)
local needs = {}
return needs
end
function match (args)
p = packet.get()
payload = p:payload()
ts = p:timestring()
for line in payload:gmatch("([^\r\n]*)[\r\n]+") do
if line == "GET /index.html HTTP/1.0" then
ipver, srcip, dstip, proto, sp, dp = p:tuple()
SCLogNotice(string.format("%s %s->%s %d->%d (pcap_cnt:%d) match! %s", ts, srcip, dstip, sp, dp, p:pcap_cnt(), line));
return 1
end
end
return 0
end
```
Methods:
`get` creates the packet object.
`payload` returns the packet payload as a buffer
`packet` returns the whole packet (includes headers)
`pcap_cnt` returns the `pcap_cnt` (pcap file mode only)
`tuple` returns various fields: srcip, dstip, proto, sp, dp
`sp` returns source port
`dp` returns destination port
`timestamp` returns time as 2 numbers: seconds and microseconds
`timestring_legacy` returns a timestamp as a string (like fastlog)
`timestring_iso8601` returns a iso8601 compat timestring (like eve)
Ticket: #7488.
The configuration of this rule is the same as for ixgbe driver except
the hash function is not RTE_ETH_HASH_FUNCTION_DEFAULT but
RTE_ETH_HASH_FUNCTION_TOEPLITZ.
The syntax in dpdk-testpmd for this rule with attributes:
port index == 0
used rx queue indices == 0 1 2 3
<hash_key> == 6d5a symmetric hash key
is as follows:
"flow create 0 ingress pattern eth / end actions rss types ipv4 ipv6
end queues 0 1 2 3 end key <hash_key> key_len 40 func toeplitz / end"
Ticket: 7337
Move and adjust the base of RSS configuration from util-dpdk-i40e.c to
a new file that can be later utilized by other cards.
RSS configuration can be configured via rte_flow rules. This is useful
for possible future features such as specific header offload
(vxlan, nvgre) also implemented via rte_flow rules, as rte_flow
rules can be chained via groups and priorities.
i40e uses multiple different rte_flow rules to setup RSS. At first,
function DeviceSetRSSFlowQueues() is used to setup rx queues.
This rule matches all types of traffic, so the equivalent
to dpdk-testpmd pattern would be "pattern end"
This rule can not contain hash types (ipv4, ipv6 etc.) nor hash key.
The hash function used here is RTE_ETH_HASH_FUNCTION_DEFAULT.
The syntax in dpdk-testpmd for this rule with attributes:
port index == 0
used rx queue indices == 0 1 2 3
is as follows:
"flow create 0 ingress pattern end actions rss queues 0 1 2 3 end
func default / end"
The other rules configured by i40eDeviceSetRSSFlowIPv4() and
i40eDeviceSetRSSFlowIPv6() match specific type of traffic by l4 protocol
(none, TCP, UDP, SCTP). For example, pattern to match l3 ipv4 with l4
tcp traffic in dpdk-testpmd syntax would be equivalent of
"pattern eth / ipv4 / tcp / end".
These rules can not have rx queues configured, but have hash types
(l3 src and dst address). This means that the traffic distribution
is affected only by l3 addresses, independent of the l4 specifics.
Also these pattern matching rules have symmetric 6d5a
hash key configured. The length of the key is dependent on DPDK version.
The hash function (either RTE_ETH_HASH_FUNCTION_SYMMETRIC_TOEPLITZ or
RTE_ETH_HASH_FUNCTION_TOEPLITZ, depending on DPKD version) used
in these rules hashes symmetricaly due to the symmetric hash key.
The syntax in dpdk-testpmd for rule to match ipv4-tcp traffic with
attributes:
port index == 0
<hash_key> == 52 bytes long 6d5a symmetric hash key
is as follows:
"flow create 0 ingress pattern eth / ipv4 / tcp / end actions rss types
ipv4-tcp l3-src-only l3-dst-only end queues end key <hash_key>
key_len 52 func toeplitz / end"
(queues need to be set to NULL)
Ticket: 7337
Use a single array of built-ins and provide 2 functions for
registering them:
- SCLuaLoadBuiltIn: for loading built-in modules in sandboxed
environments.
- SCLuaRequirefBuiltIns: registers built-in modules with the standard
package tool, allows built-ins to be loaded by output scripts that are
not restricted
I hope to refactor the sandbox so they can use SCLuaRequirefBuiltIns
as well.
This is mainly for header sanitization to avoid pulling in detect
modules into the Lua sandbox definition.
Plus if we namespace modules with names like "suricata.dataset", it
probably makes sense to keep those modules in their own files.
Implement a more compact set of trees specifically for IPv4
and IPv6 addresses. This allows for more compact data structures
and fewer memory allocations.
Based on the existing radix tree implementation.
rust-bindings.h was not being installed with "make install-headers",
and its now pulled in by a header used for plugin support, so make
sure its installed.
We first attempt to install the "dist" version if exists, otherwise
install the "gen" one. Also install the "gen" even if the "dist" one
exists, as its going to be newer.
Adds user registerable callbacks for flow initialization, flow
update and flow finish.
Some plugins, such as other DPI libraries like nDPI need a way to hook
into these flow lifecycle events.
Ticket: #7319
Ticket: #7320
Add a pure rust base64 decoder. This supports 3 modes of operation just
like the C decoder as follows.
1. RFC 2045
2. RFC 4648
3. Strict
One notable change is that "strict" mode is carried out by the rust
base64 crate instead of native Rust. This crate was already used for
encoding in a few places like datasets of string type. As a part of this
mode, now, only the strings that can be reliably converted back are
decoded.
The decoder fn is available to C via FFI.
Bug 6280
Ticket 7065
Ticket 7058
Ticket: 4863
On the way, convert some keywords to use the first-class integer
support.
And helpers for pure rust the support for multi-buffer.
Move the C unit tests about keyword mqtt.protocol_version
to unit tests for generic integer parsing, and test version 5
instead of testing twice version 3.
Also iterate all tx's messages for reason code as is done for other
keywords.
And allow detection on empty topics.
Issue: 6487
Implement the from_base64 transform:
[bytes value] [offset value] [mode strict|rfc4648|rfc2045]
The value for bytes and offset may be a byte_ variable or an
unsigned integer.
Ticket: 4863
On the way, convert unit test DetectSNMPCommunityTest to a SV test.
And also, make snmp.pdu_type use a generic uint32 for detection,
allowing operators, instead of just equality.
Lukas Sismis
Juliana Fajardini
Alice Akaki
Jason Ish
Nathan Scrivens
Jeff Lucovsky
Victor Julien
Adam Kiripolsky
Philippe Antoine
Shivani Bhardwaj
Giuseppe Longo
mmaatuq