This option allows to remove the key corresponding to the match
value from the JSON object before creating the JSON object that
will be added to the `extra` data.
For example, matching on the following JSON on the `ip` key:
```json
{"ip": "10.16.1.11", "test": "success", "context":3}
```
with a match like:
```
dataset:isset,src_ip,type ip,load src.lst,format jsonline,enrichment_key src_ip,value_key ip;
```
will produce the following:
```json
"extra": {
"src_ip": {
"ip": "10.16.1.11",
"test": "success",
"context": 3
}
```
if we add the `remove_key` option to the match:
```
dataset:isset,src_ip,type ip,load src.lst,format jsonline,enrichment_key src_ip,value_key ip, remove_key;
```
it will produce the following:
```json
"extra": {
"src_ip": {
"test": "success",
"context": 3
}
```
The option is set to false by default.
Ticket: #7372
With datajson infrastructure in place, it is now possible to
add data in the extra information section. Following an idea
by Jason Ish, this patch adds the feature for pcre extraction.
A PCRE such as pcre:"/(?P<alert_ua>[a-zA-Z]+)\//" will add the
content of the captured group to alert.extra.ua.
This patch introduces new option to dataset keyword.
Where regular dataset allows match from sets, dataset with json
format allows the same but also adds JSON data to the alert
event. This data is coming from the set definition it self.
For example, an ipv4 set will look like:
[{"ip": "10.16.1.11", "test": "success","context":3}]
The syntax is a JSON array but it can also be a JSON object
with an array inside. The idea is to directly used data coming
from the API of a threat intel management software.
The syntax of the keyword is the following:
dataset:isset,src_ip,type ip,load src.lst,format json, \
enrichment_key src_ip, value_key ip;
Compare to dataset, it just have a supplementary option key
that is used to indicate in which subobject the JSON value
should be added.
The information is added in the even under the alert.extra
subobject:
"alert": {
"extra": {
"src_ip": {
"ip": "10.6.1.11",
"test": "success",
"context": 3
},
The main interest of the feature is to be able to contextualize
a match. For example, if you have an IOC source, you can do
[
{"buffer": "value1", "actor":"APT28","Country":"FR"},
{"buffer": "value2", "actor":"APT32","Country":"NL"}
]
This way, a single dataset is able to produce context to the
event where it was not possible before and multiple signatures
had to be used.
The format introduced in datajson is an evolution of the
historical datarep format. This has some limitations. For example,
if a user fetch IOCs from a threat intel server there is a large
change that the format will be JSON or XML. Suricata has no support
for the second but can support the first one.
Keeping the key value may seem redundant but it is useful to have it
directly accessible in the extra data to be able to query it
independantly of the signature (where it can be multiple metadata
or even be a transformed metadata).
In some case, when interacting with data (mostly coming from
threat intel servers), the JSON array containing the data
to use is not at the root of the object and it is ncessary
to access a subobject.
This patch implements this with support of key in level1.level2.
This is done via the `array_key` option that contains the path
to the data.
Ticket: #7372
Remove unused TM_FLAG_STREAM_TM.
Rename TM_FLAG_DETECT_TM to TM_FLAG_FLOWWORKER_TM as it was mostly used
to check if a thread is a flow worker. TM_FLAG_DETECT_TM was always set
for a flow worker, even when there was no detection in use.
For IPS modes with a verdict thread in autofp there was an issue with
the verdict thread not shutting down, leading to a long shutdown time
until an error condition was reached.
The problem was that when the packet threads, of which the verdict
thread is one, were told to enter their flow timeout loop the verdict
thread got stuck as it immediately progressed to THV_RUNNING_DONE
instead of the expected THV_FLOW_LOOP.
This patch updates the shutdown logic to only apply the flow timeout
logic to the relevant threads, and skip the verdict thread(s).
Add TM_FLAG_VERDICT_TM to indicate a thread has a verdict module to more
explicitly shut it down.
Fixes: 12f8f03532 ("threads: fix autofp shutdown race condition")
Bug: #7681.
Allow rules in the `pre_flow` hook to disable flow tracking for a
packet:
config:packet tcp:pre_flow any any <> any 12345 ( \
config: tracking disable, type flow, scope packet; \
sid:1;)
This rule will be evaluated before a packet is handled by the flow
engine, and a match will ensure that the flow engine is skipped.
Ticket: #7715.
This also exposed a difference between the handling of TD alerts in
firewall vs non-firewall mode. In firewall mode the table/hook is also
part of the alert ordering to make sure actions from packet:td are
applied before app:td. Handle that explicitly for now.
Eric Leblond
Victor Julien
Pierre Chifflier
Alice Akaki
Jason Ish
Philippe Antoine