Commit Graph

17613 Commits (20a0575d96e936ed5d297e78d17d5c92a8faecc0)
 

Author SHA1 Message Date
Eric Leblond 20a0575d96 doc/userguide: fix some typos
Suggestions from Juliana.

Co-authored-by: Juliana Fajardini Reichow <jufajardini@gmail.com>
2 months ago
Eric Leblond 23f643a4a7 eve/schema: fix ordering 2 months ago
Eric Leblond fed7ff1e76 datasets: remove comment about unused return value 2 months ago
Eric Leblond 79c8f431d0 datajson: simplify memory handling
DatajsonAdd function is now responsible of the handling of the mem
of datajsontype passed as argument.
2 months ago
Eric Leblond 606f7ba656 datajson: use wrapper 2 months ago
Eric Leblond 7e7bd92432 datajson: fix potential leak in error handling 2 months ago
Eric Leblond 4a420085b7 datajson: remove unnecessary abstraction 2 months ago
Eric Leblond 4cadeb17e1 github: upload scan-build artifacts 2 months ago
Eric Leblond d57e8731bd datajson: use more broadly supported formatter 2 months ago
Eric Leblond 7dc683e5d9 datajson: fix formatting 2 months ago
Eric Leblond c15bcbe9d6 datasets: factorize dataset creation
Factorize DatasetGet and DatajsonGet to only have the difference
between the two in the respective function.
2 months ago
Eric Leblond 28742871d2 datasets: separate DatasetGet in 2 functions
This will be used to factorize the code with datajson.
2 months ago
Eric Leblond b766bd9438 datasets/context: rename the datajson files
They are is renamed to datasets-context-json.* so we see that
it is about context and not about a new datasets type.
2 months ago
Eric Leblond 40c545f8d9 doc/userguide: jsonline is now standard ndjson 2 months ago
Eric Leblond f7dc2a71db datajson: rename jsonline to ndjson 2 months ago
Eric Leblond f724c75cc9 doc/userguide: improve datajson doc 2 months ago
Eric Leblond b03d4f8e1a datajson: output context to "context"
Using `alert.extra` was not really reflecting the nature of what
was added. So renaming it to `alert.context`.
2 months ago
Eric Leblond 0bc21eec48 datajson: fix thread safety violation 2 months ago
Eric Leblond 40f86571d9 datajson: reduce size length 2 months ago
Eric Leblond b8e2791482 datajson: fix string format in error message 2 months ago
Eric Leblond 61ac7b46c1 eve/schema: remove reference to datajson 2 months ago
Eric Leblond a652eee508 doc/userguide: remove left over datajson reference 2 months ago
Eric Leblond 7d28758a54 doc/userguide: improve datajson doc
Patch adds ``remove_key`` option and clarifies the text.
2 months ago
Eric Leblond 3dde17fb75 datajson: add remove_key option to dataset
This option allows to remove the key corresponding to the match
value from the JSON object before creating the JSON object that
will be added to the `extra` data.

For example, matching on the following JSON on the `ip` key:

```json
{"ip": "10.16.1.11", "test": "success", "context":3}
```

with a match like:

```
dataset:isset,src_ip,type ip,load src.lst,format jsonline,enrichment_key src_ip,value_key ip;
```

will produce the following:

```json
"extra": {
  "src_ip": {
    "ip": "10.16.1.11",
    "test": "success",
    "context": 3
  }
```

if we add the `remove_key` option to the match:

```
dataset:isset,src_ip,type ip,load src.lst,format jsonline,enrichment_key src_ip,value_key ip, remove_key;
```

it will produce the following:

```json
"extra": {
  "src_ip": {
    "test": "success",
    "context": 3
  }
```

The option is set to false by default.

Ticket: #7372
2 months ago
Eric Leblond 0ae88a408a doc/userguide: basic doc for jsonline format 2 months ago
Eric Leblond 3223d8fe59 datajson: implement jsonline format
This format allows to use a one valid JSON object per line in the
data file.

Ticket: #7372
2 months ago
Eric Leblond 8058964449 datajson: prepare jsonline format
There is just a change in the iterator to go from json to jsonline
so let's factorize the parsing functions.

Ticket: #7372
2 months ago
Eric Leblond 9873c5d2e1 doc/userguide: add dataset with json 2 months ago
Eric Leblond e2d8217934 eve/schema: document datajson output 2 months ago
Eric Leblond 0e88e36020 eve/schema: pktvars is a container
It can contain any vars so need addition properties.
2 months ago
Eric Leblond 3fbc718728 detect/pcre: add extraction for alert
With datajson infrastructure in place, it is now possible to
add data in the extra information section. Following an idea
by Jason Ish, this patch adds the feature for pcre extraction.

A PCRE such as pcre:"/(?P<alert_ua>[a-zA-Z]+)\//" will add the
content of the captured group to alert.extra.ua.
2 months ago
Eric Leblond dd94dc6cc6 datajson: introduce feature
This patch introduces new option to dataset keyword.
Where regular dataset allows match from sets, dataset with json
format allows the same but also adds JSON data to the alert
event. This data is coming from the set definition it self.
For example, an ipv4 set will look like:

  [{"ip": "10.16.1.11", "test": "success","context":3}]

The syntax is a JSON array but it can also be a JSON object
with an array inside. The idea is to directly used data coming
from the API of a threat intel management software.

The syntax of the keyword is the following:

  dataset:isset,src_ip,type ip,load src.lst,format json, \
       enrichment_key src_ip, value_key ip;

Compare to dataset, it just have a supplementary option key
that is used to indicate in which subobject the JSON value
should be added.

The information is added in the even under the alert.extra
subobject:

  "alert": {
    "extra": {
      "src_ip": {
        "ip": "10.6.1.11",
        "test": "success",
        "context": 3
      },

The main interest of the feature is to be able to contextualize
a match. For example, if you have an IOC source, you can do

 [
   {"buffer": "value1", "actor":"APT28","Country":"FR"},
   {"buffer": "value2", "actor":"APT32","Country":"NL"}
 ]

This way, a single dataset is able to produce context to the
event where it was not possible before and multiple signatures
had to be used.

The format introduced in datajson is an evolution of the
historical datarep format. This has some limitations. For example,
if a user fetch IOCs from a threat intel server there is a large
change that the format will be JSON or XML. Suricata has no support
for the second but can support the first one.

Keeping the key value may seem redundant but it is useful to have it
directly accessible in the extra data to be able to query it
independantly of the signature (where it can be multiple metadata
or even be a transformed metadata).

In some case, when interacting with data (mostly coming from
threat intel servers), the JSON array containing the data
to use is not at the root of the object and it is ncessary
to access a subobject.

This patch implements this with support of key in level1.level2.
This is done via the `array_key` option that contains the path
to the data.

Ticket: #7372
2 months ago
Eric Leblond 53ac35337a util/byte: add HexToRaw function 2 months ago
Eric Leblond 61e485446a util/ip: add IPv4 and IPv6 length 2 months ago
Victor Julien d5ae9156b9 detect: replace DetectEngineCtx flag with EngineModeIsFirewall 2 months ago
Victor Julien b0b3808a55 detect/config: supported for firewall rules 2 months ago
Victor Julien 1ee9b15a6a help: group and reorder help/usage output 2 months ago
Victor Julien fa3311a253 commandline: add --help option
Acts same as -h.
2 months ago
Victor Julien 08925eac18 firewall: add --firewall and firewall.enabled
Allows for enabling firewall mode
2 months ago
Victor Julien 0e18048ef0 firewall: move config into yaml object
To make it easier to group settings or include them.
2 months ago
Pierre Chifflier adcee8d7b5 ldap: avoid unneeded renaming of variables 2 months ago
Pierre Chifflier ebc1678c5c ldap: fix clippy warnings (unneded conversions) 2 months ago
Pierre Chifflier bda22c1f4a ldap: factorize code and remove duplicated structs, use ldap_parser where relevant 2 months ago
Pierre Chifflier c152c5c7e0 ldap: update ldap-parser to 0.5.0 2 months ago
Alice Akaki 3065374314 json/schema: link file.name to email.attachment
As a Suricata keyword.

Ticket: #7683
2 months ago
Jason Ish 8e8c3040e7 doc/upgrade: note about dns address swap on responses
Document the change in DNS addresses for ticket 6400.

Ticket: https://redmine.openinfosecfoundation.org/issues/6400
2 months ago
Philippe Antoine f6856eac9c fuzz: fix -Wshorten-64-to-32 warnings
Ticket: #6186
2 months ago
Philippe Antoine 9e64555c5e util: fix -Wshorten-64-to-32 warnings for afpacket
Ticket: #6186
2 months ago
Philippe Antoine e0e91c9302 configure: add -Wshorten-64-to-32 to the flags
when configure is run with --enable-warnings

Ticket: 6186

Also add -Wimplicit-int-conversion to the flags

Both are not compatible with unit tests
2 months ago
Philippe Antoine 7b492bc83b detect/engine: fix -Wshorten-64-to-32 warnings
Ticket: #6186

Especially take care of the case where byte_extract extracts a u64
value that does not fit in a u32
2 months ago