Commit Graph

17585 Commits (e2d821793412c46bd55aae298c1f12876e56aea3)
 

Author SHA1 Message Date
Eric Leblond e2d8217934 eve/schema: document datajson output 3 months ago
Eric Leblond 0e88e36020 eve/schema: pktvars is a container
It can contain any vars so need addition properties.
3 months ago
Eric Leblond 3fbc718728 detect/pcre: add extraction for alert
With datajson infrastructure in place, it is now possible to
add data in the extra information section. Following an idea
by Jason Ish, this patch adds the feature for pcre extraction.

A PCRE such as pcre:"/(?P<alert_ua>[a-zA-Z]+)\//" will add the
content of the captured group to alert.extra.ua.
3 months ago
Eric Leblond dd94dc6cc6 datajson: introduce feature
This patch introduces new option to dataset keyword.
Where regular dataset allows match from sets, dataset with json
format allows the same but also adds JSON data to the alert
event. This data is coming from the set definition it self.
For example, an ipv4 set will look like:

  [{"ip": "10.16.1.11", "test": "success","context":3}]

The syntax is a JSON array but it can also be a JSON object
with an array inside. The idea is to directly used data coming
from the API of a threat intel management software.

The syntax of the keyword is the following:

  dataset:isset,src_ip,type ip,load src.lst,format json, \
       enrichment_key src_ip, value_key ip;

Compare to dataset, it just have a supplementary option key
that is used to indicate in which subobject the JSON value
should be added.

The information is added in the even under the alert.extra
subobject:

  "alert": {
    "extra": {
      "src_ip": {
        "ip": "10.6.1.11",
        "test": "success",
        "context": 3
      },

The main interest of the feature is to be able to contextualize
a match. For example, if you have an IOC source, you can do

 [
   {"buffer": "value1", "actor":"APT28","Country":"FR"},
   {"buffer": "value2", "actor":"APT32","Country":"NL"}
 ]

This way, a single dataset is able to produce context to the
event where it was not possible before and multiple signatures
had to be used.

The format introduced in datajson is an evolution of the
historical datarep format. This has some limitations. For example,
if a user fetch IOCs from a threat intel server there is a large
change that the format will be JSON or XML. Suricata has no support
for the second but can support the first one.

Keeping the key value may seem redundant but it is useful to have it
directly accessible in the extra data to be able to query it
independantly of the signature (where it can be multiple metadata
or even be a transformed metadata).

In some case, when interacting with data (mostly coming from
threat intel servers), the JSON array containing the data
to use is not at the root of the object and it is ncessary
to access a subobject.

This patch implements this with support of key in level1.level2.
This is done via the `array_key` option that contains the path
to the data.

Ticket: #7372
3 months ago
Eric Leblond 53ac35337a util/byte: add HexToRaw function 3 months ago
Eric Leblond 61e485446a util/ip: add IPv4 and IPv6 length 3 months ago
Victor Julien d5ae9156b9 detect: replace DetectEngineCtx flag with EngineModeIsFirewall 3 months ago
Victor Julien b0b3808a55 detect/config: supported for firewall rules 3 months ago
Victor Julien 1ee9b15a6a help: group and reorder help/usage output 3 months ago
Victor Julien fa3311a253 commandline: add --help option
Acts same as -h.
3 months ago
Victor Julien 08925eac18 firewall: add --firewall and firewall.enabled
Allows for enabling firewall mode
3 months ago
Victor Julien 0e18048ef0 firewall: move config into yaml object
To make it easier to group settings or include them.
3 months ago
Pierre Chifflier adcee8d7b5 ldap: avoid unneeded renaming of variables 3 months ago
Pierre Chifflier ebc1678c5c ldap: fix clippy warnings (unneded conversions) 3 months ago
Pierre Chifflier bda22c1f4a ldap: factorize code and remove duplicated structs, use ldap_parser where relevant 3 months ago
Pierre Chifflier c152c5c7e0 ldap: update ldap-parser to 0.5.0 3 months ago
Alice Akaki 3065374314 json/schema: link file.name to email.attachment
As a Suricata keyword.

Ticket: #7683
3 months ago
Jason Ish 8e8c3040e7 doc/upgrade: note about dns address swap on responses
Document the change in DNS addresses for ticket 6400.

Ticket: https://redmine.openinfosecfoundation.org/issues/6400
3 months ago
Philippe Antoine f6856eac9c fuzz: fix -Wshorten-64-to-32 warnings
Ticket: #6186
3 months ago
Philippe Antoine 9e64555c5e util: fix -Wshorten-64-to-32 warnings for afpacket
Ticket: #6186
3 months ago
Philippe Antoine e0e91c9302 configure: add -Wshorten-64-to-32 to the flags
when configure is run with --enable-warnings

Ticket: 6186

Also add -Wimplicit-int-conversion to the flags

Both are not compatible with unit tests
3 months ago
Philippe Antoine 7b492bc83b detect/engine: fix -Wshorten-64-to-32 warnings
Ticket: #6186

Especially take care of the case where byte_extract extracts a u64
value that does not fit in a u32
3 months ago
Victor Julien c3f054b625 time: replace usleep by SleepUsec/SleepMsec
Helps cross platform support, esp Windows
3 months ago
Victor Julien 35e711d00a threads: clean up module flags
Remove unused TM_FLAG_STREAM_TM.

Rename TM_FLAG_DETECT_TM to TM_FLAG_FLOWWORKER_TM as it was mostly used
to check if a thread is a flow worker. TM_FLAG_DETECT_TM was always set
for a flow worker, even when there was no detection in use.
3 months ago
Victor Julien bdac028fc7 threading: fix shutdown of IPS autofp modes
For IPS modes with a verdict thread in autofp there was an issue with
the verdict thread not shutting down, leading to a long shutdown time
until an error condition was reached.

The problem was that when the packet threads, of which the verdict
thread is one, were told to enter their flow timeout loop the verdict
thread got stuck as it immediately progressed to THV_RUNNING_DONE
instead of the expected THV_FLOW_LOOP.

This patch updates the shutdown logic to only apply the flow timeout
logic to the relevant threads, and skip the verdict thread(s).

Add TM_FLAG_VERDICT_TM to indicate a thread has a verdict module to more
explicitly shut it down.

Fixes: 12f8f03532 ("threads: fix autofp shutdown race condition")

Bug: #7681.
3 months ago
Philippe Antoine 6dbc421825 rust: bindgen AppLayerParserConfParserEnabled
Ticket: 7667
3 months ago
Philippe Antoine 49b2a2be5d rust: bindgen SCAppLayerParserRegisterLogger
Ticket: 7667
3 months ago
Philippe Antoine 7bfef2e1e8 rust: bindgen AppLayerParserRegisterParserAcceptableDataDirection
Ticket: 7667
3 months ago
Philippe Antoine 1ff4dbfc24 rust: bindgen AppLayerParserSetStreamDepth
Ticket: 7667
3 months ago
Philippe Antoine b29d46d81f rust: bindgen SCAppLayerParserStateIssetFlag
Ticket: 7667
3 months ago
Philippe Antoine 6d56beffef rust: bindgen AppLayerParserStateSetFlag
Ticket: 7667
3 months ago
Philippe Antoine 249131e9de src: clean includes for app-layer-parser.h
To prepare bindgening
3 months ago
Victor Julien f2faba5a23 detect/config: add flow tracking doc 3 months ago
Victor Julien 00336c45f4 detect/config: remove unused include 3 months ago
Victor Julien 67902c791d detect/config: add func docs 3 months ago
Victor Julien ec6081363c detect/config: remove filestore reference from comments 3 months ago
Victor Julien 6e0a6abb0b detect/config: add support for skipping flow tracking
Allow rules in the `pre_flow` hook to disable flow tracking for a
packet:

    config:packet tcp:pre_flow any any <> any 12345 (           \
        config: tracking disable, type flow, scope packet;      \
        sid:1;)

This rule will be evaluated before a packet is handled by the flow
engine, and a match will ensure that the flow engine is skipped.

Ticket: #7715.
3 months ago
Victor Julien f1fdc1801e detect: set detect table for non-firewall mode as well
This also exposed a difference between the handling of TD alerts in
firewall vs non-firewall mode. In firewall mode the table/hook is also
part of the alert ordering to make sure actions from packet:td are
applied before app:td. Handle that explicitly for now.
3 months ago
Victor Julien 7896148798 detect/config: allow setting a scope for action config 3 months ago
Victor Julien b554c43ab9 util/config: comment out unused types 3 months ago
Victor Julien 4836c3c46b detect/config: clean up keyword value parsing 3 months ago
Victor Julien 0de19d61aa detect: clean up signature validate logic
`SigValidate` was doing more than just validation. Break out the
function into validation steps and consolidation steps.
3 months ago
Victor Julien 12f2f3f9f1 detect: tables support per keyword
Allow keywords to specify in which detect table they can function.

E.g. the pre_flow table will not support flow keywords, as no flow is
availble at this time.
3 months ago
Victor Julien a88d58e531 detect: use accept:hook policy for pre_* hooks
Set firewall policy in scratch pad. Default to drop:packet for filter tables,
use accept:hook for pre_stream and pre_hook.
3 months ago
Victor Julien df7fe5b0ce detect: add pre_flow hook
Allows dropping of packets before a flow is created/updated.
Directionless as direction is inferred from the flow.

Ticket: #7714.
3 months ago
Victor Julien 8f2b925e09 detect: add pre_stream hook
Meant to be used from the detection engine, to allow rules to drop
traffic before it modifies the stream state.

Ticket: #7712.
3 months ago
Victor Julien ecbcccf355 detect: add tcp.wscale keyword
Allows matching on wscale option value in TCP header options.

Ticket: #7713.
3 months ago
Victor Julien 0246668de7 flow-worker: use explicit type for DetectEngineThreadCtx 3 months ago
Victor Julien 860bb11692 detect: only reset packet alert things in unittest mode 3 months ago
Victor Julien afa88757fe detect: pass de_ctx around as const 3 months ago