Commit Graph

135 Commits (master)

Author SHA1 Message Date
Jeff Lucovsky cbda276aeb output: Log ethernet type
Issue 7129

When configured with the existing "ethernet" switch, include the ether
type in the output.

This is most useful with anomaly records indicating unknown ethertypes.
4 weeks ago
Jeff Lucovsky 123b36b9f5 decode/ethertype: Event on unknown ethertype
Issue: 7129

Create a decode/engine event if unknown ethertypes are observed.
4 weeks ago
Philippe Antoine b967fcaf8f stats: remove unused pseudo_failed stat 2 months ago
Philippe Antoine b0282f49af http2: complete json schema
git grep js.set_string rust/src/http2/logger.rs | grep '"' |
cut -d'"' -f2 | sort | uniq | while read i;
do echo -n $i; git grep $i etc/schema.json | wc -l; done

shows only has_multiple was missing
2 months ago
Victor Julien 779f9d8ba3 eve/flow: add per flow TCP oob urg data counter
If TCP urgent handling is set to the OOB processing, the number of OOB
bytes is tracked for SEQ offset calculations. If this offset is
non-zero, add the field to the flow record.

Ticket: #7411.
3 months ago
Victor Julien 6882bcb3e5 stream: add TCP urgent handling options
TCP urgent handling is a complex topic due to conflicting RFCs and
implementations.

Until now the URG flag and urgent pointer values were simply ignored,
leading to an effective "inline" processing of urgent data. Many
implementations however, do not default to this behavior.

Many actual implementations use the urgent mechanism to send 1 byte of
data out of band to the application.

Complicating the matter is that the way the urgent logic is handled is
generally configurable both of the OS and the app level. So from the
network it is impossible to know with confidence what the settings are.

This patch adds the following policies:

`stream.reassembly.urgent.policy`:

- drop: drop URG packets before they affect the stream engine

- inline: ignore the urgent pointer and process all data inline

- oob (out of band): treat the last byte as out of band

- gap: skip the last byte, but do no adjust sequence offsets, leading to
       gaps in the data

For the `oob` option, tracking of a sequence number offset is required,
as the OOB data does "consume" sequence number space. This is limited to
64k. For this reason, there is a second policy:

`stream.reassembly.urgent.oob-limit-policy`:

- drop: drop URG packets before they affect the stream engine

- inline: ignore the urgent pointer and process all data inline

- gap: skip the last byte, but do no adjust sequence offsets, leading to
       gaps in the data

Bug: #7411.
3 months ago
Victor Julien ac02a71479 decode/tcp: count urg flag 3 months ago
Jason Ish 37f4c52b22 eve/dns: add truncation flags for fields that are truncated
If rrname, rdata or mname are truncated, set a flag field like
'rrname_truncated: true' to indicate that the name is truncated.

Ticket: #7280
3 months ago
Philippe Antoine f2c3776314 detect: log app-layer metadata in alert with single tx
Ticket: 7199

Uses a config parameter detect.guess-applayer-tx to enable
this behavior (off by default)

This feature is requested for use cases with signatures not
using app-layer keywords but still targetting application
layer transactions, such as pass/drop rule combination,
or lua usage.

This overrides the previous behavior of checking if the signature
has a content match, by checking if there is only one live
transaction, in addition to the config parameter being set.
3 months ago
Philippe Antoine 09ba69cfb0 output/http: log invalid status as a string
Ticket: 7311

If response_status_number is not a valid poisitive integer,
we should not try to parse it again, and fail again,
but just log the raw string.
3 months ago
Jeff Lucovsky b30df19f1a app-layer/stats: Expand memuse/memcap handling
This commit adds memcap/memuse handling to the unix-socket interface:
- ftp
- http-byterange
- host

New stats:
- ippair: memuse, memcap
- host: memuse, memcap
- http-byterange: memuse, memcap
4 months ago
Jason Ish b44fc62e60 eve/schema: add missing field "code" anomaly events 4 months ago
Juliana Fajardini 2eefc4dac8 schema/tls: add missing custom fields chain/cert
Task #7287
4 months ago
Jason Ish 6ae5ae701b doc/userguide: generate eve documentation
Add EVE documentation for QUIC and Pgsql to their respective sections of
the userguide.

Also add a complete EVE reference as an appendix.

Other protocols can be done, but its a manual process to document in the
schema, then add the glue to pull them into the documentation.

The documentation is generated during "make dist", or if it doesn't
exist, "conf.py" will attempt to generate the eve documentation for
building on Readthedocs.
5 months ago
Jeff Lucovsky a7b70ce5ce config: Update reference keys
Issue: 4974

1. Use https instead of http everywhere
2. Organize and annotate references by
    - Referenced by ET/Open and ET/Pro
        - URL resolves and works as intended (to provide supplemental
          information regarding a reference value, e.g., bug id, cve
          value)
        - URL no longer resolves
        - URL resolves but doesn't work as intended (to provide
          supplemental information)
    - Not referenced by ET/Open nor ET/Pro
        - URL resolves and works as intended (to provide supplemental
          information regarding a reference value, e.g., bug id, cve
          value)
        - URL no longer resolves
        - URL resolves but doesn't work as intended (to provide
          supplemental information)
5 months ago
Jeff Lucovsky 9bc468e63a alert/reference: Optionally add reference(s)
Issue: 4974

Optionally include rule references with the alert. Since there can be
multiple reference keywords, they are collected into an array.
5 months ago
Giuseppe Longo edf70276d6 rust/ldap: enable parser for udp
This introduces a new parser registration function for LDAP/UDP, and update
ldap configuration in order to be able to enable/disable a single parser
independently (such as dns).
Also, GAPs are accepted only for TCP parser and not for UDP.

Ticket #7203
7 months ago
Shivani Bhardwaj 638b5c4da7 eve/stats: add description for flow stats
Ticket 6434
7 months ago
Philippe Antoine 1e82e20c65 doh: implement dns over http2 app-proto
Ticket: 5773
7 months ago
Giuseppe Longo 910a5b226c rust/ldap: implement logger 8 months ago
Jason Ish fcc1b1067b eve/dns: make version required
The "eve.version" field is not always logged. Update the schema to
enforce that it is, and fix it for records that don't log it.

Ticket: #7167
8 months ago
Jason Ish df656324ba dns: new v3 style logging for alerts
V3 style DNS logging fixes the discrepancies between request and
response logging better dns records and alert records.

The main change is that queries and answers are always logged as
arrays, and header fields are not logged in array items.

For alerts this means that answers are now logged as arrays, queries
already were.

DNS records will get this new format as well, but with a configuration
parameter.

Bug: #6281
8 months ago
Nathan Scrivens 9ecc3573a7 dns: parse and populate OPT rdata struct
Feature: 7017
Add DNSRDataOPT struct and DNSRData enum type OPT.
Add OPT parsing function and test function.
Add DNSRData OPT type to lua.rs match.
Log OPT rdata.
8 months ago
Nathan Scrivens 4598ca164d dns log: add additional section
Feature: 7011
dns_log_json_answer: log additional section records.
update schema.json with new "additionals" section.
8 months ago
Philippe Antoine c9ce43b31e output: configurable payload_length field for alerts
Ticket: 7098
8 months ago
Victor Julien 869d5492dc eve/schema: update for alpn 8 months ago
Victor Julien 7f474af1d0 eve/schema: minor enip reformat 8 months ago
Philippe Antoine 82c03f72c3 enip: convert to rust
Ticket: 3958

- transactions are now bidirectional
- there is a logger
- gap support is improved with probing for resync
- frames support
- app-layer events
- enip_command keyword accepts now string enumeration as values.
- add enip.status keyword
- add keywords :
    enip.product_name, enip.protocol_version, enip.revision,
    enip.identity_status, enip.state, enip.serial, enip.product_code,
    enip.device_type, enip.vendor_id, enip.capabilities,
    enip.cip_attribute, enip.cip_class, enip.cip_instance,
    enip.cip_status, enip.cip_extendedstatus
9 months ago
Shivani Bhardwaj de1de53e2f eve/stats: add description for flow mgr & recycler
Ticket 6434
9 months ago
Philippe Antoine 9332bc2c45 dns: adds missing NS field in json schema 9 months ago
Philippe Antoine 8d4699fbba eve/schema: complete and reorder smtp fields
received and cc were missing
9 months ago
Victor Julien 1190e426f9 defrag: remove trackers on lookup
When looking up a tracker, remove any timed out / completed trackers.
9 months ago
Victor Julien 75b78d7643 defrag: add defrag.memuse counter
Gives a current snapshot of the memory in use by the defrag engine.
9 months ago
Victor Julien 83dc703d1f defrag: add various counters 9 months ago
Victor Julien fc05d253d2 defrag: add defrag.mgr.tracker_timeout counter
Updated by flow manager.
9 months ago
Victor Julien 76e05c72f6 eve/schema: reformat 9 months ago
Jason Ish 10e6028175 lua: track memory limit exceede errors
Update the Lua allocated to set a code on memory allocation limit
exceeded errors so an appropriate error message can be logged and a
state incremented.

Fixes the tracking of the allocated size by using the difference
between original size, and new size and toss in some debug
validations.
9 months ago
Jason Ish 5a1cba72f0 lua: add logging and counter for instruction limit being exceeded 9 months ago
Jason Ish c8fa454cb2 lua: add blocked functions as a special log type plus stat
Distinguish between a generic Lua script error and an error created by a
function being blocked, so each is logged once respective of each other.

Also add a stat that is incremented when a script fails due to a
blocked function.

NOTE: This does not catch calls to functions that are blocked by not
having the library loaded, such as "io.open", as they are blocked by
not even loading the "io" library.
9 months ago
Shivani Bhardwaj f073cf2350 eve/schema: add tls.subjectaltname fields
Feature 5234
9 months ago
Jason Ish 224f55ba21 detect/lua: don't treat a crashed script as no match
If a rule script crashed, the return value was treated as a no
match. This would make a negation of the rule match and alert.

Instead cleanup and exit early if the rule script crashed and don't
run negation logic.

A stat, detect.lua.errors has been added to count how many times a
script crashes.

Also consolidates the running of the Lua script and return value
handling to a common function.

Bug: #6940
10 months ago
Philippe Antoine 2c305ba37e pop3: protocol detection
Ticket: #6366
10 months ago
Giuseppe Longo 01586d884d output-json/arp: implement logger
This adds a logger for ARP, disabled by default.

Ticket #6827
10 months ago
Giuseppe Longo 5219a5da5f decode/arp: implement decoder
This adds a decoder for ARP.

Ticket #6827
10 months ago
Shivani Bhardwaj 329ac61961 eve/stats: add description for ips
Ticket 6434
10 months ago
Shivani Bhardwaj 861ffff972 eve/stats: add description for transactions
Ticket 6434
10 months ago
Giuseppe Longo bff790b6ac rust/sdp: implement logger
This implements a logger for the SDP protocol.
Given that SDP is encapsulated within other protocols (such as SIP),
enabling it separately is not necessary.

Ticket #6627
10 months ago
Philippe Antoine 0291d37009 websocket: configurable logging of payload in alerts 11 months ago
Philippe Antoine 44b6aa5e4b app-layer: websockets protocol support
Ticket: 2695
11 months ago
Sascha Steinbiss 120313f4da ja4: implement for TLS and QUIC
Ticket: OISF#6379
11 months ago