mirror of https://github.com/OISF/suricata
You cannot select more than 25 topics
Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
477 lines
15 KiB
ReStructuredText
477 lines
15 KiB
ReStructuredText
.. _eve-json-output:
|
|
|
|
Eve JSON Output
|
|
===============
|
|
|
|
The EVE output facility outputs alerts, anomalies, metadata, file info and protocol
|
|
specific records through JSON.
|
|
|
|
The most common way to use this is through 'EVE', which is a firehose approach
|
|
where all these logs go into a single file.
|
|
|
|
.. literalinclude:: ../../partials/eve-log.yaml
|
|
|
|
Each alert, http log, etc will go into this one file: 'eve.json'. This file
|
|
can then be processed by 3rd party tools like Logstash (ELK) or jq.
|
|
|
|
Output types
|
|
~~~~~~~~~~~~
|
|
|
|
EVE can output to multiple methods. ``regular`` is a normal file. Other
|
|
options are ``syslog``, ``unix_dgram``, ``unix_stream`` and ``redis``.
|
|
|
|
Output types::
|
|
|
|
filetype: regular #regular|syslog|unix_dgram|unix_stream|redis
|
|
filename: eve.json
|
|
#prefix: "@cee: " # prefix to prepend to each log entry
|
|
# the following are valid when type: syslog above
|
|
#identity: "suricata"
|
|
#facility: local5
|
|
#level: Info ## possible levels: Emergency, Alert, Critical,
|
|
## Error, Warning, Notice, Info, Debug
|
|
#redis:
|
|
# server: 127.0.0.1
|
|
# port: 6379
|
|
# async: true ## if redis replies are read asynchronously
|
|
# mode: list ## possible values: list|lpush (default), rpush, channel|publish
|
|
# ## lpush and rpush are using a Redis list. "list" is an alias for lpush
|
|
# ## publish is using a Redis channel. "channel" is an alias for publish
|
|
# key: suricata ## key or channel to use (default to suricata)
|
|
# Redis pipelining set up. This will enable to only do a query every
|
|
# 'batch-size' events. This should lower the latency induced by network
|
|
# connection at the cost of some memory. There is no flushing implemented
|
|
# so this setting as to be reserved to high traffic suricata.
|
|
# pipelining:
|
|
# enabled: yes ## set enable to yes to enable query pipelining
|
|
# batch-size: 10 ## number of entry to keep in buffer
|
|
|
|
Alerts
|
|
~~~~~~
|
|
|
|
Alerts are event records for rule matches. They can be amended with
|
|
metadata, such as the application layer record (HTTP, DNS, etc) an
|
|
alert was generated for, and elements of the rule.
|
|
|
|
Metadata::
|
|
|
|
- alert:
|
|
#payload: yes # enable dumping payload in Base64
|
|
#payload-buffer-size: 4kb # max size of payload buffer to output in eve-log
|
|
#payload-printable: yes # enable dumping payload in printable (lossy) format
|
|
#packet: yes # enable dumping of packet (without stream segments)
|
|
#http-body: yes # Requires metadata; enable dumping of http body in Base64
|
|
#http-body-printable: yes # Requires metadata; enable dumping of http body in printable format
|
|
|
|
# metadata:
|
|
|
|
# Include the decoded application layer (ie. http, dns)
|
|
#app-layer: true
|
|
|
|
# Log the the current state of the flow record.
|
|
#flow: true
|
|
|
|
#rule:
|
|
# Log the metadata field from the rule in a structured
|
|
# format.
|
|
#metadata: true
|
|
|
|
# Log the raw rule text.
|
|
#raw: false
|
|
|
|
Anomaly
|
|
~~~~~~~
|
|
|
|
Anomalies are event records created when packets with unexpected or anomalous
|
|
values are handled. These events include conditions such as incorrect protocol
|
|
values, incorrect protocol length values, and other conditions which render the
|
|
packet suspect. Other conditions may occur during the normal progression of a stream;
|
|
these are termed ``stream`` events are include control sequences with incorrect
|
|
values or that occur out of expected sequence.
|
|
|
|
Anomalies are reported by and configured by type:
|
|
|
|
- Decode
|
|
- Stream
|
|
- Application layer
|
|
|
|
Metadata::
|
|
|
|
- anomaly:
|
|
# Anomaly log records describe unexpected conditions such as truncated packets,
|
|
# packets with invalid IP/UDP/TCP length values, and other events that render
|
|
# the packet invalid for further processing or describe unexpected behavior on
|
|
# an established stream. Networks which experience high occurrences of
|
|
# anomalies may experience packet processing degradation.
|
|
#
|
|
# Anomalies are reported for the following:
|
|
# 1. Decode: Values and conditions that are detected while decoding individual
|
|
# packets. This includes invalid or unexpected values for low-level protocol
|
|
# lengths as well.
|
|
# 2. Stream: This includes stream related events (TCP 3-way handshake issues,
|
|
# unexpected sequence number, etc).
|
|
# 3. Application layer: These denote application layer specific conditions that
|
|
# are unexpected, invalid or are unexpected given the application monitoring
|
|
# state.
|
|
#
|
|
# By default, anomaly logging is disabled. When anomaly logging is enabled,
|
|
# application-layer anomaly reporting is enabled.
|
|
#
|
|
# Choose one or both types of anomaly logging and whether to enable
|
|
# logging of the packet header for packet anomalies.
|
|
types:
|
|
#decode: no
|
|
#stream: no
|
|
#applayer: yes
|
|
#packethdr: no
|
|
|
|
HTTP
|
|
~~~~
|
|
|
|
HTTP transaction logging.
|
|
|
|
Config::
|
|
|
|
- http:
|
|
extended: yes # enable this for extended logging information
|
|
# custom allows additional http fields to be included in eve-log
|
|
# the example below adds three additional fields when uncommented
|
|
#custom: [Accept-Encoding, Accept-Language, Authorization]
|
|
# set this value to one among {both, request, response} to dump all
|
|
# http headers for every http request and/or response
|
|
# dump-all-headers: [both, request, response]
|
|
|
|
List of custom fields:
|
|
|
|
====================== ======================
|
|
Yaml Option HTTP Header
|
|
====================== ======================
|
|
accept accept
|
|
accept_charset accept-charset
|
|
accept_encoding accept-encoding
|
|
accept_language accept-language
|
|
accept_datetime accept-datetime
|
|
authorization authorization
|
|
cache_control cache-control
|
|
cookie cookie
|
|
from from
|
|
max_forwards max-forwards
|
|
origin origin
|
|
pragma pragma
|
|
proxy_authorization proxy-authorization
|
|
range range
|
|
te te
|
|
via via
|
|
x_requested_with x-requested-with
|
|
dnt dnt
|
|
x_forwarded_proto x-forwarded-proto
|
|
x_authenticated_user x-authenticated-user
|
|
x_flash_version x-flash-version
|
|
accept_range accept-range
|
|
age age
|
|
allow allow
|
|
connection connection
|
|
content_encoding content-encoding
|
|
content_language content-language
|
|
content_length content-length
|
|
content_location content-location
|
|
content_md5 content-md5
|
|
content_range content-range
|
|
content_type content-type
|
|
date date
|
|
etag etags
|
|
expires expires
|
|
last_modified last-modified
|
|
link link
|
|
location location
|
|
proxy_authenticate proxy-authenticate
|
|
referrer referrer
|
|
refresh refresh
|
|
retry_after retry-after
|
|
server server
|
|
set_cookie set-cookie
|
|
trailer trailer
|
|
transfer_encoding transfer-encoding
|
|
upgrade upgrade
|
|
vary vary
|
|
warning warning
|
|
www_authenticate www-authenticate
|
|
true_client_ip true-client-ip
|
|
org_src_ip org-src-ip
|
|
x_bluecoat_via x-bluecoat-via
|
|
====================== ======================
|
|
|
|
In the ``custom`` option values from both columns can be used. The
|
|
``HTTP Header`` column is case insensitive.
|
|
|
|
DNS
|
|
~~~
|
|
|
|
.. note:: As of Suricata 5.0, the version 2 format of the EVE DNS log
|
|
is the default.
|
|
|
|
DNS records are logged as one entry for the request, and one entry for
|
|
the response.
|
|
|
|
YAML::
|
|
|
|
- dns:
|
|
# As of Suricata 5.0, version 2 of the eve dns output
|
|
# format is the default.
|
|
#version: 2
|
|
|
|
# Enable/disable this logger. Default: enabled.
|
|
#enabled: yes
|
|
|
|
# Control logging of requests and responses:
|
|
# - requests: enable logging of DNS queries
|
|
# - responses: enable logging of DNS answers
|
|
# By default both requests and responses are logged.
|
|
#requests: no
|
|
#responses: no
|
|
|
|
# Format of answer logging:
|
|
# - detailed: array item per answer
|
|
# - grouped: answers aggregated by type
|
|
# Default: all
|
|
#formats: [detailed, grouped]
|
|
|
|
# Types to log, based on the query type.
|
|
# Default: all.
|
|
#types: [a, aaaa, cname, mx, ns, ptr, txt]
|
|
|
|
DNS v1 Format
|
|
~~~~~~~~~~~~~
|
|
|
|
The version 1 DNS output has been obsoleted by the version 2 output
|
|
above. The v1 format logs a record per answer in the response possibly
|
|
resulting in much more than 2 log records per request and response.
|
|
|
|
YAML::
|
|
|
|
- dns:
|
|
# Must set the version to 1 to get the old style format.
|
|
version: 1
|
|
# control logging of queries and answers
|
|
# default yes, no to disable
|
|
query: yes # enable logging of DNS queries
|
|
answer: yes # enable logging of DNS answers
|
|
# control which RR types are logged
|
|
# all enabled if custom not specified
|
|
#custom: [a, aaaa, cname, mx, ns, ptr, txt]
|
|
|
|
To reduce verbosity the output can be filtered by supplying the record types
|
|
to be logged under ``custom``.
|
|
|
|
TLS
|
|
~~~
|
|
|
|
TLS records are logged one record per session.
|
|
|
|
YAML::
|
|
|
|
- tls:
|
|
extended: yes # enable this for extended logging information
|
|
# custom allows to control which tls fields that are included
|
|
# in eve-log
|
|
#custom: [subject, issuer, serial, fingerprint, sni, version, not_before, not_after, certificate, chain, ja3, ja3s]
|
|
|
|
The default is to log certificate subject and issuer. If ``extended`` is
|
|
enabled, then the log gets more verbose.
|
|
|
|
By using ``custom`` it is possible to select which TLS fields to log.
|
|
|
|
Date modifiers in filename
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
It is possible to use date modifiers in the eve-log filename.
|
|
|
|
::
|
|
|
|
outputs:
|
|
- eve-log:
|
|
filename: eve-%s.json
|
|
|
|
The example above adds epoch time to the filename. All the date modifiers from the
|
|
C library should be supported. See the man page for ``strftime`` for all supported
|
|
modifiers.
|
|
|
|
.. _output_eve_rotate:
|
|
|
|
Rotate log file
|
|
~~~~~~~~~~~~~~~
|
|
|
|
Eve-log can be configured to rotate based on time.
|
|
|
|
::
|
|
|
|
outputs:
|
|
- eve-log:
|
|
filename: eve-%Y-%m-%d-%H:%M.json
|
|
rotate-interval: minute
|
|
|
|
The example above creates a new log file each minute, where the filename contains
|
|
a timestamp. Other supported ``rotate-interval`` values are ``hour`` and ``day``.
|
|
|
|
In addition to this, it is also possible to specify the ``rotate-interval`` as a
|
|
relative value. One example is to rotate the log file each X seconds.
|
|
|
|
::
|
|
|
|
outputs:
|
|
- eve-log:
|
|
filename: eve-%Y-%m-%d-%H:%M:%S.json
|
|
rotate-interval: 30s
|
|
|
|
The example above rotates eve-log each 30 seconds. This could be replaced with
|
|
``30m`` to rotate every 30 minutes, ``30h`` to rotate every 30 hours, ``30d``
|
|
to rotate every 30 days, or ``30w`` to rotate every 30 weeks.
|
|
|
|
Multiple Logger Instances
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
It is possible to have multiple 'EVE' instances, for example the following is valid:
|
|
|
|
::
|
|
|
|
outputs:
|
|
- eve-log:
|
|
enabled: yes
|
|
type: file
|
|
filename: eve-ips.json
|
|
types:
|
|
- alert
|
|
- drop
|
|
|
|
- eve-log:
|
|
enabled: yes
|
|
type: file
|
|
filename: eve-nsm.json
|
|
types:
|
|
- http
|
|
- dns
|
|
- tls
|
|
|
|
So here the alerts and drops go into 'eve-ips.json', while http, dns and tls go into 'eve-nsm.json'.
|
|
|
|
With the exception of ``drop``, you can specify multiples of the same
|
|
logger type, however, ``drop`` can only be used once.
|
|
|
|
.. note:: The use of independent json loggers such as alert-json-log,
|
|
dns-json-log, etc. has been deprecated and will be removed
|
|
by June 2020. Please use multiple eve-log instances as
|
|
documented above instead. Please see the `deprecation
|
|
policy`_ for more information.
|
|
|
|
File permissions
|
|
~~~~~~~~~~~~~~~~
|
|
|
|
Log file permissions can be set individually for each logger. ``filemode`` can be used to
|
|
control the permissions of a log file, e.g.:
|
|
|
|
::
|
|
|
|
outputs:
|
|
- eve-log:
|
|
enabled: yes
|
|
filename: eve.json
|
|
filemode: 600
|
|
|
|
The example above sets the file permissions on ``eve.json`` to 600, which means that it is
|
|
only readable and writable by the owner of the file.
|
|
|
|
JSON flags
|
|
~~~~~~~~~~
|
|
|
|
Several flags can be specified to control the JSON output in EVE:
|
|
|
|
::
|
|
|
|
outputs:
|
|
- eve-log:
|
|
json:
|
|
# Sort object keys in the same order as they were inserted
|
|
preserve-order: yes
|
|
|
|
# Make the output more compact
|
|
compact: yes
|
|
|
|
# Escape all unicode characters outside the ASCII range
|
|
ensure-ascii: yes
|
|
|
|
# Escape the '/' characters in string with '\/'
|
|
escape-slash: yes
|
|
|
|
All these flags are enabled by default, and can be modified per EVE instance.
|
|
|
|
Community Flow ID
|
|
~~~~~~~~~~~~~~~~~
|
|
|
|
Often Suricata is used in combination with other tools like Bro/Zeek. Enabling
|
|
the community-id option in the eve-log section adds a new ``community_id``
|
|
field to each output.
|
|
|
|
Example::
|
|
|
|
{
|
|
"timestamp": "2003-12-16T13:21:44.891921+0000",
|
|
"flow_id": 1332028388187153,
|
|
"pcap_cnt": 1,
|
|
"event_type": "alert",
|
|
...
|
|
"community_id": "1:LQU9qZlK+B5F3KDmev6m5PMibrg=",
|
|
"alert": {
|
|
"action": "allowed",
|
|
"gid": 1,
|
|
"signature_id": 1,
|
|
},
|
|
}
|
|
{
|
|
"timestamp": "2003-12-16T13:21:45.037333+0000",
|
|
"flow_id": 1332028388187153,
|
|
"event_type": "flow",
|
|
"flow": {
|
|
"pkts_toserver": 5,
|
|
"pkts_toclient": 4,
|
|
"bytes_toserver": 338,
|
|
"bytes_toclient": 272,
|
|
"start": "2003-12-16T13:21:44.891921+0000",
|
|
"end": "2003-12-16T13:21:45.346457+0000",
|
|
"age": 1,
|
|
"state": "closed",
|
|
"reason": "shutdown",
|
|
"alerted": true
|
|
},
|
|
"community_id": "1:LQU9qZlK+B5F3KDmev6m5PMibrg=",
|
|
}
|
|
|
|
Options
|
|
"""""""
|
|
|
|
The output can be enabled per instance of the EVE logger.
|
|
|
|
The ``community-id`` option is boolean. If set to ``true`` it is enabled.
|
|
The ``community-id-seed`` option specifies a unsigned 16 bit value that
|
|
is used a seed to the hash that is calculated for the ``community-id``
|
|
output. This must be set to the same value on all tools that output this
|
|
record.
|
|
|
|
YAML::
|
|
|
|
- eve-log:
|
|
# Community Flow ID
|
|
# Adds a 'community_id' field to EVE records. These are meant to give
|
|
# a records a predictable flow id that can be used to match records to
|
|
# output of other tools such as Bro.
|
|
#
|
|
# Takes a 'seed' that needs to be same across sensors and tools
|
|
# to make the id less predictable.
|
|
|
|
# enable/disable the community id feature.
|
|
community-id: false
|
|
# Seed value for the ID output. Valid values are 0-65535.
|
|
community-id-seed: 0
|
|
|
|
|
|
.. _deprecation policy: https://suricata-ids.org/about/deprecation-policy/
|