Under eve/alert, introduce a new metadata configuration
section. If no provided, or simply yes defaults will be used.
Otherwise this a map with fields that can be toggled on and
off. The defaults are:
outputs:
- eve-log:
types:
- alert:
metadata:
app-layer: true
flow: true
rule:
raw: false
metadata: true
To enable something that is disabled by default, or to disable
something that is enabled by default, only that key need to
be changed, everything else will keep its default value.
For SIEM analysis it is often useful to refer to the actual rules to
find out why a specific alert has been triggered when the signature
message does not convey enough information.
Turn on the new rule flag to include the rule text in eve alert output.
The feature is turned off by default.
With a rule like this:
alert dns $HOME_NET any -> 8.8.8.8 any (msg:"Google DNS server contacted"; sid:42;)
The eve alert output might look something like this (pretty-printed for
readability):
{
"timestamp": "2017-08-14T12:35:05.830812+0200",
"flow_id": 1919856770919772,
"in_iface": "eth0",
"event_type": "alert",
"src_ip": "10.20.30.40",
"src_port": 50968,
"dest_ip": "8.8.8.8",
"dest_port": 53,
"proto": "UDP",
"alert": {
"action": "allowed",
"gid": 1,
"signature_id": 42,
"rev": 0,
"signature": "Google DNS server contacted",
"category": "",
"severity": 3,
"rule": "alert dns $HOME_NET any -> 8.8.8.8 any (msg:\"Google DNS server contacted\"; sid:43;)"
},
"app_proto": "dns",
"flow": {
"pkts_toserver": 1,
"pkts_toclient": 0,
"bytes_toserver": 81,
"bytes_toclient": 0,
"start": "2017-08-14T12:35:05.830812+0200"
}
}
Feature #2020
Metadata of the signature can now conditionaly put in the alert
events. This will allow user to get more context about the events
generated by the alert.
detect-metadata: conditional parsing
Only parses metadata if an output module will use the information.
Patch also adds a unittest to check metadata is not parsed if not
asked to.
output-json-alert: optional output keys as array
Update rule metadata configuration to have an option to output
value as array. Also adds an option to log only a series of keys
as array. This is useful in the case of some ruleset where from
instance the `tag` key is used multiple time.
(Jason Ish) rule metadata: always log as lists
After review of rule metadata, we can't make assumptions
on what should be a list or not. So log everything as a list.
By default log metadata.
Remove toggles for individual protocol types and just use a
single toggle to control including the app-layer with the
alert.
The metadata (currently app-layer and flow) can be disabled
by setting metadata to a falsey value, but its removed
from the default configuration (but wil be in docs)
The new OutputInitResult is a struct return type that allows
logger init functions to return a NULL context without
raising error.
Instead of returning NULL to signal error, the "ok" field will
be set to false. If ok, but the ctx is NULL, then silently
move on to the next logger.
Use case: multiple versions of a specific logger, and one
implementation decides the configuration is not for that
implemenation. It can return NULL, ok.
Use expectation to be able to identify connections that are
ftp data. It parses the PASV response, STOR message and the
RETR message to provide extraction of files.
Implementation in Rust of FTP messages parsing is available.
Also this patch changes some var name prefixed by ssh to ftp.
Clean up option parsing. Allow options to be disabled as well as
enabled.
E.g.
metadata: true
flow: false
The metadata setting will enable all. Then flow is disabled.
This patch adds a partial flow entry in the alert event
(if applayer or flow is selected) or simply app_proto if
it is not.
app_proto is useful as filter and aggregation field. And
the partial flow entry contains more information about the
proto as well as some volumetry info.
Use metadata provided information to output the Source and Target
in the definition of IDMEF.
The output is now the following:
"alert": {
"action": "allowed",
"gid": 1,
"signature_id": 1,
"rev": 1,
"signature": "connection to home",
"category": "",
"severity": 3,
"source": {
"ip": "2001:31d0:000a:f68a:0000:0000:0000:0001",
"port": 80
},
"target": {
"ip": "2a01:0e34:ee97:b130:c685:08ff:dab3:c9c8",
"port": 48390
}
Set flags by default:
-Wmissing-prototypes
-Wmissing-declarations
-Wstrict-prototypes
-Wwrite-strings
-Wcast-align
-Wbad-function-cast
-Wformat-security
-Wno-format-nonliteral
-Wmissing-format-attribute
-funsigned-char
Fix minor compiler warnings for these new flags on gcc and clang.
Log src_ip, dst_ip and proto for root packet (p->root) if the
packet that triggered is inside a tunnel, as JSON object
'tunnel'. Also log recursion depth to indicate the depth of
the tunnel.
Fix rate_filter issues: if action was modified it wouldn't be logged
in EVE. To address this pass the PacketAlert structure to the threshold
code so it can flag the PacketAlert as modified. Use this in logging.
Update API to use const where possible. Fix a timout issue that this
uncovered.
A warning log is already emitted if eve-log is enabled in the
configuration but json support is not built so the logger
registration functions can be silent.
As the logging modules are no longer threading modules, rename
them so they don't look like they are being registered as
threading modules.
Also, move the registration to the output.c which will handle
registration of the loggers.
Create a new eve event type, "packet" for logging packets that
are tagged as part of an event. The packet is still at the top
level to keep it consistent with alert event types.
In addition to the packet being logged, a packet_info object
is created to hold the linktype and any future meta data
we may want to add about the packet.
When the string representation of a JSON message grew bigger than
64k, the JSON record would just be truncated. This lead to errors
in the parser(s) of the JSON stream.
This patch changes the buffer logic to grow the buffer on demand.
In the EVE stream payload logging the IPS path logged the wrong dir.
Both IDS and IPS can take the same path as the detection engine
inspects in the same direction in both cases, so the alert is also
generated in the same direction.
Bug #1684
The cast of data to AlertJsonThread was not correct as the real
type of the void pointer is a OutputJsonCtx. This was working by
luck because they both have a file_ctx as first element.
This patch adds the capabilities to log the TLS information the
same way it is currently possible to do with HTTP. As it is
quite hard to read ASN.1 directly in the stream, this will help
people to understand why suricata is firing on alert relative
to TLS.
This patches is fixing a issue in the OutputJSONBuffer function. It
was writing to file the content of the buffer starting from the start
to the final offset. But as the writing is done for each JSON string
we are duplicating the previous events if we are reusing the same
buffer.
Duplication was for example triggered when we have multiple alerts
attached to a packet. In the case of two alerts, the first one was
logged twice more as the second one.
- Added the suricata.yaml configurations and updated the comments
- Renamed the field in the configuration structure to something generic
- Added two new constants and the warning codes
- Created app-layer-htp-xff.c and app-layer-htp-xff.h
- Added entries in the Makefile.am
- Added the necessary configuration options to EVE alert section
- Updated Unified2 XFF configuration comments and removed unnecessary whitespace
- Created a generic function to parse the configuration
- Release the flow locks sooner and remove debug logging
- Added XFF support to EVE alert output
- Removed unnecessary assignment of the data field
- Removed else condition (same function called for IPv4 and IPV6)
- Fixed constants to be a power of two (used in bitwise operations)
Instead of error phrone externs with macro's, use functions with a local
static enum var instead.
- EngineModeIsIPS(): in IPS mode
- EngineModeIsIDS(): in IDS mode
To set the modes:
- EngineModeSetIDS(): IDS mode (default)
- EngineModeSetIPS(): IPS mode
Bug #1177.
This patch updates the timestamp format used in eve loggin.
It uses a ISO 8601 comptatible string. This allow tools parsing
the output to easily detect adn/or use the timestamp.
In the EVE JSON output, the value of the timestamp key has been
changed to 'timestamp' (instead of 'time'). This allows tools
like Splunk to detect the timestamp and use it without configuration.
Logstash configuration is simple:
input {
file {
path => [ "/usr/local/var/log/suricata/eve.json" ]
codec => json
type => "suricata-log"
}
}
filter {
if [type] == "suricata-log" {
date {
match => [ "timestamp", "ISO8601" ]
}
}
}
In splunk, auto detection of the fle format is failling and it seems
you need to define a type to parse JSON in
$SPLUNK_DIR/etc/system/local/props.conf:
[suricata]
KV_MODE = json
NO_BINARY_CHECK = 1
TRUNCATE = 0
Then you can simply declare the log file in
$SPLUNK_DIR/etc/system/local/inputs.conf:
[monitor:///usr/local/var/log/suricata/eve.json]
sourcetype = suricata
In both cases the timestamp are correctly imported by
the tools.
The TM_FLAG_LOGAPI_TM flag indicates that a module is run by the log
api, not by the 'regular' thread module call functions.
Set flag in all all Log API users' registration code.
Purpose of this flag is in profiling. In profiling output it will be
used to list log api thread modules separately.
This patch adds an event_type key to the generated events. Current
value is one of "dns", "alert, "file", "tls", "http", "drop". It is
then easy to differentiate in log analysis tools the events based on
source inside Suricata.
This patch is synchronizing key name with Common Information Model.
It updates key name following what is proposed in:
http://docs.splunk.com/Documentation/PCI/2.0/DataSource/CommonInformationModelFieldReference
The interest of these modifications is that using the same key name
as other software will provide an easy to correlate and improve
data. For example, geoip setting in logstash can be applied on
all src_ip fields allowing geoip tagging of data.
Victor Julien
Jason Ish
Martin Natano
Eric Leblond
Mats Klepsland
maxtors
Tom DeCanio
Andreas Herz
Alexander Gozman
Duarte Silva
Giuseppe Longo
Ken Steele
Alexander Gozman
Matt Carothers