A filetype plugin is a plugin that implements an eve filetype. Most
of the current filetypes could likely be implemented as such a plugin.
Such a plugin must implement Open, Close and Write, where Write
is provided the formatted JSON to be logged.
This commit also includes the plumbing for plugin loading. Example
plugin to come.
Plugins are loaded by the "plugin" section in the configuration
file:
plugins:
- /path/to/directory/plugins
- /path/to/plugin_file.so
This can also be done on the command line with:
--set plugins.0=/path/plugin_file.so
This commit adds MAC address output to the EVE-JSON format. We follow the
remarks made in Redmine ticket #962: for packets, log MAC src/dst as a
scalar field in EVE; for flows, log MAC src/dst as lists in EVE. Field names
are different between flow and packet context to avoid type confusion
(src_mac vs. src_macs). Configuration approach and JSON representation is
taken from previous GitHub PR #2700.
This commit modifies the JSON loggers with changes necessary to support
multi-threaded EVE output.
Each "thread-init" function sets up the per-thread log file context for
subsequent calls to the JSON output to buffer function.
This commit creates a common file output helper function based on the
logic in output-file-info.c:BuildBuildFileInfoRecord
The refactored helper will be used to create "fileinfo" information
during the alert output path.
Convert alert Eve logging JsonBuilder. Currently
makes heavy use of JsonBuilder being able to log Jansson's json_t
which is a temporary measure until all protocols loggers can be
converted to JsonBuilder.
New functions that replace Jansson versions with JsonBuilder
variations use "Eve" instead of "JSON".
Update the source/target logging to use the cached address info
instead of fetching it from the constructed json_t object.
This is required for migration to JsonBuilder which does not
have the ability to retrieve already set fields.
Currently alert logging relies on the ability to change existing
values in the json_t structure to overwrite addresses with xff
data. This feature is also used for the "target" logging.
As we can't do this with JsonBuilder, create a new struct to
hold the 5 tuple, with the values swapped as needed, and
overwritten with XFF data if needed. This struct will now
be used to write out the 5 tuple, as well as cache the information
for log fields to be written out later on in the log path.
Add support for community flow id, meant to give a records a
predictable flow id that can be used to match records to
output of other tools.
Takes a 'seed' that needs to be same across sensors and tools
to make the id less predictable.
Substitute json_string with SCJsonString custom function.
SCJsonString will ensure string passed is json-encodable (utf-8).
If it's not, the string will be converted in such a way that any
non-printable character will be encoded in its hex form.
The resulting json object will be returned.
rust modification will encode any non-printable character during its
conversion in to_cstring.
The new OutputInitResult is a struct return type that allows
logger init functions to return a NULL context without
raising error.
Instead of returning NULL to signal error, the "ok" field will
be set to false. If ok, but the ctx is NULL, then silently
move on to the next logger.
Use case: multiple versions of a specific logger, and one
implementation decides the configuration is not for that
implemenation. It can return NULL, ok.
Many were using AlertJsonThread instead of OutputJsonCtx,
but as the datatypes were similar enough no harm was done.
Now that they are using their proper datatype, removed
AlertJsonThread from output.h as its no longer used.
Move code to get 5-tuple in JSON object to own function 'JsonFiveTuple'.
This enables this code to be reused when printing 'parent' JSON object in
output-json-alert.
As the logging modules are no longer threading modules, rename
them so they don't look like they are being registered as
threading modules.
Also, move the registration to the output.c which will handle
registration of the loggers.
When the string representation of a JSON message grew bigger than
64k, the JSON record would just be truncated. This lead to errors
in the parser(s) of the JSON stream.
This patch changes the buffer logic to grow the buffer on demand.
To be able to correlate between events, it is better to have the
tx_id information in the root object. This function adds a new
function to automate the addition of the field.
When running on a TILEncore-Gx PCIe card, setting the filetype of fast.log
to pcie, will open a connection over PCIe to a host application caleld
tile-pcie-logd, that receives the alert strings and writes them to a file
on the host. The file name to open is also passed over the PCIe link.
This allows running Suricata on the TILEncore-Gx PCIe card, but have the
alerts logged to the host system's file system efficiently. The PCIe API that
is used is the Tilera Packet Queue (PQ) API which can access PCIe from User
Space, thus avoiding system calls.
Created util-logopenfile-tile.c and util-logopen-tile.h for the TILE
specific PCIe logging functionality.
Using Write() and Close() function pointers in LogFileCtx, which
default to standard write and close for files and sockets, but are
changed to PCIe write and close functions when a PCIe channel is
openned for logging.
Moved Logging contex out of tm-modules.h into util-logopenfile.h,
where it makes more sense. This required including util-logopenfile.h
into a couple of alert-*.c files, which previously were getting the
definitions from tm-modules.h.
The source and Makefile for tile-pcie-logd are added in contrib/tile-pcie-logd.
By default, the file name for fast.log specified in suricata.yaml is used as
the filename on the host. An optional argument to tile-pcie-logd, --prefix=,
can be added to prepend the supplied file path. For example, is the file
in suricata.yaml is specified as "/var/log/fast.log" and --prefix="/tmp",
then the file will be written to "/tmp/var/log/fast.log".
Check for TILERA_ROOT environment variable before building tile_pcie_logd
Building tile_pcie_logd on x86 requires the Tilera MDE for its PCIe libraries
and API header files. Configure now checs for TILERA_ROOT before enabling
builing tile_pcie_logd in contrib/tile_pcie_logd
This patch adds an event_type key to the generated events. Current
value is one of "dns", "alert, "file", "tls", "http", "drop". It is
then easy to differentiate in log analysis tools the events based on
source inside Suricata.
Jason Ish
Sascha Steinbiss
Jeff Lucovsky
Victor Julien
Maurizio Abba
Jason Ish
Eric Leblond
Mats Klepsland
Andreas Herz
Ken Steele