Commit Graph

79 Commits (00b6e628d17f16789bfba1c4e8ca3029cef4f021)

Author SHA1 Message Date
Jason Ish 00b6e628d1 logging: hook into flow worker thread 9 years ago
Jason Ish 04a44a077d logging: convert pcap log to non-thread module 9 years ago
Jason Ish fc35a78ba1 logging: use a single entry point for all loggers
Introduces a new thread module, TMM_LOGGER, which is the
root most logger.

Only handles loggers in the packet path, stats and flow
logging are not included.

The loggers are made up of a hierarchy of loggers. At the top we
have the root logger which is the main entry point to
logging. Under the root there exists parent loggers that are the
entry point for specific types of loggers such as packet logger,
transaction loggers, etc. Each parent logger may have 0 or more
loggers that actual handle the job of producing output to something
like a file.
9 years ago
Jason Ish 9489d5b9e3 logging: remove dead code from output-json
The "parent" json logger was setup like a real logger, but
some of that code was never being called.
9 years ago
Jason Ish 42b8f30272 logging: convert lua output to non-thread module 9 years ago
Jason Ish 7a0737b9a9 logging: convert tls log to non-thread module 9 years ago
Jason Ish 7cb16bc90d logging: convert alert debug log to non-thread module 9 years ago
Jason Ish 7a8e8343e5 logging: convert tcp data logging to non-thread module 9 years ago
Jason Ish 4d8b8ca046 logging: convert tls store logging to non-thread module 9 years ago
Jason Ish 60b6ccc3c4 logging: convert file data logging to non-thread module 9 years ago
Jason Ish f9bb9029c5 logging: convert file logging to non-thread module 9 years ago
Jason Ish 669827ae16 logging: convert unified2 to non-thread module 9 years ago
Jason Ish b580016c80 logging: convert stats loggers to non-thread module 9 years ago
Jason Ish 9475c83713 logging: convert http log to non-thread module 9 years ago
Jason Ish e00dcd52a0 logging: convert alert syslog to non-thread module 9 years ago
Jason Ish 5bbb4fd134 logging: convert json template output to non-thread module 9 years ago
Jason Ish bac65f09e8 logging: convert json drop output to non-thread module 9 years ago
Jason Ish 38354479b7 logging: convert json smtp output to non-thread module 9 years ago
Jason Ish 3fea12d7b3 logging: convert json ssh output to non-thread module 9 years ago
Jason Ish 01cc508257 logging: convert json netflow output to non-thread module 9 years ago
Jason Ish 983a619ff0 logging: convert json flow output to non-thread module 9 years ago
Jason Ish ad15ac8297 logging: convert json alert output to non-thread module 9 years ago
Jason Ish aaa65f3d16 logging: convert json tls output to non-thread module 9 years ago
Jason Ish 31663f1627 logging: convert prelude output to non-thread module 9 years ago
Jason Ish dedda33f01 logging: convert eve http to non-thread module 9 years ago
Jason Ish 687602c0ca logging: convert eve dns logging to non-thread module 9 years ago
Jason Ish b1200dba54 logging: convert fast log to a non-thread module 9 years ago
Jason Ish 637aa34610 logging: convert dns log to a non-thread module 9 years ago
Victor Julien 7dfdcdc770 thread modules: remove unused id's 9 years ago
Victor Julien a8f257e05f detect: no longer a thread module
Like stream, detect is now invoked directly by the FlowWorker.
9 years ago
Victor Julien 4a96820320 stream-tcp: more cleanups 9 years ago
Victor Julien 52d500c670 flowworker: initial support
Initial version of the 'FlowWorker' thread module. This module
combines Flow handling, TCP handling, App layer handling and
Detection in a single module. It does all flow related processing
under a single flow lock.
9 years ago
Jason Ish bcda92134d app-layer: template for application layer tx logger 10 years ago
Victor Julien eb09118d64 detect: create loader threads
To speed up startup with many tenants, tenant loading will be parallelized.
As no tempary threads should be used for these memory allocation heavy
tasks, this patch adds new type of 'command' thread that can be used to
load and reload tenants.

This patch hardcodes the number of loaders to 4. Future work will make it
dynamic.

The loader thread essentially sleeps constantly. When a tasks is sent to
it, it will wake up and execute it.
10 years ago
Eric Leblond 4db0a35f25 tls-store: now a separate module
An design error was made when doing the TLS storage module which
has been made dependant of the TLS logging. At the time there was
only one TLS logging module but there is now two different ones.

By putting the TLS store module in a separate module, we can now
use EVE output and TLS store at the same time.
10 years ago
Tom DeCanio 117eed0385 eve-log: add JSON stats logging
Support for counters in stats.log in eve output JSON stream.
10 years ago
Victor Julien df5e9d44ca unix-manager: convert to thread module
Sync command thread for unix manager with other managers and make
it a full thread module.
10 years ago
gureedo 10104066e1 netmap support 11 years ago
Victor Julien e98346b555 Introduce stats log API, convert existing output
Convert regular 'stats.log' output to this new API.

In addition to the current stats value, also give the last value. This
makes it easy to display the difference.
11 years ago
Tom DeCanio 31f8f5cf20 eve-log: SMTP JSON logger 11 years ago
Victor Julien db30ed8c3e output: Lua HTTP log initial implementation
Initial version of a HTTP LUA logger. Execute lua scripts from the
Tx-log API.
11 years ago
Victor Julien 3dec0e96f8 tcp-data: new streaming logger
tcp-data logs out reassembled stream data in a streaming fashion.

Records type to log into different directories.
11 years ago
Victor Julien 9d9ef983dd output-streaming: a Log API for streaming data
This patch adds a new Log API for streaming data such as TCP reassembled
data and HTTP body data. It could also replace Filedata API.

Each time a new chunk of data is available, the callback will be called.
11 years ago
Victor Julien e0841218f0 flow-manager: support multiple instances
Use new management API to run the flow manager.

Support multiple flow managers, where each of them works with it's
own part of the flow hash.

Make number of threads configurable:

flow:
  memcap: 64mb
  hash-size: 65536
  prealloc: 10000
  emergency-recovery: 30
  managers: 2

This sets up 2 flow managers.

Handle misc tasks only in instance 1: Handle defrag hash timeout
handing, host hash timeout handling and flow spare queue updating
only from the first instance.
11 years ago
Victor Julien db15339f47 netflow-json: initial version
Initial version of netflow module, a flow logger that logs each
direction in a completely separate record (line).
11 years ago
Victor Julien 3c7af02067 flow-json-log: stub
Stub for JSON flow logger.
11 years ago
Giuseppe Longo d213d89981 Updating the Tmm Id for declaration of nflog capture mode 11 years ago
Victor Julien b877cf6158 ssh: add json logger
Sub module of eve-log, but can also run separately as ssh-json-log. Only
one at a time though.
12 years ago
Ken Steele 497575d38e Add option on Tile-Gx for logging for fast.log alerts over PCIe
When running on a TILEncore-Gx PCIe card, setting the filetype of fast.log
to pcie, will open a connection over PCIe to a host application caleld
tile-pcie-logd, that receives the alert strings and writes them to a file
on the host. The file name to open is also passed over the PCIe link.

This allows running Suricata on the TILEncore-Gx PCIe card, but have the
alerts logged to the host system's file system efficiently. The PCIe API that
is used is the Tilera Packet Queue (PQ) API which can access PCIe from User
Space, thus avoiding system calls.

Created util-logopenfile-tile.c and util-logopen-tile.h for the TILE
specific PCIe logging functionality.

Using Write() and Close() function pointers in LogFileCtx, which
default to standard write and close for files and sockets, but are
changed to PCIe write and close functions when a PCIe channel is
openned for logging.

Moved Logging contex out of tm-modules.h into util-logopenfile.h,
where it makes more sense. This required including util-logopenfile.h
into a couple of alert-*.c files, which previously were getting the
definitions from tm-modules.h.

The source and Makefile for tile-pcie-logd are added in contrib/tile-pcie-logd.

By default, the file name for fast.log specified in suricata.yaml is used as
the filename on the host. An optional argument to tile-pcie-logd, --prefix=,
can be added to prepend the supplied file path. For example, is the file
in suricata.yaml is specified as "/var/log/fast.log" and --prefix="/tmp",
then the file will be written to "/tmp/var/log/fast.log".

Check for TILERA_ROOT environment variable before building tile_pcie_logd

Building tile_pcie_logd on x86 requires the Tilera MDE for its PCIe libraries
and API header files. Configure now checs for TILERA_ROOT before enabling
builing tile_pcie_logd in contrib/tile_pcie_logd
12 years ago
Victor Julien 3fc63d3656 jansson file log: make file log module
Turn the libjansson based file logger into a file module, as a child
of eve-log.
12 years ago