Add option on Tile-Gx for logging for fast.log alerts over PCIe
When running on a TILEncore-Gx PCIe card, setting the filetype of fast.log
to pcie, will open a connection over PCIe to a host application caleld
tile-pcie-logd, that receives the alert strings and writes them to a file
on the host. The file name to open is also passed over the PCIe link.
This allows running Suricata on the TILEncore-Gx PCIe card, but have the
alerts logged to the host system's file system efficiently. The PCIe API that
is used is the Tilera Packet Queue (PQ) API which can access PCIe from User
Space, thus avoiding system calls.
Created util-logopenfile-tile.c and util-logopen-tile.h for the TILE
specific PCIe logging functionality.
Using Write() and Close() function pointers in LogFileCtx, which
default to standard write and close for files and sockets, but are
changed to PCIe write and close functions when a PCIe channel is
openned for logging.
Moved Logging contex out of tm-modules.h into util-logopenfile.h,
where it makes more sense. This required including util-logopenfile.h
into a couple of alert-*.c files, which previously were getting the
definitions from tm-modules.h.
The source and Makefile for tile-pcie-logd are added in contrib/tile-pcie-logd.
By default, the file name for fast.log specified in suricata.yaml is used as
the filename on the host. An optional argument to tile-pcie-logd, --prefix=,
can be added to prepend the supplied file path. For example, is the file
in suricata.yaml is specified as "/var/log/fast.log" and --prefix="/tmp",
then the file will be written to "/tmp/var/log/fast.log".
Check for TILERA_ROOT environment variable before building tile_pcie_logd
Building tile_pcie_logd on x86 requires the Tilera MDE for its PCIe libraries
and API header files. Configure now checs for TILERA_ROOT before enabling
builing tile_pcie_logd in contrib/tile_pcie_logd
13 years ago
|
|
|
/* Copyright (C) 2007-2014 Open Information Security Foundation
|
|
|
|
|
*
|
|
|
|
|
* You can copy, redistribute or modify this Program under the terms of
|
|
|
|
|
* the GNU General Public License version 2 as published by the Free
|
|
|
|
|
* Software Foundation.
|
|
|
|
|
*
|
|
|
|
|
* This program is distributed in the hope that it will be useful,
|
|
|
|
|
* but WITHOUT ANY WARRANTY; without even the implied warranty of
|
|
|
|
|
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
|
|
|
|
* GNU General Public License for more details.
|
|
|
|
|
*
|
|
|
|
|
* You should have received a copy of the GNU General Public License
|
|
|
|
|
* version 2 along with this program; if not, write to the Free Software
|
|
|
|
|
* Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA
|
|
|
|
|
* 02110-1301, USA.
|
|
|
|
|
*/
|
|
|
|
|
|
|
|
|
|
/**
|
|
|
|
|
* \file
|
|
|
|
|
*
|
|
|
|
|
* \author Victor Julien <victor@inliniac.net>
|
|
|
|
|
*/
|
|
|
|
|
|
|
|
|
|
#ifndef __TM_MODULES_H__
|
|
|
|
|
#define __TM_MODULES_H__
|
|
|
|
|
|
Add per packet profiling.
Per packet profiling uses tick based accounting. It has 2 outputs, a summary
and a csv file that contains per packet stats.
Stats per packet include:
1) total ticks spent
2) ticks spent per individual thread module
3) "threading overhead" which is simply calculated by subtracting (2) of (1).
A number of changes were made to integrate the new code in a clean way:
a number of generic enums are now placed in tm-threads-common.h so we can
include them from any part of the engine.
Code depends on --enable-profiling just like the rule profiling code.
New yaml parameters:
profiling:
# packet profiling
packets:
# Profiling can be disabled here, but it will still have a
# performance impact if compiled in.
enabled: yes
filename: packet_stats.log
append: yes
# per packet csv output
csv:
# Output can be disabled here, but it will still have a
# performance impact if compiled in.
enabled: no
filename: packet_stats.csv
Example output of summary stats:
IP ver Proto cnt min max avg
------ ----- ------ ------ ---------- -------
IPv4 6 19436 11448 5404365 32993
IPv4 256 4 11511 49968 30575
Per Thread module stats:
Thread Module IP ver Proto cnt min max avg
------------------------ ------ ----- ------ ------ ---------- -------
TMM_DECODEPCAPFILE IPv4 6 19434 1242 47889 1770
TMM_DETECT IPv4 6 19436 1107 137241 1504
TMM_ALERTFASTLOG IPv4 6 19436 90 1323 155
TMM_ALERTUNIFIED2ALERT IPv4 6 19436 108 1359 138
TMM_ALERTDEBUGLOG IPv4 6 19436 90 1134 154
TMM_LOGHTTPLOG IPv4 6 19436 414 5392089 7944
TMM_STREAMTCP IPv4 6 19434 828 1299159 19438
The proto 256 is a counter for handling of pseudo/tunnel packets.
Example output of csv:
pcap_cnt,ipver,ipproto,total,TMM_DECODENFQ,TMM_VERDICTNFQ,TMM_RECEIVENFQ,TMM_RECEIVEPCAP,TMM_RECEIVEPCAPFILE,TMM_DECODEPCAP,TMM_DECODEPCAPFILE,TMM_RECEIVEPFRING,TMM_DECODEPFRING,TMM_DETECT,TMM_ALERTFASTLOG,TMM_ALERTFASTLOG4,TMM_ALERTFASTLOG6,TMM_ALERTUNIFIEDLOG,TMM_ALERTUNIFIEDALERT,TMM_ALERTUNIFIED2ALERT,TMM_ALERTPRELUDE,TMM_ALERTDEBUGLOG,TMM_ALERTSYSLOG,TMM_LOGDROPLOG,TMM_ALERTSYSLOG4,TMM_ALERTSYSLOG6,TMM_RESPONDREJECT,TMM_LOGHTTPLOG,TMM_LOGHTTPLOG4,TMM_LOGHTTPLOG6,TMM_PCAPLOG,TMM_STREAMTCP,TMM_DECODEIPFW,TMM_VERDICTIPFW,TMM_RECEIVEIPFW,TMM_RECEIVEERFFILE,TMM_DECODEERFFILE,TMM_RECEIVEERFDAG,TMM_DECODEERFDAG,threading
1,4,6,172008,0,0,0,0,0,0,47889,0,0,48582,1323,0,0,0,0,1359,0,1134,0,0,0,0,0,8028,0,0,0,49356,0,0,0,0,0,0,0,14337
First line of the file contains labels.
2 example gnuplot scripts added to plot the data.
15 years ago
|
|
|
#include "tm-threads-common.h"
|
|
|
|
|
#include "threadvars.h"
|
|
|
|
|
|
|
|
|
|
/* thread flags */
|
|
|
|
|
#define TM_FLAG_RECEIVE_TM 0x01
|
|
|
|
|
#define TM_FLAG_DECODE_TM 0x02
|
|
|
|
|
#define TM_FLAG_STREAM_TM 0x04
|
|
|
|
|
#define TM_FLAG_DETECT_TM 0x08
|
|
|
|
|
#define TM_FLAG_LOGAPI_TM 0x10 /**< TM is run by Log API */
|
|
|
|
|
#define TM_FLAG_MANAGEMENT_TM 0x20
|
|
|
|
|
#define TM_FLAG_COMMAND_TM 0x40
|
|
|
|
|
|
|
|
|
|
typedef TmEcode (*ThreadInitFunc)(ThreadVars *, const void *, void **);
|
|
|
|
|
typedef TmEcode (*ThreadDeinitFunc)(ThreadVars *, void *);
|
|
|
|
|
typedef void (*ThreadExitPrintStatsFunc)(ThreadVars *, void *);
|
|
|
|
|
|
|
|
|
|
typedef struct TmModule_ {
|
|
|
|
|
const char *name;
|
|
|
|
|
|
|
|
|
|
/** thread handling */
|
|
|
|
|
TmEcode (*ThreadInit)(ThreadVars *, const void *, void **);
|
|
|
|
|
void (*ThreadExitPrintStats)(ThreadVars *, void *);
|
|
|
|
|
TmEcode (*ThreadDeinit)(ThreadVars *, void *);
|
|
|
|
|
|
|
|
|
|
/** the packet processing function */
|
|
|
|
|
TmEcode (*Func)(ThreadVars *, Packet *, void *);
|
|
|
|
|
|
|
|
|
|
TmEcode (*PktAcqLoop)(ThreadVars *, void *, void *);
|
|
|
|
|
|
|
|
|
|
/** terminates the capture loop in PktAcqLoop */
|
|
|
|
|
TmEcode (*PktAcqBreakLoop)(ThreadVars *, void *);
|
|
|
|
|
|
|
|
|
|
TmEcode (*Management)(ThreadVars *, void *);
|
|
|
|
|
|
|
|
|
|
/** global Init/DeInit */
|
|
|
|
|
TmEcode (*Init)(void);
|
|
|
|
|
TmEcode (*DeInit)(void);
|
|
|
|
|
#ifdef UNITTESTS
|
|
|
|
|
void (*RegisterTests)(void);
|
|
|
|
|
#endif
|
|
|
|
|
uint8_t cap_flags; /**< Flags to indicate the capability requierment of
|
|
|
|
|
the given TmModule */
|
|
|
|
|
/* Other flags used by the module */
|
|
|
|
|
uint8_t flags;
|
|
|
|
|
} TmModule;
|
|
|
|
|
|
|
|
|
|
extern TmModule tmm_modules[TMM_SIZE];
|
|
|
|
|
|
|
|
|
|
/**
|
|
|
|
|
* Structure that output modules use to maintain private data.
|
|
|
|
|
*/
|
|
|
|
|
typedef struct OutputCtx_ {
|
|
|
|
|
|
|
|
|
|
/** Pointer to data private to the output. */
|
|
|
|
|
void *data;
|
|
|
|
|
|
|
|
|
|
/** Pointer to a cleanup function. */
|
|
|
|
|
void (*DeInit)(struct OutputCtx_ *);
|
|
|
|
|
|
|
|
|
|
TAILQ_HEAD(, OutputModule_) submodules;
|
|
|
|
|
} OutputCtx;
|
|
|
|
|
|
|
|
|
|
TmModule *TmModuleGetByName(const char *name);
|
|
|
|
|
TmModule *TmModuleGetById(int id);
|
|
|
|
|
int TmModuleGetIdByName(const char *name);
|
|
|
|
|
int TmModuleGetIDForTM(TmModule *tm);
|
|
|
|
|
TmEcode TmModuleRegister(char *name, int (*module_func)(ThreadVars *, Packet *, void *));
|
|
|
|
|
void TmModuleDebugList(void);
|
|
|
|
|
void TmModuleRegisterTests(void);
|
Add per packet profiling.
Per packet profiling uses tick based accounting. It has 2 outputs, a summary
and a csv file that contains per packet stats.
Stats per packet include:
1) total ticks spent
2) ticks spent per individual thread module
3) "threading overhead" which is simply calculated by subtracting (2) of (1).
A number of changes were made to integrate the new code in a clean way:
a number of generic enums are now placed in tm-threads-common.h so we can
include them from any part of the engine.
Code depends on --enable-profiling just like the rule profiling code.
New yaml parameters:
profiling:
# packet profiling
packets:
# Profiling can be disabled here, but it will still have a
# performance impact if compiled in.
enabled: yes
filename: packet_stats.log
append: yes
# per packet csv output
csv:
# Output can be disabled here, but it will still have a
# performance impact if compiled in.
enabled: no
filename: packet_stats.csv
Example output of summary stats:
IP ver Proto cnt min max avg
------ ----- ------ ------ ---------- -------
IPv4 6 19436 11448 5404365 32993
IPv4 256 4 11511 49968 30575
Per Thread module stats:
Thread Module IP ver Proto cnt min max avg
------------------------ ------ ----- ------ ------ ---------- -------
TMM_DECODEPCAPFILE IPv4 6 19434 1242 47889 1770
TMM_DETECT IPv4 6 19436 1107 137241 1504
TMM_ALERTFASTLOG IPv4 6 19436 90 1323 155
TMM_ALERTUNIFIED2ALERT IPv4 6 19436 108 1359 138
TMM_ALERTDEBUGLOG IPv4 6 19436 90 1134 154
TMM_LOGHTTPLOG IPv4 6 19436 414 5392089 7944
TMM_STREAMTCP IPv4 6 19434 828 1299159 19438
The proto 256 is a counter for handling of pseudo/tunnel packets.
Example output of csv:
pcap_cnt,ipver,ipproto,total,TMM_DECODENFQ,TMM_VERDICTNFQ,TMM_RECEIVENFQ,TMM_RECEIVEPCAP,TMM_RECEIVEPCAPFILE,TMM_DECODEPCAP,TMM_DECODEPCAPFILE,TMM_RECEIVEPFRING,TMM_DECODEPFRING,TMM_DETECT,TMM_ALERTFASTLOG,TMM_ALERTFASTLOG4,TMM_ALERTFASTLOG6,TMM_ALERTUNIFIEDLOG,TMM_ALERTUNIFIEDALERT,TMM_ALERTUNIFIED2ALERT,TMM_ALERTPRELUDE,TMM_ALERTDEBUGLOG,TMM_ALERTSYSLOG,TMM_LOGDROPLOG,TMM_ALERTSYSLOG4,TMM_ALERTSYSLOG6,TMM_RESPONDREJECT,TMM_LOGHTTPLOG,TMM_LOGHTTPLOG4,TMM_LOGHTTPLOG6,TMM_PCAPLOG,TMM_STREAMTCP,TMM_DECODEIPFW,TMM_VERDICTIPFW,TMM_RECEIVEIPFW,TMM_RECEIVEERFFILE,TMM_DECODEERFFILE,TMM_RECEIVEERFDAG,TMM_DECODEERFDAG,threading
1,4,6,172008,0,0,0,0,0,0,47889,0,0,48582,1323,0,0,0,0,1359,0,1134,0,0,0,0,0,8028,0,0,0,49356,0,0,0,0,0,0,0,14337
First line of the file contains labels.
2 example gnuplot scripts added to plot the data.
15 years ago
|
|
|
const char * TmModuleTmmIdToString(TmmId id);
|
|
|
|
|
|
|
|
|
|
void TmModuleRunInit(void);
|
|
|
|
|
void TmModuleRunDeInit(void);
|
|
|
|
|
|
|
|
|
|
#endif /* __TM_MODULES_H__ */
|
|
|
|
|
|
