Commit Graph

112 Commits (c27304451e30c67b6b370974c89948ab7e115b51)

Author SHA1 Message Date
Eric Leblond 9961520316 output: clean file desc at exit.
This is a beginning of implementation for bug #1660:
 https://redmine.openinfosecfoundation.org/issues/1160

This patch adds a cleaning function for each logger of new type
(packet, tx and file). These functions are called in RunModeShutDown().

The state of this patch is that it is crashing suricata when sending
pcap to analyse:
 - At first pcap if tx and file cleaning function are called
 - At second pcap if only packet cleaning function is called

The cause in first case is unknown. In second case this is due to
the necessity of cleaning the list of logger registered to a logging
type.
11 years ago
Victor Julien a3020b5306 eve-log: output cleanup
Suppress debug messages and print in a nicer way which modules are
being enabled.
12 years ago
Victor Julien c95df281e3 eve-log: add warning if enabled but not supported
If we're not compiled against libjansson, the eve-log output is not
available. This patch adds a warning.
12 years ago
Victor Julien a9eab06593 output: simple name space support for sub modules
To avoid module name clashes, a submode abc of parent xyz, will now
register itself as xyz.abc.
12 years ago
Victor Julien f830cb8026 output: introduce concept of sub-modules
To support the 'eve-log' idea, we need to be able to force all log
modules to be enabled by the master eve-log module, and need to be
able to make all logs go into a single file. This didn't fit the
API so far, so added the sub-module concept.

A sub-module is a regular module, that registers itself as a sub-
module of another module:

    OutputRegisterTxSubModule("eve-log", "JsonHttpLog", "http",
            OutputHttpLogInitSub, ALPROTO_HTTP, JsonHttpLogger);

The first argument is the name of the parent. The 4th argument is
the OutputCtx init function. It differs slightly from the non-sub
one. The different is that in addition to it's ConfNode, it gets
the OutputCtx from the parent. This way it can set the parents
LogFileCtx in it's own OutputCtx.

The runmode setup code will take care of all the extra setup. It's
possible to register a module both as a normal module and as a sub-
module, which can operate at the same time.

Only the TxLogger API is handled in this patch, the rest will be
updated later.
12 years ago
Victor Julien 9ff6608668 Introduce Filedata Logger API
A new logger API for registering file storage handlers. Where the
FileLog handler is called once per file, this handler will be called
for each data chunk so that storing the entire file is possible.

The logger call in the API is as follows:
    typedef int (*FiledataLogger)(ThreadVars *, void *thread_data,
        const Packet *, const File *, const FileData *, uint8_t flags);

All data is const, thus should be read only. The final flags field
is used to indicate to the caller that the file is new, or if it's
being closed.

Files use an internal unique id 'file_id' which can be used by the
loggers to create unique file names. This id can use the 'waldo'
feature of the log-filestore module. This patch moves that waldo
loading and storing logic to this API's implementation. A new
configuration directive 'file-store-waldo: <filename>' is added,
but the existing waldo settings will also continue to work.
12 years ago
Victor Julien ee2a8a9cda Introduce 'file' logging API
This patch introduces a new logging API for logging extracted file info.
It allows for registration of a callback that is called once per file:
when it's considered 'closed'.

Users of this API register their Log Function through:
    OutputRegisterFileModule()

The API uses a magic settings globally. This might be changed later.
12 years ago
Victor Julien ad70793f78 Introduce TX logging API
This patch introduces a new API for logging transactions from
tx-aware app layer protocols. It runs all the registered loggers
from a single thread module. This thread module takes care of the
transaction handling and flow locking. The logger just gets a
transaction to log out.

All loggers for a protocol will be run at the same time, so there
will not be any timing differences.

Loggers will no longer act as Thread Modules in the strictest sense.
The Func is NULL, and SetupOuputs no longer attaches them to the
thread module chain individually. Instead, after registering through
OutputRegisterTxModule, the setup data is used in the single logging
module.

The logger (LogFunc) is called for each transaction once, at the end
of the transaction.
12 years ago
Victor Julien d43ac9ae98 Introduce packet logging output API
This patch introduces a new API for outputs that log based on the
packet, such as alert outputs. In converts fast-log to the new API.

The API gets rid of the concept of each logger being a thread module,
but instead there is one thread module that runs all packet loggers.
Through the registration function OutputRegisterPacketModule a log
module can register itself to be considered for each packet.

Each logger registers itself to this new API with 2 functions and the
OutputCtx object that was already used in the old implementation.
The function pointers are:

LogFunc:       the log function

ConditionFunc: this function is called before the LogFunc and only
               if this returns TRUE the LogFunc is called.

For a simple alert logger like fast-log, the condition function will
simply return TRUE if p->alerts.cnt > 0.
12 years ago
Victor Julien 0e08f4b6fc update runmodes to handle detect-less
In runmodes setup, consider a NULL de_ctx to mean detect is disabled.
12 years ago
Eric Leblond 1f07d1521e Fix realloc error handling
This patch is fixing realloc error handling. In case of a realloc
failure, it free the initial memory and continue existing error
handling.

The patch has been obtained via the following semantic patch and
a bit oh hand editing:

@@
expression x, E;
identifier f;
@@

f(...)
{
+ void *ptmp;
<+...
- x = SCRealloc(x, E);
+ ptmp = SCRealloc(x, E);
... when != x
- if (x == NULL)
+ if (ptmp == NULL)
{
+ SCFree(x);
+ x = NULL;
...
- }
+ } else {
+     x = ptmp;
+ }
...+>
}

@@
expression x, E;
identifier f;
statement ES;
@@

f(...) {
+ void *ptmp;

<+...
- x = SCRealloc(x, E);
+ ptmp = SCRealloc(x, E);
... when != x
- if (x == NULL) ES
+ if (ptmp == NULL) {
+ SCFree(x);
+ x = NULL;
+ ES
+ } else {
+     x = ptmp;
+ }
...+>

}

@@
expression x, E;
identifier f;
@@

f(...)
{
+ void *ptmp;
<+...
- x = SCRealloc(x, E);
+ ptmp = SCRealloc(x, E);
... when != x
- if (unlikely(x == NULL))
+ if (unlikely(ptmp == NULL))
{
+ SCFree(x);
+ x = NULL;
...
- }
+ } else {
+     x = ptmp;
+ }
...+>
}

@@
expression x, E;
identifier f;
statement ES;
@@

f(...) {
+ void *ptmp;

<+...
- x = SCRealloc(x, E);
+ ptmp = SCRealloc(x, E);
... when != x
- if (unlikely(x == NULL)) ES
+ if (unlikely(ptmp == NULL)) {
+ SCFree(x);
+ x = NULL;
+ ES
+ } else {
+     x = ptmp;
+ }
...+>

}
12 years ago
Eric Leblond 28c5c68192 error checking: add missing alloc error treatment
The return of some malloc like functions was not treated in some
places of the code.
12 years ago
Victor Julien 435f99409f Fix small memleak in runmode setup
[src/runmodes.c:338]: (error) Memory leak: custom_mode
12 years ago
Anoop Saldanha b90a56b626 fix for #927.
Print an error message when the user supplies an invalid value for
detect-thread-ratio in the conf file.
12 years ago
Eric Leblond 80542816cd add internal running mode 12 years ago
Eric Leblond 135ef0186b runmodes: fix comment 12 years ago
Ken Steele 316190c6b9 Add TILE-Gx mPIPE packet processing support.
The TILE-Gx processor includes a packet processing engine, called
mPIPE, that can deliver packets directly into user space memory. It
handles buffer allocation and load balancing (either static 5-tuple
hashing, or dynamic flow affinity hashing are used here). The new
packet source code is in source-mpipe.c and source-mpipe.h

A new Tile runmode is added that configures the Suricata pipelines in
worker mode, where each thread does the entire packet processing
pipeline.  It scales across all the Gx chips sizes of 9, 16, 36 or 72
cores. The new runmode is in runmode-tile.c and runmode-tile.h

The configure script detects the TILE-Gx architecture and defines
HAVE_MPIPE, which is then used to conditionally enable the code to
support mPIPE packet processing. Suricata runs on TILE-Gx even without
mPIPE support enabled.

The Suricata Packet structures are allocated by the mPIPE hardware by
allocating the Suricata Packet structure immediatley before the mPIPE
packet buffer and then pushing the mPIPE packet buffer pointer onto
the mPIPE buffer stack.  This way, mPIPE writes the packet data into
the buffer, returns the mPIPE packet buffer pointer, which is then
converted into a Suricata Packet pointer for processing inside
Suricata. When the Packet is freed, the buffer is returned to mPIPE's
buffer stack, by setting ReleasePacket to an mPIPE release specific
function.

The code checks for the largest Huge page available in Linux when
Suricata is started. TILE-Gx supports Huge pages sizes of 16MB, 64MB,
256MB, 1GB and 4GB. Suricata then divides one of those page into
packet buffers for mPIPE.

The code is not yet optimized for high performance. Performance
improvements will follow shortly.

The code was originally written by Tom Decanio and then further
modified by Tilera.

This code has been tested with Tilera's Multicore Developement
Environment (MDE) version 4.1.5. The TILEncore-Gx36 (PCIe card) and
TILEmpower-Gx (1U Rack mount).
12 years ago
Anoop Saldanha 602c91ed41 Minor cosmetic changes to the cuda code.
Moved a couple of functions to more cuda relevant files;
Re-structured some data types.
12 years ago
Anoop Saldanha b787da5643 Remove all cuda related code in the engine except for the cuda api wrappers 12 years ago
Eric Leblond 74a9fc4b66 Add function to display current capture mode
This patch adds a function to display the capture mode.
13 years ago
Eric Leblond 20a8b9dbe5 unix-manager: add unix command socket and associated script
This patch introduces a unix command socket. JSON formatted messages
can be exchanged between suricata and a program connecting to a
dedicated socket.
The protocol is the following:
 * Client connects to the socket
 * It sends a version message: { "version": "$VERSION_ID" }
 * Server answers with { "return": "OK|NOK" }
If server returns OK, the client is now allowed to send command.

The format of command is the following:
 {
   "command": "pcap-file",
   "arguments": { "filename": "smtp-clean.pcap", "output-dir": "/tmp/out" }
 }
The server will try to execute the "command" specified with the
(optional) provided "arguments".
The answer by server is the following:
 {
   "return": "OK|NOK",
   "message": JSON_OBJECT or information string
 }

A simple script is provided and is available under scripts/suricatasc. It
is not intended to be enterprise-grade tool but it is more a proof of
concept/example code.  The first command line argument of suricatasc is
used to specify the socket to connect to.

Configuration of the feature is made in the YAML under the 'unix-command'
section:
  unix-command:
    enabled: yes
    filename: custom.socket
The path specified in 'filename' is not absolute and is relative to the
state directory.

A new running mode called 'unix-socket' is also added.
When starting in this mode, only a unix socket manager
is started. When it receives a 'pcap-file' command, the manager
start a 'pcap-file' running mode which does not really leave at
the end of file but simply exit. The manager is then able to start
a new running mode with a new file.

To start this mode, Suricata must be started with the --unix-socket
 option which has an optional argument which fix the file name of the
socket. The path is not absolute and is relative to the state directory.

THe 'pcap-file' command adds a file to the list of files to treat.
For each pcap file, a pcap file running mode is started and the output
directory is changed to what specified in the command. The running
mode specified in the 'runmode' YAML setting is used to select which
running mode must be use for the pcap file treatment.

This requires modification in suricata.c file where initialisation code
is now conditional to the fact 'unix-socket' mode is not used.

Two other commands exists to get info on the remaining tasks:
 * pcap-file-number: return the number of files in the waiting queue
 * pcap-file-list: return the list of waiting files
'pcap-file-list' returns a structured object as message. The
structure is the following:
 {
  'count': 2,
  'files': ['file1.pcap', 'file2.pcap']
 }
13 years ago
Matt Keeler 37e3de8425 Refactor Napatech 3GD to just Napatech as Suricata is only going to support 3GD.
Signed-off-by: Matt Keeler <mk@npulsetech.com>
13 years ago
Matt Keeler 5786a32d0f Remove Napatech 2GD support
Removed the Napatech 2GD support

runmode-napatech-3gd.c had an include from runmode-napatech.h which was erroneous and has been removed as well.

Signed-off-by: Matt Keeler <mk@npulsetech.com>
13 years ago
Matt Keeler 844e4dba11 Napatech 3GD Support
For use with Network Cards from Napatech utilizing the 3GD driver/api.

    - Implemented new run modes in runmode-napatech-3gd.*
    - Implemented capture/decode threads in source-napatech-3gd.*
    - Integrated the new run modes and source into the build infrastructure.

    New configure switches
    --enabled-napatech-3gd : Turns on the NT 3GD support
    --with-napatech-3gd-includes : The directory containing the NT 3GD header files
    --with-napatech-3gd-libraries : The directory containing the NT 3GD libraries to link against.

    New CLI switch
    --napatech-3gd : Uses the Napatech 3GD run mode

    Runmodes Supported:
    - auto
    - autofp
    - workers

    Notes:
    - tested with 1 Gbps sustained traffic (no drops)

Signed-off-by: Matt Keeler <mk@npulsetech.com>
13 years ago
Eric Leblond e176be6fcc Use unlikely for error treatment.
When handling error case on SCMallog, SCCalloc or SCStrdup
we are in an unlikely case. This patch adds the unlikely()
expression to indicate this to gcc.

This patch has been obtained via coccinelle. The transformation
is the following:

@istested@
identifier x;
statement S1;
identifier func =~ "(SCMalloc|SCStrdup|SCCalloc)";
@@

x = func(...)
... when != x
- if (x == NULL) S1
+ if (unlikely(x == NULL)) S1
13 years ago
Eric Leblond d292004880 Add some missing checks of SCStrdup return. 13 years ago
Eric Leblond a3465fb971 Rename 'worker' running mode to 'workers'
This patch renamed the 'worker' running mode into 'workers'. Thus,
there is only one name in Suricata for the same thing. Backward
compatibility is ensured by replacing "worker" by "workers" when
the old name is used. A warning is printed in the log when the old
name is used.
13 years ago
Victor Julien 79d5ef3707 Improve warning if prelude output is selected but support not compiled in. #320. 13 years ago
Anoop Saldanha 32183faa82 free flowvar entries in flow after live rule swap. Sync flowbits entries into packet struct to be used by alert debuglog when alert debuglog is enabled 13 years ago
Nikolay Denev 139768dd58 Do not use underscored config vars internally. 14 years ago
Victor Julien 1d9f6ff8f2 Initial Napatech support by Randy Caldejon / nPulse. 14 years ago
Eric Leblond ef3951d914 runmode: export running mode
This will permit to put some optimisation in different components.
This is done via the RunmodeGetActive() function.
14 years ago
Victor Julien 1df3304655 Clean up for unittests code: only compile unittest api code when unittests are enabled. Fix unittest code that wasn't wrapped in the proper UNITTESTS ifdefs. 14 years ago
Eric Leblond 24f15fa321 Don't warn about non enable non existing output module
This patch modifies output module loading to only trigger alert
message for non existing modules when they are loaded. It also
warn about unified1 removal.
14 years ago
Eric Leblond 391d813c82 Remove unified1 output module. 14 years ago
Eric Leblond c45d898572 af-packet: basic support for AF_PACKET socket
This patch provides basic support for AF_PACKET socket. It is
completed by a subsequent patches prodiding extended features
and bugfixes.
14 years ago
Anoop Saldanha 4f7df1029d Unify the use of slots to a single struct for threading API. Remove separate slot append functions for 1slot and varslot 14 years ago
Jason Ish 7257fed0f3 Fix bug 288, accept true in output configuration.
Refactor a bit to run checks for truth through a common function
that takes yes, true, on and 1 as true values.
14 years ago
Victor Julien 169104a803 Slightly clean up --list-runmodes output. 14 years ago
Anoop Saldanha e4d890e186 modify runmode api to accept conf runmode paramter as a char string, instead of an interger id 14 years ago
Anoop Saldanha 229f7281ea list runmodes. Allow specification of runmode id from cof file. Also allow for command line override 14 years ago
Anoop Saldanha 6fceeda8c5 move erf dag runmode into its own file runmode-erf-dag.[ch] 14 years ago
Anoop Saldanha f51cf34210 move erf file runmode into its own file runmode-erf-file.[ch] 14 years ago
Anoop Saldanha 86eabbc2f5 move ipfw runmode into its own file runmode-ipfw.[ch] 14 years ago
Anoop Saldanha 036015d6b9 move nfq runmode into its own file runmode-nfq.[ch] 14 years ago
Anoop Saldanha 9affa39b29 move pfring runmode into its own file runmode-pfring.[ch] 14 years ago
Anoop Saldanha e7ac1d7c4c move pcap file runmode into its own file runmode-pcap-file.[ch] 14 years ago
Anoop Saldanha f6af567ce0 move pcap live runmode into its own file runmode-pcap.[ch] 14 years ago
Victor Julien 07776c113b Fix valgrind error on pfring_recv, rename threads from RecvPfring to RxPfring so the name still looks right for 100+ threads. Add --pfring commandline option that just enables pfring, then takes interface from config. 15 years ago
Victor Julien 1c9e48ae98 Fix compilation error on non-pfring systems. 15 years ago