suricata/src/output-filedata.h

/* Copyright (C) 2007-2014 Open Information Security Foundation
 *
 * You can copy, redistribute or modify this Program under the terms of
 * the GNU General Public License version 2 as published by the Free
 * Software Foundation.
 *
 * This program is distributed in the hope that it will be useful,
 * but WITHOUT ANY WARRANTY; without even the implied warranty of
 * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 * GNU General Public License for more details.
 *
 * You should have received a copy of the GNU General Public License
 * version 2 along with this program; if not, write to the Free Software
 * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA
 * 02110-1301, USA.
 */

/**
 * \file
 *
 * \author Victor Julien <victor@inliniac.net>
 *
 * AppLayer Filedata Logger Output registration functions
 */

#ifndef __OUTPUT_FILEDATA_H__
#define __OUTPUT_FILEDATA_H__

#include "decode.h"
#include "util-file.h"

#define OUTPUT_FILEDATA_FLAG_OPEN  0x01
#define OUTPUT_FILEDATA_FLAG_CLOSE 0x02

/** filedata logger function pointer type */
typedef int (*FiledataLogger)(ThreadVars *, void *thread_data, const Packet *,
        File *, const uint8_t *, uint32_t, uint8_t);

/** packet logger condition function pointer type,
 *  must return true for packets that should be logged
 */
//typedef int (*TxLogCondition)(ThreadVars *, const Packet *);

int OutputRegisterFiledataLogger(LoggerId id, const char *name,
    FiledataLogger LogFunc, OutputCtx *, ThreadInitFunc ThreadInit,
    ThreadDeinitFunc ThreadDeinit,
    ThreadExitPrintStatsFunc ThreadExitPrintStats);

void OutputFiledataLoggerRegister(void);

void OutputFiledataShutdown(void);

#endif /* __OUTPUT_FILE_H__ */
Introduce Filedata Logger API A new logger API for registering file storage handlers. Where the FileLog handler is called once per file, this handler will be called for each data chunk so that storing the entire file is possible. The logger call in the API is as follows: typedef int (FiledataLogger)(ThreadVars , void thread_data, const Packet , const File , const FileData , uint8_t flags); All data is const, thus should be read only. The final flags field is used to indicate to the caller that the file is new, or if it's being closed. Files use an internal unique id 'file_id' which can be used by the loggers to create unique file names. This id can use the 'waldo' feature of the log-filestore module. This patch moves that waldo loading and storing logic to this API's implementation. A new configuration directive 'file-store-waldo: <filename>' is added, but the existing waldo settings will also continue to work. 12 years ago			`/* Copyright (C) 2007-2014 Open Information Security Foundation`
			`*`
			`* You can copy, redistribute or modify this Program under the terms of`
			`* the GNU General Public License version 2 as published by the Free`
			`* Software Foundation.`
			`*`
			`* This program is distributed in the hope that it will be useful,`
			`* but WITHOUT ANY WARRANTY; without even the implied warranty of`
			`* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the`
			`* GNU General Public License for more details.`
			`*`
			`* You should have received a copy of the GNU General Public License`
			`* version 2 along with this program; if not, write to the Free Software`
			`* Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA`
			`* 02110-1301, USA.`
			`*/`

			`/**`
			`* \file`
			`*`
			`* \author Victor Julien <victor@inliniac.net>`
			`*`
			`* AppLayer Filedata Logger Output registration functions`
			`*/`

			`#ifndef __OUTPUT_FILEDATA_H__`
			`#define __OUTPUT_FILEDATA_H__`

			`#include "decode.h"`
			`#include "util-file.h"`

			`#define OUTPUT_FILEDATA_FLAG_OPEN 0x01`
			`#define OUTPUT_FILEDATA_FLAG_CLOSE 0x02`

			`/** filedata logger function pointer type */`
			`typedef int (FiledataLogger)(ThreadVars , void thread_data, const Packet ,`
filestore: avoid open write close sequence Current file storing approach is using a open file, write data, close file logic. If this technic is fixing the problem of getting too much open files in Suricata it is not optimal. Test on a loop shows that open, write, close on a single file is two time slower than a single open, loop of write, close. This patch updates the logic by storing the fd in the File structure. This is done for a certain number of files. If this amount is exceeded then the previous logic is used. This patch also adds two counters. First is the number of currently open files. The second one is the number of time the open, write, close sequence has been used due to too much open files. In EVE, the entries are: stats {file_store: {"open_files_max_hit":0,"open_files":5}} 8 years ago			`File , const uint8_t , uint32_t, uint8_t);`
Introduce Filedata Logger API A new logger API for registering file storage handlers. Where the FileLog handler is called once per file, this handler will be called for each data chunk so that storing the entire file is possible. The logger call in the API is as follows: typedef int (FiledataLogger)(ThreadVars , void thread_data, const Packet , const File , const FileData , uint8_t flags); All data is const, thus should be read only. The final flags field is used to indicate to the caller that the file is new, or if it's being closed. Files use an internal unique id 'file_id' which can be used by the loggers to create unique file names. This id can use the 'waldo' feature of the log-filestore module. This patch moves that waldo loading and storing logic to this API's implementation. A new configuration directive 'file-store-waldo: <filename>' is added, but the existing waldo settings will also continue to work. 12 years ago
			`/** packet logger condition function pointer type,`
			`* must return true for packets that should be logged`
			`*/`
			`//typedef int (TxLogCondition)(ThreadVars , const Packet *);`

logging: add profiling back for non-tmm loggers The loggers moved away from a TMM required a new profiling support. 9 years ago			`int OutputRegisterFiledataLogger(LoggerId id, const char *name,`
			`FiledataLogger LogFunc, OutputCtx *, ThreadInitFunc ThreadInit,`
			`ThreadDeinitFunc ThreadDeinit,`
logging: convert file data logging to non-thread module 9 years ago			`ThreadExitPrintStatsFunc ThreadExitPrintStats);`
Introduce Filedata Logger API A new logger API for registering file storage handlers. Where the FileLog handler is called once per file, this handler will be called for each data chunk so that storing the entire file is possible. The logger call in the API is as follows: typedef int (FiledataLogger)(ThreadVars , void thread_data, const Packet , const File , const FileData , uint8_t flags); All data is const, thus should be read only. The final flags field is used to indicate to the caller that the file is new, or if it's being closed. Files use an internal unique id 'file_id' which can be used by the loggers to create unique file names. This id can use the 'waldo' feature of the log-filestore module. This patch moves that waldo loading and storing logic to this API's implementation. A new configuration directive 'file-store-waldo: <filename>' is added, but the existing waldo settings will also continue to work. 12 years ago
logging: rename registration functions to not have tmm As the logging modules are no longer threading modules, rename them so they don't look like they are being registered as threading modules. Also, move the registration to the output.c which will handle registration of the loggers. 9 years ago			`void OutputFiledataLoggerRegister(void);`
Introduce Filedata Logger API A new logger API for registering file storage handlers. Where the FileLog handler is called once per file, this handler will be called for each data chunk so that storing the entire file is possible. The logger call in the API is as follows: typedef int (FiledataLogger)(ThreadVars , void thread_data, const Packet , const File , const FileData , uint8_t flags); All data is const, thus should be read only. The final flags field is used to indicate to the caller that the file is new, or if it's being closed. Files use an internal unique id 'file_id' which can be used by the loggers to create unique file names. This id can use the 'waldo' feature of the log-filestore module. This patch moves that waldo loading and storing logic to this API's implementation. A new configuration directive 'file-store-waldo: <filename>' is added, but the existing waldo settings will also continue to work. 12 years ago
output api: complete shutdown functions Add missing function for Filedata API. Clean up list in all functions. 11 years ago			`void OutputFiledataShutdown(void);`

Introduce Filedata Logger API A new logger API for registering file storage handlers. Where the FileLog handler is called once per file, this handler will be called for each data chunk so that storing the entire file is possible. The logger call in the API is as follows: typedef int (FiledataLogger)(ThreadVars , void thread_data, const Packet , const File , const FileData , uint8_t flags); All data is const, thus should be read only. The final flags field is used to indicate to the caller that the file is new, or if it's being closed. Files use an internal unique id 'file_id' which can be used by the loggers to create unique file names. This id can use the 'waldo' feature of the log-filestore module. This patch moves that waldo loading and storing logic to this API's implementation. A new configuration directive 'file-store-waldo: <filename>' is added, but the existing waldo settings will also continue to work. 12 years ago			`#endif /* __OUTPUT_FILE_H__ */`