104 Commits (60d9eb6790d8f7d9fd2faf3d5b415d818a84591c)

Author	SHA1	Message	Date
Victor Julien	b34c6dc93a	counters: remove references to SCPerfCounterAddDouble ... They were all in comments anyway.	10 years ago
Victor Julien	e9b067c1eb	counters: make increment call take threadvars ... This hides the implementation from the caller.	10 years ago
Victor Julien	9a8bff7d96	counters: threadvars s/sc_perf_pca/perf_private_ctx/g	10 years ago
Victor Julien	04ccfda639	pcap: implement LINKTYPE_NULL ... Implement LINKTYPE_NULL for pcap live and pcap file. From: http://www.tcpdump.org/linktypes.html "BSD loopback encapsulation; the link layer header is a 4-byte field, in host byte order, containing a PF_ value from socket.h for the network-layer protocol of the packet. Note that ``host byte order'' is the byte order of the machine on which the packets are captured, and the PF_ values are for the OS of the machine on which the packets are captured; if a live capture is being done, ``host byte order'' is the byte order of the machine capturing the packets, and the PF_ values are those of the OS of the machine capturing the packets, but if a ``savefile'' is being read, the byte order and PF_ values are not necessarily those of the machine reading the capture file." Feature ticket #1445	10 years ago
Eric Leblond	ee7422de0a	pcap-file: add missing atomic init ... It is mandatory to init all atomic to avoid problem on system without atomic support.	11 years ago
Victor Julien	3499d682c4	flow timeout: cleanups ... Rename FlowForceReassemblyForFlowV2 to just FlowForceReassemblyForFlow as there is no V1.	11 years ago
Ken Steele	a781fc5c2e	Make suricata_ctl_flags be volatile ... The global variable suricata_ctl_flags needs to volatile, otherwise the compiler might not cause the variable to be read every time because it doesn't know other threads might write the variable. This was causing Suricata to not exit under some conditions.	11 years ago
Ken Steele	8f1d75039a	Enforce function coding standard ... Functions should be defined as: int foo(void) { } Rather than: int food(void) { } All functions where changed by a script to match this standard.	11 years ago
Victor Julien	98c88d5170	decode: pass ThreadVars to DecodeThreadVarsFree ... Flow output thread data deinit function which will be called from DecodeThreadVarsFree will need it.	11 years ago
Victor Julien	b5d3b7e92a	Fix pcap packet acquisition methods ... Fix pcap packet acquisition methods passing 0 to pcap_dispatch. Previously they passed the packet pool size, but the packet_q_len variable was now hardcoded at 0. This patch sets packet_q_len to 64. If packet pool is empty, we fall back to direct alloc. As the pcap_dispatch function is only called when packet pool is not empty, we alloc at most 63 packets.	11 years ago
Ken Steele	28ccea51d3	Add error checking for pthread_setspecific() and pthread_key_create().	11 years ago
Ken Steele	3c6e01f653	Replace ringbuffer in Packet Pool with a stack for better cache locality ... Using a stack for free Packet storage causes recently freed Packets to be reused quickly, while there is more likelihood of the data still being in cache. The new structure has a per-thread private stack for allocating Packets which does not need any locking. Since Packets can be freed by any thread, there is a second stack (return stack) for freeing packets by other threads. The return stack is protected by a mutex. Packets are moved from the return stack to the private stack when the private stack is empty. Returning packets back to their "home" stack keeps the stacks from getting out of balance. The PacketPoolInit() function is now called by each thread that will be allocating packets. Each thread allocates max_pending_packets, which is a change from before, where that was the total number of packets across all threads.	11 years ago
Victor Julien	bb2e9af40f	pcap-file: clean up decode thread local storage ... Clean up the thread local data the decode portion of pcap-file use. Bug #978.	11 years ago
Victor Julien	f7b1aefaf4	Bug 1107: decoders: bail out on pseudo packets ... Flow-timeout code injects pseudo packets into the decoders, leading to various issues. For a full explanation, see: https://redmine.openinfosecfoundation.org/issues/1107 This patch works around the issues with a hack. It adds a check to each of the decoder entry points to bail out as soon as a pseudo packet from the flow timeout is encountered. Ticket #1107.	12 years ago
Victor Julien	5f307acace	Pass ThreadVars ptr to various thread init funcs ... To be able to register counters from AppLayerGetCtxThread, the ThreadVars pointer needs to be available in it and thus in it's callers: - AppLayerGetCtxThread - DecodeThreadVarsAlloc - StreamTcpReassembleInitThreadCtx	12 years ago
Eric Leblond	afbb2eb32b	capture: display exit stats at default verbosity ... This patch updates capture modes not using LiveDecice counters to display per-thread exit statistics with default verbosity.	12 years ago
Eric Leblond	1bdc39fe9b	cmdline: add -k to specify checksum validation ... This patch adds a '-k' option to suricata to be able to specify the checksum validation to use. If '-k all' is used, checksum validation is forced. If '-k none' is used, no checksum validation is made. Message output in case of detection of a pcap file with a probable cheksum issue has been updated to indicate that '-k' is a solution.	12 years ago
Eric Leblond	8b5be26f49	pcap-file: add checksum-checks configuration variable ... This patch adds support for checksum-checks in the pcap-file running mode. This is the same functionnality as the one already existing for live interface. It can be setup in the YAML: pcap-file: checksum-checks: auto A message is displayed for small pcap to warn that invalid checksum rate is big on the pcap file and that checksum-check could be set to no.	12 years ago
Eric Leblond	3088b6ac34	Add invalid pkt counter. ... This patch adds and increments a invalid packet counter. It does this by introducing PacketDecodeFinalize function This function is incrementing the invalid counter and is also signalling the packet to CUDA.	12 years ago
Eric Leblond	d4b7ecfbe3	decode: update API to return error ... In some cases, the decoding is not possible and some really invalid packet can be created. This is in particular the case of tunnel. In that case, it is more interesting to forget about the tunneled packet and only consider the original packet. DecodeTunnel function is maked as warn_unused_result because it is meaningful for the decoder to know if the underlying data were not correct. And in this case, only focus detection on the content.	12 years ago
Victor Julien	677cd03e52	Counters: more unused code removal	12 years ago
Victor Julien	698ff4e4aa	Counters: remove all unused parts of the API	12 years ago
Anoop Saldanha	602c91ed41	Minor cosmetic changes to the cuda code. ... Moved a couple of functions to more cuda relevant files; Re-structured some data types.	12 years ago
Anoop Saldanha	17c763f855	Version 1 of AC Cuda.	12 years ago
Eric Leblond	5b067e1abb	pcap-file: treat the case of unsupported pcap link ... In unix socket mode, Suricata was stopping processing pcap files when a pcap file with an unsupported datalink was treated. This patch updates error handling to allow Suricata to treat other pcap files.	13 years ago
Eric Leblond	6b81430bcb	pcap-file: don't kill engine in unix socket mode ... This patch updates the cleaning code to avoid to exit from suricata in unix socket mode when a invalid pcap is given.	13 years ago
Eric Leblond	f8921d8a28	unix-socket: introduce API to add commands and tasks ... This patch transforms the unix socket into a flexible system to add commands (triggered by user) and taks (run periodically). It introduces two functions UnixManagerRegisterCommand and UnixManagerRegisterBackroundTask to registed commands and tasks. Other part of Suricata can then declare a new command via a simple call of the function. In the case of a command the caller is responsible of building the answer message using Jansson API. The sending of the message is made by unix manager code.	13 years ago
Eric Leblond	20a8b9dbe5	unix-manager: add unix command socket and associated script ... This patch introduces a unix command socket. JSON formatted messages can be exchanged between suricata and a program connecting to a dedicated socket. The protocol is the following: * Client connects to the socket * It sends a version message: { "version": "$VERSION_ID" } * Server answers with { "return": "OK|NOK" } If server returns OK, the client is now allowed to send command. The format of command is the following: { "command": "pcap-file", "arguments": { "filename": "smtp-clean.pcap", "output-dir": "/tmp/out" } } The server will try to execute the "command" specified with the (optional) provided "arguments". The answer by server is the following: { "return": "OK|NOK", "message": JSON_OBJECT or information string } A simple script is provided and is available under scripts/suricatasc. It is not intended to be enterprise-grade tool but it is more a proof of concept/example code. The first command line argument of suricatasc is used to specify the socket to connect to. Configuration of the feature is made in the YAML under the 'unix-command' section: unix-command: enabled: yes filename: custom.socket The path specified in 'filename' is not absolute and is relative to the state directory. A new running mode called 'unix-socket' is also added. When starting in this mode, only a unix socket manager is started. When it receives a 'pcap-file' command, the manager start a 'pcap-file' running mode which does not really leave at the end of file but simply exit. The manager is then able to start a new running mode with a new file. To start this mode, Suricata must be started with the --unix-socket option which has an optional argument which fix the file name of the socket. The path is not absolute and is relative to the state directory. THe 'pcap-file' command adds a file to the list of files to treat. For each pcap file, a pcap file running mode is started and the output directory is changed to what specified in the command. The running mode specified in the 'runmode' YAML setting is used to select which running mode must be use for the pcap file treatment. This requires modification in suricata.c file where initialisation code is now conditional to the fact 'unix-socket' mode is not used. Two other commands exists to get info on the remaining tasks: * pcap-file-number: return the number of files in the waiting queue * pcap-file-list: return the list of waiting files 'pcap-file-list' returns a structured object as message. The structure is the following: { 'count': 2, 'files': ['file1.pcap', 'file2.pcap'] }	13 years ago
Eric Leblond	84f2645e3e	pcap-file: free thread var at deinit.	13 years ago
Victor Julien	829238e49c	OpenBSD 5.2 build fixes, Unit test fix.	13 years ago
Anoop Saldanha	b33986c887	Add a packet src for every packet generated inside suricata.	13 years ago
Eric Leblond	e176be6fcc	Use unlikely for error treatment. ... When handling error case on SCMallog, SCCalloc or SCStrdup we are in an unlikely case. This patch adds the unlikely() expression to indicate this to gcc. This patch has been obtained via coccinelle. The transformation is the following: @istested@ identifier x; statement S1; identifier func =~ "(SCMalloc|SCStrdup|SCCalloc)"; @@ x = func(...) ... when != x - if (x == NULL) S1 + if (unlikely(x == NULL)) S1	13 years ago
Eric Leblond	c4f9d0e0e1	Fix invalid usage of operator.	13 years ago
Anoop Saldanha	34581ce902	rx TMs shouldn't return TM_ECODE_FAILED if engine is in shutdown mode + minor cleanup	13 years ago
Anoop Saldanha	bc6cf43840	#482 - use decode_flag for all decode TMs. Use the flag as a way to retrieve decode TMs from ThreadVars	13 years ago
Victor Julien	41e9dba20b	Profile pcap file callback.	14 years ago
Victor Julien	0150e66ede	flow engine: improve scalability ... Major redesign of the flow engine. Remove the flow queues that turned out to be major choke points when using many threads. Flow manager now walks the hash table directly. Simplify the way we get a new flow in case of emergency.	14 years ago
Victor Julien	28e15be526	Clean up default output. Use simpler output format for releases.	14 years ago
Anoop Saldanha	5133098bd6	Accomodate pcap-file mode to signal flow mgr to wakeup when it exceeds a certain time interval. This let's the flow mgr keep in sync with pcap timestamp changes	14 years ago
Victor Julien	67cea09911	Handle failing thread modules that are called by the Pcap file callback.	14 years ago
Anoop Saldanha	d68f182ebd	introduce SCPerfSyncCounters/SCPerfSyncCounters macro to synchronize counters	14 years ago
Anoop Saldanha	f7b1972263	update broken stats.log. Use pktacqloop funcs in pcap-file, pfring, pcap-live, af-pkt to sync counters - bug #343	14 years ago
Anoop Saldanha	3f1c4efceb	Add new flags var to tm module. TMs can now set flags to identify special properties. Also use these to identify receive TMs	14 years ago
Eric Leblond	de1d002ea6	Return OK when leaving cleanly.	14 years ago
Victor Julien	820b0ded82	Add per packet profiling. ... Per packet profiling uses tick based accounting. It has 2 outputs, a summary and a csv file that contains per packet stats. Stats per packet include: 1) total ticks spent 2) ticks spent per individual thread module 3) "threading overhead" which is simply calculated by subtracting (2) of (1). A number of changes were made to integrate the new code in a clean way: a number of generic enums are now placed in tm-threads-common.h so we can include them from any part of the engine. Code depends on --enable-profiling just like the rule profiling code. New yaml parameters: profiling: # packet profiling packets: # Profiling can be disabled here, but it will still have a # performance impact if compiled in. enabled: yes filename: packet_stats.log append: yes # per packet csv output csv: # Output can be disabled here, but it will still have a # performance impact if compiled in. enabled: no filename: packet_stats.csv Example output of summary stats: IP ver Proto cnt min max avg ------ ----- ------ ------ ---------- ------- IPv4 6 19436 11448 5404365 32993 IPv4 256 4 11511 49968 30575 Per Thread module stats: Thread Module IP ver Proto cnt min max avg ------------------------ ------ ----- ------ ------ ---------- ------- TMM_DECODEPCAPFILE IPv4 6 19434 1242 47889 1770 TMM_DETECT IPv4 6 19436 1107 137241 1504 TMM_ALERTFASTLOG IPv4 6 19436 90 1323 155 TMM_ALERTUNIFIED2ALERT IPv4 6 19436 108 1359 138 TMM_ALERTDEBUGLOG IPv4 6 19436 90 1134 154 TMM_LOGHTTPLOG IPv4 6 19436 414 5392089 7944 TMM_STREAMTCP IPv4 6 19434 828 1299159 19438 The proto 256 is a counter for handling of pseudo/tunnel packets. Example output of csv: pcap_cnt,ipver,ipproto,total,TMM_DECODENFQ,TMM_VERDICTNFQ,TMM_RECEIVENFQ,TMM_RECEIVEPCAP,TMM_RECEIVEPCAPFILE,TMM_DECODEPCAP,TMM_DECODEPCAPFILE,TMM_RECEIVEPFRING,TMM_DECODEPFRING,TMM_DETECT,TMM_ALERTFASTLOG,TMM_ALERTFASTLOG4,TMM_ALERTFASTLOG6,TMM_ALERTUNIFIEDLOG,TMM_ALERTUNIFIEDALERT,TMM_ALERTUNIFIED2ALERT,TMM_ALERTPRELUDE,TMM_ALERTDEBUGLOG,TMM_ALERTSYSLOG,TMM_LOGDROPLOG,TMM_ALERTSYSLOG4,TMM_ALERTSYSLOG6,TMM_RESPONDREJECT,TMM_LOGHTTPLOG,TMM_LOGHTTPLOG4,TMM_LOGHTTPLOG6,TMM_PCAPLOG,TMM_STREAMTCP,TMM_DECODEIPFW,TMM_VERDICTIPFW,TMM_RECEIVEIPFW,TMM_RECEIVEERFFILE,TMM_DECODEERFFILE,TMM_RECEIVEERFDAG,TMM_DECODEERFDAG,threading 1,4,6,172008,0,0,0,0,0,0,47889,0,0,48582,1323,0,0,0,0,1359,0,1134,0,0,0,0,0,8028,0,0,0,49356,0,0,0,0,0,0,0,14337 First line of the file contains labels. 2 example gnuplot scripts added to plot the data.	14 years ago
Eric Leblond	88559901d4	pcap-file: Allocated packet must be free if there's error	14 years ago
Victor Julien	b753ecce50	Implement a pkt acq loop infra with support for pcap-file.	14 years ago
Eric Leblond	dd038c1906	Modify files to avoid direct pckt payload access ... This patch implements the needed modification of payload access in a Packet structure to support the abstraction introduced by the extended data system.	15 years ago
Victor Julien	4cacb1e970	Disable adding to unregistered mbit/s counter.	15 years ago
Victor Julien	6943a7eb8c	Move updating the time from the pcap callback to the decoding stage in file mode.	15 years ago