9691 Commits (e9fec043b51374ee3927e2111d1ac5f5b59d79ca)

Author	SHA1	Message	Date
Victor Julien	107ad95c41	fuzz/pcap: enable http2	6 years ago
Victor Julien	928d4820f9	plugins: remove unused func, suppressing compile warning	6 years ago
Victor Julien	9902413745	plugins: add missing guards	6 years ago
Jeff Lucovsky	abc8bd11b9	output/ssh: Use correct file context ... This commit corrects an issue with the SSH output module that resulted in a SEGV when SSH output is logged.	6 years ago
Jason Ish	e10d107415	plugins: support for capture plugins ... Allow a plugin to register itself as a capture source. This isn't that much different than how current sources register, it just happens a little later on during startup. One "slot" is reserved for capture plugins, but multiple plugins implementing a capture can be loaded. The --capture-plugin command line option must be used to tell Suricata which plugin to use. This is still very much a work in progress, but can load PF_RING as a capture plugin.	6 years ago
Jason Ish	8fb35236e6	plugins: initial support for a filetype plugin ... A filetype plugin is a plugin that implements an eve filetype. Most of the current filetypes could likely be implemented as such a plugin. Such a plugin must implement Open, Close and Write, where Write is provided the formatted JSON to be logged. This commit also includes the plumbing for plugin loading. Example plugin to come. Plugins are loaded by the "plugin" section in the configuration file: plugins: - /path/to/directory/plugins - /path/to/plugin_file.so This can also be done on the command line with: --set plugins.0=/path/plugin_file.so	6 years ago
Jason Ish	f35c25cef2	util-error: define generic plugin error code	6 years ago
Jason Ish	900f1522b4	plugins: config.h: move into src and rename to autoconf.h ... While fixing files that include config.h, just remove the include if possible.	6 years ago
Jason Ish	f26d6eaf98	http2: log as http to abstract http and http2 a little ... This commit logs http2 as an http event. The idea is to somewhat normalize http/http2 so common info can be version agnostic. This puts the http2 specific fields in an "http2" object inside the "http" object. HTTP2 headers/values that are in common with HTTP1 are logged under the "http" object to be compatible with HTTP1 logging.	6 years ago
Philippe Antoine	1422b18a99	http2: initial support	6 years ago
Philippe Antoine	0507d1e8f8	detect: generic structures for mpm with lists	6 years ago
Victor Julien	ee41c0e293	flow/spare: implement pool shrinking ... Remove at most one block per run, so it shrinks slowly.	6 years ago
Victor Julien	b3599507f4	flow: redesign of flow timeout handling ... Goals: - reduce locking - take advantage of 'hot' caches - better locality Locking reduction New flow spare pool. The global pool is implmented as a list of blocks, where each block has a 100 spare flows. Worker threads fetch a block at a time, storing the block in the local thread storage. Flow Recycler now returns flows to the pool is blocks as well. Flow Recycler fetches all flows to be processed in one step instead of one at a time. Cache 'hot'ness Worker threads now check the timeout of flows they evaluate during lookup. The worker will have to read the flow into cache anyway, so the added overhead of checking the timeout value is minimal. When a flow is considered timed out, one of 2 things happens: - if the flow is 'owned' by the thread it is handled locally. Handling means checking if the flow needs 'timeout' work. - otherwise, the flow is added to a special 'evicted' list in the flow bucket where it will be picked up by the flow manager. Flow Manager timing By default the flow manager now tries to do passes of the flow hash in smaller steps, where the goal is to do full pass in 8 x the lowest timeout value it has to enforce. So if the lowest timeout value is 30s, a full pass will take 4 minutes. The goal here is to reduce locking overhead and not get in the way of the workers. In emergency mode each pass is full, and lower timeouts are used. Timing of the flow manager is also no longer relying on pthread condition variables, as these generally cause waking up much quicker than the desired timout. Instead a simple (u)sleep loop is used. Both changes reduce the number of hash passes a lot. Emergency behavior In emergency mode there a number of changes to the workers. In this scenario the flow memcap is fully used up and it is unavoidable that some flows won't be tracked. 1. flow spare pool fetches are reduced to once a second. This avoids locking overhead, while the chance of success was very low. 2. getting an active flow directly from the hash skips flows that had very recent activity to avoid the scenario where all flows get only into the NEW state before getting reused. Rather allow some to have a chance of completing. 3. TCP packets that are not SYN packets will not get a used flow, unless stream.midstream is enabled. The goal here is again to avoid evicting active flows unnecessarily. Better Localily Flow Manager injects flows into the worker threads now, instead of one or two packets. Advantage of this is that the worker threads can get packets from their local packet pools, avoiding constant overhead of packets returning to 'foreign' pools. Counters A lot of flow counters have been added and some have been renamed. Overall the worker threads increment 'flow.wrk.*' counters, while the flow manager increments 'flow.mgr.*'. Additionally, none of the counters are snapshots anymore, they all increment over time. The flow.memuse and flow.spare counters are exceptions. Misc FlowQueue has been split into a FlowQueuePrivate (unlocked) and FlowQueue. Flow no longer has 'prev' pointers and used a unified 'next' pointer for both hash and queue use.	6 years ago
Victor Julien	e0aa7c1dbc	unittests: check for flow memuse	6 years ago
Victor Julien	f50c7b6d11	flow-manager: call other timeouts max once a second ... Call Defrag and others only once per second. Flow Manager may wake up (much) more often when flow engine is under resource pressure. As this does not affect Defrag and others, it only unnecessarily adds load.	6 years ago
Victor Julien	6814f08e93	flow-manager: only update FlowBucket::next_ts if it changed	6 years ago
Victor Julien	1d6d7f0858	flow: unref flow at end of flow worker	6 years ago
Victor Julien	2fd7c87f22	flow: do timeout checks before tuple compare	6 years ago
Victor Julien	48605f4867	flow: don't reorder list on lookup ... Reduces cache misses.	6 years ago
Victor Julien	2a872ccb86	flow: timeout check on flow lookup	6 years ago
Victor Julien	7583a6c37c	flow: simplify hash lookup logic ... Remove double compare paths in favor of a single unified path.	6 years ago
Victor Julien	8b016cff4b	flow: only move lastts forward ... Pcaps with timestamps jumping around could confuse flow timeout handling otherwise.	6 years ago
Victor Julien	afd4a8012e	flow/worker: check pkt src using DEBUG_VALIDATE_BUG_ON	6 years ago
Jason Ish	fd5d8b78d0	alert/eve: add snmp metadata for rdp alerts ... Redmine issue: https://redmine.openinfosecfoundation.org/issues/3441	6 years ago
Jason Ish	ef0ebc9550	alert/eve: add snmp metadata for snmp alerts ... Redmine issue: https://redmine.openinfosecfoundation.org/issues/3441	6 years ago
Shivani Bhardwaj	4c7f55e636	dcerpc: handle gap for TCP streams	6 years ago
James Dutrisac	db5b73d9d6	pcap: read directories recursively ... Describe Changes - Added ability to recursively read pcap directories - src/suricata.c: addition of new command line parameter --pcap-file-recursive - src/source-pcap-file.c: parsing of the command line argument - src/source-pcap-file-directory-helper.h: two thread vars tracking directory depth and should recurse - src/util-error.c / src/util-error.h: Added new warning code "SC_WARN_PATH_READ_ERROR" - Redmine ticket: https://redmine.openinfosecfoundation.org/issues/2363 Ticket: #2363	6 years ago
James Dutrisac	5a92d0a704	path: introduce path handling util funcs ... This commit provides changes to util-path.c and util-path.h to support the recursive reading of directories. It adds 4 functions. - SCIsRegularFile to provide OS independent file info. - SCIsRegularDirectory to provide OS independent directory info. - SCRealPath is an OS independent wrapper for realpath. - PathJoin to manage path resolution logic.	6 years ago
Philippe Antoine	3cfc1fcb07	stream/app-layer: break loop on proto change	6 years ago
Shivani Bhardwaj	c9a637d854	datasets: fix null pointer deref	6 years ago
Shivani Bhardwaj	a8f147d17a	datasets: Init even in socket mode ... Closes redmine ticket 3476.	6 years ago
Sascha Steinbiss	93eef1da84	detect-mqtt: unify error handling in rule parsing ... This is meant to provide a single path to the error case. This might help make things more clear for static checkers.	6 years ago
Sascha Steinbiss	5dc21b0e09	detect-engine: initialize struct fields	6 years ago
Philippe Antoine	9a954e944d	http: merge duplicated code ... HtpRequestBodyHandlePUT and HtpRequestBodyHandlePOST	6 years ago
Philippe Antoine	fe6950de08	http: use more precise parameter in HtpRequestBodySetupMultipart ... So that we can see that it does not have to handle gaps	6 years ago
Philippe Antoine	e13b319beb	http: remove unused code ... HtpRequestBodySetupPUT function So that we can see that we do not use data=NULL in there	6 years ago
Philippe Antoine	28050c967f	file: handles gaps natively ... ie data=NULL and len>0 parameters	6 years ago
Philippe Antoine	f0159b2fd2	util: PrintRawDataFp handles null ... for gaps which are data=NULL and len>0	6 years ago
Philippe Antoine	6343920dfc	applayer: allow rust parsers to have only one probe	6 years ago
Sascha Steinbiss	4e1a41a17d	output-json: add MAC address output ... This commit adds MAC address output to the EVE-JSON format. We follow the remarks made in Redmine ticket #962: for packets, log MAC src/dst as a scalar field in EVE; for flows, log MAC src/dst as lists in EVE. Field names are different between flow and packet context to avoid type confusion (src_mac vs. src_macs). Configuration approach and JSON representation is taken from previous GitHub PR #2700.	6 years ago
Jeff Lucovsky	c42574169e	output/anomaly: Restrict anomaly logger count ... This commit restricts the anomaly logger count. The restriction is necessary due to state maintenance in the logger that doesn't scale beyond a single logger. Until that issue's solved, when multiple anomaly loggers are configured, an error message will be emitted to highlight the restriction.	6 years ago
Philippe Antoine	61c327dd80	signature: checks for integer overflow in limits propagation	6 years ago
Victor Julien	1c748f394b	fuzz/pcap: enable MQTT parser	6 years ago
Shivani Bhardwaj	e9fe5ada7f	datasets: reload static sets	6 years ago
Sascha Steinbiss	c31360070b	rust/mqtt: add MQTT parser	6 years ago
Philippe Antoine	a5572890a9	detect: adds engine for u8 keywords	6 years ago
Jeff Lucovsky	30ae98f658	output/json: Multi-threaded EVE logging support ... This commit modifies the JSON loggers with changes necessary to support multi-threaded EVE output. Each "thread-init" function sets up the per-thread log file context for subsequent calls to the JSON output to buffer function.	6 years ago
Jeff Lucovsky	aa20770277	log: Support multi-threaded eve output.	6 years ago
Jeff Lucovsky	15b4554ab3	output: Check for fwrite_unlocked ... This commit creates a macro for fwrite_unlocked which is probed during configuration time.	6 years ago
Jeff Lucovsky	3d0f353ee5	output: Correct typos	6 years ago
Jeff Lucovsky	1b791f34a5	output: Remove unused variables/define	6 years ago
Jeff Lucovsky	92e2e2ec8d	log: remove unused include files	6 years ago
Jeff Lucovsky	60658cbe01	output/flow: Eliminate unnecessary parameter ... This commit removes a parameter to an internal-only function call. Removing the parameter allows an JSON builder optimization to be used.	6 years ago
Jeff Lucovsky	4aa7c988e8	output/netflow: Eliminate unneeded parameter ... This commit changes an internal-only function to remove a parameter that's invariant in all use cases. This allows an JSON builder optimization to be used.	6 years ago
Philippe Antoine	1569f3e349	transform: adds url_decode keyword ... Fixes https://redmine.openinfosecfoundation.org/issues/2689 Adds a new source file to handle this keyword. And modifies documentation, Makefile, and registration accordingly. url_decode decodes url-encoded data, ie replacing '+' with space and '%HH' with its value.	6 years ago
Victor Julien	4c8af9cb96	stream: fix endless loop in traffic with gaps	6 years ago
Victor Julien	9b13c1b804	flow: avoid double state update on reuse ... Avoids an unnecessary atomic operation.	6 years ago
Victor Julien	fd2dff8542	flow: minor cleanups	6 years ago
Victor Julien	7bf000731c	flow: validate emergency timeout settings ... Make sure they are below the regular values.	6 years ago
Victor Julien	611c991f27	flow: improve performance in emergency mode ... When the flow engine enters emergency mode, 3 things happen: 1. a different set of (lower) timeout values are applied 2. the flow manager runs more often 3. worker threads go get a flow directly from the hash table Testing showed that performance went down significantly due to concurrency issues: 1. worker threads would fight each other over the hash access 2. flow manager would get in the way of workers This patch changes the behavior in 2 ways: 1. it makes the flow manager slightly less aggressive. It will still try to run ~3 times per second, but no longer 10 times. This should be reducing the contention. At the same time flows won't time out faster if they are checked many times per second. 2. The 'get a used flow' logic optimizes the use of atomics by only doing an atomic operation once, and while doing so reserving a slice of the hash per worker. The worker will also give up much quicker, to avoid the overhead of hash walking and taking and releasing locks. These combined changes show much better 'under stress' behavior, esp on multi-NUMA systems.	6 years ago
Philippe Antoine	0da4dc0dea	enip: use status for probing parser	6 years ago
Philippe Antoine	a99ad4c1e4	signature: checks for integer overflow in limits propagation	6 years ago
Philippe Antoine	5c31383d1c	detect: fix read overflow in DetectGetLastSMByListId	6 years ago
Victor Julien	b99ffd9ece	eve: remove unused jansson code	6 years ago
Victor Julien	a8e2399ea9	eve/metadata: create preformatted json string at start up ... Avoid runtime overhead of assembling metadata json string by pre-creating it at rule parsing time.	6 years ago
Victor Julien	1a18081a59	detect/profile: convert match dumps to jsonbuilder ... Remove unused code and do minor misc cleanups as well.	6 years ago
Victor Julien	1639dfa36e	pfring: fix compile warning	6 years ago
Victor Julien	38fe11f3b2	eve: remove unused json_t common functions ... These are no longer used as all callers have switched to the JsonBuilder equivalents.	6 years ago
Victor Julien	7ccfa177de	eve/tls: minor cleanups	6 years ago
Victor Julien	04dad483c1	eve/metadata: convert to jsonbuilder	6 years ago
Victor Julien	556bee2adc	stream: call parser with 0 data on EOF ... This way both sides can call the EOF logic.	6 years ago
Victor Julien	616d7f256b	app-layer/tcp: don't use un-ACK'd data ... Still use un-ACK'd data in unclean shutdown. This means any state before TCP_CLOSED, or TCP_CLOSED that was caused by a RST.	6 years ago
Victor Julien	42205006d1	flow/timeout: flag last pseudo packet ... Flag the last flow timeout pseudo packet so that we can force TX logging w/o setting both app-layer flags. Case this fixes: 1. flow times out when only TS TCP data received, but non of it is ACK'd. So there is no app-layer proto yet, or app state or Flow::alparser. So EOF flags can't be set. 2. Flow timeout sees no reason to create pseudo packet in TC direction. 3. TS pseudo packet finds HTTP, creates HTTP state, flag EOF TS. 4. TX logging skips HTTP logging because: - TC progress not reached - EOF TC flag not set. The solution has been to flag the very last packet for the flow as such and use it has a master-EOF flag.	6 years ago
Victor Julien	c825f83633	stream/tcp: track if ssn has been closed with RST	6 years ago
Victor Julien	1b3582325b	app-layer: set EOFs on app-layer disable	6 years ago
Victor Julien	1cbbc82647	flow/worker: set EOF flags on change proto	6 years ago
Victor Julien	a9f2540203	flow-timeout: set app-layer EOF flag	6 years ago
Victor Julien	ecd7862c36	app-layer: add debug	6 years ago
Victor Julien	d8d59ac9b5	stream: minor debug fixup	6 years ago
Victor Julien	5fd9386665	app-layer/pd: improve size check in bail conditions	6 years ago
Victor Julien	4f73943df9	app-layer: split EOF flag per direction	6 years ago
Victor Julien	57b75f89da	stream: app update from loop ... When the stream engine has data ready for the app-layer it will call this API from a loop instead of just once. The loop is to ensure that if we have a very lossy stream where between 'app_progress' and 'last_ack' there are multiple chunks of data and multiple gaps we process all the chunks.	6 years ago
Victor Julien	e822b30cc2	stream: improve gap handling with 'incomplete' ... Make sure stream requiring more data because of 'incomplete' records properly move ahead if there is a GAP in the window of required data.	6 years ago
Victor Julien	c7d59a61ea	stream: fix IDS mode using un-ACK'd data	6 years ago
Victor Julien	49eba6ac23	stream: code cleanup	6 years ago
Victor Julien	f65bf4c7ea	flow/tcp: consider pkts established based on 3whs	6 years ago
Victor Julien	7309c97eda	detect/flow: test cleanup	6 years ago
Jeff Lucovsky	52cb1b8167	detect/dns-query: Splice UT to rust	6 years ago
Victor Julien	0025467f90	sources: hide RegisterTests behind ifdef UNITTESTS ... Update callers.	6 years ago
Victor Julien	085eb9fc8e	eve/ssh: minor cleanup	6 years ago
Philippe Antoine	0c92b8f7e4	dcerpc: adds invalid signature unit test	6 years ago
Philippe Antoine	b8069365f5	dcerpc: check app proto for signature keywords	6 years ago
Victor Julien	6ab323d323	detect: hide RegisterTests behind ifdef UNITTESTS ... Update all callers to more aggressively use UNITTESTS guards as well.	6 years ago
Xiaofan Wang	071f55dcd7	ftp: fix direction of expectation for STOR command ... Fix direction in active mode.	6 years ago
Victor Julien	0d24066876	sip: minor cleanup	6 years ago
Victor Julien	db3b637ada	htp: minor UNITTESTS guarding cleanup	6 years ago
Victor Julien	7c364017da	ftp: small code cleanup	6 years ago
Jeff Lucovsky	72e2f36f9b	ftp: Restrict file name lengths ... Restrict file name lengths to PATH_MAX - 1 to avoid over subscribing memory to FTP file name tracking.	6 years ago
Shivani Bhardwaj	c169cfe0a3	bytetest: use ByteExtractString instead of StringParse	6 years ago
Shivani Bhardwaj	6f84515dd9	util: fix trailing char check with ByteExtractString	6 years ago