9751 Commits (c41e64d637fc0ce7a95e68c2de08f4d4ea24f49f)

Author	SHA1	Message	Date
Jason Ish	c4d0a61eca	datasets: fix dataset load path construction ... Test the full path instead of just the filename provided in the rule to see if it exists. Fixes the case where a rule file is loaded from a directory other than the default-rule-directory. Redmine issue: https://redmine.openinfosecfoundation.org/issues/3916	5 years ago
Philippe Antoine	9788b2ec8d	signature: frees transform options in SigMatchPrepare	5 years ago
Philippe Antoine	1674239442	detect: checks for overflow when comparing signatures priorities	5 years ago
Victor Julien	abca451901	flow: suppress Coverity FP	5 years ago
Victor Julien	204302cbac	flow: minor code cleanup	5 years ago
Victor Julien	42ce297e0e	flow: turn BUG_ON into debug check	5 years ago
Eric Leblond	6494abc6b1	ebpf: fix invalid description in doc string	5 years ago
Victor Julien	6492fe0841	detect/app-layer-events: improve warnings/errors ... Improve handling of outdated events that are no longer supported by the engine.	5 years ago
Victor Julien	8cd82486e2	flow/bypass: don't bypass on flow timeout pseudo packets	5 years ago
Jason Ish	cc2c7b731a	rule parsing: valid that input rule string is UTF8 ... Before parsing a rule string, validate that it is UTF-8 first. Related Redmine issue: https://redmine.openinfosecfoundation.org/issues/3850	5 years ago
Jason Ish	583593aa1b	http/eve: use set_string_from_bytes where appropriate ... The JsonBuilder set_string_from_bytes will escape unprintable characters in the output stream. Using BytesToStringBuffer can generate invalid UTF-8 which prevents the conversion from a C string to a Rust string.	5 years ago
Victor Julien	d6cf4b3335	detect/dataset: error if set couldn't be fully loaded	5 years ago
Victor Julien	017c038bcb	datasets: free old data when reusing a hash container	5 years ago
Jeff Lucovsky	00f77f9643	log: Log errors while writing log info ... This commit adds logic to log errors during output. Errors are logged once and the number of errors is maintained.	5 years ago
Jeff Lucovsky	b9458adf8a	log: Add log output error code	5 years ago
Jeff Lucovsky	6cdd87f5ad	log: Use unlocked variants of stdio functions ... This commit uses the unlocked variants of additional stdio functions	5 years ago
Jeff Lucovsky	3c91e14352	log: Add more stdio_unlocked macros ... This commit adds additional macros for interfaces in stdio_unlocked according to their local availability.	5 years ago
Victor Julien	fa0b91b18f	error: reformat enum	5 years ago
Victor Julien	fbdc776525	app-layer: handle parser return code issues more gracefully	5 years ago
Victor Julien	5155982ba5	pcap: fix minor scan-build warnings	5 years ago
Victor Julien	aac2b9dbb9	commandline: add static analyzer hint for -r parsing	5 years ago
Victor Julien	71297f575f	commandline: minor formatting fixes	5 years ago
Jeff Lucovsky	9efb936697	general: Improve grammar in error messages ... This commit corrects a minor grammar issue in address/port error messages.	5 years ago
Ali Jad Khalil	6be08637c3	decode/teredo: Modified/refactored Teredo logic ... This is just a slight refactor to make analagous decoding/encapsulation schemes - Geneve, Teredo, and VXLAN - be implemented as similarly as possible.	5 years ago
Ali Jad Khalil	66452dd38a	decode/vxlan: Modified/refactored VXLAN logic ... This is just a slight refactor to make analagous decoding/encapsulation schemes - Geneve, Teredo, and VXLAN - be implemented as similarly as possible.	5 years ago
Ali Jad Khalil	5d955c1836	decode/geneve: Add Geneve decoding functionality ... These changes are in response to feature request 3063. Geneve is very similar to VXLAN, but uses a slightly different encapsulation scheme.	5 years ago
Victor Julien	77bafe13a3	decode: reformat event table	5 years ago
Jason Ish	665328b29e	plugins: require registration function SCPluginRegister ... Instead of looking for a symbol, "PluginSpec" look for a function named SCPluginRegister that returns a SCPlugin. This makes it much easier to create Rust plugins without having to deal with dlopen constructors and such, which is rather straight forward in C, but a bit of advanced boilerplate in Rust that can be eliminated by simply calling a registration function.	5 years ago
Jason Ish	335e4e728f	suricata: expose the SuricataContext with a function ... Expose the "SuricataContext" required by Rust as a function. During normal startup we register this context with the Rust code, but plugins written in Rust will need to get the same registration done, but to do this in a plugin, the plugin code must call and set the context within its address space.	5 years ago
Jason Ish	87a91e6aa8	logging: expose the log level with a function ... The log level needs to exposed so Rust plugins can bootstrap themselves with the correct login to SCLogNotice!(), etc work as expected.	5 years ago
Jason Ish	17776e098e	rust-context: remove unused opaque type Store	5 years ago
Joshua Lumb	6dbc6d6669	runmodes: memory leak on runmode single	5 years ago
Jeff Lucovsky	ff29345527	output: Remove unused output functions ... This commit removes registration, initialization, and de-initialization functions no longer needed	5 years ago
Jeff Lucovsky	15caf3eea5	log: Remove standalone output registration ... Remove standalone output logger registration since eve is multi-instance.	5 years ago
Victor Julien	703de67086	flow: fix multi-manager hash range calculation	5 years ago
Victor Julien	260a20bf91	flow: suppress noisy messages	5 years ago
Philippe Antoine	c06d8f2463	dcerpc: validate signature with dcerpc keywords ... so that they do not use another protocol's keywords	5 years ago
Philippe Antoine	d509a78074	http: handles gaps ... ie data=NULL and len>0 from libhtp callbacks	5 years ago
Victor Julien	23c0efa2ec	flowbits: convert flowbits dumping to json builder	5 years ago
Victor Julien	7facf5785f	detect/analyzer: convert to jsonbuilder	5 years ago
Victor Julien	d3cf2c21df	threshold: fix potential memory leak	5 years ago
Victor Julien	6444f738af	reject: minor code cleanup ... Use enum for direction to assist compiler.	5 years ago
Victor Julien	b6fed6d431	stream: fix GAP size ... Fix GAPs sizes sometimes exceeding the ACK'd data size.	5 years ago
Victor Julien	6978d8dac6	stream: fix deinit after incomplete init	5 years ago
Victor Julien	a015a31073	output/flow: fix deinit after incomplete init	5 years ago
Victor Julien	c39101bbec	nfq: suppress debug message using info level	5 years ago
Victor Julien	797a128256	plugins: suppress coverity toctou warning	5 years ago
Victor Julien	bd68bd5037	flow: work around harmless coverity warnings	5 years ago
Victor Julien	29148fb563	flow: fix spare sync incomplete counter	5 years ago
Victor Julien	50c7b0d1c3	reject: work around coverity warning	5 years ago
Vadym Malakhatko	65455208a5	plugin: fix typo in long_opts struct	5 years ago
Jeff Lucovsky	cc93638d33	napatech: Style -- remove extra space	5 years ago
Jeff Lucovsky	67529bd25a	napatech: Use proper parser for type ... This commit uses the proper parser call for the value being parsed.	5 years ago
Jeff Lucovsky	45b055aca5	napatech: Improve configuration range handling ... This commit corrects issues parsing ranges from the Napatech section of the configuration file.	5 years ago
Jeff Lucovsky	c408b15c51	napatech: Fix compiler issues w/out bypass ... This commit fixes compiler errors when Napatech bypass is not configured	5 years ago
Carl Smith	81d7a7aa82	threshold: Change rule parsing to use pcre_copy_substring ... Fixes memory leak when parsing threshold rules. All parsed strings are less than 16 characters except for the IP address which could be up to 48 characters. Remove redefinition of MAX_SUBSTRINGS	5 years ago
Jason Ish	b8994cdaca	plugins: track all loaded plugins in a list ... Track the pointer returned from dlopen in a list to prevent a resource leak by the pointer going out of scope. Found by Coverity, CID 1465661. Redmine issue: https://redmine.openinfosecfoundation.org/issues/3864	5 years ago
Jason Ish	dbb5dcb1dd	plugins: use closedir to close open directory (not free) ... Found by Coverity, CID 1465665: ALLOC_FREE_MISMATCH. Redmine issue: https://redmine.openinfosecfoundation.org/issues/3864	5 years ago
Jason Ish	f2a1626b51	output-json: fix Coverity USE_AFTER_FREE ... Return error if plugin open fails. Fixes Coverity CID 1465664 USE_AFTER_FREE error. Redmine issue: https://redmine.openinfosecfoundation.org/issues/3864	5 years ago
Victor Julien	ac491c6e8d	fuzz/pcap: add missing flow queue	5 years ago
Victor Julien	107ad95c41	fuzz/pcap: enable http2	5 years ago
Victor Julien	928d4820f9	plugins: remove unused func, suppressing compile warning	5 years ago
Victor Julien	9902413745	plugins: add missing guards	5 years ago
Jeff Lucovsky	abc8bd11b9	output/ssh: Use correct file context ... This commit corrects an issue with the SSH output module that resulted in a SEGV when SSH output is logged.	5 years ago
Jason Ish	e10d107415	plugins: support for capture plugins ... Allow a plugin to register itself as a capture source. This isn't that much different than how current sources register, it just happens a little later on during startup. One "slot" is reserved for capture plugins, but multiple plugins implementing a capture can be loaded. The --capture-plugin command line option must be used to tell Suricata which plugin to use. This is still very much a work in progress, but can load PF_RING as a capture plugin.	5 years ago
Jason Ish	8fb35236e6	plugins: initial support for a filetype plugin ... A filetype plugin is a plugin that implements an eve filetype. Most of the current filetypes could likely be implemented as such a plugin. Such a plugin must implement Open, Close and Write, where Write is provided the formatted JSON to be logged. This commit also includes the plumbing for plugin loading. Example plugin to come. Plugins are loaded by the "plugin" section in the configuration file: plugins: - /path/to/directory/plugins - /path/to/plugin_file.so This can also be done on the command line with: --set plugins.0=/path/plugin_file.so	5 years ago
Jason Ish	f35c25cef2	util-error: define generic plugin error code	5 years ago
Jason Ish	900f1522b4	plugins: config.h: move into src and rename to autoconf.h ... While fixing files that include config.h, just remove the include if possible.	5 years ago
Jason Ish	f26d6eaf98	http2: log as http to abstract http and http2 a little ... This commit logs http2 as an http event. The idea is to somewhat normalize http/http2 so common info can be version agnostic. This puts the http2 specific fields in an "http2" object inside the "http" object. HTTP2 headers/values that are in common with HTTP1 are logged under the "http" object to be compatible with HTTP1 logging.	5 years ago
Philippe Antoine	1422b18a99	http2: initial support	5 years ago
Philippe Antoine	0507d1e8f8	detect: generic structures for mpm with lists	5 years ago
Victor Julien	ee41c0e293	flow/spare: implement pool shrinking ... Remove at most one block per run, so it shrinks slowly.	5 years ago
Victor Julien	b3599507f4	flow: redesign of flow timeout handling ... Goals: - reduce locking - take advantage of 'hot' caches - better locality Locking reduction New flow spare pool. The global pool is implmented as a list of blocks, where each block has a 100 spare flows. Worker threads fetch a block at a time, storing the block in the local thread storage. Flow Recycler now returns flows to the pool is blocks as well. Flow Recycler fetches all flows to be processed in one step instead of one at a time. Cache 'hot'ness Worker threads now check the timeout of flows they evaluate during lookup. The worker will have to read the flow into cache anyway, so the added overhead of checking the timeout value is minimal. When a flow is considered timed out, one of 2 things happens: - if the flow is 'owned' by the thread it is handled locally. Handling means checking if the flow needs 'timeout' work. - otherwise, the flow is added to a special 'evicted' list in the flow bucket where it will be picked up by the flow manager. Flow Manager timing By default the flow manager now tries to do passes of the flow hash in smaller steps, where the goal is to do full pass in 8 x the lowest timeout value it has to enforce. So if the lowest timeout value is 30s, a full pass will take 4 minutes. The goal here is to reduce locking overhead and not get in the way of the workers. In emergency mode each pass is full, and lower timeouts are used. Timing of the flow manager is also no longer relying on pthread condition variables, as these generally cause waking up much quicker than the desired timout. Instead a simple (u)sleep loop is used. Both changes reduce the number of hash passes a lot. Emergency behavior In emergency mode there a number of changes to the workers. In this scenario the flow memcap is fully used up and it is unavoidable that some flows won't be tracked. 1. flow spare pool fetches are reduced to once a second. This avoids locking overhead, while the chance of success was very low. 2. getting an active flow directly from the hash skips flows that had very recent activity to avoid the scenario where all flows get only into the NEW state before getting reused. Rather allow some to have a chance of completing. 3. TCP packets that are not SYN packets will not get a used flow, unless stream.midstream is enabled. The goal here is again to avoid evicting active flows unnecessarily. Better Localily Flow Manager injects flows into the worker threads now, instead of one or two packets. Advantage of this is that the worker threads can get packets from their local packet pools, avoiding constant overhead of packets returning to 'foreign' pools. Counters A lot of flow counters have been added and some have been renamed. Overall the worker threads increment 'flow.wrk.*' counters, while the flow manager increments 'flow.mgr.*'. Additionally, none of the counters are snapshots anymore, they all increment over time. The flow.memuse and flow.spare counters are exceptions. Misc FlowQueue has been split into a FlowQueuePrivate (unlocked) and FlowQueue. Flow no longer has 'prev' pointers and used a unified 'next' pointer for both hash and queue use.	5 years ago
Victor Julien	e0aa7c1dbc	unittests: check for flow memuse	5 years ago
Victor Julien	f50c7b6d11	flow-manager: call other timeouts max once a second ... Call Defrag and others only once per second. Flow Manager may wake up (much) more often when flow engine is under resource pressure. As this does not affect Defrag and others, it only unnecessarily adds load.	5 years ago
Victor Julien	6814f08e93	flow-manager: only update FlowBucket::next_ts if it changed	5 years ago
Victor Julien	1d6d7f0858	flow: unref flow at end of flow worker	5 years ago
Victor Julien	2fd7c87f22	flow: do timeout checks before tuple compare	5 years ago
Victor Julien	48605f4867	flow: don't reorder list on lookup ... Reduces cache misses.	5 years ago
Victor Julien	2a872ccb86	flow: timeout check on flow lookup	5 years ago
Victor Julien	7583a6c37c	flow: simplify hash lookup logic ... Remove double compare paths in favor of a single unified path.	5 years ago
Victor Julien	8b016cff4b	flow: only move lastts forward ... Pcaps with timestamps jumping around could confuse flow timeout handling otherwise.	5 years ago
Victor Julien	afd4a8012e	flow/worker: check pkt src using DEBUG_VALIDATE_BUG_ON	5 years ago
Jason Ish	fd5d8b78d0	alert/eve: add snmp metadata for rdp alerts ... Redmine issue: https://redmine.openinfosecfoundation.org/issues/3441	5 years ago
Jason Ish	ef0ebc9550	alert/eve: add snmp metadata for snmp alerts ... Redmine issue: https://redmine.openinfosecfoundation.org/issues/3441	5 years ago
Shivani Bhardwaj	4c7f55e636	dcerpc: handle gap for TCP streams	5 years ago
James Dutrisac	db5b73d9d6	pcap: read directories recursively ... Describe Changes - Added ability to recursively read pcap directories - src/suricata.c: addition of new command line parameter --pcap-file-recursive - src/source-pcap-file.c: parsing of the command line argument - src/source-pcap-file-directory-helper.h: two thread vars tracking directory depth and should recurse - src/util-error.c / src/util-error.h: Added new warning code "SC_WARN_PATH_READ_ERROR" - Redmine ticket: https://redmine.openinfosecfoundation.org/issues/2363 Ticket: #2363	5 years ago
James Dutrisac	5a92d0a704	path: introduce path handling util funcs ... This commit provides changes to util-path.c and util-path.h to support the recursive reading of directories. It adds 4 functions. - SCIsRegularFile to provide OS independent file info. - SCIsRegularDirectory to provide OS independent directory info. - SCRealPath is an OS independent wrapper for realpath. - PathJoin to manage path resolution logic.	5 years ago
Philippe Antoine	3cfc1fcb07	stream/app-layer: break loop on proto change	5 years ago
Shivani Bhardwaj	c9a637d854	datasets: fix null pointer deref	5 years ago
Shivani Bhardwaj	a8f147d17a	datasets: Init even in socket mode ... Closes redmine ticket 3476.	5 years ago
Sascha Steinbiss	93eef1da84	detect-mqtt: unify error handling in rule parsing ... This is meant to provide a single path to the error case. This might help make things more clear for static checkers.	5 years ago
Sascha Steinbiss	5dc21b0e09	detect-engine: initialize struct fields	5 years ago
Philippe Antoine	9a954e944d	http: merge duplicated code ... HtpRequestBodyHandlePUT and HtpRequestBodyHandlePOST	5 years ago
Philippe Antoine	fe6950de08	http: use more precise parameter in HtpRequestBodySetupMultipart ... So that we can see that it does not have to handle gaps	5 years ago
Philippe Antoine	e13b319beb	http: remove unused code ... HtpRequestBodySetupPUT function So that we can see that we do not use data=NULL in there	5 years ago
Philippe Antoine	28050c967f	file: handles gaps natively ... ie data=NULL and len>0 parameters	5 years ago
Philippe Antoine	f0159b2fd2	util: PrintRawDataFp handles null ... for gaps which are data=NULL and len>0	5 years ago
Philippe Antoine	6343920dfc	applayer: allow rust parsers to have only one probe	5 years ago
Sascha Steinbiss	4e1a41a17d	output-json: add MAC address output ... This commit adds MAC address output to the EVE-JSON format. We follow the remarks made in Redmine ticket #962: for packets, log MAC src/dst as a scalar field in EVE; for flows, log MAC src/dst as lists in EVE. Field names are different between flow and packet context to avoid type confusion (src_mac vs. src_macs). Configuration approach and JSON representation is taken from previous GitHub PR #2700.	5 years ago