96 Commits (2696fda04168cb82bedc8920fb8a3cc7d55289de)

Author	SHA1	Message	Date
Victor Julien	58ac4027ef	detect: clean up inspect engine registration	10 years ago
Victor Julien	9e35fa7f41	detect: remove empty app registration table	10 years ago
Victor Julien	5bde86b0e8	detect-engine: new registration call ... Make it more in line with MPM registration.	10 years ago
Victor Julien	5f994756e6	detect-engine: improved inspect engines ... Inspect engines are called per signature per sigmatch list. Most wrap around DetectEngineContentInspection, but it's more generic. Until now, the inspect engines were setup in a large per ipproto, per alproto, per direction table. For stateful inspection each engine needed a global flag. This approach had a number of issues: 1. inefficient: each inspection round walked the table and then checked if the inspect engine was even needed for the current rule. 2. clumsy registration with global flag registration. 3. global flag space was approaching the need for 64 bits 4. duplicate registration for alprotos supporting both TCP and TCP (DNS). This patch introduces a new approach. First, it does away with the per ipproto engines. This wasn't used. Second, it adds a per signature list of inspect engine containing only those engines that actually apply to the rule. Third, it gets rid of the global flags and replaces it with flags assigned per rule per engine.	10 years ago
Mats Klepsland	d91664d67a	detect-dns: move DetectEngineInspectGenericList to detect-engine.c ... Move DetectEngineInspectGenericList from detect-engine-dns.c to detect-engine.c to enable it to be used other places as well.	10 years ago
Victor Julien	36535cbc61	yaml: remove conf_filename global ... conf_filename was a global pointer to the filename of the yaml. Move into SCInstance. This reduces it's scope and cleans up the code.	11 years ago
Victor Julien	07d8617b3e	multi-detect: improve error handling	11 years ago
Eric Leblond	a4089873c7	rules-reload: fix reload with -s or -S ... When using the -S or -s option, the reload was causing the specified rules file to be forgotten and the default rules to be loaded at reload time.	11 years ago
Victor Julien	b7b27684c2	multi-detect: detect loader for unix socket ... Move the tenant load and reload commands to be executed by the detect loader thread(s). Limitation: no yaml parsing in parallel. The Conf API is currently not thread safe, so don't load the tenant config (yaml) in parallel.	11 years ago
Victor Julien	eb09118d64	detect: create loader threads ... To speed up startup with many tenants, tenant loading will be parallelized. As no tempary threads should be used for these memory allocation heavy tasks, this patch adds new type of 'command' thread that can be used to load and reload tenants. This patch hardcodes the number of loaders to 4. Future work will make it dynamic. The loader thread essentially sleeps constantly. When a tasks is sent to it, it will wake up and execute it.	11 years ago
Victor Julien	4e0683c3f5	multi-detect: cleanup, reuse tenant loading code ... Reuse tenant loading from YAML code for Unix Socket.	11 years ago
Victor Julien	216638c342	multi-detect: implement unregister-tenant-handler ... Remove a tenant handler from the list and apply it.	11 years ago
Victor Julien	6d92e8d220	unix-socket: implement register-tenant-handler ... Register tenant handlers/selectors based on what the unix command "register-tenant-handler" tells. Check traffic id before adding it. No duplicated registrations for a traffic id are allowed.	11 years ago
Victor Julien	1893c5edb1	multi-detect: initial selectors for tenants ... The Detection Thread has the TenantGetId pointer which allows it to select a tenant id based on the packet.	11 years ago
Victor Julien	0ff6d3dcfd	detect: select detect engine at Detect entry ... Limited to Pcap only currently.	11 years ago
Victor Julien	b653479815	detect: make multi tenancy a global switch ... At start up we will set this flag based on "multi-detect.enabled".	11 years ago
Victor Julien	def2b58725	detect: initial MT lookup logic ... In the DetectEngineThreadCtx, store another DetectEngineThreadCtx per tenant. Currently it's just a simple array indexed by the tenant id.	11 years ago
Victor Julien	147a6d2bfd	multi-detect: (un)register-tenant unix socket commands ... Make available to live mode and unix socket mode. register-tenant: Loads a new YAML, does basic validation. Loads a new detection engine Loads rules Add new de_ctx to master store and stores tenant id in the de_ctx so we can look it up by tenant id later. unregister-tenant: Gets the de_ctx, moves it to the freelist Removes config Introduce DetectEngineGetByTenantId, which gets a reference to the detect engine by tenant id.	11 years ago
Victor Julien	8949054212	detect: remove unused match_flags from inspect engines	12 years ago
Victor Julien	7c9e015748	unix-socket: implement reload-rules ... Implement the reload-rules unix socket command. The unix command thread signals the main thread to do the reload and it waits for it to complete.	12 years ago
Victor Julien	71d01f06b9	detect reload: load config ... Load the YAML into a prefix "detect-engine-reloads.N" where N is the reload counter. This way we can load the updated config w/o overwriting the current one.	12 years ago
Victor Julien	7108085d33	detect: initialize detection engine by prefix ... Initalize detection engine by configuration prefix. DetectEngineCtxInitWithPrefix(const char *prefix) Takes the detection engine configuration from: <prefix>.<config> If prefix is NULL the regular config will be used. Update sure that DetectLoadCompleteSigPath considers the prefix when retrieving the configuration.	12 years ago
Victor Julien	55e7370fc5	detect reload: allow master update during reload ... Add DetectEngineReference, which takes a reference to a detect engine, and make DetectEngineThreadCtxInitForLiveRuleSwap use it. This way reload will not depend on master staying the same. This allows master to be updated in between w/o affecting the reload that is in progress.	12 years ago
Victor Julien	b1c54a8673	detect: remove old live reload implementation ... Remove code that ran the reload in it's own thread. Simplify the signal handling.	12 years ago
Victor Julien	e7882da178	detect: introduce 'minimal' detect engine ... The minimal detect engine has only the minimal memory use and setup time. It's to be used for 'delayed' detect where the first detection engine is essentially empty. The threads setup are also minimal.	12 years ago
Victor Julien	38b349af1e	runmodes: remove DetectEngineCtx passing from API ... No longer pass a pointer to the current detection engine to the runmode API calls. Note: breaks delayed detect. Will be fixed in a future commit.	12 years ago
Victor Julien	d66fa1add1	detect: update detect engine management ... Update detect engine management to make it easier to reload the detect engine. Core of the new approach is a 'master' ctx, that keeps a list of one or more detect engines. The detect engines will not be passed to any thread directly, but instead will only be accessed through the detect engine thread contexts. As we can replace those atomically, replacing a detect engine becomes easier. Each thread keeps a reference to its detect context. When a detect engine is replaced or removed, it's added to a free list. Once its reference count reaches 0, it is freed.	12 years ago
Victor Julien	f5f148805c	app layer: uint16_t alproto -> AppProto alproto ... This conversion was missing in a couple of places.	13 years ago
Victor Julien	5cdeadb33d	Use u8 for ipproto ... In a few places in app layer and unittests u16 was used.	13 years ago
Anoop Saldanha	429c6388f6	App layer API rewritten. The main files in question are: ... app-layer.[ch], app-layer-detect-proto.[ch] and app-layer-parser.[ch]. Things addressed in this commit: - Brings out a proper separation between protocol detection phase and the parser phase. - The dns app layer now is registered such that we don't use "dnstcp" and "dnsudp" in the rules. A user who previously wrote a rule like this - "alert dnstcp....." or "alert dnsudp....." would now have to use, alert dns (ipproto:tcp;) or alert udp (app-layer-protocol:dns;) or alert ip (ipproto:udp; app-layer-protocol:dns;) The same rules extend to other another such protocol, dcerpc. - The app layer parser api now takes in the ipproto while registering callbacks. - The app inspection/detection engine also takes an ipproto. - All app layer parser functions now take direction as STREAM_TOSERVER or STREAM_TOCLIENT, as opposed to 0 or 1, which was taken by some of the functions. - FlowInitialize() and FlowRecycle() now resets proto to 0. This is needed by unittests, which would try to clean the flow, and that would call the api, AppLayerParserCleanupParserState(), which would try to clean the app state, but the app layer now needs an ipproto to figure out which api to internally call to clean the state, and if the ipproto is 0, it would return without trying to clean the state. - A lot of unittests are now updated where if they are using a flow and they need to use the app layer, we would set a flow ipproto. - The "app-layer" section in the yaml conf has also been updated as well.	13 years ago
Victor Julien	9419b9cbb6	detect: add tostring function for DETECT_SM_LIST_ enum.	13 years ago
Anoop Saldanha	d4d18e3136	Transaction engine redesigned. ... Improved accuracy, improved performance. Performance improvement noticeable with http heavy traffic and ruleset. A lot of other cosmetic changes carried out as well. Wrappers introduced for a lot of app layer functions. Failing dce unittests disabled. Will be reintroduced in the updated dce engine. Cross transaction matching taken care of. FPs emanating from these matches have now disappeared. Double inspection of transactions taken care of as well.	14 years ago
Anoop Saldanha	b99f9fe890	New app inspection engine introduced. Moved existing inspecting engines to use it.	14 years ago
Anoop Saldanha	2bc7d0792d	update clean up of old detection engine contexts for live rule swap	15 years ago
Anoop Saldanha	ecad4a24fa	live rule support added ... To reload ruleset during engine runtime, send the USR2 signal to the engine, and the ruleset would be reloaded from the same yaml file supplied at engine startup	15 years ago
Victor Julien	820b0ded82	Add per packet profiling. ... Per packet profiling uses tick based accounting. It has 2 outputs, a summary and a csv file that contains per packet stats. Stats per packet include: 1) total ticks spent 2) ticks spent per individual thread module 3) "threading overhead" which is simply calculated by subtracting (2) of (1). A number of changes were made to integrate the new code in a clean way: a number of generic enums are now placed in tm-threads-common.h so we can include them from any part of the engine. Code depends on --enable-profiling just like the rule profiling code. New yaml parameters: profiling: # packet profiling packets: # Profiling can be disabled here, but it will still have a # performance impact if compiled in. enabled: yes filename: packet_stats.log append: yes # per packet csv output csv: # Output can be disabled here, but it will still have a # performance impact if compiled in. enabled: no filename: packet_stats.csv Example output of summary stats: IP ver Proto cnt min max avg ------ ----- ------ ------ ---------- ------- IPv4 6 19436 11448 5404365 32993 IPv4 256 4 11511 49968 30575 Per Thread module stats: Thread Module IP ver Proto cnt min max avg ------------------------ ------ ----- ------ ------ ---------- ------- TMM_DECODEPCAPFILE IPv4 6 19434 1242 47889 1770 TMM_DETECT IPv4 6 19436 1107 137241 1504 TMM_ALERTFASTLOG IPv4 6 19436 90 1323 155 TMM_ALERTUNIFIED2ALERT IPv4 6 19436 108 1359 138 TMM_ALERTDEBUGLOG IPv4 6 19436 90 1134 154 TMM_LOGHTTPLOG IPv4 6 19436 414 5392089 7944 TMM_STREAMTCP IPv4 6 19434 828 1299159 19438 The proto 256 is a counter for handling of pseudo/tunnel packets. Example output of csv: pcap_cnt,ipver,ipproto,total,TMM_DECODENFQ,TMM_VERDICTNFQ,TMM_RECEIVENFQ,TMM_RECEIVEPCAP,TMM_RECEIVEPCAPFILE,TMM_DECODEPCAP,TMM_DECODEPCAPFILE,TMM_RECEIVEPFRING,TMM_DECODEPFRING,TMM_DETECT,TMM_ALERTFASTLOG,TMM_ALERTFASTLOG4,TMM_ALERTFASTLOG6,TMM_ALERTUNIFIEDLOG,TMM_ALERTUNIFIEDALERT,TMM_ALERTUNIFIED2ALERT,TMM_ALERTPRELUDE,TMM_ALERTDEBUGLOG,TMM_ALERTSYSLOG,TMM_LOGDROPLOG,TMM_ALERTSYSLOG4,TMM_ALERTSYSLOG6,TMM_RESPONDREJECT,TMM_LOGHTTPLOG,TMM_LOGHTTPLOG4,TMM_LOGHTTPLOG6,TMM_PCAPLOG,TMM_STREAMTCP,TMM_DECODEIPFW,TMM_VERDICTIPFW,TMM_RECEIVEIPFW,TMM_RECEIVEERFFILE,TMM_DECODEERFFILE,TMM_RECEIVEERFDAG,TMM_DECODEERFDAG,threading 1,4,6,172008,0,0,0,0,0,0,47889,0,0,48582,1323,0,0,0,0,1359,0,1134,0,0,0,0,0,8028,0,0,0,49356,0,0,0,0,0,0,0,14337 First line of the file contains labels. 2 example gnuplot scripts added to plot the data.	15 years ago
Anoop Saldanha	bc99328ec8	define a new conf paramter detect-engine:inspection-recursion-limit; Defines a recursion limit for content inspection code	16 years ago
William Metcalf	ce01927515	Import of GPLv2 Header 050410	17 years ago
Pablo Rincon	b6a3395c08	Adding unittest helper functions for building generic packets, checking arrays of expected match results, perform generic tests, etc. Look at util-unittest-helper.c and detect-ipproto.c for references	17 years ago
Victor Julien	2d0e9658f8	Speed up per sgh content maxlen calc. Remove mpm ptrs from mpm ctx. Add unittests testing the detection engine internals.	17 years ago
Victor Julien	e4c98c562c	Merge DetectAddressData and DetectAddressGroup	17 years ago
Victor Julien	cfb605aa8a	Put the precooked runmodes in a separate file.	17 years ago
Brian Rectanus	fa5939ca91	64 bit cleanup part2	17 years ago
Victor Julien	5df5b35e90	Put all globals in the detection engine ctx. Add HashList type, a hash that also stores the items ina list to it can be traversed. Many cleanups.	18 years ago
Victor Julien	54ffe2053e	Large detection engine update.	18 years ago
Victor Julien	dc48c58473	Switch to using a detection engine ctx.	18 years ago