Commit Graph

756 Commits (723e90a174fd6f4df6677eff348fa46c25e36f65)

Author SHA1 Message Date
Anoop Saldanha 4810ee9c5f All uricontent modified patterns now are DETECT_CONTENT and not DETECT_URICONTENT. Step towards unifying all content based patterns. Makes way for easier management of patterns 14 years ago
Anoop Saldanha e682796d03 feature #414 - support listing supported keywords. Remove support for dummy keywords __address__, __proto__, __port__. Remove support for recursive keyword and all references to it 14 years ago
Anoop Saldanha 35f1f7e8d9 unify payload detection engines + fix other bugs in pcre init 14 years ago
Anoop Saldanha 0677190960 rebase commit for hscd and hsmd patches 14 years ago
Anoop Saldanha 09313cf9bd Support http stat code detection engine, fast pattern(mpm engine included). Fix http stat code setup function. Fix pcre option for stat msg keyword. With this the pcre options for server_body is Q, for stat_msg is Y and for stat_code is S 14 years ago
Anoop Saldanha 2007c2711c Support http stat msg detection engine, fast pattern(mpm engine included). Fix http stat msg setup function. Fix pcre option for stat msg keyword 14 years ago
Victor Julien 9dc153c8f4 Fix path handling for including rule files on win32. 14 years ago
Anoop Saldanha 716afac5a2 fix debug messages that have references to the old mpm contexts 14 years ago
Anoop Saldanha 9a665e035b code cleanup over last 2 commits 14 years ago
Anoop Saldanha 419cdc8558 support splitting mpm ctxs based on direction v2 14 years ago
Victor Julien 28e15be526 Clean up default output. Use simpler output format for releases. 14 years ago
Victor Julien 6eeab37ab3 Add post-match list, move flowbits set, etc functions to it. Move flowint set, etc functions to it as well. 14 years ago
Victor Julien aac2d91bcc Set DROP flag for reject action so in addition to sending the rst, in IPS mode also drop the offending packet. 14 years ago
Victor Julien f084874998 Fix HTTP state and raw stream not being inspected at the same time. Adds an exception to transaction id handling for HTTP. 14 years ago
Anoop Saldanha 9c4954ff42 support http response raw header inspection + carry out hrhd mpm on both request/response headers + add unittests for the same 14 years ago
Anoop Saldanha 30247dce8c bug 389 - support http response header inspection + fix bug with stateful inspection for sigs that would have both request/response inpection 14 years ago
Victor Julien 21ee59e6f3 Add signature direction (flow:toserver/flow:toclient) as a signature flag. 14 years ago
Victor Julien d5402d33d4 Simplify detection loop. Inspect packet keywords before the state. 14 years ago
Anoop Saldanha eea5ab4a7a Support for app layer decoder events added + app_layer_event keyword added 14 years ago
Anoop Saldanha 517040c4af indentation fix 14 years ago
Anoop Saldanha 37b223645a fix detection engine for alert stability. Fix cases where we have multiple rules having same pattern. We should see good perf increase(~5%) with this change, now that we avoid unnecessary inspection" 14 years ago
Anoop Saldanha 92643f6110 introduce separate mpm ctxs for tcp/udp/other_protos 14 years ago
Victor Julien 18d79c4215 file store: respect flowbits and other keywords
The filestore keyword until now flagged a file, tx or ssn for storage as soon
as the keyword was inspected. This happens before flowbits and some other
keywords, so files were stored that weren't supposed to.

This patch makes the filestore keyword fill an array in the detect engine
thread ctx. Then if the full signature matches, a post-match filestore
function makes the store final.
14 years ago
Victor Julien 7adac3048d file-data: create initial keyword registration. 14 years ago
Victor Julien dd9da1a56f Merge all http mpm related signature flags into a single set: SIG_FLAG_MPM_HTTP and SIG_FLAG_MPM_HTTP_NEG. 14 years ago
Victor Julien 2650551192 Rename signature init flags to indicate they are init flags. 14 years ago
Victor Julien 291ddd95f2 Detection engine -- mpm
Each signature is in one mpm ctx at max, but there were 3 separate
id's in use: packet, stream, http. Merged them all into one.

Could shrink the SignatureHeader structure with 8 bytes because of this,
should lead to better caching performance.
14 years ago
Victor Julien 7db72bce75 Optimize detection engine prefiltering logic. 14 years ago
Victor Julien 89f83e714c Introduce http_server_body keyword.
The http_server_body content modifier modifies the previous content to inspect
the normalized (dechunked, unzipped) http_server_body. The workings are similar
to http_client_body. Additionally, a new pcre flag was introduced "/S".

To facilitate this change the signature flags field was changed to be 64 bit.
14 years ago
Anoop Saldanha b970273163 fix broken unittest 14 years ago
Anoop Saldanha d40fb5b933 Remove unnecessary flow NULL check 14 years ago
Anoop Saldanha cc7db6315c Move setting packet iponly flags from decode section to stream section 14 years ago
Victor Julien 117d51c965 Fix a compile warning when debug is enabled. 14 years ago
Victor Julien 042fd850fc Make sure we check the sgh for no magic and no store once per flow direction. 14 years ago
Victor Julien d59ca75e46 file extract: split toserver and toclient tracking
Split toserver and toclient file tracking for the http state.
14 years ago
Victor Julien 04ea70ccf7 file extract: pruning
Add pruning of files in memory so we keep only memory what we really need.
Fix magic logic.
Reset file part of the de_state on receiving another file in the same tx.
14 years ago
Victor Julien e1022ee5ae file-extraction: Disconnect file handling from flow and move into the app layer state. 14 years ago
Victor Julien 9b62ec65ab Make sure filemagic works properly regardless of filestore being in use for a flow. 14 years ago
Victor Julien 5945e652d6 Initial implementation of filemagic keyword. 14 years ago
Victor Julien 23e01d23d3 Implement filestore keyword, including a way for the stateful detection engine to conclude that a file will never have to be stored. 14 years ago
Pablo Rincon 6d60b3a747 filename and fileext keywords 14 years ago
Anoop Saldanha 9a6aef459e modify all relevant app layer API calls to accomodate passing parser local storage argument 14 years ago
Anoop Saldanha 2b356dadff Support for tos keyword added 14 years ago
Victor Julien 262a7300d7 flow: shrink Flow datatype
Introduce a separate FlowAddress structure for holding the ipv4 or ipv6 address
that doesn't have the family in it like the Address structure. Instead, the
family is stored in the flow as a flag: FLOW_IPV4 and FLOW_IPV6.

Add macro's to check the family, copy the address, etc.

Update many unittests to reflect these changes. Introduce unittest helper
functions for creating and initializing a flow and freeing it again.

On 64 bit this shrinks the flow with 8 bytes.
14 years ago
Victor Julien 06904c9024 App Layer cleanup
Removal of per flow 'aldata' array. It contained a ptr for each ALPROTO. Instead now we have 2 ptrs in the flow: alparser and alstate.
Various cleanups and dead code removal from the app layer API.
Should safe 100+ bytes memory per flow on 64 bit.
Updated lots of unittests to reflect these changes.
14 years ago
Victor Julien 3d845b6c77 Consider Windows new line chars as well when parsing rule files. Bug #374. 14 years ago
Victor Julien 8cc82c7241 Add -S commandline option that loads a rule file exclusively. Issue #338. 14 years ago
Victor Julien 2d16abcf8b Minor code cleanups fixing all GCC 4.6 compiler warnings for default, debug and unittests mode. 14 years ago
Anoop Saldanha 0acfcc206c fix unittests. fix replace unittests that allow alproto keywords with replace 14 years ago
Anoop Saldanha a0eec3d846 fix detection code that handles cases when we use recursion(from recursive keyword) 14 years ago
Anoop Saldanha 7433d92dd2 undo this commit -
commit eff08f93d8
Author: Anoop Saldanha <poonaatsoc@gmail.com>
Date:   Thu Nov 3 14:31:24 2011 +0530

    update failing unittest to reflect the mpm design update

Fixed a bug in the mpm code that would make all the changes in the commit just undone wrong.
14 years ago
Anoop Saldanha 1b1332fff0 fix mpm bug on running stream mpm for packets not added to stream mpm 14 years ago
Anoop Saldanha eff08f93d8 update failing unittest to reflect the mpm design update 14 years ago
Victor Julien af51493da2 Mpm update: Toss out signatures that mix pkt and stream/state. Update profiling code to track new mpm. 14 years ago
Anoop Saldanha 17f3f36d38 packet keywords only added for packet mpm. Rest in stream mpm. Update detection engine to handle the same 14 years ago
Anoop Saldanha c7b9d3fecb Remove broken dsize_sm in SigMatch used by dsize in detection engine 14 years ago
Eric Leblond 60a99915c1 doc: create http support group
This patch create an httplayer group and adds related files to
it. It also fixes some typo in documentation string and format.
14 years ago
Eric Leblond 6bf15bac31 Fix various packet access.
The coccinelle based tests have detected invalid uses of access to
Packet data. This patch fixes the detected problems.
14 years ago
Victor Julien 7ef34b7bcc Exlcude DSIZE LT case from setting the 'need payload' mask bit as it can include 0, which means no payload. 14 years ago
Victor Julien 2e2e80b812 Add packet alert flag to indicate a match happened (partly) in the app layer state. Make unified2 use this flag. 14 years ago
Victor Julien a1f68bf411 Fix detection engine informational message misrepresenting decoder only signatures. 14 years ago
Eric Leblond 3f75b10f79 Suppress useless code.
This code was making a warning for some time now. This patch kills
it.
14 years ago
Eric Leblond a85dc9b0e2 Add support for replace keyword.
This patch adds support for the replace keyword. It is used with
content to change selected part of the payload. The major point
with this patch is that having a replace keyword made necessary
to avoid all stream level check because we need to access to the
could-be-modified packet payload.

One of the main difficulty is to handle complex signature. If there is
other content check, we must do the substitution when we're sure all
match are valid. The patch adds an attribute to the thread context
variable to be able to deal with recursivity of the match function.

Replace is only activated in IPS mode and apply only to raw match.
14 years ago
Anoop Saldanha 8028392e9a fix mpm segv. Use sgh flags to check if the sgh has packet or stream mpm set or not 14 years ago
Victor Julien e13181496c ip-only: added support for matching on ports. 14 years ago
Victor Julien 751a77a9be Make sure stream/engine-event signatures are recognized as such. 14 years ago
Victor Julien 63f834d9a7 Add profiling to various HTTP buffer MPM calls. 14 years ago
Victor Julien 3693a7a9ee Profiling: add accounting for several detection phases. 14 years ago
Eric Leblond e3a6d8955e Introduce engine-event keyword
This patch renames the 'decode-event' keyword to 'engine-event' and
keep it for backword compatibility of rulesets. All *DecodeEvent*
references in the code are replaced by EngineEvent version.
14 years ago
Eric Leblond 2ac8755382 Rename detect-decode-event to detect-engine-event
This patch does a simple renaming of detect-decode-event file to
the more global detect-engine-event name.
14 years ago
Victor Julien 820b0ded82 Add per packet profiling.
Per packet profiling uses tick based accounting. It has 2 outputs, a summary
and a csv file that contains per packet stats.

Stats per packet include:
 1) total ticks spent
 2) ticks spent per individual thread module
 3) "threading overhead" which is simply calculated by subtracting (2) of (1).

A number of changes were made to integrate the new code in a clean way:
a number of generic enums are now placed in tm-threads-common.h so we can
include them from any part of the engine.

Code depends on --enable-profiling just like the rule profiling code.

New yaml parameters:

profiling:
  # packet profiling
  packets:

    # Profiling can be disabled here, but it will still have a
    # performance impact if compiled in.
    enabled: yes
    filename: packet_stats.log
    append: yes

    # per packet csv output
    csv:

      # Output can be disabled here, but it will still have a
      # performance impact if compiled in.
      enabled: no
      filename: packet_stats.csv

Example output of summary stats:

IP ver   Proto   cnt        min      max          avg
------   -----   ------     ------   ----------   -------
 IPv4       6     19436      11448      5404365     32993
 IPv4     256         4      11511        49968     30575

Per Thread module stats:

Thread Module              IP ver   Proto   cnt        min      max          avg
------------------------   ------   -----   ------     ------   ----------   -------
TMM_DECODEPCAPFILE          IPv4       6     19434       1242        47889      1770
TMM_DETECT                  IPv4       6     19436       1107       137241      1504
TMM_ALERTFASTLOG            IPv4       6     19436         90         1323       155
TMM_ALERTUNIFIED2ALERT      IPv4       6     19436        108         1359       138
TMM_ALERTDEBUGLOG           IPv4       6     19436         90         1134       154
TMM_LOGHTTPLOG              IPv4       6     19436        414      5392089      7944
TMM_STREAMTCP               IPv4       6     19434        828      1299159     19438

The proto 256 is a counter for handling of pseudo/tunnel packets.

Example output of csv:

pcap_cnt,ipver,ipproto,total,TMM_DECODENFQ,TMM_VERDICTNFQ,TMM_RECEIVENFQ,TMM_RECEIVEPCAP,TMM_RECEIVEPCAPFILE,TMM_DECODEPCAP,TMM_DECODEPCAPFILE,TMM_RECEIVEPFRING,TMM_DECODEPFRING,TMM_DETECT,TMM_ALERTFASTLOG,TMM_ALERTFASTLOG4,TMM_ALERTFASTLOG6,TMM_ALERTUNIFIEDLOG,TMM_ALERTUNIFIEDALERT,TMM_ALERTUNIFIED2ALERT,TMM_ALERTPRELUDE,TMM_ALERTDEBUGLOG,TMM_ALERTSYSLOG,TMM_LOGDROPLOG,TMM_ALERTSYSLOG4,TMM_ALERTSYSLOG6,TMM_RESPONDREJECT,TMM_LOGHTTPLOG,TMM_LOGHTTPLOG4,TMM_LOGHTTPLOG6,TMM_PCAPLOG,TMM_STREAMTCP,TMM_DECODEIPFW,TMM_VERDICTIPFW,TMM_RECEIVEIPFW,TMM_RECEIVEERFFILE,TMM_DECODEERFFILE,TMM_RECEIVEERFDAG,TMM_DECODEERFDAG,threading
1,4,6,172008,0,0,0,0,0,0,47889,0,0,48582,1323,0,0,0,0,1359,0,1134,0,0,0,0,0,8028,0,0,0,49356,0,0,0,0,0,0,0,14337

First line of the file contains labels.

2 example gnuplot scripts added to plot the data.
14 years ago
Victor Julien 1bd1a62526 Rename profile macro's and variables to reflect that they are for rule profiling. 14 years ago
Anoop Saldanha 7e5c52c80b add flowbits:set; only sigs to be treated as ip only 14 years ago
Victor Julien 16b41a5eff Use p->proto in detect to determine TCP/UDP/SCTP. 14 years ago
Victor Julien f7f037c1d1 Make sure stateful detection engine inspecting HTTP streams works well for to_client rules as well. 14 years ago
Anoop Saldanha 35f3eafa5e byte extract added to the engine. Detection support added for packet payload, uri and dce detection engines 14 years ago
Victor Julien 73efb4c70f Add a app layer state and stateful detection engine counter that makes sure the stateful inspection is only done when the state changes. 14 years ago
Victor Julien 0768ca9806 Fix SIMD mask checking on 64 bit systems. 14 years ago
Victor Julien 350215966b Fix signature mask bitorder. 14 years ago
Victor Julien aa822c0ac1 Always reset alert cnt and always increment det_ctx->pkts. 14 years ago
Victor Julien 1e0b050a54 Add more mask flags. 14 years ago
Victor Julien 4b52823ab6 Use 64 bit mask on 64-bit systems. 14 years ago
Victor Julien e5b6c0f518 Check 32 masks per run instead of 16 in the SIMD code. 14 years ago
Victor Julien 2dbfdd40af Clean up new SIMD mask checking code, improve non-SIMD checks. 14 years ago
Victor Julien b421019cef Match packet mask against 16 signature masks at once using SIMD instructions for SSE3 and up. 14 years ago
Victor Julien e1d4e16645 Simplify packet decoding macro's. 14 years ago
Victor Julien 75439863ed Shrink PacketAlerts structure so that Packet structure is a lot smaller. Reduce max events per packet from 256 to 15. 14 years ago
Victor Julien 6aa551c558 Small optimizations to IPV4 and TCP header parsing. 15 years ago
Anoop Saldanha 966119b6aa support for http_raw_uri keyword + mpm engine 15 years ago
Victor Julien 864c8718e1 Store matching stream msg (ptr) in packets alert structure so it's available to the output plugins. 15 years ago
Victor Julien 140eb4fde8 Fix decode-event keyword parsing. Fix code that indicates a signature is decode-event only. Add 'pkthdr' protocol as an alias for any/ip to be used by decode-event signatures. 15 years ago
Victor Julien e197f50727 Fix IP-Only unittests failing on Big Endian. 15 years ago
Eric Leblond 9beebf621a Add support for 'nfq_set_mark' keyword
This patch introduces 'nfq_set_mark' which is new rules option. If a packet
matches a rule using nfq_set_mark in NFQ mode, it is marked with the mark/mask
specified in the option during the verdict.
It is thus possible to trigger different behaviour on the packet inside
Linux/Netfilter.
15 years ago
Anoop Saldanha c105a739e9 support for ssl_state keyword added 15 years ago
Victor Julien b9fd978253 Fix compiler warnings in two unittests. 15 years ago
Eric Leblond 005dc599a6 detect.c: Fix usage of integer standing for protocol
This patch fixes direct usage of integer to code protocol value.
15 years ago
Eric Leblond 2c80f18dc9 detect: Add sctp detection and parsing.
This patch adds the support of SCTP in signature subsystem.
15 years ago
Gurvinder Singh 8f8b1212af support for ssl_version keyword 15 years ago
Victor Julien ec5b622553 Inspect all stream msgs at any time when running in stream-inline mode. Skip detection for packets flagged for dropping before detect. 15 years ago
Victor Julien 1d971b53a6 Update all unittests 15 years ago
Victor Julien f606621e8c Fix the pseudo packet having the wrong proto set, causing massive fp's. Flag packets to be part of the established phase of a tcp session, so we won't prematurely inspect the app layer state. 15 years ago
Victor Julien 24f071cabb Make sure http_cookie inspects all HTTP transactions. Clean up error messages. Get rid of unused code and dead comments. 15 years ago
Anoop Saldanha c9897a44a4 fast pattern support for http_cookie. Also support relative modifiers 15 years ago
Anoop Saldanha bbbedaf963 fast pattern support for http_method. Also support relative modifiers 15 years ago
Eric Leblond 49adc264bc Don't print message after SCMalloc failure.
This patch generated via coccinelle is getting rid of logging
message after a SCMalloc failure. They were useless as SCMalloc
already displays a message.
15 years ago
Eric Leblond 1db4aadd16 Supress usage of Packet declaration in tests.
For convenience, a massive usage of 'Packet p;' declaration has
been done in the tests function. Although this was completely
legal, this is not possible anymore because of the new Packet
allocation structure. This massive patch modifies all suricata
files to use a SCMalloc allocated pointer to Packet instead.

This patch has been done using coccinelle (http://coccinelle.lip6.fr)
which is a semantic patching tool. This ensures that things like call
to SCFree() should have not been forget because the semantic patch
explicitly forces the call to SCFree(p) before each return. With this
patch all unittests are running fine with a small and a big default
packet size.
15 years ago
Eric Leblond dd038c1906 Modify files to avoid direct pckt payload access
This patch implements the needed modification of payload access
in a Packet structure to support the abstraction introduced by
the extended data system.
15 years ago
Victor Julien 18b4e3380f Make mpm-algo use the mpm_table that has the actual mpm's registered. Clean up dead code. 15 years ago
Victor Julien 3f47eade6b Fix couple of cases where incorrect handling of keyword parsing errors would lead to access of uninitialized memory. Found by clang. 15 years ago
Victor Julien 3d60e9bfeb Clean up output. 15 years ago
Anoop Saldanha d241e51b32 adapt fast pattern engine analysis to reflect the new changes made to your mpm design 15 years ago
Anoop Saldanha c734cd1bdd make cuda mpm parameters configurable 15 years ago
Victor Julien 0e8e8e3728 Don't stop stream reassembly if protocol detection failed, only stop/prevent app layer parsing. 15 years ago
Victor Julien 39dea56a84 Remove flowbits as a mask prefilter as they are dynamic. Add a dynamic check. 15 years ago
Anoop Saldanha 8c21511c99 don't buffer raw headers. Retrieve them individually from htp_state during mpm stage and content valiadation stage 15 years ago
Victor Julien 55ca988222 Change locking of http_header, http_raw_header and http_client_body so that flow isn't accessed without lock anywhere. 15 years ago
Victor Julien 435d0fb327 Clean up signature flags creating room for merging flags and mpm_flags. Merge flags and mpm_flags. Move new mpm id's into signature header. Get rid of full signature access in signature prefiltering. 15 years ago
Victor Julien 169aa5581a Rename SIG_FLAG_AMATCH flag to SIG_FLAG_STATE_MATCH to better reflects its purpose. 15 years ago
Victor Julien d7b92d9bfe Consolidate several signature flags into one. 15 years ago
Victor Julien 2102a54c26 Cleanup and rearrange detection code slightly. 15 years ago
Anoop Saldanha 25588b6910 comment out hrhd flags that we were using previously. Also remove the de_mpm_ based flags inside detect.h used by uri|hcbd|hhd|hrhd mpms. indentation fix as well 15 years ago
Anoop Saldanha e50503e820 cleanup/remove dead code 15 years ago
Anoop Saldanha 93fa7ea828 modify detection engine to run hrhd mpm before building the match array 15 years ago
Anoop Saldanha b140ed1c9c modify detection engine to run hhd mpm before building the match array 15 years ago
Anoop Saldanha 4e273f2c8b modify detection engine to carry out hcbd mpm run before build match array if alproto is http and if sgh has atleast one sig with hcbd mpm set 15 years ago
Anoop Saldanha 72b0fcf419 modify detection engine to carry out uri mpm run before build match array if alproto is http and if sgh has atleast one sig with uri mpm set 15 years ago
Victor Julien 6a5d2cb40d Fix potential locking issue in out of memory conditions in the http_header, http_raw_header code. Fix other potential small issues in http_ code. 15 years ago
Anoop Saldanha 7ec0382774 support fast pattern for http raw header. Also support relative modifiers for http raw header 15 years ago
Victor Julien 0c806f70bb Fix --enable-debug compilation, just unittest with --enable-debug-validation enabled. 15 years ago
Victor Julien 1a32d9b5ec Fix printing unprintable characters in the engine-analysis fast_pattern mode. 15 years ago
Anoop Saldanha c61c68fd36 mpm and fast pattern support for http_header. Also support relative modifiers for http_header 15 years ago
Anoop Saldanha 778ec0939c make client body buffer limit configurable. Also some minor changes 15 years ago
Anoop Saldanha 0aa5cffb12 fast pattern support for http_client_body keyword added. Also mpm support for http_client_body added 15 years ago
Anoop Saldanha c227aeeacb remove support for skipping reinspecting fast pattern contents once again during packet payload inspection. Also make some changes to our detection engine 15 years ago
Anoop Saldanha bbd0c5056b store the content added for mpm inside Signature. also carry out an unconditional cleanup of packet pattern matcher pmq det_ctx->pmq 15 years ago
Anoop Saldanha 6df051321f fix fp when content is negated and also added to mpm 15 years ago
Anoop Saldanha 5c6a65dc58 support relative modifiers for http_client_body. Introduce body processing engine in detect-engine-hcbd.[ch] 15 years ago
Anoop Saldanha 3d2f81d978 replace all Signature->dmatch instances in the engine with Signature->sm_lists[DETECT_SM_LIST_DMATCH] 15 years ago
Anoop Saldanha a7353be20d replace all Signature->amatch instances in the engine with Signature->sm_lists[DETECT_SM_LIST_AMATCH] 15 years ago
Anoop Saldanha e0476242c6 replace all Signature->umatch instances in the engine with Signature->sm_lists[DETECT_SM_LIST_UMATCH] 15 years ago
Anoop Saldanha e54358a9e1 replace all Signature->pmatch instances in the engine with Signature->sm_lists[DETECT_SM_LIST_PMATCH] 15 years ago
Anoop Saldanha 82fd581b64 replace all sm lists (match, pmatch, dmatch, umatch, amatch, tmatch) with an array Signature->sm_lists[]. Replace all Signature->match instances in the engine with Signature->sm_lists[DETECT_SM_LIST_MATCH] 15 years ago
Victor Julien 001f91056e Add http_raw_header as an alias to the http_header keyword as that actually inspects the raw headers (see issue #243). Closes issue #242. 15 years ago
Gurvinder Singh b7da115e6d support for http_stat_code keyword has been added to detection module 15 years ago
Gurvinder Singh 1deae70cf7 added http_stat_msg keyword support for detection module 15 years ago
Anoop Saldanha 2cdb5be391 Print out file name for fast_pattern engine_analysis. Also add some info logs 15 years ago
Anoop Saldanha a2d04a94b5 selecting auto for detect-engine.sgh_mpm_context now uses single if the mpm is ac, full otherwise 15 years ago
Anoop Saldanha 174048544d fix hash generation in b2g and ac addpattern. Brings down the no of patterns added from close to a million to a couple of thousands 15 years ago
Anoop Saldanha 0ef684705c support single mpm context distribution across sghs in staging. Also see to it that ac works fine with this setup 15 years ago
Anoop Saldanha a85fa6b792 support for fast_pattern only and fast_pattern:offset,length. Also support the new option for engine-analysis 15 years ago
Anoop Saldanha 0d741b9a55 fix for bug 227. For negated contents that have been added to mpm we might have pmq.pattern_id_array_cnt as 0. We can't ignore inspecting sigs if this is 0, in case the content added is negated 15 years ago
Victor Julien cbd4c298ed Initial version of a new bitmask based signature pre-filtering method. 15 years ago
Victor Julien 94898a91cc Reorganize SigMatchSignatures. 15 years ago
Victor Julien fc248ca7a1 Many small performance updates. 15 years ago
Pablo Rincon 5c43db85ce Drop streams on inline mode when a drop rule match from a reassembled stream and/or app layer inspection 15 years ago
Anoop Saldanha f094523eb1 clang fix - some minor fixes for unittests 15 years ago
Pablo Rincon 9d7baa7a9f Adding ssh app layer module with two new keywords: ssh.protoversion and ssh.softwareversion 15 years ago
Victor Julien 04d3832d8f Remove ports check and fix small typo. 15 years ago
Victor Julien a492518e7a Properly detect detect-event-only sigs. 15 years ago
Pablo Rincon 21d79b05ad Fix for bug221 (avoid considering sig as "decoder event only" if ports are specified). Now the sig gets grouped to get a sgh at SigMatchSignatures 15 years ago
Victor Julien 6299fbfb0f Fix stream msg content inspection not inspecting the correct id. 15 years ago
Victor Julien 1071a53210 Fix unittests after ip_proto keyword change. 15 years ago
Pablo Rincon 70bda6506d Fix for bug 180 (check proto specified at the IP hdr) 15 years ago
Victor Julien 7acb97da9d Use same mpm prepare procedure for uricontent as for normal content. More cleanups. 15 years ago
Victor Julien 9ba11dbfbd Clean up detection engine mpm initialization phase. 15 years ago
Victor Julien 0d008c8135 Change stateful detection engine to be able to start the stateful detection separate from other sigs. Fixes bugs #213, #214, #215. 15 years ago
Victor Julien 689d05b10b Add missing protocol check in the sig matching process. This prevents FP's such as the one reported in bug #209. 15 years ago
Victor Julien 0219b767b8 Fix a content pattern matching bug related to signature grouping and mpm_ctx sharing. In certain conditions (signature combinations) the mpm_stream_ctx (the ctx that handles stream pattern scanning) wasn't properly setup. 15 years ago
Victor Julien 102092a89c Make signature address matching more cache efficient. 15 years ago
Victor Julien 1eec149f5e Use Address structure in DetectAddress struct. 15 years ago
Victor Julien c6ddcda7f8 Improve out of memory handling during initialization. 15 years ago
Victor Julien bfd167521e Fix DCERPC over SMB/SMB2 detection issues. Fix not updating transaction id in a stream direction if there was no sgh. 15 years ago
Anoop Saldanha 33f4beb0bc batching of packets support for cuda b2g mpm. Supported for both 32 and 64 bit platforms 15 years ago
Victor Julien b3c22cd512 Improve app layer proto check. 15 years ago
Victor Julien 39cb1bdbda Fix app layer sigs being recognized as decoder event only or ip only. 15 years ago
Victor Julien d41b5645ef Make sure decoder event rules are inspected even if the packet is invalid and has no addesses or proto. Update fast log and alert debug log to display the alerts. Fixes #179. 15 years ago
Victor Julien e685579231 Add optional structure validation code. 15 years ago
Victor Julien 393acd77d2 Detection improvements: uricontent escaping now working, better negated pattern (content) handling. 15 years ago
Anoop Saldanha 9ecade76b9 in case of duplicate signatures used the one with the latest revision 15 years ago
Gurvinder Singh 8852b83fa7 flowbits, flowvars, pktvars, flow flags and app layer info added to alert-debug.log 15 years ago
Victor Julien 580b09c2b8 Make sure we inspect all outstanding reassembled stream chunks (smsg) if the stream is shutting down. Make sure to do inspect signatures that use dsize against the tcp packet payload, even if that payload was already added to the stream. Likewise, the dsize signatures are not inspected against the reassembled stream. 15 years ago
Victor Julien a3ff0e7210 Don't scan TCP packet payload if it was added to the stream. Inspect the tcp stream with the correct packet. Should fix #184 and #185. 15 years ago
Pablo Rincon a8cb8d830b Fix for bug 186 and thresholding issue handling ip versions 15 years ago
Pablo Rincon eed0ef6e69 Adding tag keyword support 15 years ago
Victor Julien ca7f54de25 Make sure ICMP unreach packets are not inspected against the flow sgh as it's for the original protocol, not for the ICMP packet. Fixes #174. 15 years ago
Victor Julien b8fec77f37 Fix tcp connections that are reset (RST packet) not always inspecting the reassembled stream. Update transaction id code to make sure both directions of a transaction are inspected before incrementing the inspect_id. 15 years ago
Victor Julien cdc9570f0e Have the detect.alerts counter count actual alerts. 15 years ago
William Metcalf 0e4235cc94 FLOW_DESTROY added to clean-up UT's that init flow 15 years ago
Victor Julien 2f29b8a724 Improve detection of app layer, making sure we only handle app layer on 'established' packets. Should really fix #166. 15 years ago
Victor Julien 37442a8a84 Prefilter signatures before fully scanning them. 15 years ago
Victor Julien d6709b0961 Fix a segv caused by invalidly accessing the smsg_pmq array. 15 years ago
Victor Julien 8cea3779fa Move dce payload inspection to stateful detection engine. 15 years ago
Anoop Saldanha 45ea0d914e dce stub content keywords support using dcepayload.c support for all dce related content keywords 15 years ago
Victor Julien 83b2c8abdb Improve stateful uri detection code. 15 years ago
Victor Julien 9dd753b5f3 Scan uricontent mpm on demand. 15 years ago
Victor Julien e8fce5f7fa Convert uricontent scanning to use the detect engine state. 15 years ago
Pablo Rincon 8cc525c939 UDP support at AppLayer message handling 15 years ago
William Metcalf cc76aa4bc6 properly init flows inside of unit-tests caused lock-up when falling back to using mutex locks 15 years ago
Victor Julien a24f288074 Moving the stream content scanning to have it's own mpm ctx. 15 years ago
Victor Julien 9a08d6c11c Fixes to stream pattern matching. 15 years ago
Victor Julien a0c1209a44 Inspect the reassembled stream together with the packet payload in the same direction. 15 years ago
Victor Julien 81f2499834 Store stream msgs processed by the app layer in the tcp session so they can be inspected by the detection module as well. The detection module returns them to the pool. 15 years ago
Victor Julien c26434fef1 Move flow use cnt to atomic and outside of the flow mutex protection. 15 years ago
Victor Julien 2fd31a1a11 Remove dsize grouping from detection engine grouping reducing memory usage. Store sgh in flow to reduce lookups. Reduce locking in alert handling. Increase default grouping values as we use less memory. 15 years ago
Victor Julien dff6795df5 Detect cleanups. 15 years ago
Gerardo Iglesias Galvan 55dfa36963 Add support for http_uri keyword 15 years ago
Jason Ish ea4b7cc33b add profiling to stateful detection engine + other fixups. 15 years ago
Victor Julien 4e7df60b2f Make pcap file mode read multiple packets per 'read'. Update threading model to deal with this. 15 years ago
Pablo Rincon 3fa3229e01 ASN1 decoder and keyword implementation 15 years ago
Victor Julien 70b32f7380 First stab at creating a stateful detection engine.
Stateful detection for app layer detection keywords, except uricontent. Stores it's partial results in the flow structure. Other modifications:

- Generalize transaction tracking, logging and inspection.
- Adapt http and dcerpc to use the new transaction handling.
- Stream engine now always notifies app layer of a stream eof.

This commit fixes bug #124.
15 years ago
Jason Ish 18e5ac8cde Basic rule profiling even though the results may be skewed by a bad rule in a grouping of rules. 15 years ago
Victor Julien 42eeb84c9a Properly lock flow before setting IP only action flags. Small alert api cleanups. 15 years ago
Pablo Rincon 9bae6a8628 Moving alert logic to detect-engine-alert.c 15 years ago
Gerardo Iglesias Galvan 9f4fae5b1a Fix inconsistent use of dynamic memory allocation 15 years ago
William Metcalf 8d66323f62 clang fixes for null derefrences 15 years ago
Victor Julien e27cefa6f7 Complete conversion of pattern id mpm storage vs sig id storage. 15 years ago
Victor Julien 46831e0f8f Fix signature grouping bug for protocols without ports. Add debugging code. 15 years ago
Victor Julien 7a427ec7f4 Switch to pattern id based results checking in the mpm. Move app layer proto detection towards a more signature based approach. 15 years ago
Pablo Rincon 46187bfe73 Fix action logic after last pass changes 15 years ago
Gurvinder Singh 3721037de5 unittests for bug 134&139 and some typo correction 16 years ago
Victor Julien a372c1d14e Fix/workaround a strange detection issue. 16 years ago
Victor Julien ce90e87304 Fix failing thresholding unittests 16 years ago
Pablo Rincon e18e2ec998 Changing threshold logic 16 years ago
Pablo Rincon 8bcdf29ab7 Small fix on pass action handling and added more unittests 16 years ago
Pablo Rincon 1238668961 Adding actions order and suport for rule action "pass" 16 years ago
William Metcalf ce01927515 Import of GPLv2 Header 050410 16 years ago
Victor Julien 070ed778b8 Libcap-ng support by Gurvinder Singh and myself. Basic support for per thread caps is added, but not activated as it doesn't seem to work yet. Work around for incompatibility between libnet 1.1 and libcap-ng added. 16 years ago
Pablo Rincon ab02ab9ead adding http_header keyword support 16 years ago
Pablo Rincon 224a33f19e Moving inline functions to the .h files, so gcc can inline them correctly 16 years ago
Victor Julien eeb98c6900 Move SCSetThreadName to proper functions. 16 years ago
Gerardo Iglesias Galvan 9f35a24a1f Set threads name. Fix bug #83 16 years ago
Gurvinder Singh 69a4fee757 fixed the API and logic error reported by clang tool 16 years ago
Victor Julien 78e15ea7fa Explicitly test for ipv6 in the htp personalities code. Update all affected unittests to set addr family to the flow. 16 years ago
Anoop Saldanha 47037ef9ec fix for bug 115 16 years ago
Victor Julien fe7ece997a Different approach to the reference keyword. Lots of cleanups, bug fixes in reference keyword code and tests. 16 years ago
Breno Silva 89baf93a40 Reference Support 16 years ago
Pablo Rincon e7a989e305 IP Only Engine using radix trees 16 years ago
Anoop Saldanha 97d49d8f5e support for http_client_body keyword 16 years ago
Victor Julien 26e8a0a06a Cleanup global threshold code. 16 years ago
Breno Silva 67f2026279 Global Threshold config 16 years ago
Victor Julien 08600df6b1 Small uri cleanups. 16 years ago
Pablo Rincon c7350a8ac6 Fixing some naming convention issues and incorrect error messages 16 years ago
Pablo Rincon b708d7f65d Adding Uricontent inspection with spm. Modifiers for uricontent are now supported 16 years ago
Anoop Saldanha c54b91ed94 fix for bug 113 16 years ago
Victor Julien 6be0778532 Comment SigMatchSignatures a bit. 16 years ago
Gerardo Iglesias Galvan ef2ae76c42 Add support for detection_filter keyword 16 years ago
Victor Julien 297001c6d9 Only process a app layer sig if it has the proper state. Make sure a sig can't have conflicting sigmatches, such as ftpbouce and uricontent. 16 years ago
Victor Julien ec47f840f3 Remove more scan references. 16 years ago
Victor Julien bee4e04664 More scan/search related cleanups. 16 years ago
Victor Julien 6990d9c91b Fix thresholding signature unittests. Because of the bug fix that made thresholding compatible to ip-only sigs the test sigs needed to be made non-ip-only. 16 years ago
Victor Julien b259e362cd Convert uricontent to use new scanning methods as well. Move http_method and http_cookie keywords out of pmatch list for now. 16 years ago
Victor Julien bef70a04ce First stage of detect engine redesign: equal patterns share id's, search phase no longer used, new match verification phase. 16 years ago
Pablo Rincon 25a3a5c6d8 Adding mem wrapper to debug runtime alloc()/free() functions. Fixing some memory leaks. 16 years ago
Pablo Rincon ebcbc859e3 Fix on IPOnly match at flows, for inline mode 16 years ago
Gurvinder Singh 50f7d0a887 app layer htp logging and better htp request handling. removed recent_in_tx. 16 years ago
Gurvinder Singh 4768e42159 bug 95 patch 16 years ago
Pablo Rincon 38dc7ffebc Adding settings for detect engine group config 16 years ago
Anoop Saldanha 1a5ee37bd3 Added cuda logs for the engine, which shows device info and memory usage 16 years ago
Anoop Saldanha c26e92733d handle the cuda cleanup at shutdown. should get rid of any errors from the call to SigGroupCleanup 16 years ago
Anoop Saldanha 41e6735b92 mpm b2g cuda support added 16 years ago
Pablo Rincon 34216fd3e2 bug87 Fix IPOnly veredicts on flows 16 years ago
Pablo Rincon 0165b3f0d8 pcre P modifier support (pcre match over http body requests) 16 years ago
Pablo Rincon ad2c136e8f Renaming errors (naming conventions) 16 years ago
Breno Silva 7e299834d2 FragOffset Rule Keyword 16 years ago
Victor Julien 6a53ab9c5a Stream engine memory handling update
The stream engine memory handling needed updating as it didn't scale. Changes:

- pools can now be initialized to size 0, meaning unlimited
- stream engine uses a memcap setting. Sessions, segments and aldata is part
  of this, app layer state isn't.
- memory is accounted using a global int that is spinlocked.
- a counter for sessions that have not been picked up because of memcap was
  added.
- all reassembly errors are converted to debug msgs.
16 years ago
Pablo Rincon 51dc773eec Changing the veredict actions to flags to allow simultaneous veredict 16 years ago
Victor Julien b3bcba077f Only inspect http flows against uri sigs, clean up uri scanning code. 16 years ago
Gurvinder Singh 0cb43d27e9 uricontent new design 16 years ago
Gurvinder Singh 356a8bf385 applayer uri match and modified http handling 16 years ago
Victor Julien 0d34990d7f Add OpenBSD's strlcpy and strlcat and replace all strcat/strcpy/strncat/strncpy by those calls. 16 years ago
Gurvinder Singh f6b0c481b0 urilen support for engine 16 years ago
Victor Julien 4284276b11 Merge applayer detect function into normal match function. Should speed up detection. 16 years ago
Victor Julien bcd0682150 Make engine startup a little less verbose. 16 years ago
Breno Silva 1d055b0e09 ICMP Seq Rule Keyword 16 years ago
Pablo Rincon c80160b96d More examples of unittest helper functions usage reference 16 years ago
Pablo Rincon b6a3395c08 Adding unittest helper functions for building generic packets, checking arrays of expected match results, perform generic tests, etc. Look at util-unittest-helper.c and detect-ipproto.c for references 16 years ago
Victor Julien eb67bb442e Fixup unittests that use buffers that simulate configuration files. They now include the YAML header. 16 years ago
Gerardo Iglesias Galvan 40c514f295 Fix bug in logging msg when using --init-errors-fatal 16 years ago
Gerardo Iglesias Galvan 5eb819b0f4 Add signature line no. to error message when parsing fails 16 years ago
Gerardo Iglesias Galvan fae92f8d7b Fix bug#30. Fix logging call from prev patch 16 years ago
Gerardo Iglesias Galvan 988dc5520b Improve output when loading rules 16 years ago
Victor Julien 6b36e23e45 Fix not decreasing the flow use_cnt reference counter in some cases from the app layer detection code. This caused some streams to never fully time out and thus clutter up the flow table and session pool. 16 years ago
Brian Rectanus c22d42693a Added http_method rule keyword. 16 years ago
Victor Julien b7bac14040 Fixup code to compile with -Wall -Werror -Wextra -Wno-unused-parameter compiler options. 16 years ago
Victor Julien 2b66667a76 Make sure we can't overflow our packet alert storage 16 years ago
Anoop Saldanha f684989f98 dce_iface, dce_opnum, dce_stub_data keyword support 16 years ago
Victor Julien ecab1fae36 Remove contents of VRT classification.config. 16 years ago
Anoop Saldanha 011b74df63 Modify the classification config tests to use the buffer than a temp file and also fix an invalid free 16 years ago
Anoop Saldanha bc4df59414 Support for Classtype keyword and Classification Config file 16 years ago
Victor Julien d5c732f1f9 Add tag keyword stub 16 years ago
Jason Ish ce20c33634 multiline rule support. 16 years ago
Will Metcalf 23aa6cf642 more fixes for exit on sig init failure 16 years ago
Victor Julien 778228d1c5 Flags keyword fix. Fatal init fix. 16 years ago
Victor Julien 35e884f303 Make sure offset modifies depth. 16 years ago
William Metcalf c63b1e0f67 failing unit test depth doesn't take into account offset 16 years ago
Gurvinder Singh 8cfdf6c666 bug 18 patch update 16 years ago
Gurvinder Singh b92886a79a bug#18 and some minor changes 16 years ago
Victor Julien 10cc9d5b6a Add icmp flow handling. 16 years ago
Victor Julien 71ed2d38f5 Fix scan patterns sometimes not being added to the scan ctx. Should fix bug #9. 16 years ago
William Metcalf c40e81d42b failing unittest rules with same content match fail 16 years ago
Victor Julien 4824868766 Application layer detection improvements
- improve locking of application layer handling, making sure that the flow cannot be freed/cleared when the detection engine is still working with it.
- add a check to the app layer detection to make sure that a match function will only inspect an app layer state if it's of the right type.
16 years ago
Pablo Rincon f2f9b83280 Adding FTP app layer parser and ftpbounce detection at L7 16 years ago
Gerardo Iglesias Galvan 7e87f373b9 Add icmp_id keyword support 16 years ago
Victor Julien 01976a0b74 Support for sigs with both pkt and applayer detect
Sets a flowbit with the sig id if the packet matches match. Checks
on that if the app layer matches match. Currently misuses the
flowbits api for this in a way that needs fixing.
16 years ago
Victor Julien 493715c0d2 Implement alert sid storage in the flow so we can check previous alerts in the flow. 16 years ago
Victor Julien 9fd46e9425 Support for sigs with both pkt and applayer detect
Sets a flowbit with the sig id if the packet matches match. Checks
on that if the app layer matches match. Currently misuses the
flowbits api for this in a way that needs fixing.
16 years ago
Victor Julien 4f843ff8e9 Improve matching of packet and app layer sigs. 16 years ago
Pablo Rincon 992aaa3d79 Small fix on SigMatchSignaturesAppLayer() and SigMatchSignatures() 16 years ago
Breno Silva 69eb869cc9 Threshold Rule 16 years ago
Victor Julien ecf86f9c23 Rename to Suricata. 16 years ago
Gurvinder Singh a0f184866c http_cookie keywork support 16 years ago
Pablo Rincon a67bd2457b Loading rules from config support 16 years ago
Pablo Rincon c816af822e Adding support to load rule files from config 16 years ago
Pablo Rincon 1ad6d75dfe Added rpc keyword support at packet level 16 years ago
Pablo Rincon a8d7b71490 First version of flowints 16 years ago
Gerardo Iglesias 991d421394 Changed printf's to logging API functions 16 years ago
Victor Julien 2cfa284999 Fix app layer detect to actually work. 16 years ago
Victor Julien a1d33e2d0b Compilation fixes after merge. 16 years ago
Victor Julien f1f7df0766 First iteration of doing app layer detection. 16 years ago
Anoop Saldanha 7a10ddc07b Fixes for the fast-pattern tests and a couple of other minor changes 16 years ago
Anoop Saldanha 6ca5dbc9e9 Support fast_pattern modifier keyword for content 16 years ago
Victor Julien 07bcc8cc9a Fix signatures with ports and/or addresses but without sigmatches. 16 years ago
Anoop Saldanha 1c7ac13c25 Support for negated content 16 years ago
Victor Julien 241db64dd7 Fix a few memory issues. 16 years ago
Victor Julien 981ca859cb Fixes for distance and within content modifiers. 16 years ago
William Metcalf 884b1034b4 failed unittest for within distance 16 years ago
Anoop Saldanha dc44700ce5 Support vars lookup from conf file. Current patch support address and port group vars lookup 16 years ago
Victor Julien 951b4d5cf4 Fix within in some corner cases and add some more tests. 16 years ago
Victor Julien 4f2164677a - Fix pattern matchers b2g and b3g not being able to deal with a single pattern of the max pattern length (32 bytes by default).
- Fix the setting of the correct pattern matcher when it was set in the detection ctx.
- Add tests for the fixes.
16 years ago
Victor Julien d6be6ceb19 Fixup artifact from automatic renames: rename DetectAddresssHead to DetectAddressHead. 16 years ago
Anoop Saldanha 78db80ea15 Unittests and style fix for detect-engine-siggroup.[ch] 16 years ago
Gurvinder Singh cacbf31aad support for ttl keyword 16 years ago
Victor Julien de8caa7964 Rename DetectAddressGroup* to DetectAddress* 16 years ago
Victor Julien 13dca5e8ce Cleanup of address functions. 16 years ago
Victor Julien 3c7a038477 Further memory cleanups. Split out init only vars out of the sig group head. 16 years ago
Victor Julien 32e51e5e5a Replace sgh refcnt by a flag. 16 years ago
Victor Julien 2d0e9658f8 Speed up per sgh content maxlen calc. Remove mpm ptrs from mpm ctx. Add unittests testing the detection engine internals. 16 years ago
Victor Julien e4c98c562c Merge DetectAddressData and DetectAddressGroup 16 years ago
Victor Julien 15ab5d7003 More engine init memleaks fixed. HashListTable remove function fixed. 16 years ago
Victor Julien 7a7bb7a390 Get rid of global mpm_ctx. 16 years ago
Victor Julien fbe87a3ad5 Bunch of mostly unittest related memleak fixes. 16 years ago
Victor Julien 227cc082a1 Fix small memleak at engine init. 16 years ago
Brian Rectanus ed30067bd7 Ack/Seq Keywords 16 years ago
Brian Rectanus ec6c5258b6 Sameip Keyword 16 years ago
Victor Julien f76a740fe2 Small layout fix. 16 years ago
Victor Julien b470e7b40f Fixup merge artifact. 16 years ago
Breno Silva 15a8f34d36 Gid Keyword
Signed-off-by: Brian Rectanus <brectanu@gmail.com>
16 years ago
Breno Silva 6100a7f610 FragBits Keyword
Signed-off-by: Brian Rectanus <brectanu@gmail.com>
16 years ago
Victor Julien d4733e3a76 Move pcre test out of detect.c 16 years ago
William Metcalf 86380b9d45 split out pcre parsing fixed some errors 16 years ago