aiden
/
suricata
mirror of https://github.com/OISF/suricata
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
9,115 Commits
7 Branches
161 Tags
210 MiB
master
main-7.0.x
master-6.0.x
master-5.0.x
master-4.1.x
master-4.0.x
master-3.2.x
suricata-8.0.0
suricata-7.0.11
suricata-8.0.0-rc1
suricata-8.0.0-beta1
suricata-7.0.10
suricata-7.0.9
suricata-7.0.8
suricata-7.0.7
suricata-7.0.6
suricata-6.0.20
suricata-7.0.5
suricata-6.0.19
suricata-6.0.18
suricata-7.0.4
suricata-6.0.17
suricata-7.0.3
suricata-6.0.16
suricata-7.0.2
suricata-6.0.15
suricata-7.0.1
suricata-6.0.14
suricata-7.0.0
suricata-7.0.0-rc2
suricata-6.0.13
suricata-6.0.12
suricata-6.0.11
suricata-7.0.0-rc1
suricata-6.0.10
suricata-6.0.9
suricata-7.0.0-beta1
suricata-6.0.8
suricata-6.0.7
suricata-6.0.6
suricata-5.0.10
suricata-6.0.5
suricata-5.0.9
suricata-6.0.4
suricata-5.0.8
suricata-6.0.3
suricata-5.0.7
suricata-6.0.2
suricata-5.0.6
suricata-6.0.1
suricata-5.0.5
suricata-4.1.10
suricata-4.1.9
suricata-5.0.4
suricata-6.0.0
suricata-6.0.0-rc1
suricata-6.0.0-beta1
suricata-5.0.3
suricata-4.1.8
suricata-4.1.7
suricata-5.0.2
suricata-4.1.6
suricata-5.0.1
suricata-5.0.0
suricata-5.0.0-rc1
suricata-4.1.5
suricata-5.0.0-beta1
suricata-4.1.4
suricata-4.1.3
suricata-4.0.7
suricata-4.1.2
suricata-4.1.1
suricata-4.1.0
suricata-4.0.6
suricata-4.1.0-rc2
suricata-4.1.0-rc1
suricata-4.0.5
suricata-4.1.0-beta1
suricata-4.0.4
suricata-4.0.3
suricata-4.0.2
suricata-3.2.5
suricata-4.0.1
suricata-3.2.4
suricata-4.0.0
suricata-4.0.0-rc2
suricata-3.2.3
suricata-4.0.0-rc1
suricata-4.0.0-beta1
suricata-3.2.2
suricata-3.1.4
suricata-3.2.1
suricata-3.2
suricata-3.2RC1
suricata-3.1.3
suricata-3.2beta1
suricata-3.1.2
suricata-3.1.1
suricata-3.1
suricata-3.0.2
suricata-3.1RC1
suricata-3.0.1
suricata-3.0.1RC1
suricata-3.0
suricata-2.0.11
suricata-3.0RC3
suricata-3.0RC2
suricata-3.0RC1
suricata-2.0.10
suricata-2.0.9
suricata-2.1beta4
suricata-2.0.8
suricata-2.0.7
suricata-2.1beta3
suricata-2.0.6
suricata-2.0.5
suricata-2.1beta2
suricata-2.0.4
suricata-2.1beta1
suricata-2.0.3
suricata-2.0.2
suricata-2.0.1
suricata-2.0.1rc1
suricata-2.0
suricata-2.0rc3
suricata-2.0rc2
suricata-2.0rc1
suricata-2.0beta2
suricata-1.4.7
suricata-1.4.6
suricata-1.4.5
suricata-2.0beta1
suricata-1.4.4
suricata-1.4.3
suricata-1.4.2
suricata-1.4.1
suricata-1.3.6
suricata-1.4
suricata-1.3.5
suricata-1.4rc1
suricata-1.3.4
suricata-1.4beta3
suricata-1.3.3
suricata-1.4beta2
suricata-1.3.2
suricata-1.4beta1
suricata-1.3.1
suricata-1.3
suricata-1.3rc1
suricata-1.3beta2
suricata-1.3beta1
suricata-1.2.1
suricata-1.2
suricata-1.2rc1
suricata-1.2beta1
suricata-1.1.1
suricata-0.8.2
suricata-1.0.0
suricata-1.0.1
suricata-1.0.2
suricata-1.0.3
suricata-1.0.4
suricata-1.0.5
suricata-1.1
suricata-1.1beta1
suricata-1.1beta2
suricata-1.1beta3
suricata-1.1rc1
Branches Tags
${ item.name }
Create tag ${ searchTerm }
Create branch ${ searchTerm }
from '5879dafe55'
${ noResults }
Commit Graph

937 Commits (5879dafe55852191cd71d2a6f8ba24cce918195c)

Author SHA1 Message Date
Victor Julien cc4f7a4b96 detect: add profiling for non-mpm list build & filter 11 years ago
Victor Julien 4c10635dc1 detect: optimize non-mpm mask checking 
Store id and mask in a single array of type SignatureNonMpmStore so
that both are loaded into the same cache line.
 11 years ago
Victor Julien b5a3127151 detect: add mask check prefilter for non mpm list 
Add mask array for non_mpm sigs, so that we can exclude many sigs before
we merge sort.

Shows 50% less non mpm sigs inspected on average.
 11 years ago
Ken Steele 904441327c Conditionalize SigMatch performance counters. 
Only include the counters when PROFILING.
 11 years ago
Victor Julien 30b7fdcb49 Detect perf counters 11 years ago
Victor Julien ef6875d583 detect: Disable unused SignatureHeader code 11 years ago
Ken Steele 4bd280f196 Indentation clean up 11 years ago
Ken Steele 403b5a4645 Further optimize merging mpm and non-mpm rule ID lists. 
When reaching the end of either list, merging is no longer required,
simply walk down the other list.

If the non-MPM list can't have duplicates, it would be worth removing
the duplicate check for the non-MPM list when it is the only non-empty list
remaining.
 11 years ago
Ken Steele 86f4c6c47b Custom Quick Sort for Signature IDs 
Use an in place Quick Sort instead of qsort(), which does merge sort and
calls memcpy().

Improves performance on my tests.
 11 years ago
Ken Steele 736ac6a459 Use SigIntId as the type for storing signature IDs (Internal) 
Previously using uint32_t, but SigIntId is currently uint16_t, so arrays
will take less memory.
 11 years ago
Ken Steele 1874784c10 Create optimized sig_arrays from sig_lists 
Create a copy of the SigMatch data in the sig_lists linked-lists and store
it in an array for faster access and not next and previous pointers. The
array is then used when calling the Match() functions.

Gives a 7.7% speed up on one test.
 11 years ago
Ken Steele 923a77e952 Change Match() function to take const SigMatchCtx* 
The Match functions don't need a pointer to the SigMatch object, just the
context pointer contained inside, so pass the Context to the Match function
rather than the SigMatch object. This allows for further optimization.

Change SigMatch->ctx to have type SigMatchCtx* rather than void* for better
type checking. This requires adding type casts when using or assigning it.

The SigMatch contex should not be changed by the Match() funciton, so pass it
as a const SigMatchCtx*.
 11 years ago
Ken Steele 1c76fa50b1 Prefetch the next signature pointer 
Read one signature pointer ahead to prefetch the value.
Use a variable, sflags, for s->flags, since it is used many times and the
compiles doesn't know that the signatures structure doesn't change, so it
will reload s->flags.
 11 years ago
Ken Steele d31db4ed1c Fix clang warning 
Clang doesn't seem to like defining a function within a function.
 11 years ago
Ken Steele 23d2a1c422 Optimize DetectPrefilterMergeSort 
Fixup rebase changes to remove debug code
 11 years ago
Victor Julien 7876277119 detect: move checks from prefilter to rule detect 
Move the prefilter checks to the main detect loop.
 11 years ago
Victor Julien d1d895a884 Replace build match array with new filter logic 
Use MPM and non-MPM lists to build our match array. Both lists are
sorted, and are merged and sorted into the match array.

This disables the old match array building code and thus also bypasses
the mask checking.
 11 years ago
Victor Julien f5df526f9b Detect: create per sgh non-MPM rule array 
Array of rule id's that are not using MPM prefiltering. These will be
merged with the MPM results array. Together these should lead to a
list of all the rules that can possibly match.
 11 years ago
Victor Julien e49d0a5924 MPM: build sid list from MPM matches 
Pmq add rule list: Array of uint32_t's to store (internal) sids from the MPM.

AC: store sids in the pattern list, append to Pmq::rule_id_array on match.

Detect: sort rule_id_array after it was set up by the MPM. Rule id's
(Signature::num) are ordered, and the rule's with the lowest id are to
be inspected first. As the MPM doesn't fill the array in order, but instead
'randomly' we need this sort step to assure proper inspection order.
 11 years ago
Victor Julien 43a1007788 detect: add test for memcmp issue 11 years ago
Victor Julien d951de2f19 Bug 977: -T / --init-errors-fatal to process all rules 
Have -T / --init-errors-fatal process all rules so that it's easier
to debug problems in ruleset. Otherwise it can be a lengthy fix, test
error cycle if multiple rules have issues.

Convert empty rulefile error into a warning.

Bug #977
 11 years ago
DIALLO David b3bf2f9939 Detect: Add Modbus keyword management 
Add the modbus.function and subfunction) keywords for public function match in rules (Modbus layer).
Matching based on code function, and if necessary, sub-function code
or based on category (assigned, unassigned, public, user or reserved)
and negation is permitted.

Add the modbus.access keyword for read/write Modbus function match in rules (Modbus layer).
Matching based on access type (read or write),
and/or function type (discretes, coils, input or holding)
and, if necessary, read or write address access,
and, if necessary, value to write.
For address and value matching, "<", ">" and "<>" is permitted.

Based on TLS source code and file size source code (address and value matching).

Signed-off-by: David DIALLO <diallo@et.esia.fr>
 11 years ago
Anoop Saldanha b334b8a6e9 CUDA: Update the inspection engine to inform the cuda module that it 
doesn't need the gpu results and to release the packet for the next run.

Previously the inspection engine wouldn't inform the cuda module, if it
didn't need the results.  As a consequence, when the packet is next taken
for re-use, and if the packet is still being processed by the cuda module,
the engine would wait till the cuda module frees the packet.

This commits updates this functionality to inform the cuda module to
release the packet for the afore-mentioned case.
 11 years ago
Ken Steele 60c46170b0 Check replist is not NULL inline before doing any processing. 
The replist is often NULL, so it is worth checking that case before making
the function call do perform work on the list.
 11 years ago
Victor Julien f23399d672 Rename Lua code to just Lua 
As we support regular Lua as well as LuaJIT, it makes more sense to call
it all Lua.
 11 years ago
Ken Steele 8f1d75039a Enforce function coding standard 
Functions should be defined as:

int foo(void)
{
}

Rather than:
int food(void) {
}

All functions where changed by a script to match this standard.
 11 years ago
Victor Julien 96adcf6829 refactor IDS/IPS engine mode logic 
Instead of error phrone externs with macro's, use functions with a local
static enum var instead.

- EngineModeIsIPS(): in IPS mode
- EngineModeIsIDS(): in IDS mode

To set the modes:

- EngineModeSetIDS(): IDS mode (default)
- EngineModeSetIPS(): IPS mode

Bug #1177.
 11 years ago
Victor Julien fc559ce227 detect: fix alstate handling 
Previously, the alstate use in the main detect loop was unsafe. The
alstate pointer would be set duing a lock, but it would again be used
after one or more lock/unlock cycles. If the data pointed to would
disappear, a dangling pointer would be the result.

Due to they way flows are cleaned up using reference counting and
such, changes of this happening were very small. However, at least
one path can lead to this situation. So it had to be fixed.
 11 years ago
Victor Julien b6e2a6f525 detect: locking update continued 
Make DeStateDetectContinueDetection get it's own alstate pointer instead
of using the one that was passed to it. We now get and use it only
inside a flow lock.
 11 years ago
Victor Julien cf31e2cc74 detect: locking update 
Make DeStateDetectStartDetection get it's own alstate pointer instead
of using the one that was passed to it. We now get and use it only
inside a flow lock.
 11 years ago
Victor Julien 5e1bc99e5b detect: cleanup 
Remove unused alstate and app layer flags arguments from
DetectEngineInspectPacketPayload()
 11 years ago
Victor Julien f00703acc1 Fix False Positive of rules with ports on portless protocols 
In case of 'alert ip' rules that have ports, the port checks would
be bypassed for non-port protocols, such as ICMP. This would lead to
a rule matching: a false positive.

This patch adds a check. If the rule has a port setting other than
'any' and the protocol is not TCP, UDP or SCTP, then we rule won't
match.

Rules with 'alert ip' and ports are rare, so the impact should be
minimal.

Bug #611.
 12 years ago
Victor Julien b5afe2b51f ssh: reenable ssh.protoversion keyword 12 years ago
Victor Julien 6c0162bf26 ssh: reenable ssh.softwareversion keyword 12 years ago
Victor Julien df927f7ea8 unittest: fix mutex unlock w/o a lock 
Fixes an error in a test. SCMutexUnlock was called w/o a prior
SCMutexLock.
 12 years ago
Victor Julien 2c3a92a1c9 profiling: conditional rule profiling 
Add support for conditional rule profiling. Currently only simple
rate limiting is supported, but hardcoded to inspecting rules for
each packet.
 12 years ago
Victor Julien 54d64a1237 detect: use pflow pointer 
Use pflow pointer in SigMatchSignatures consistently. Also, when
needing access to the ipproto, use p->proto, not p->flow->proto.
 12 years ago
Victor Julien 0ec375d95a stream msg: remove structure 12 years ago
Victor Julien 8dbf7a0d78 Update tests to use AppLayerParserThreadCtx ptr instead of void. Fix a few bugs uncovered by this. 12 years ago
Victor Julien fdefb65be4 app-layer: rename AppLayerThreadCtx funcs 
AppLayerParserGetCtxThread -> AppLayerParserThreadCtxAlloc
AppLayerParserDestroyCtxThread -> AppLayerParserThreadCtxFree
 12 years ago
Victor Julien f5f148805c app layer: uint16_t alproto -> AppProto alproto 
This conversion was missing in a couple of places.
 12 years ago
Victor Julien 8527b8e08e App Layer: cleanup state func naming 
Rename functions related to AppLayerState to be more consistent.
 12 years ago
Victor Julien e111401c10 detect unittests: clang build fix and cleanups 
A number of unittests would lead to clang build errors because
of unsafe det_ctx ptr usage. This patch fixes these and inits
det_ctx to NULL in the other detect tests.
 12 years ago
Anoop Saldanha 429c6388f6 App layer API rewritten. The main files in question are: 
app-layer.[ch], app-layer-detect-proto.[ch] and app-layer-parser.[ch].

Things addressed in this commit:
- Brings out a proper separation between protocol detection phase and the
  parser phase.
- The dns app layer now is registered such that we don't use "dnstcp" and
  "dnsudp" in the rules.  A user who previously wrote a rule like this -

  "alert dnstcp....." or
  "alert dnsudp....."

  would now have to use,

  alert dns (ipproto:tcp;) or
  alert udp (app-layer-protocol:dns;) or
  alert ip (ipproto:udp; app-layer-protocol:dns;)

  The same rules extend to other another such protocol, dcerpc.
- The app layer parser api now takes in the ipproto while registering
  callbacks.
- The app inspection/detection engine also takes an ipproto.
- All app layer parser functions now take direction as STREAM_TOSERVER or
  STREAM_TOCLIENT, as opposed to 0 or 1, which was taken by some of the
  functions.
- FlowInitialize() and FlowRecycle() now resets proto to 0.  This is
  needed by unittests, which would try to clean the flow, and that would
  call the api, AppLayerParserCleanupParserState(), which would try to
  clean the app state, but the app layer now needs an ipproto to figure
  out which api to internally call to clean the state, and if the ipproto
  is 0, it would return without trying to clean the state.
- A lot of unittests are now updated where if they are using a flow and
  they need to use the app layer, we would set a flow ipproto.
- The "app-layer" section in the yaml conf has also been updated as well.
 12 years ago
Anoop Saldanha abded4200a Disabling the ssh parser temporarily, since we are moving away from some 
of the archaic features we use in the app layer. We will reintroduce this
parser shortly. Also do note that keywords that rely on the ssh parser
would now be disabled.
 12 years ago
Victor Julien 2913a4a860 debug: fix realloc error checking on flowbit print 
detect.c:1074:17: warning: Potential leak of memory pointed to by \
                                field 'debuglog_flowbits_names'
                return;

Bug #1062.
 12 years ago
Ken Steele 882d98733f Fix pfring so that zero-copy mode can work. 
Detect when default_packet_size is zero, which enables zero-copy mode for
pfring and in that case, do what AF Packet does and set pkt_ext pointer to
the data and set PKT_ZERO_COPY flag.
 12 years ago
Ken Steele f9705377ae Remove pkt variable from Packet structure. 
The uint8_t *pkt in the Packet structure always points to the memory
immediately following the Packet structure. It is better to simply
calculate that value every time than store the 8 byte pointer.
 12 years ago
Ken Steele 5a1bc025e5 Mark pflow as a constant pointer. 
Address review comment from Victor that the pflow pointer is constant, so
it can be marked as such.
 12 years ago
Ken Steele bee24d8909 Use pflow variable in place of p->flow to prevent reloading. 
In SigMatchSignatures, the value p->flow doens't change, but GCC can't
figure that out, so it reloads p->flow many times during the function.
When p->flow is loaded into the variable pflow once at the start of the
function, the compile then doesn't need to reload it.
 12 years ago
Victor Julien 0b0e9340dc rule setup: cleanup 
Remove rule preparation logic that ran, but it's results were not
used.
 12 years ago
Victor Julien d397ed94c5 detect: use macro for max rule size 12 years ago
Victor Julien e7f6107e79 signature address parsing improvements and tests 
Fix sigatures not supporting [10.0.0.0/24, !10.1.1.1] notation when
used directly in a rule instead of through a variable.

Add tests for Bugs #815 and #920.
 12 years ago
Victor Julien 5330b1cae1 detect: don't consider smsgs for no inspect flag 
When the PKT_NOPAYLOAD_INSPECTION flag is set, don't apply it to smsgs.
This way we can still inspect the outstanding smsgs.

The PKT_NOPAYLOAD_INSPECTION is set for encrypted traffic, and is combined
with disabling stream reassembly. So we only inspect the smsgs up to the
point of the disable detection point.
 12 years ago
Ken Steele c6a8d0ab6b Share Packet checksum values for TCP, UDP, IPv6. ICMPv4 and ICMPv6 
Keep a separate checksum for IPV4, since a packet can have both an IPV4
checksum and a TCPV4 checksum, or IPV4 and UDPV4 checksum.

This will allow future sharing of more values.

Use PACKET_RESET_CHECKSUMS() in Unit Tests in place of setting the
individual checksum values.
 12 years ago
Victor Julien 72a147b6f1 detect: only inspect smsg for valid tcp packets 
Packets that are rejected by the stream engine are not considered
part of an established tcp session. By allowing them to inspect
an smsg, some smsgs would not be properly inspected.
 12 years ago
Ken Steele dfc2117e80 Add Unit test to check TCPv4 and IPV4 checksums together. 
Copied SigTest26TCPV4Keyword and added check for invalid IPV4 checksums.
Created new SigTest26TCPV4AndIPV4Keyword test with a new packet with valid
IPV4 checksums.
 12 years ago
Victor Julien 5686c673ec profiling: per buffer profiling 12 years ago
Victor Julien 97bfcac444 profiling: introduce per keyword profiling 
Initial version of per keyword profiling. Prints stats about
how ofter a keyword was checked and what the costs were.
 12 years ago
Victor Julien 1b11165864 Reset both sides of the de_state on rule reload. Bug #998. 12 years ago
Victor Julien edeeb7ed44 Store TX id with alerts 
When generating an alert and storing it in the packet, store the tx_id
as well. This way the output modules can log the tx_id and access the
proper tx for logging.

Issue #904.
 12 years ago
Anoop Saldanha b1dffdfbe0 Add app layer protocol packet event detection support. 12 years ago
Anoop Saldanha 94e40907e2 feature #727 - Add support for app-layer-protocol:<protocol> keyword 12 years ago
Victor Julien 48b5513ed9 Properly clean up decoder event rules 
Addresses:
~~Dr.M~~ Error #3: LEAK 120 direct bytes 0x08a26ac8-0x08a26b40 + 1871 indirect bytes
~~Dr.M~~ # 0 replace_malloc                               [/work/drmemory_package/common/alloc_replace.c:2292]
~~Dr.M~~ # 1 SigGroupHeadAlloc                            [/home/victor/dev/oisf/src/detect-engine-siggroup.c:144]
~~Dr.M~~ # 2 SigGroupHeadAppendSig                        [/home/victor/dev/oisf/src/detect-engine-siggroup.c:1014]
~~Dr.M~~ # 3 DetectEngineAddDecoderEventSig               [/home/victor/dev/oisf/src/detect.c:3026]
~~Dr.M~~ # 4 SigAddressPrepareStage2                      [/home/victor/dev/oisf/src/detect.c:3075]
~~Dr.M~~ # 5 SigGroupBuild                                [/home/victor/dev/oisf/src/detect.c:4311]
~~Dr.M~~ # 6 SigLoadSignatures                            [/home/victor/dev/oisf/src/detect.c:464]
~~Dr.M~~ # 7 LoadSignatures                               [/home/victor/dev/oisf/src/suricata.c:1706]
~~Dr.M~~ # 8 main                                         [/home/victor/dev/oisf/src/suricata.c:1994]
 12 years ago
Victor Julien 8080494e9a counters: consolidate counters after all ThreadInit functions of a thread have run. This prevents duplicate and overwriting memory allocations. 12 years ago
Victor Julien 3470b07ea5 Fix several compile and runtime warnings found by clang 3.2 with the -fsanitize=address option. 12 years ago
Victor Julien c82ecf553a Tag: document in the code that 'tag' is compatible with ip only 12 years ago
Ken Steele 50f859e9f2 Move SIMD implementations out of detect.c 
Move SIMD the implementations of SigMatchSignaturesBuildMatchArray()
for SSE3 and Tile out of detect.c to reduce the size of the file.

Also moved SIMD unit tests to detect-simd.c
 12 years ago
Ken Steele 784843b146 Use Tilera SIMD for Signature matching ala SSE3 
Makes use of 8-wide byte compare instructions in signature matching.

For allocating aligned memory, _mm_malloc() is SSE only, so added
check for __tile__ to use memalign() instead.

Shows a 13% speed up.
 12 years ago
Eric Leblond cd3e32ce19 unittests: some functions needs a flow lock. 
In debug validation mode, it is required to call application layer
parsing and other functions with a lock on flow. This patch updates
the code to do so.
 12 years ago
Eric Leblond c5bd04f102 unittest: recycle packet before exit 
To avoid an issue with flow validation, we need to recycle the packet
before cleaning the flow.
 12 years ago
Eric Leblond 75fa1e20d7 engine analysis is a running mode 12 years ago
Eric Leblond 325462d396 Export IsRuleReloadSet and use it. 12 years ago
Victor Julien 055b422c28 Remove obsolete code: flow alert sid storage 12 years ago
Victor Julien 9faa4b740d Add --unittests-coverage option to list how many code modules have tests 12 years ago
Victor Julien f4dcba6de3 In case of fragments, don't consider ports. Bug #847. 12 years ago
Anoop Saldanha 48cf0585fb Suricata upgrade to libhtp 0.5.x. 
Remove the support for now unsupported personalities from libhtp -
TOMCAT_6_0, APACHE and APACHE_2_2.  We instead use the APACHE_2
personality.
 12 years ago
Anoop Saldanha fba95e9125 Remove mpm ctxs in the wrong direction. 
A lot of http mpm ctxs have now been removed as a result of this.
 12 years ago
Victor Julien 9dc04d9fab app layer: add support for per TX decoder events 12 years ago
Victor Julien 6645620c03 Merge SIG_FLAG_MPM_HTTP and SIG_FLAG_MPM_DNS into SIG_FLAG_MPM_APPLAYER, do the same for the _NEG variant. 12 years ago
Victor Julien 43ba5a677e DNS: enable mpm/fast_pattern support for dns_query 12 years ago
Victor Julien f10dd603ff DNS: adding dns_request content modifier 12 years ago
Eric Leblond 4521de2dfd Use PACKET_* macro instead of UPDATE 
Setting the ACTION_DROP flag can be done via PACKET_DROP instead
of using PACKET_UPDATE_ACTION.
 12 years ago
Eric Leblond 3f107fa130 decode: Packet action start with PACKET 
Rename all Packet action macro to have them prefixed by PACKET.
 12 years ago
Anoop Saldanha 602c91ed41 Minor cosmetic changes to the cuda code. 
Moved a couple of functions to more cuda relevant files;
Re-structured some data types.
 12 years ago
Anoop Saldanha 17c763f855 Version 1 of AC Cuda. 12 years ago
Anoop Saldanha 557cab3dc9 We call packet and stream mpm as late as possible now. Won't affect the working of the engine. 
The rationale behind this is, if we have pkt buffered to the gpu, we'd want
to delay processing the pkt as much as possible.
 12 years ago
Anoop Saldanha f4c719b83a code refactoring. Call mpmprefilter slightly later than where it's called atm 12 years ago
Anoop Saldanha b787da5643 Remove all cuda related code in the engine except for the cuda api wrappers 12 years ago
Anoop Saldanha e2a6cfb6a6 update cuda API wrappers 12 years ago
Eric Leblond a35c367942 action handling: use macro for test. 
Use test macro instead of direct access to action field.

This patch has been obtained by using the following
spatch file:

  @@
  Packet *p;
  expression E;
  @@

  - p->action & E
  + TEST_PACKET_ACTION(p, E)
 12 years ago
Eric Leblond efaa9a7302 action handling: define and use macros 
The action field in Packet structure should not be accessed
directly as the tunneled packet needs to update the root packet
and not the initial packet.

This patch is fixing issue #819 where suricata was not able to
drop fragmented packets in AF_PACKET IPS mode. It also fixes
drop capability for tunneled packets.
 12 years ago
Victor Julien 6e18ed0489 luajit flowvar support 
This patch adds flowvar support to luajit. It does so by exposing two special
C functions to the luajit scripts: ScFlowvarGet and ScFlowvarSet.
 12 years ago
Anoop Saldanha ab1f8afbc3 Removed Signature->order_id and replaced it with Signature->num. 12 years ago
Anoop Saldanha 43d1229dfa 1. Fix assignment of signums, which affected how we used read 
sigs(priority wise) inside staging.

   Previously we would assign signums before sig ordering, and hence the
   order didn't actually reflect the order of the sig in the
   sig_list(assuming sig reordering changed the sig_list).  Staging would
   use the old sig_nums to decide the priority of sigs.
2. Fix sig ordering for flowvar, flowbits, flowint, pktvar sigs.   We have
   introduced a new priority to treat sigs with set + read as lower
   priority compared to set only sigs.
3. Previously we treated sigs with a "priority(keyword)" > another sig's
   priority, as a sig with greater priority than the later.  We have
   reversed it.  Now the sig priority ordering is 1,2,.etc.  Updated
   sigordering unittests to reflect the same.
 12 years ago
Anoop Saldanha a490176c8a More lock fixes for the transaction update. Issues reported by Coverity. 12 years ago
Anoop Saldanha d4d18e3136 Transaction engine redesigned. 
Improved accuracy, improved performance.  Performance improvement
noticeable with http heavy traffic and ruleset.

A lot of other cosmetic changes carried out as well.  Wrappers introduced
for a lot of app layer functions.

Failing dce unittests disabled.  Will be reintroduced in the updated dce
engine.

Cross transaction matching taken care of.  FPs emanating from these
matches have now disappeared.  Double inspection of transactions taken
care of as well.
 12 years ago
Anoop Saldanha 6dcde9d7e9 hsbd mpm and packet mpm share same mpm ctx id. 
This is a bug emanating from we having a var reference for hsbd mpm,
but failing to initialize it, and we default to using the packet mpm.
 12 years ago
Victor Julien 3b68a9d1c6 UDP: inspection app layer state as soon as we have it. 12 years ago
Victor Julien 4cd736fcc9 flowvar: fix deadlock with http buffers 
Bug #802

Flowvars are set from pcre, and lock the flow when being set. However
when HTTP buffers were inspected, flow was already locked: deadlock.

This patch introduces a post-match list in the detection engine thread
ctx, where store candidates are kept. Then a post-match function is used
to finalize the storing if the rule matches.

Solves the deadlock and brings the handling of flowvars more in line
with flowbits and flowints.
 12 years ago
Anoop Saldanha a3212f6a0f Minor fixes against the last set of patches for #564, 565, 581 + fp automation. 
Rename struct DetectFigureFPAndId_t_ to DetectFPAndItsId_ and move it's
definition from inside the function where it's used to the global namespace,
as requested on #suricata.

Rename DetectEngineContentModifiedBufferSetup to DetectEngineContentModifierBufferSetup.

Also rename DetectFigureFPAndId() to DetectSetFastPatternAndItsId().

Updated DetectSetFastPatternAndItsId() to not exit on failure and return error.
 13 years ago
Anoop Saldanha 6de8b1ed53 fix for #564. 
Get rid of the hash table, and use a single-one_time_alloc'ed array for
pattern id assignment.
 13 years ago
Anoop Saldanha e77fd1c883 We now assign ids to fp patterns only. Rest of them don't need one. 13 years ago
Anoop Saldanha 4c6efa2d40 Update content id assignment. 
All fp id assignment now happens in one go.
Also noticing a slight perf increase, probably emanating from improved cache
perf.
Removed irrelevant unittests as well.
 13 years ago
Anoop Saldanha 601836d831 Fast pattern setup now configurable in our code. 
You can either enable/disable fp for a particular type + set priority.
 13 years ago
Victor Julien 37c80ea508 If an IP-only pass rule matches, set the no inspect flag for that flow. Bug #718. 13 years ago
Anoop Saldanha 3511f91bba Add support for the new keyword - http_raw_host header. 
The corresponding pcre modifier would be 'Z'.
 13 years ago
Anoop Saldanha c4ce19a1be Add support for a new keyword to inspect http_host header. 
The corresponding content keyword would now be - http_host.
The corresponding pcre modifier would be W.
 13 years ago
Victor Julien afb2d4eddf Fix stateful inspection not always inspecting at stream end. 13 years ago
Ignacio Sanchez d771e08156 Adds support for the geoip keyword 
Adds support for match-on conditions (src, dst, any, both)
Uses GEOIP_MEMORY_CACHE for performance reasons
Adds support for negation and multiple countries in the same rule

Bug fixes

Changed to take flow direction from rule, if present

Comments addressed. Unit tests added.
 13 years ago
Victor Julien 538a941486 decoder events: fix bug causing some rules not to be inspected if the decoder completed with warnings 13 years ago
Anoop Saldanha b1ce94babe Temporary fix for bug #599. 
Treat sigs with negated addresses as non ip-only.

This fix exposes bug #608, which results in 2 failed unittest which
have now been disabled by this commit.  Would be reenabled when we
have #608 fix in.
 13 years ago
Last G 8ae11f73b2 Added parentheses to fix Eclipse static code analysis 
Fixed bug in action priority (REJECT_DST had lowest prio)
 13 years ago
Eric Leblond 40891223e9 list-keyword: detect non built keyword 
This patch update the glafs list to be able to indicate that a
flag is not supported. This information is used by list-keyword to
display information to the user.
 13 years ago
Eric Leblond 6842545331 Add documentation url in list-keyword output. 
The output of the list-keyword is modified to include the url to
the keyword documentation when this is available. All documented
keywords should have their link set.

list-keyword can be used with an optional value:
 no option or short: display list of keywords
 csv: display a csv output on info an all keywords
 all: display a human readable output of keywords info
 $KWD: display the info about one keyword.
 13 years ago
Eric Leblond 5e4552fdcd suricata: update list-keyword command 
This patch update the list-keyword command. Without any option,
the previous behavior is conserved. If 'all' is used as option,
suricata print a csv formatted output of keyword information:
	name;features;description
If a keyword name is used as argument, suricata print a readable
message:
tls.subject
Features: state inspecting
Description: Match TLS/SSL certificate Subject field
 13 years ago
Eric Leblond 86709f5e9d rule analyser: display message for invalid signatures 13 years ago
Victor Julien e30b1bfe64 Simple IP reputation implementation 13 years ago
Victor Julien 84bad6db77 Silence compiler warnings found by clang 13 years ago
Xavier Lange 234922f3c6 Keyword pkt_data 13 years ago
Victor Julien 70bc9e2494 filestore: fix logic flag in continued stateful detection 13 years ago
Anoop Saldanha 7a7cd6999e feature #558. 
Print FP info in rule analysis + other cleanup.
 13 years ago
Victor Julien a3f963f630 filestore: fix a case where a matching non-filestore sig could trigger the store of a partially matching filestore sig. 13 years ago
Victor Julien 80d62b59ec Fix drop (and other actions) not being applied to thresholded packets. Bug #613. 13 years ago
Anoop Saldanha b99f9fe890 New app inspection engine introduced. Moved existing inspecting engines to use it. 13 years ago
Anoop Saldanha 10a6e6a3eb Engine cleanup. Remove all old engine inspection and mpm functions. 13 years ago
Anoop Saldanha b0e20a486c update client/server/http_header to use a different form of 
buffering/buffer_retrieval.

Now it happens per tx, based on tx id.  Also notice a perf improvement with
this.
 13 years ago
Victor Julien 10a11b750d Add dsize check to prefilter stage 
Many sigs with dsize have a weak fast_pattern. Those patterns
are likely to match. By filtering on dsize early, we safe a lot
of cycles later.
 13 years ago
Victor Julien 45cbef0735 For signatures with the dsize option set depth on any content match in that sig. 13 years ago
Anoop Saldanha 3d74fa964a Update all flow referencing to use the new FlowReference and FlowDeReference 
macros.
 13 years ago
Eric Leblond e176be6fcc Use unlikely for error treatment. 
When handling error case on SCMallog, SCCalloc or SCStrdup
we are in an unlikely case. This patch adds the unlikely()
expression to indicate this to gcc.

This patch has been obtained via coccinelle. The transformation
is the following:

@istested@
identifier x;
statement S1;
identifier func =~ "(SCMalloc|SCStrdup|SCCalloc)";
@@

x = func(...)
... when != x
- if (x == NULL) S1
+ if (unlikely(x == NULL)) S1
 13 years ago
Eric Leblond d292004880 Add some missing checks of SCStrdup return. 13 years ago
Victor Julien 525367113a Fix compilation if luajit is disabled. 13 years ago
Victor Julien 98484ffdcc luajit: prealloc lua states to increases chances of alloc success. Luajit requires them to be in memory <2GB. 13 years ago
Victor Julien 9a4b612126 app layer events: prefilter sigs that need an event 13 years ago
Victor Julien 575c87aeba engine events: prefilter sigs that need a event 13 years ago
Victor Julien ec7e79c748 Rule profiling update 
- Remove usage of counters api.
- Store stats in detect engine thread ctx to remove locking
- Support rule reloads
 13 years ago
Eric Leblond d51dd6a30e Fix warning about unused return of SC_ATOMIC func. 13 years ago
Victor Julien f58e828c5e luajit: stub detection keyword 13 years ago
Eric Leblond 71b4257bc2 sig: add l3_proto keyword 
This patch adds a l3_proto keyword to the signature language. It
can be used to specify if the signature has to match on IPv4, IPv6
or both. For example, one can write:
  alert http any any -> any 22 (msg: "HTTP v6"; l3_proto:ip6; sid:14;)

This should close #494.
 13 years ago
Eric Leblond fd7b6db22d sig: Add ipv6 and ipv4 to list of protocols 
With this patch it is possible to do:
 alert ipv6 any any -> any any
or
 alert ip4 any any -> any any
to match on IPv4 or IPv6 packets.
 13 years ago
Victor Julien 5d27518bbd Make sure we never underflow len in DetectLoadSigFile 13 years ago
Victor Julien 8f71333e12 file: implement filesize keyword. #489. 13 years ago
pi-rho 0df4c5838d spelling corrections documented in redmine bug#533 13 years ago
Victor Julien bd6b865473 rule analyzer: fix fast pattern analyzer reporting wrong filename (same as rule analyzer). 13 years ago
Victor Julien 6841171882 profiling: fix 'match' counter sometimes not incrementing. #460. 13 years ago
Anoop Saldanha 64fad5b36e Update fast_pattern engine to not use negated content as fast_pattern if we have non-negated content in the sig. 
Noticing a good spike in perf with et_pro ruleset.

Thanks to Will Metcalf for the suggestion.
 13 years ago
Victor Julien d8356c5ebd Windows build and other misc fixes. 13 years ago
Victor Julien c9e93ec52c filemd5: add support code for md5 handling for signatures. 13 years ago
Anoop Saldanha 0c24bbab0c code cleanup for live swap 13 years ago
Anoop Saldanha 32183faa82 free flowvar entries in flow after live rule swap. Sync flowbits entries into packet struct to be used by alert debuglog when alert debuglog is enabled 13 years ago
Victor Julien 4cde2355bd Simplify flow resetting on de_ctx update. Detect ctx id starts at 1. So in a flow 0 means uninitialized (thus set) and if we detect flow is not equal to detect id, we reset the sgh storage and de_state. 13 years ago
Anoop Saldanha ecad4a24fa live rule support added 
To reload ruleset during engine runtime, send the USR2 signal to the engine, and the ruleset would be reloaded from the same yaml file supplied at engine startup
 13 years ago
Anoop Saldanha f4ce9011d2 make mpm ctx container de_ctx specific. Also introduce global variable in mpm_ctx. this is a workaround for cleaning non global mpm_ctx's since we now don't supply the de_ctx around the detection engine API 13 years ago
Victor Julien 60c3af9303 detect: Only run mpm on HTTP buffers in the proper direction. Fixes a file_data FN. 13 years ago
Anoop Saldanha 0d602d9cde we now support offset, depth inspection against all packet payloads and stream messages 13 years ago
Anoop Saldanha db8500bb26 fast pattern cleanup - Remove FastPatternSupportEnabledForSigMatchList() and all it's associated structures 13 years ago
Anoop Saldanha 988c92f71c http user agent keyword + mpm + inspection + fast pattern support added 13 years ago
Eileen Donlon c81020e9a3 feature 349 rule analyzer v1 13 years ago
Victor Julien 3df573219b Fix compilation warning. 13 years ago
Victor Julien 9f7588a756 Add filemd5 keyword that loads a list of md5's to match a file's md5 against. 13 years ago
Eric Leblond a0e57f58e5 OpenBSD: introduce SCLocalTime function. 
This function is a wrapper to localtime_r. It is needed to avoid
a compilation warning on OpenBSD. I'm forced to type the function
to a non pointer first parameter. If not we will have to use two
differents functions in OpenBSD where tv->tv_sec is a long
(different from time_t).
 13 years ago
Victor Julien cae46ab5eb pcre: print filename and line number for JIT warning. 13 years ago
Victor Julien ea0d172693 No longer pass StreamMsg to output for alert logging, instead use the same callback code as is used for state alerts. 13 years ago
Anoop Saldanha 64a04fc721 code cleanup 13 years ago
Anoop Saldanha 493c3db413 fix FNs for flow- only_stream and no_stream options 13 years ago
Anoop Saldanha ad36d55771 code cleanup - indentation fix 13 years ago
Anoop Saldanha 046819e1b8 bug 452 - fix detection bug for sigs that don't have a content but need payload inspection 13 years ago
Victor Julien 98c30be2db ipv6: improve handling of packets with duplicate (or more) ipv6 extension headers. 13 years ago
Victor Julien d378b76c04 http: body inspection improvement 
Improve http_client_body and file_data performance when request and
response body limits are set to high values.
 13 years ago
Victor Julien 40fcae3aa0 Minor unittest fixes to make Coverity happy. 14 years ago
Victor Julien 0a80e362aa Fix some minor clang scan-build warnings. 14 years ago
Victor Julien 19a7e7f395 flow: create a flow lock macro API, implement it for mutex and rwlocks. Mutex remains the default. 14 years ago
Pierre Chifflier 4be65fd016 TLS handshake: decode the SERVER_CERTIFICATE message 
Add a decoder for the SERVER_CERTIFICATE during a TLS handshake, extracts the
certificates and keep the subject name.
Add the tls.subject keyword for substring match in rules (TLS layer).

Signed-off-by: Pierre Chifflier <pierre.chifflier@ssi.gouv.fr>
 14 years ago
Victor Julien 8b1333a277 Add more flow lock assertions to the debug validation code. 14 years ago
Anoop Saldanha 059ee217ff All http_http_raw_uri modified patterns now are DETECT_CONTENT and not DETECT_AL_HTTP_RAW_URI 14 years ago
Anoop Saldanha b1a0d35106 All http_http_cookie modified patterns now are DETECT_CONTENT and not DETECT_AL_HTTP_COOKIE 14 years ago
Anoop Saldanha 49bdad9345 All http_http_method modified patterns now are DETECT_CONTENT and not DETECT_AL_HTTP_METHOD 14 years ago
Anoop Saldanha 97d8fc9cba All http_http_raw_header modified patterns now are DETECT_CONTENT and not DETECT_AL_HTTP_RAW_HEADER 14 years ago
Anoop Saldanha 97308674ee All http_http_header modified patterns now are DETECT_CONTENT and not DETECT_AL_HTTP_HEADER 14 years ago
Anoop Saldanha a5b46e727c All http_client_body modified patterns now are DETECT_CONTENT and not DETECT_AL_HTTP_CLIENT_BODY 14 years ago
Anoop Saldanha 4810ee9c5f All uricontent modified patterns now are DETECT_CONTENT and not DETECT_URICONTENT. Step towards unifying all content based patterns. Makes way for easier management of patterns 14 years ago
Anoop Saldanha e682796d03 feature #414 - support listing supported keywords. Remove support for dummy keywords __address__, __proto__, __port__. Remove support for recursive keyword and all references to it 14 years ago
Anoop Saldanha 35f1f7e8d9 unify payload detection engines + fix other bugs in pcre init 14 years ago
Anoop Saldanha 0677190960 rebase commit for hscd and hsmd patches 14 years ago
Anoop Saldanha 09313cf9bd Support http stat code detection engine, fast pattern(mpm engine included). Fix http stat code setup function. Fix pcre option for stat msg keyword. With this the pcre options for server_body is Q, for stat_msg is Y and for stat_code is S 14 years ago
Anoop Saldanha 2007c2711c Support http stat msg detection engine, fast pattern(mpm engine included). Fix http stat msg setup function. Fix pcre option for stat msg keyword 14 years ago
Victor Julien 9dc153c8f4 Fix path handling for including rule files on win32. 14 years ago
Anoop Saldanha 716afac5a2 fix debug messages that have references to the old mpm contexts 14 years ago
Anoop Saldanha 9a665e035b code cleanup over last 2 commits 14 years ago
Anoop Saldanha 419cdc8558 support splitting mpm ctxs based on direction v2 14 years ago
Victor Julien 28e15be526 Clean up default output. Use simpler output format for releases. 14 years ago
Victor Julien 6eeab37ab3 Add post-match list, move flowbits set, etc functions to it. Move flowint set, etc functions to it as well. 14 years ago
Victor Julien aac2d91bcc Set DROP flag for reject action so in addition to sending the rst, in IPS mode also drop the offending packet. 14 years ago
Victor Julien f084874998 Fix HTTP state and raw stream not being inspected at the same time. Adds an exception to transaction id handling for HTTP. 14 years ago
Anoop Saldanha 9c4954ff42 support http response raw header inspection + carry out hrhd mpm on both request/response headers + add unittests for the same 14 years ago
Anoop Saldanha 30247dce8c bug 389 - support http response header inspection + fix bug with stateful inspection for sigs that would have both request/response inpection 14 years ago
Victor Julien 21ee59e6f3 Add signature direction (flow:toserver/flow:toclient) as a signature flag. 14 years ago
Victor Julien d5402d33d4 Simplify detection loop. Inspect packet keywords before the state. 14 years ago
Anoop Saldanha eea5ab4a7a Support for app layer decoder events added + app_layer_event keyword added 14 years ago
Anoop Saldanha 517040c4af indentation fix 14 years ago
Anoop Saldanha 37b223645a fix detection engine for alert stability. Fix cases where we have multiple rules having same pattern. We should see good perf increase(~5%) with this change, now that we avoid unnecessary inspection" 14 years ago
Anoop Saldanha 92643f6110 introduce separate mpm ctxs for tcp/udp/other_protos 14 years ago
Victor Julien 18d79c4215 file store: respect flowbits and other keywords 
The filestore keyword until now flagged a file, tx or ssn for storage as soon
as the keyword was inspected. This happens before flowbits and some other
keywords, so files were stored that weren't supposed to.

This patch makes the filestore keyword fill an array in the detect engine
thread ctx. Then if the full signature matches, a post-match filestore
function makes the store final.
 14 years ago
Victor Julien 7adac3048d file-data: create initial keyword registration. 14 years ago
Victor Julien dd9da1a56f Merge all http mpm related signature flags into a single set: SIG_FLAG_MPM_HTTP and SIG_FLAG_MPM_HTTP_NEG. 14 years ago
Victor Julien 2650551192 Rename signature init flags to indicate they are init flags. 14 years ago
Victor Julien 291ddd95f2 Detection engine -- mpm 
Each signature is in one mpm ctx at max, but there were 3 separate
id's in use: packet, stream, http. Merged them all into one.

Could shrink the SignatureHeader structure with 8 bytes because of this,
should lead to better caching performance.
 14 years ago
Victor Julien 7db72bce75 Optimize detection engine prefiltering logic. 14 years ago
Victor Julien 89f83e714c Introduce http_server_body keyword. 
The http_server_body content modifier modifies the previous content to inspect
the normalized (dechunked, unzipped) http_server_body. The workings are similar
to http_client_body. Additionally, a new pcre flag was introduced "/S".

To facilitate this change the signature flags field was changed to be 64 bit.
 14 years ago
Anoop Saldanha b970273163 fix broken unittest 14 years ago
Anoop Saldanha d40fb5b933 Remove unnecessary flow NULL check 14 years ago
Anoop Saldanha cc7db6315c Move setting packet iponly flags from decode section to stream section 14 years ago
Victor Julien 117d51c965 Fix a compile warning when debug is enabled. 14 years ago
Victor Julien 042fd850fc Make sure we check the sgh for no magic and no store once per flow direction. 14 years ago
Victor Julien d59ca75e46 file extract: split toserver and toclient tracking 
Split toserver and toclient file tracking for the http state.
 14 years ago
Victor Julien 04ea70ccf7 file extract: pruning 
Add pruning of files in memory so we keep only memory what we really need.
Fix magic logic.
Reset file part of the de_state on receiving another file in the same tx.
 14 years ago
Victor Julien e1022ee5ae file-extraction: Disconnect file handling from flow and move into the app layer state. 14 years ago
Victor Julien 9b62ec65ab Make sure filemagic works properly regardless of filestore being in use for a flow. 14 years ago
Victor Julien 5945e652d6 Initial implementation of filemagic keyword. 14 years ago
Victor Julien 23e01d23d3 Implement filestore keyword, including a way for the stateful detection engine to conclude that a file will never have to be stored. 14 years ago
Pablo Rincon 6d60b3a747 filename and fileext keywords 14 years ago
Anoop Saldanha 9a6aef459e modify all relevant app layer API calls to accomodate passing parser local storage argument 14 years ago
Anoop Saldanha 2b356dadff Support for tos keyword added 14 years ago
Victor Julien 262a7300d7 flow: shrink Flow datatype 
Introduce a separate FlowAddress structure for holding the ipv4 or ipv6 address
that doesn't have the family in it like the Address structure. Instead, the
family is stored in the flow as a flag: FLOW_IPV4 and FLOW_IPV6.

Add macro's to check the family, copy the address, etc.

Update many unittests to reflect these changes. Introduce unittest helper
functions for creating and initializing a flow and freeing it again.

On 64 bit this shrinks the flow with 8 bytes.
 14 years ago
Victor Julien 06904c9024 App Layer cleanup 
Removal of per flow 'aldata' array. It contained a ptr for each ALPROTO. Instead now we have 2 ptrs in the flow: alparser and alstate.
Various cleanups and dead code removal from the app layer API.
Should safe 100+ bytes memory per flow on 64 bit.
Updated lots of unittests to reflect these changes.
 14 years ago
Victor Julien 3d845b6c77 Consider Windows new line chars as well when parsing rule files. Bug #374. 14 years ago
Victor Julien 8cc82c7241 Add -S commandline option that loads a rule file exclusively. Issue #338. 14 years ago
Victor Julien 2d16abcf8b Minor code cleanups fixing all GCC 4.6 compiler warnings for default, debug and unittests mode. 14 years ago
Anoop Saldanha 0acfcc206c fix unittests. fix replace unittests that allow alproto keywords with replace 14 years ago
Anoop Saldanha a0eec3d846 fix detection code that handles cases when we use recursion(from recursive keyword) 14 years ago
Anoop Saldanha 7433d92dd2 undo this commit - 
commit eff08f93d8
Author: Anoop Saldanha <poonaatsoc@gmail.com>
Date:   Thu Nov 3 14:31:24 2011 +0530

    update failing unittest to reflect the mpm design update

Fixed a bug in the mpm code that would make all the changes in the commit just undone wrong.
 14 years ago
Anoop Saldanha 1b1332fff0 fix mpm bug on running stream mpm for packets not added to stream mpm 14 years ago
Anoop Saldanha eff08f93d8 update failing unittest to reflect the mpm design update 14 years ago
Victor Julien af51493da2 Mpm update: Toss out signatures that mix pkt and stream/state. Update profiling code to track new mpm. 14 years ago
Anoop Saldanha 17f3f36d38 packet keywords only added for packet mpm. Rest in stream mpm. Update detection engine to handle the same 14 years ago
Anoop Saldanha c7b9d3fecb Remove broken dsize_sm in SigMatch used by dsize in detection engine 14 years ago
Eric Leblond 60a99915c1 doc: create http support group 
This patch create an httplayer group and adds related files to
it. It also fixes some typo in documentation string and format.
 14 years ago
Eric Leblond 6bf15bac31 Fix various packet access. 
The coccinelle based tests have detected invalid uses of access to
Packet data. This patch fixes the detected problems.
 14 years ago
Victor Julien 7ef34b7bcc Exlcude DSIZE LT case from setting the 'need payload' mask bit as it can include 0, which means no payload. 14 years ago
Victor Julien 2e2e80b812 Add packet alert flag to indicate a match happened (partly) in the app layer state. Make unified2 use this flag. 14 years ago
Victor Julien a1f68bf411 Fix detection engine informational message misrepresenting decoder only signatures. 14 years ago
Eric Leblond 3f75b10f79 Suppress useless code. 
This code was making a warning for some time now. This patch kills
it.
 14 years ago
Eric Leblond a85dc9b0e2 Add support for replace keyword. 
This patch adds support for the replace keyword. It is used with
content to change selected part of the payload. The major point
with this patch is that having a replace keyword made necessary
to avoid all stream level check because we need to access to the
could-be-modified packet payload.

One of the main difficulty is to handle complex signature. If there is
other content check, we must do the substitution when we're sure all
match are valid. The patch adds an attribute to the thread context
variable to be able to deal with recursivity of the match function.

Replace is only activated in IPS mode and apply only to raw match.
 14 years ago
Anoop Saldanha 8028392e9a fix mpm segv. Use sgh flags to check if the sgh has packet or stream mpm set or not 14 years ago
Victor Julien e13181496c ip-only: added support for matching on ports. 14 years ago
Victor Julien 751a77a9be Make sure stream/engine-event signatures are recognized as such. 14 years ago
Victor Julien 63f834d9a7 Add profiling to various HTTP buffer MPM calls. 14 years ago
Victor Julien 3693a7a9ee Profiling: add accounting for several detection phases. 14 years ago
Eric Leblond e3a6d8955e Introduce engine-event keyword 
This patch renames the 'decode-event' keyword to 'engine-event' and
keep it for backword compatibility of rulesets. All *DecodeEvent*
references in the code are replaced by EngineEvent version.
 14 years ago
Eric Leblond 2ac8755382 Rename detect-decode-event to detect-engine-event 
This patch does a simple renaming of detect-decode-event file to
the more global detect-engine-event name.
 14 years ago
Victor Julien 820b0ded82 Add per packet profiling. 
Per packet profiling uses tick based accounting. It has 2 outputs, a summary
and a csv file that contains per packet stats.

Stats per packet include:
 1) total ticks spent
 2) ticks spent per individual thread module
 3) "threading overhead" which is simply calculated by subtracting (2) of (1).

A number of changes were made to integrate the new code in a clean way:
a number of generic enums are now placed in tm-threads-common.h so we can
include them from any part of the engine.

Code depends on --enable-profiling just like the rule profiling code.

New yaml parameters:

profiling:
  # packet profiling
  packets:

    # Profiling can be disabled here, but it will still have a
    # performance impact if compiled in.
    enabled: yes
    filename: packet_stats.log
    append: yes

    # per packet csv output
    csv:

      # Output can be disabled here, but it will still have a
      # performance impact if compiled in.
      enabled: no
      filename: packet_stats.csv

Example output of summary stats:

IP ver   Proto   cnt        min      max          avg
------   -----   ------     ------   ----------   -------
 IPv4       6     19436      11448      5404365     32993
 IPv4     256         4      11511        49968     30575

Per Thread module stats:

Thread Module              IP ver   Proto   cnt        min      max          avg
------------------------   ------   -----   ------     ------   ----------   -------
TMM_DECODEPCAPFILE          IPv4       6     19434       1242        47889      1770
TMM_DETECT                  IPv4       6     19436       1107       137241      1504
TMM_ALERTFASTLOG            IPv4       6     19436         90         1323       155
TMM_ALERTUNIFIED2ALERT      IPv4       6     19436        108         1359       138
TMM_ALERTDEBUGLOG           IPv4       6     19436         90         1134       154
TMM_LOGHTTPLOG              IPv4       6     19436        414      5392089      7944
TMM_STREAMTCP               IPv4       6     19434        828      1299159     19438

The proto 256 is a counter for handling of pseudo/tunnel packets.

Example output of csv:

pcap_cnt,ipver,ipproto,total,TMM_DECODENFQ,TMM_VERDICTNFQ,TMM_RECEIVENFQ,TMM_RECEIVEPCAP,TMM_RECEIVEPCAPFILE,TMM_DECODEPCAP,TMM_DECODEPCAPFILE,TMM_RECEIVEPFRING,TMM_DECODEPFRING,TMM_DETECT,TMM_ALERTFASTLOG,TMM_ALERTFASTLOG4,TMM_ALERTFASTLOG6,TMM_ALERTUNIFIEDLOG,TMM_ALERTUNIFIEDALERT,TMM_ALERTUNIFIED2ALERT,TMM_ALERTPRELUDE,TMM_ALERTDEBUGLOG,TMM_ALERTSYSLOG,TMM_LOGDROPLOG,TMM_ALERTSYSLOG4,TMM_ALERTSYSLOG6,TMM_RESPONDREJECT,TMM_LOGHTTPLOG,TMM_LOGHTTPLOG4,TMM_LOGHTTPLOG6,TMM_PCAPLOG,TMM_STREAMTCP,TMM_DECODEIPFW,TMM_VERDICTIPFW,TMM_RECEIVEIPFW,TMM_RECEIVEERFFILE,TMM_DECODEERFFILE,TMM_RECEIVEERFDAG,TMM_DECODEERFDAG,threading
1,4,6,172008,0,0,0,0,0,0,47889,0,0,48582,1323,0,0,0,0,1359,0,1134,0,0,0,0,0,8028,0,0,0,49356,0,0,0,0,0,0,0,14337

First line of the file contains labels.

2 example gnuplot scripts added to plot the data.
 14 years ago
Victor Julien 1bd1a62526 Rename profile macro's and variables to reflect that they are for rule profiling. 14 years ago
Anoop Saldanha 7e5c52c80b add flowbits:set; only sigs to be treated as ip only 14 years ago
Victor Julien 16b41a5eff Use p->proto in detect to determine TCP/UDP/SCTP. 14 years ago
Victor Julien f7f037c1d1 Make sure stateful detection engine inspecting HTTP streams works well for to_client rules as well. 14 years ago
Anoop Saldanha 35f3eafa5e byte extract added to the engine. Detection support added for packet payload, uri and dce detection engines 14 years ago
Victor Julien 73efb4c70f Add a app layer state and stateful detection engine counter that makes sure the stateful inspection is only done when the state changes. 14 years ago
Victor Julien 0768ca9806 Fix SIMD mask checking on 64 bit systems. 14 years ago
Victor Julien 350215966b Fix signature mask bitorder. 14 years ago
Victor Julien aa822c0ac1 Always reset alert cnt and always increment det_ctx->pkts. 14 years ago
Victor Julien 1e0b050a54 Add more mask flags. 14 years ago
Victor Julien 4b52823ab6 Use 64 bit mask on 64-bit systems. 14 years ago
Victor Julien e5b6c0f518 Check 32 masks per run instead of 16 in the SIMD code. 14 years ago
Victor Julien 2dbfdd40af Clean up new SIMD mask checking code, improve non-SIMD checks. 14 years ago
Victor Julien b421019cef Match packet mask against 16 signature masks at once using SIMD instructions for SSE3 and up. 14 years ago
Victor Julien e1d4e16645 Simplify packet decoding macro's. 14 years ago
Victor Julien 75439863ed Shrink PacketAlerts structure so that Packet structure is a lot smaller. Reduce max events per packet from 256 to 15. 14 years ago
Victor Julien 6aa551c558 Small optimizations to IPV4 and TCP header parsing. 14 years ago
Anoop Saldanha 966119b6aa support for http_raw_uri keyword + mpm engine 15 years ago
Victor Julien 864c8718e1 Store matching stream msg (ptr) in packets alert structure so it's available to the output plugins. 15 years ago
Victor Julien 140eb4fde8 Fix decode-event keyword parsing. Fix code that indicates a signature is decode-event only. Add 'pkthdr' protocol as an alias for any/ip to be used by decode-event signatures. 15 years ago
Victor Julien e197f50727 Fix IP-Only unittests failing on Big Endian. 15 years ago
Eric Leblond 9beebf621a Add support for 'nfq_set_mark' keyword 
This patch introduces 'nfq_set_mark' which is new rules option. If a packet
matches a rule using nfq_set_mark in NFQ mode, it is marked with the mark/mask
specified in the option during the verdict.
It is thus possible to trigger different behaviour on the packet inside
Linux/Netfilter.
 15 years ago
Anoop Saldanha c105a739e9 support for ssl_state keyword added 15 years ago
Victor Julien b9fd978253 Fix compiler warnings in two unittests. 15 years ago
Eric Leblond 005dc599a6 detect.c: Fix usage of integer standing for protocol 
This patch fixes direct usage of integer to code protocol value.
 15 years ago
Eric Leblond 2c80f18dc9 detect: Add sctp detection and parsing. 
This patch adds the support of SCTP in signature subsystem.
 15 years ago
Gurvinder Singh 8f8b1212af support for ssl_version keyword 15 years ago
Victor Julien ec5b622553 Inspect all stream msgs at any time when running in stream-inline mode. Skip detection for packets flagged for dropping before detect. 15 years ago
Victor Julien 1d971b53a6 Update all unittests 15 years ago
Victor Julien f606621e8c Fix the pseudo packet having the wrong proto set, causing massive fp's. Flag packets to be part of the established phase of a tcp session, so we won't prematurely inspect the app layer state. 15 years ago
Victor Julien 24f071cabb Make sure http_cookie inspects all HTTP transactions. Clean up error messages. Get rid of unused code and dead comments. 15 years ago
Anoop Saldanha c9897a44a4 fast pattern support for http_cookie. Also support relative modifiers 15 years ago
Anoop Saldanha bbbedaf963 fast pattern support for http_method. Also support relative modifiers 15 years ago
Eric Leblond 49adc264bc Don't print message after SCMalloc failure. 
This patch generated via coccinelle is getting rid of logging
message after a SCMalloc failure. They were useless as SCMalloc
already displays a message.
 15 years ago
Eric Leblond 1db4aadd16 Supress usage of Packet declaration in tests. 
For convenience, a massive usage of 'Packet p;' declaration has
been done in the tests function. Although this was completely
legal, this is not possible anymore because of the new Packet
allocation structure. This massive patch modifies all suricata
files to use a SCMalloc allocated pointer to Packet instead.

This patch has been done using coccinelle (http://coccinelle.lip6.fr)
which is a semantic patching tool. This ensures that things like call
to SCFree() should have not been forget because the semantic patch
explicitly forces the call to SCFree(p) before each return. With this
patch all unittests are running fine with a small and a big default
packet size.
 15 years ago
Eric Leblond dd038c1906 Modify files to avoid direct pckt payload access 
This patch implements the needed modification of payload access
in a Packet structure to support the abstraction introduced by
the extended data system.
 15 years ago
Victor Julien 18b4e3380f Make mpm-algo use the mpm_table that has the actual mpm's registered. Clean up dead code. 15 years ago
Victor Julien 3f47eade6b Fix couple of cases where incorrect handling of keyword parsing errors would lead to access of uninitialized memory. Found by clang. 15 years ago
Victor Julien 3d60e9bfeb Clean up output. 15 years ago
Anoop Saldanha d241e51b32 adapt fast pattern engine analysis to reflect the new changes made to your mpm design 15 years ago
Anoop Saldanha c734cd1bdd make cuda mpm parameters configurable 15 years ago
Victor Julien 0e8e8e3728 Don't stop stream reassembly if protocol detection failed, only stop/prevent app layer parsing. 15 years ago
Victor Julien 39dea56a84 Remove flowbits as a mask prefilter as they are dynamic. Add a dynamic check. 15 years ago
Anoop Saldanha 8c21511c99 don't buffer raw headers. Retrieve them individually from htp_state during mpm stage and content valiadation stage 15 years ago
Victor Julien 55ca988222 Change locking of http_header, http_raw_header and http_client_body so that flow isn't accessed without lock anywhere. 15 years ago
Victor Julien 435d0fb327 Clean up signature flags creating room for merging flags and mpm_flags. Merge flags and mpm_flags. Move new mpm id's into signature header. Get rid of full signature access in signature prefiltering. 15 years ago
Victor Julien 169aa5581a Rename SIG_FLAG_AMATCH flag to SIG_FLAG_STATE_MATCH to better reflects its purpose. 15 years ago
Victor Julien d7b92d9bfe Consolidate several signature flags into one. 15 years ago
Victor Julien 2102a54c26 Cleanup and rearrange detection code slightly. 15 years ago
Anoop Saldanha 25588b6910 comment out hrhd flags that we were using previously. Also remove the de_mpm_ based flags inside detect.h used by uri|hcbd|hhd|hrhd mpms. indentation fix as well 15 years ago
Anoop Saldanha e50503e820 cleanup/remove dead code 15 years ago
Anoop Saldanha 93fa7ea828 modify detection engine to run hrhd mpm before building the match array 15 years ago
Anoop Saldanha b140ed1c9c modify detection engine to run hhd mpm before building the match array 15 years ago
Anoop Saldanha 4e273f2c8b modify detection engine to carry out hcbd mpm run before build match array if alproto is http and if sgh has atleast one sig with hcbd mpm set 15 years ago
Anoop Saldanha 72b0fcf419 modify detection engine to carry out uri mpm run before build match array if alproto is http and if sgh has atleast one sig with uri mpm set 15 years ago
Victor Julien 6a5d2cb40d Fix potential locking issue in out of memory conditions in the http_header, http_raw_header code. Fix other potential small issues in http_ code. 15 years ago
Anoop Saldanha 7ec0382774 support fast pattern for http raw header. Also support relative modifiers for http raw header 15 years ago
Victor Julien 0c806f70bb Fix --enable-debug compilation, just unittest with --enable-debug-validation enabled. 15 years ago
Victor Julien 1a32d9b5ec Fix printing unprintable characters in the engine-analysis fast_pattern mode. 15 years ago
Anoop Saldanha c61c68fd36 mpm and fast pattern support for http_header. Also support relative modifiers for http_header 15 years ago
Anoop Saldanha 778ec0939c make client body buffer limit configurable. Also some minor changes 15 years ago
Anoop Saldanha 0aa5cffb12 fast pattern support for http_client_body keyword added. Also mpm support for http_client_body added 15 years ago
Anoop Saldanha c227aeeacb remove support for skipping reinspecting fast pattern contents once again during packet payload inspection. Also make some changes to our detection engine 15 years ago
Anoop Saldanha bbd0c5056b store the content added for mpm inside Signature. also carry out an unconditional cleanup of packet pattern matcher pmq det_ctx->pmq 15 years ago
Anoop Saldanha 6df051321f fix fp when content is negated and also added to mpm 15 years ago
Anoop Saldanha 5c6a65dc58 support relative modifiers for http_client_body. Introduce body processing engine in detect-engine-hcbd.[ch] 15 years ago
Anoop Saldanha 3d2f81d978 replace all Signature->dmatch instances in the engine with Signature->sm_lists[DETECT_SM_LIST_DMATCH] 15 years ago
Anoop Saldanha a7353be20d replace all Signature->amatch instances in the engine with Signature->sm_lists[DETECT_SM_LIST_AMATCH] 15 years ago
Anoop Saldanha e0476242c6 replace all Signature->umatch instances in the engine with Signature->sm_lists[DETECT_SM_LIST_UMATCH] 15 years ago
Anoop Saldanha e54358a9e1 replace all Signature->pmatch instances in the engine with Signature->sm_lists[DETECT_SM_LIST_PMATCH] 15 years ago
Anoop Saldanha 82fd581b64 replace all sm lists (match, pmatch, dmatch, umatch, amatch, tmatch) with an array Signature->sm_lists[]. Replace all Signature->match instances in the engine with Signature->sm_lists[DETECT_SM_LIST_MATCH] 15 years ago
Victor Julien 001f91056e Add http_raw_header as an alias to the http_header keyword as that actually inspects the raw headers (see issue #243). Closes issue #242. 15 years ago
Gurvinder Singh b7da115e6d support for http_stat_code keyword has been added to detection module 15 years ago
Gurvinder Singh 1deae70cf7 added http_stat_msg keyword support for detection module 15 years ago
Anoop Saldanha 2cdb5be391 Print out file name for fast_pattern engine_analysis. Also add some info logs 15 years ago
Anoop Saldanha a2d04a94b5 selecting auto for detect-engine.sgh_mpm_context now uses single if the mpm is ac, full otherwise 15 years ago
Anoop Saldanha 174048544d fix hash generation in b2g and ac addpattern. Brings down the no of patterns added from close to a million to a couple of thousands 15 years ago
Anoop Saldanha 0ef684705c support single mpm context distribution across sghs in staging. Also see to it that ac works fine with this setup 15 years ago
Anoop Saldanha a85fa6b792 support for fast_pattern only and fast_pattern:offset,length. Also support the new option for engine-analysis 15 years ago
Anoop Saldanha 0d741b9a55 fix for bug 227. For negated contents that have been added to mpm we might have pmq.pattern_id_array_cnt as 0. We can't ignore inspecting sigs if this is 0, in case the content added is negated 15 years ago
Victor Julien cbd4c298ed Initial version of a new bitmask based signature pre-filtering method. 15 years ago
Victor Julien 94898a91cc Reorganize SigMatchSignatures. 15 years ago
Victor Julien fc248ca7a1 Many small performance updates. 15 years ago
Pablo Rincon 5c43db85ce Drop streams on inline mode when a drop rule match from a reassembled stream and/or app layer inspection 15 years ago
Anoop Saldanha f094523eb1 clang fix - some minor fixes for unittests 15 years ago
Pablo Rincon 9d7baa7a9f Adding ssh app layer module with two new keywords: ssh.protoversion and ssh.softwareversion 15 years ago
Victor Julien 04d3832d8f Remove ports check and fix small typo. 15 years ago
Victor Julien a492518e7a Properly detect detect-event-only sigs. 15 years ago
Pablo Rincon 21d79b05ad Fix for bug221 (avoid considering sig as "decoder event only" if ports are specified). Now the sig gets grouped to get a sgh at SigMatchSignatures 15 years ago
Victor Julien 6299fbfb0f Fix stream msg content inspection not inspecting the correct id. 15 years ago
Victor Julien 1071a53210 Fix unittests after ip_proto keyword change. 15 years ago
Pablo Rincon 70bda6506d Fix for bug 180 (check proto specified at the IP hdr) 15 years ago
Victor Julien 7acb97da9d Use same mpm prepare procedure for uricontent as for normal content. More cleanups. 15 years ago
Victor Julien 9ba11dbfbd Clean up detection engine mpm initialization phase. 15 years ago
Victor Julien 0d008c8135 Change stateful detection engine to be able to start the stateful detection separate from other sigs. Fixes bugs #213, #214, #215. 15 years ago
Victor Julien 689d05b10b Add missing protocol check in the sig matching process. This prevents FP's such as the one reported in bug #209. 15 years ago
Victor Julien 0219b767b8 Fix a content pattern matching bug related to signature grouping and mpm_ctx sharing. In certain conditions (signature combinations) the mpm_stream_ctx (the ctx that handles stream pattern scanning) wasn't properly setup. 15 years ago
Victor Julien 102092a89c Make signature address matching more cache efficient. 15 years ago