aiden
/
suricata
mirror of https://github.com/OISF/suricata
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
4,477 Commits
7 Branches
161 Tags
210 MiB
master
main-7.0.x
master-6.0.x
master-5.0.x
master-4.1.x
master-4.0.x
master-3.2.x
suricata-8.0.0
suricata-7.0.11
suricata-8.0.0-rc1
suricata-8.0.0-beta1
suricata-7.0.10
suricata-7.0.9
suricata-7.0.8
suricata-7.0.7
suricata-7.0.6
suricata-6.0.20
suricata-7.0.5
suricata-6.0.19
suricata-6.0.18
suricata-7.0.4
suricata-6.0.17
suricata-7.0.3
suricata-6.0.16
suricata-7.0.2
suricata-6.0.15
suricata-7.0.1
suricata-6.0.14
suricata-7.0.0
suricata-7.0.0-rc2
suricata-6.0.13
suricata-6.0.12
suricata-6.0.11
suricata-7.0.0-rc1
suricata-6.0.10
suricata-6.0.9
suricata-7.0.0-beta1
suricata-6.0.8
suricata-6.0.7
suricata-6.0.6
suricata-5.0.10
suricata-6.0.5
suricata-5.0.9
suricata-6.0.4
suricata-5.0.8
suricata-6.0.3
suricata-5.0.7
suricata-6.0.2
suricata-5.0.6
suricata-6.0.1
suricata-5.0.5
suricata-4.1.10
suricata-4.1.9
suricata-5.0.4
suricata-6.0.0
suricata-6.0.0-rc1
suricata-6.0.0-beta1
suricata-5.0.3
suricata-4.1.8
suricata-4.1.7
suricata-5.0.2
suricata-4.1.6
suricata-5.0.1
suricata-5.0.0
suricata-5.0.0-rc1
suricata-4.1.5
suricata-5.0.0-beta1
suricata-4.1.4
suricata-4.1.3
suricata-4.0.7
suricata-4.1.2
suricata-4.1.1
suricata-4.1.0
suricata-4.0.6
suricata-4.1.0-rc2
suricata-4.1.0-rc1
suricata-4.0.5
suricata-4.1.0-beta1
suricata-4.0.4
suricata-4.0.3
suricata-4.0.2
suricata-3.2.5
suricata-4.0.1
suricata-3.2.4
suricata-4.0.0
suricata-4.0.0-rc2
suricata-3.2.3
suricata-4.0.0-rc1
suricata-4.0.0-beta1
suricata-3.2.2
suricata-3.1.4
suricata-3.2.1
suricata-3.2
suricata-3.2RC1
suricata-3.1.3
suricata-3.2beta1
suricata-3.1.2
suricata-3.1.1
suricata-3.1
suricata-3.0.2
suricata-3.1RC1
suricata-3.0.1
suricata-3.0.1RC1
suricata-3.0
suricata-2.0.11
suricata-3.0RC3
suricata-3.0RC2
suricata-3.0RC1
suricata-2.0.10
suricata-2.0.9
suricata-2.1beta4
suricata-2.0.8
suricata-2.0.7
suricata-2.1beta3
suricata-2.0.6
suricata-2.0.5
suricata-2.1beta2
suricata-2.0.4
suricata-2.1beta1
suricata-2.0.3
suricata-2.0.2
suricata-2.0.1
suricata-2.0.1rc1
suricata-2.0
suricata-2.0rc3
suricata-2.0rc2
suricata-2.0rc1
suricata-2.0beta2
suricata-1.4.7
suricata-1.4.6
suricata-1.4.5
suricata-2.0beta1
suricata-1.4.4
suricata-1.4.3
suricata-1.4.2
suricata-1.4.1
suricata-1.3.6
suricata-1.4
suricata-1.3.5
suricata-1.4rc1
suricata-1.3.4
suricata-1.4beta3
suricata-1.3.3
suricata-1.4beta2
suricata-1.3.2
suricata-1.4beta1
suricata-1.3.1
suricata-1.3
suricata-1.3rc1
suricata-1.3beta2
suricata-1.3beta1
suricata-1.2.1
suricata-1.2
suricata-1.2rc1
suricata-1.2beta1
suricata-1.1.1
suricata-0.8.2
suricata-1.0.0
suricata-1.0.1
suricata-1.0.2
suricata-1.0.3
suricata-1.0.4
suricata-1.0.5
suricata-1.1
suricata-1.1beta1
suricata-1.1beta2
suricata-1.1beta3
suricata-1.1rc1
Branches Tags
${ item.name }
Create tag ${ searchTerm }
Create branch ${ searchTerm }
from 'cc61a190ce'
${ noResults }
Commit Graph

49 Commits (cc61a190ce882a3696feaed76bec944c82b5e32b)

Author SHA1 Message Date
Ken Steele e05034f5dd New Multi-pattern matcher, ac-tile, optimized for Tile architecture. 
Aho-Corasick mpm optimized for Tilera Tile-Gx architecture. Based on the
util-mpm-ac.c code base. The primary optimizations are:
1) Matching function used Tilera specific instructions.
2) Alphabet compression to reduce delta table size to increase cache
   utilization  and performance.

The basic observation is that not all 256 ASCII characters are used by
the set of multiple patterns in a group for which a DFA is
created. The first reason is that Suricata's pattern matching is
case-insensitive, so all uppercase characters are converted to
lowercase, leaving a hole of 26 characters in the
alphabet. Previously, this hole was simply left in the middle of the
alphabet and thus in the generated Next State (delta) tables.

A new, smaller, alphabet is created using a translation table of 256
bytes per mpm group. Previously, there was one global translation
table for converting upper case to lowercase.

Additional, unused characters are found by creating a histogram of all
the characters in all the patterns. Then all the characters with zero
counts are mapped to one character (0) in the new alphabet. Since
These characters appear in no pattern, they can all be mapped to a
single character and still result in the same matches being
found. Zero was chosen for the value in the new alphabet since this
"character" is more likely to appear in the input. The unused
character always results in the next state being state zero, but that
fact is not currently used by the code, since special casing takes
additional instructions.

The characters that do appear in some pattern are mapped to
consecutive characters in the new alphabet, starting at 1. This
results in a dense packing of next state values in the delta tables
and additionally can allow for a smaller number of columns in that
table, thus using less memory and better packing into the cache. The
size of the new alphabet is the number of used characters plus 1 for
the unused catch-all character.

The alphabet size is rounded up to the next larger power-of-2 so that
multiplication by the alphabet size can be done with a shift.  It
might be possible to use a multiply instruction, so that the exact
alphabet size could be used, which would further reduce the size of
the delta tables, increase cache density and not require the
specialized search functions. The multiply would likely add 1 cycle to
the inner search loop.

Since the multiply by alphabet-size is cleverly merged with a mask
instruction (in the SINDEX macro), specialized versions of the
SCACSearch function are generated for alphabet sizes 256, 128, 64, 32
and 16.  This is done by including the file util-mpm-ac-small.c
multiple times with a redefined SINDEX macro. A function pointer is
then stored in the mpm context for the search function. For alpha bit
sizes of 8 or smaller, the number of states usually small, so the DFA
is already very small, so there is little difference using the 16
state search function.

The SCACSearch function is also specialized by the size of the value
stored in the next state (delta) tables, either 16-bits or 32-bits.
This removes a conditional inside the Search function. That
conditional is only called once, but doesn't hurt to remove
it. 16-bits are used for up to 32K states, with the sign bit set for
states with matches.

Future optimization:

The state-has-match values is only needed per state, not per next
state, so checking the next-state sign bit could be replaced with
reading a different value, at the cost of an additional load, but
increasing the 16-bit next state span to 64K.

Since the order of the characters in the new alphabet doesn't matter,
the new alphabet could be sorted by the frequency of the characters in
the expected input stream for that multi-pattern matcher. This would
group more frequent characters into the same cache lines, thus
increasing the probability of reusing a cache-line.

All the next state values for each state live in their own set of
cache-lines. With power-of-two sizes alphabets, these don't overlap.
So either 32 or 16 character's next states are loaded in each cache
line load. If the alphabet size is not an exact power-of-2, then the
last cache-line is not completely full and up to 31*2 bytes of that
line could be wasted per state.

The next state table could be transposed, so that all the next states
for a specific character are stored sequentially, this could be better
if some characters, for example the unused character, are much more
frequent.
 12 years ago
Anoop Saldanha cdaa13012a fix for #882. 
Refactor the code that initializes the cuda mpm environment.
 12 years ago
Anoop Saldanha 3c2ddf04c1 Update mpm init ctx to not accept the final cuda_rc_module argument. 
It was a part of our older architecture and is no longer used.
 12 years ago
Anoop Saldanha 17c763f855 Version 1 of AC Cuda. 12 years ago
Anoop Saldanha b787da5643 Remove all cuda related code in the engine except for the cuda api wrappers 12 years ago
Anoop Saldanha f4ce9011d2 make mpm ctx container de_ctx specific. Also introduce global variable in mpm_ctx. this is a workaround for cleaning non global mpm_ctx's since we now don't supply the de_ctx around the detection engine API 13 years ago
Anoop Saldanha 419cdc8558 support splitting mpm ctxs based on direction v2 14 years ago
Anoop Saldanha 199288309d Support for new MPM ac-bs added 14 years ago
Anoop Saldanha 1389cf6913 update cuda mpm to support per proto mpm contexts. Fix faulty stream mpm usage of cuda 14 years ago
Martin Beyer b1c577f829 cuda streams support in b2g-cuda MPM 15 years ago
Martin Beyer 621815ded0 cuda-packet-batcher timeout supports float values 15 years ago
Anoop Saldanha c734cd1bdd make cuda mpm parameters configurable 15 years ago
Anoop Saldanha 3c73854d2d completely remove populate_mpm_flags. Some indentation changes. Also disable support to avoid double checks inside payload inspection for patterns added to mpm. Also add support to MpmFactory to reclaim a mpm_ctx 15 years ago
Victor Julien 344ea14695 Change mpm hash_size config setting highest to higher as highest wasn't the... highest. Max was higher. Leaving highest as an alias to higher for backwards compatibility. 15 years ago
Anoop Saldanha 0ef684705c support single mpm context distribution across sghs in staging. Also see to it that ac works fine with this setup 15 years ago
Anoop Saldanha 658ff5753d aho-corasick for the cpu. We have 2 versions of ac. The first MPM_AC uses the delta table and the secone one MPM_AC_GFBS uses the goto-failure table 15 years ago
Victor Julien 87f88867f4 Further improve B2gc. Add B2gm. Improve memory layout. 15 years ago
Victor Julien 9dfbab42f8 WIP B2gc 15 years ago
Victor Julien 31261e7583 Improve B2g performance by merging pattern array and hash. 15 years ago
Victor Julien a0c1209a44 Inspect the reassembled stream together with the packet payload in the same direction. 15 years ago
William Metcalf 2eef905c07 GPL and Copyright header updates. 15 years ago
Victor Julien e27cefa6f7 Complete conversion of pattern id mpm storage vs sig id storage. 15 years ago
Victor Julien 7a427ec7f4 Switch to pattern id based results checking in the mpm. Move app layer proto detection towards a more signature based approach. 15 years ago
Pablo Rincon e18e2ec998 Changing threshold logic 15 years ago
William Metcalf ce01927515 Import of GPLv2 Header 050410 15 years ago
Pablo Rincon d94d929b04 Moving inline functions to the .h files, so gcc can inline them correctly 15 years ago
Victor Julien ec47f840f3 Remove more scan references. 16 years ago
Victor Julien f0d68b633e Remove nosearch flag from pattern api and add a generic bitwise flags field. 16 years ago
Victor Julien 1e01fd613c Remove all references to the scan phase from the pattern matchers and it's api. 16 years ago
Victor Julien dd846c9b0e Remove all search code from the pattern matchers, cleanup mpm api, remove unused http code, more cleanups. 16 years ago
Anoop Saldanha 41e6735b92 mpm b2g cuda support added 16 years ago
Anoop Saldanha 84df26d3fd cuda interface 16 years ago
Gurvinder Singh 999a200bc9 pattern matcher options support 16 years ago
Pablo Rincon f233f9fa31 Adding detect_content chunks handling for max_pattern_length and unittests. Updating modifiers to use it. 16 years ago
Victor Julien 4f2164677a - Fix pattern matchers b2g and b3g not being able to deal with a single pattern of the max pattern length (32 bytes by default). 
- Fix the setting of the correct pattern matcher when it was set in the detection ctx.
- Add tests for the fixes.
 16 years ago
Victor Julien 3c7a038477 Further memory cleanups. Split out init only vars out of the sig group head. 16 years ago
Victor Julien 2d0e9658f8 Speed up per sgh content maxlen calc. Remove mpm ptrs from mpm ctx. Add unittests testing the detection engine internals. 16 years ago
Victor Julien 7a7bb7a390 Get rid of global mpm_ctx. 16 years ago
Victor Julien b26b5aa462 More cleanups. 16 years ago
Victor Julien 4369816cdd Improvements to content keyword memory handling. 
First version of a simple pattern based L7 proto detection engine. Currently just works by matching a single pattern in the initial data. Implemented HTTP, SSL, MSN, JABBER, SMTP and a few more.

Couple of pattern matcher cleanups.
 16 years ago
Brian Rectanus fa5939ca91 64 bit cleanup part2 16 years ago
Victor Julien 689bbfdc45 Rename all structure definitions in the "typedef struct _SomeStruct" format to "typedef struct SomeStruct_" to make the Doxygen output more useful. 
Remove the Trie multi pattern matcher code. It wasn't used anymore.
 16 years ago
Victor Julien 657be002d1 Big detection engine update: scan improvements, b2g/b3g updates, bloom fixes, iponly detection implementation, dsize/flow grouping. 16 years ago
Victor Julien b2eb954099 Add b3g 3gram BNDM pattern matcher. Fix multi queue nfq initialization. Improve speed of b2g and wumanber. 16 years ago
Victor Julien 1c0ad1d415 Add implementation of the Simple BNDM 2gram pattern matcher algorithm. 16 years ago
Victor Julien efb10fc0d6 big update 16 years ago
Victor Julien 54ffe2053e Large detection engine update. 16 years ago
Victor Julien dc224cb2d2 Large update containing the first step to making the detection engine use rule groups. Address based rule groups are now implemented. 16 years ago
Victor Julien bab4b62376 Initial add of the files. 16 years ago