Commit Graph

47 Commits (4dff903b356bd155896eab516cda5bccd63b8a4f)

Author SHA1 Message Date
Victor Julien 4dff903b35 detect: introduce pkt mpm engines
Instead of the hardcode L4 matching in MPM that was recently introduced,
add an API similar to the AppLayer MPM and inspect engines.

Share part of the registration code with the AppLayer.

Implement for the tcp.hdr and udp.hdr keywords.
6 years ago
Victor Julien 429ca858dc rust/gen: turn *mut*const T into const T ** 6 years ago
Victor Julien 32fb7d773a detect/content-inspect: turn void arg into Packet
Replace the 'void *data' argument by a 'Packet *p' as this was
the only user left of the data pointer.
6 years ago
Jason Ish 67b2692d34 dns: remove as much C DNS code as possible
As some of the C code is still used it can't all be removed.

Redmine issue:
https://redmine.openinfosecfoundation.org/issues/2850
6 years ago
Victor Julien 6974eb3116 detect/dns: add dns.query for dns_query keyword
Improve error checking.

Part of #2283.
6 years ago
Victor Julien 3eec088d31 detect/parse: error out on unused sticky buffers 7 years ago
Victor Julien 0b3220a0df detect: improve inspect buffer handling
Fix and Optimize cleanup. For the simple single inspect buffer optimize
the cleanup by keeping track of the actually used buffers. This avoid
looping over unused buffers.

Fix the case of cleaning not being done after a tx if the next tx is
also inspected in the context of the same packet.

Fix cleanup of the multi-inspect buffers. Optimize in 2 ways. First
like with single keep track of which multi-inspect buffers have been
used. Second, keep a max of the buffers within a multi-inspect buffer.
Use this max to limit (nested) looping.
7 years ago
Victor Julien d14e51a4aa detect/content: pass START/END flags to inspection 8 years ago
Victor Julien 82ffba20f4 detect/dns_query: move to API v2. Supports transforms. 8 years ago
Jason Ish 922a27ed97 detect-dns-query: use unit test macros 8 years ago
Victor Julien ab1200fbd7 compiler: more strict compiler warnings
Set flags by default:

    -Wmissing-prototypes
    -Wmissing-declarations
    -Wstrict-prototypes
    -Wwrite-strings
    -Wcast-align
    -Wbad-function-cast
    -Wformat-security
    -Wno-format-nonliteral
    -Wmissing-format-attribute
    -funsigned-char

Fix minor compiler warnings for these new flags on gcc and clang.
8 years ago
Victor Julien d304be5bc3 detect: register progress in inspect engines
Register required progress so we can stop inspecting as soon
as the progress isn't far enough yet.
8 years ago
Victor Julien dfac5276b8 detect: remove unused SIGMATCH_PAYLOAD flag 9 years ago
Victor Julien 775e182531 detect: remove AppLayerMatch API call 9 years ago
Victor Julien 8bd1422948 detect: detect engine registration cleanup 9 years ago
Victor Julien d9b3ae6cd6 dns: use dynamic buffers 9 years ago
Victor Julien f370e88135 detect: move init only Signature members to init_data 9 years ago
Sascha Steinbiss e6044aaf1c mpm/spm: check for SSSE3 and enable/disable HS
The new Hyperscan 4.4 API provides a function to check for SSSE3
presence at runtime. This allows us to fall back to non-Hyperscan
matchers on systems without SSSE3 even when the suricata executable
is built with Hyperscan support. Addresses Redmine issue #2010.

Signed-off-by: Sascha Steinbiss <sascha@steinbiss.name>
Tested-by: Arturo Borrero Gonzalez <arturo@debian.org>
9 years ago
Victor Julien 2db094ab7a dns detect: register inspect engine from keyword 9 years ago
Victor Julien 960461f4db fast_pattern: register app layer mpms automatically
Allow for duplicate registrations for the same list. After the first
registration new calls will be ignored.
9 years ago
Victor Julien 57ae3c43e5 dns_query: register mpm from keyword 9 years ago
Giuseppe Longo 675fa56497 app-layer: add ThreadVars to AppLayerParserParse
To be able to add a transaction counter we will need a ThreadVars
in the AppLayerParserParse function.
This function is massively used in unittests
and this result in an long commit.
9 years ago
Victor Julien 6530c3d0d8 unittests: replace SCMutex* calls by FLOWLOCK_* 9 years ago
Victor Julien 9030e89c94 detect: don't set alproto while registering keyword
The field is not used except for some printing, and is wrong for
many keywords.
9 years ago
maxtors 9d3fd82849 Removed duplicate include statements. 9 years ago
Jason Ish 796dd5223b tests: no longer necessary to provide successful return code
1 pass, 0 is fail.
9 years ago
Victor Julien 6bb2b001a3 mpm: cleanup: move mpm funcs into buffer specific files 9 years ago
Victor Julien 2c8e8c2516 dns: rename type so it's purpose is more clear 10 years ago
Victor Julien 02529b13a8 rule parser: set flag for optionless keywords
If a keyword doesn't have an argument, it should set the SIGMATCH_NOOPT
flag so the parser knows.
11 years ago
Ken Steele 8f1d75039a Enforce function coding standard
Functions should be defined as:

int foo(void)
{
}

Rather than:
int food(void) {
}

All functions where changed by a script to match this standard.
11 years ago
Victor Julien a77b9b36e5 app-layer: parser cleanup
Use f->protomap instead of calling FlowGetProtoMapping. Don't use
TcpSession *ssn ptr for anything other than TCP
12 years ago
Victor Julien 8dbf7a0d78 Update tests to use AppLayerParserThreadCtx ptr instead of void. Fix a few bugs uncovered by this. 12 years ago
Victor Julien fdefb65be4 app-layer: rename AppLayerThreadCtx funcs
AppLayerParserGetCtxThread -> AppLayerParserThreadCtxAlloc
AppLayerParserDestroyCtxThread -> AppLayerParserThreadCtxFree
12 years ago
Anoop Saldanha 429c6388f6 App layer API rewritten. The main files in question are:
app-layer.[ch], app-layer-detect-proto.[ch] and app-layer-parser.[ch].

Things addressed in this commit:
- Brings out a proper separation between protocol detection phase and the
  parser phase.
- The dns app layer now is registered such that we don't use "dnstcp" and
  "dnsudp" in the rules.  A user who previously wrote a rule like this -

  "alert dnstcp....." or
  "alert dnsudp....."

  would now have to use,

  alert dns (ipproto:tcp;) or
  alert udp (app-layer-protocol:dns;) or
  alert ip (ipproto:udp; app-layer-protocol:dns;)

  The same rules extend to other another such protocol, dcerpc.
- The app layer parser api now takes in the ipproto while registering
  callbacks.
- The app inspection/detection engine also takes an ipproto.
- All app layer parser functions now take direction as STREAM_TOSERVER or
  STREAM_TOCLIENT, as opposed to 0 or 1, which was taken by some of the
  functions.
- FlowInitialize() and FlowRecycle() now resets proto to 0.  This is
  needed by unittests, which would try to clean the flow, and that would
  call the api, AppLayerParserCleanupParserState(), which would try to
  clean the app state, but the app layer now needs an ipproto to figure
  out which api to internally call to clean the state, and if the ipproto
  is 0, it would return without trying to clean the state.
- A lot of unittests are now updated where if they are using a flow and
  they need to use the app layer, we would set a flow ipproto.
- The "app-layer" section in the yaml conf has also been updated as well.
12 years ago
Anoop Saldanha d0c5f51293 Update rule engine relationship with regard to setting ip protocol between specifying protocol after action, ip_proto and app-layer-protocol.
Now we can specify alproto, ip_proto combinations this way

alert dns (ip_proto:[tcp/udp];)
alert ip (app-layer-protocol:dns;)
alert ip (app-layer-protocol:dns; ip_proto:tcp;)
alert tcp (app-layer-protocol:dns:)

so on.  Neater than using dnstcp/dnsudp.

This is related to feature #424.
12 years ago
Anoop Saldanha f592c481dc Introduce a separate inspection engine for app events. 12 years ago
Anoop Saldanha 6f8cfd999f Allow detection ports for alproto to be specified via the conf file.
To understand the option have a look at the option

app-layer.protocols.tls.detection-ports
12 years ago
Anoop Saldanha ddde572fba Introduce new options into the conf file to enable/disable -
1. Proto detection
2. Parsers

For app layer protocols.

libhtp has now been moved to the section under app-layer.protocols.http,
but we still provide backward compatibility with older conf files.
12 years ago
Ken Steele e05034f5dd New Multi-pattern matcher, ac-tile, optimized for Tile architecture.
Aho-Corasick mpm optimized for Tilera Tile-Gx architecture. Based on the
util-mpm-ac.c code base. The primary optimizations are:
1) Matching function used Tilera specific instructions.
2) Alphabet compression to reduce delta table size to increase cache
   utilization  and performance.

The basic observation is that not all 256 ASCII characters are used by
the set of multiple patterns in a group for which a DFA is
created. The first reason is that Suricata's pattern matching is
case-insensitive, so all uppercase characters are converted to
lowercase, leaving a hole of 26 characters in the
alphabet. Previously, this hole was simply left in the middle of the
alphabet and thus in the generated Next State (delta) tables.

A new, smaller, alphabet is created using a translation table of 256
bytes per mpm group. Previously, there was one global translation
table for converting upper case to lowercase.

Additional, unused characters are found by creating a histogram of all
the characters in all the patterns. Then all the characters with zero
counts are mapped to one character (0) in the new alphabet. Since
These characters appear in no pattern, they can all be mapped to a
single character and still result in the same matches being
found. Zero was chosen for the value in the new alphabet since this
"character" is more likely to appear in the input. The unused
character always results in the next state being state zero, but that
fact is not currently used by the code, since special casing takes
additional instructions.

The characters that do appear in some pattern are mapped to
consecutive characters in the new alphabet, starting at 1. This
results in a dense packing of next state values in the delta tables
and additionally can allow for a smaller number of columns in that
table, thus using less memory and better packing into the cache. The
size of the new alphabet is the number of used characters plus 1 for
the unused catch-all character.

The alphabet size is rounded up to the next larger power-of-2 so that
multiplication by the alphabet size can be done with a shift.  It
might be possible to use a multiply instruction, so that the exact
alphabet size could be used, which would further reduce the size of
the delta tables, increase cache density and not require the
specialized search functions. The multiply would likely add 1 cycle to
the inner search loop.

Since the multiply by alphabet-size is cleverly merged with a mask
instruction (in the SINDEX macro), specialized versions of the
SCACSearch function are generated for alphabet sizes 256, 128, 64, 32
and 16.  This is done by including the file util-mpm-ac-small.c
multiple times with a redefined SINDEX macro. A function pointer is
then stored in the mpm context for the search function. For alpha bit
sizes of 8 or smaller, the number of states usually small, so the DFA
is already very small, so there is little difference using the 16
state search function.

The SCACSearch function is also specialized by the size of the value
stored in the next state (delta) tables, either 16-bits or 32-bits.
This removes a conditional inside the Search function. That
conditional is only called once, but doesn't hurt to remove
it. 16-bits are used for up to 32K states, with the sign bit set for
states with matches.

Future optimization:

The state-has-match values is only needed per state, not per next
state, so checking the next-state sign bit could be replaced with
reading a different value, at the cost of an additional load, but
increasing the 16-bit next state span to 64K.

Since the order of the characters in the new alphabet doesn't matter,
the new alphabet could be sorted by the frequency of the characters in
the expected input stream for that multi-pattern matcher. This would
group more frequent characters into the same cache lines, thus
increasing the probability of reusing a cache-line.

All the next state values for each state live in their own set of
cache-lines. With power-of-two sizes alphabets, these don't overlap.
So either 32 or 16 character's next states are loaded in each cache
line load. If the alphabet size is not an exact power-of-2, then the
last cache-line is not completely full and up to 31*2 bytes of that
line could be wasted per state.

The next state table could be transposed, so that all the next states
for a specific character are stored sequentially, this could be better
if some characters, for example the unused character, are much more
frequent.
12 years ago
Eric Leblond cd3e32ce19 unittests: some functions needs a flow lock.
In debug validation mode, it is required to call application layer
parsing and other functions with a lock on flow. This patch updates
the code to do so.
12 years ago
Victor Julien f353fb630c DNS: convert dns_query to sticky buffer 12 years ago
Victor Julien 72e35efbc6 Reset app layer events when we start inspecting a new TX 12 years ago
Victor Julien 28a6c1d9f8 DNS: add test for app layer event match 12 years ago
Victor Julien 43ba5a677e DNS: enable mpm/fast_pattern support for dns_query 12 years ago
Victor Julien 4817e1305f DNS: add /F modifier to pcre to inspect DNS query name 12 years ago
Victor Julien e567e12230 DNS: add unittests for UDP and TCP for dns_query keyword 12 years ago
Victor Julien f10dd603ff DNS: adding dns_request content modifier 12 years ago