Commit Graph

405 Commits (1b1332fff03839268ed9c633d31692acbaa89f47)

Author SHA1 Message Date
Anoop Saldanha 1b1332fff0 fix mpm bug on running stream mpm for packets not added to stream mpm 15 years ago
Anoop Saldanha eff08f93d8 update failing unittest to reflect the mpm design update 15 years ago
Victor Julien af51493da2 Mpm update: Toss out signatures that mix pkt and stream/state. Update profiling code to track new mpm. 15 years ago
Anoop Saldanha 17f3f36d38 packet keywords only added for packet mpm. Rest in stream mpm. Update detection engine to handle the same 15 years ago
Anoop Saldanha c7b9d3fecb Remove broken dsize_sm in SigMatch used by dsize in detection engine 15 years ago
Eric Leblond 60a99915c1 doc: create http support group
This patch create an httplayer group and adds related files to
it. It also fixes some typo in documentation string and format.
15 years ago
Eric Leblond 6bf15bac31 Fix various packet access.
The coccinelle based tests have detected invalid uses of access to
Packet data. This patch fixes the detected problems.
15 years ago
Victor Julien 7ef34b7bcc Exlcude DSIZE LT case from setting the 'need payload' mask bit as it can include 0, which means no payload. 15 years ago
Victor Julien 2e2e80b812 Add packet alert flag to indicate a match happened (partly) in the app layer state. Make unified2 use this flag. 15 years ago
Victor Julien a1f68bf411 Fix detection engine informational message misrepresenting decoder only signatures. 15 years ago
Eric Leblond 3f75b10f79 Suppress useless code.
This code was making a warning for some time now. This patch kills
it.
15 years ago
Eric Leblond a85dc9b0e2 Add support for replace keyword.
This patch adds support for the replace keyword. It is used with
content to change selected part of the payload. The major point
with this patch is that having a replace keyword made necessary
to avoid all stream level check because we need to access to the
could-be-modified packet payload.

One of the main difficulty is to handle complex signature. If there is
other content check, we must do the substitution when we're sure all
match are valid. The patch adds an attribute to the thread context
variable to be able to deal with recursivity of the match function.

Replace is only activated in IPS mode and apply only to raw match.
15 years ago
Anoop Saldanha 8028392e9a fix mpm segv. Use sgh flags to check if the sgh has packet or stream mpm set or not 15 years ago
Victor Julien e13181496c ip-only: added support for matching on ports. 15 years ago
Victor Julien 751a77a9be Make sure stream/engine-event signatures are recognized as such. 15 years ago
Victor Julien 63f834d9a7 Add profiling to various HTTP buffer MPM calls. 15 years ago
Victor Julien 3693a7a9ee Profiling: add accounting for several detection phases. 15 years ago
Eric Leblond e3a6d8955e Introduce engine-event keyword
This patch renames the 'decode-event' keyword to 'engine-event' and
keep it for backword compatibility of rulesets. All *DecodeEvent*
references in the code are replaced by EngineEvent version.
15 years ago
Eric Leblond 2ac8755382 Rename detect-decode-event to detect-engine-event
This patch does a simple renaming of detect-decode-event file to
the more global detect-engine-event name.
15 years ago
Victor Julien 820b0ded82 Add per packet profiling.
Per packet profiling uses tick based accounting. It has 2 outputs, a summary
and a csv file that contains per packet stats.

Stats per packet include:
 1) total ticks spent
 2) ticks spent per individual thread module
 3) "threading overhead" which is simply calculated by subtracting (2) of (1).

A number of changes were made to integrate the new code in a clean way:
a number of generic enums are now placed in tm-threads-common.h so we can
include them from any part of the engine.

Code depends on --enable-profiling just like the rule profiling code.

New yaml parameters:

profiling:
  # packet profiling
  packets:

    # Profiling can be disabled here, but it will still have a
    # performance impact if compiled in.
    enabled: yes
    filename: packet_stats.log
    append: yes

    # per packet csv output
    csv:

      # Output can be disabled here, but it will still have a
      # performance impact if compiled in.
      enabled: no
      filename: packet_stats.csv

Example output of summary stats:

IP ver   Proto   cnt        min      max          avg
------   -----   ------     ------   ----------   -------
 IPv4       6     19436      11448      5404365     32993
 IPv4     256         4      11511        49968     30575

Per Thread module stats:

Thread Module              IP ver   Proto   cnt        min      max          avg
------------------------   ------   -----   ------     ------   ----------   -------
TMM_DECODEPCAPFILE          IPv4       6     19434       1242        47889      1770
TMM_DETECT                  IPv4       6     19436       1107       137241      1504
TMM_ALERTFASTLOG            IPv4       6     19436         90         1323       155
TMM_ALERTUNIFIED2ALERT      IPv4       6     19436        108         1359       138
TMM_ALERTDEBUGLOG           IPv4       6     19436         90         1134       154
TMM_LOGHTTPLOG              IPv4       6     19436        414      5392089      7944
TMM_STREAMTCP               IPv4       6     19434        828      1299159     19438

The proto 256 is a counter for handling of pseudo/tunnel packets.

Example output of csv:

pcap_cnt,ipver,ipproto,total,TMM_DECODENFQ,TMM_VERDICTNFQ,TMM_RECEIVENFQ,TMM_RECEIVEPCAP,TMM_RECEIVEPCAPFILE,TMM_DECODEPCAP,TMM_DECODEPCAPFILE,TMM_RECEIVEPFRING,TMM_DECODEPFRING,TMM_DETECT,TMM_ALERTFASTLOG,TMM_ALERTFASTLOG4,TMM_ALERTFASTLOG6,TMM_ALERTUNIFIEDLOG,TMM_ALERTUNIFIEDALERT,TMM_ALERTUNIFIED2ALERT,TMM_ALERTPRELUDE,TMM_ALERTDEBUGLOG,TMM_ALERTSYSLOG,TMM_LOGDROPLOG,TMM_ALERTSYSLOG4,TMM_ALERTSYSLOG6,TMM_RESPONDREJECT,TMM_LOGHTTPLOG,TMM_LOGHTTPLOG4,TMM_LOGHTTPLOG6,TMM_PCAPLOG,TMM_STREAMTCP,TMM_DECODEIPFW,TMM_VERDICTIPFW,TMM_RECEIVEIPFW,TMM_RECEIVEERFFILE,TMM_DECODEERFFILE,TMM_RECEIVEERFDAG,TMM_DECODEERFDAG,threading
1,4,6,172008,0,0,0,0,0,0,47889,0,0,48582,1323,0,0,0,0,1359,0,1134,0,0,0,0,0,8028,0,0,0,49356,0,0,0,0,0,0,0,14337

First line of the file contains labels.

2 example gnuplot scripts added to plot the data.
15 years ago
Victor Julien 1bd1a62526 Rename profile macro's and variables to reflect that they are for rule profiling. 15 years ago
Anoop Saldanha 7e5c52c80b add flowbits:set; only sigs to be treated as ip only 16 years ago
Victor Julien 16b41a5eff Use p->proto in detect to determine TCP/UDP/SCTP. 16 years ago
Victor Julien f7f037c1d1 Make sure stateful detection engine inspecting HTTP streams works well for to_client rules as well. 16 years ago
Anoop Saldanha 35f3eafa5e byte extract added to the engine. Detection support added for packet payload, uri and dce detection engines 16 years ago
Victor Julien 73efb4c70f Add a app layer state and stateful detection engine counter that makes sure the stateful inspection is only done when the state changes. 16 years ago
Victor Julien 0768ca9806 Fix SIMD mask checking on 64 bit systems. 16 years ago
Victor Julien 350215966b Fix signature mask bitorder. 16 years ago
Victor Julien aa822c0ac1 Always reset alert cnt and always increment det_ctx->pkts. 16 years ago
Victor Julien 1e0b050a54 Add more mask flags. 16 years ago
Victor Julien 4b52823ab6 Use 64 bit mask on 64-bit systems. 16 years ago
Victor Julien e5b6c0f518 Check 32 masks per run instead of 16 in the SIMD code. 16 years ago
Victor Julien 2dbfdd40af Clean up new SIMD mask checking code, improve non-SIMD checks. 16 years ago
Victor Julien b421019cef Match packet mask against 16 signature masks at once using SIMD instructions for SSE3 and up. 16 years ago
Victor Julien e1d4e16645 Simplify packet decoding macro's. 16 years ago
Victor Julien 75439863ed Shrink PacketAlerts structure so that Packet structure is a lot smaller. Reduce max events per packet from 256 to 15. 16 years ago
Victor Julien 6aa551c558 Small optimizations to IPV4 and TCP header parsing. 16 years ago
Anoop Saldanha 966119b6aa support for http_raw_uri keyword + mpm engine 16 years ago
Victor Julien 864c8718e1 Store matching stream msg (ptr) in packets alert structure so it's available to the output plugins. 16 years ago
Victor Julien 140eb4fde8 Fix decode-event keyword parsing. Fix code that indicates a signature is decode-event only. Add 'pkthdr' protocol as an alias for any/ip to be used by decode-event signatures. 16 years ago
Victor Julien e197f50727 Fix IP-Only unittests failing on Big Endian. 16 years ago
Eric Leblond 9beebf621a Add support for 'nfq_set_mark' keyword
This patch introduces 'nfq_set_mark' which is new rules option. If a packet
matches a rule using nfq_set_mark in NFQ mode, it is marked with the mark/mask
specified in the option during the verdict.
It is thus possible to trigger different behaviour on the packet inside
Linux/Netfilter.
16 years ago
Anoop Saldanha c105a739e9 support for ssl_state keyword added 16 years ago
Victor Julien b9fd978253 Fix compiler warnings in two unittests. 16 years ago
Eric Leblond 005dc599a6 detect.c: Fix usage of integer standing for protocol
This patch fixes direct usage of integer to code protocol value.
16 years ago
Eric Leblond 2c80f18dc9 detect: Add sctp detection and parsing.
This patch adds the support of SCTP in signature subsystem.
16 years ago
Gurvinder Singh 8f8b1212af support for ssl_version keyword 16 years ago
Victor Julien ec5b622553 Inspect all stream msgs at any time when running in stream-inline mode. Skip detection for packets flagged for dropping before detect. 16 years ago
Victor Julien 1d971b53a6 Update all unittests 16 years ago
Victor Julien f606621e8c Fix the pseudo packet having the wrong proto set, causing massive fp's. Flag packets to be part of the established phase of a tcp session, so we won't prematurely inspect the app layer state. 16 years ago