Commit Graph

108 Commits (d292f1a529f6c238a8f9b576a5d2bc99ccb3dabf)

Author SHA1 Message Date
Eric Leblond 34abd818dd Prefix util-conf function with Config 12 years ago
Eric Leblond 54006de40c Use new function GetLogDirectory() 12 years ago
Eric Leblond 3f107fa130 decode: Packet action start with PACKET
Rename all Packet action macro to have them prefixed by PACKET.
12 years ago
Eric Leblond a35c367942 action handling: use macro for test.
Use test macro instead of direct access to action field.

This patch has been obtained by using the following
spatch file:

  @@
  Packet *p;
  expression E;
  @@

  - p->action & E
  + TEST_PACKET_ACTION(p, E)
12 years ago
Ken Steele 9c7b411a5d More PacketGetFromMalloc() to allocate packets. 12 years ago
Victor Julien 4b3166b193 unified2: more udp fixes 12 years ago
Victor Julien b54a19937f unified2: only call stream callback for TCP 12 years ago
Jake Gionet 1ac8938787 Adding support for Feature #667 13 years ago
Victor Julien 72443a0d62 unified2: append open instead of trucate open so that in case we rotate within a second we don't overwrite files. Instead we violate the limit. 13 years ago
Eric Leblond e176be6fcc Use unlikely for error treatment.
When handling error case on SCMallog, SCCalloc or SCStrdup
we are in an unlikely case. This patch adds the unlikely()
expression to indicate this to gcc.

This patch has been obtained via coccinelle. The transformation
is the following:

@istested@
identifier x;
statement S1;
identifier func =~ "(SCMalloc|SCStrdup|SCCalloc)";
@@

x = func(...)
... when != x
- if (x == NULL) S1
+ if (unlikely(x == NULL)) S1
13 years ago
Victor Julien 2a4992e7a0 inline: fix unified2 alert direction selection 13 years ago
Victor Julien 1906d317ec unified2: minor cleanups 13 years ago
Victor Julien 66856831fa unified2: big rewrite to clean up code that deals with tcp segment logging. 13 years ago
Victor Julien 02e19502c7 unified2: minor cleanups. 13 years ago
Victor Julien ea0d172693 No longer pass StreamMsg to output for alert logging, instead use the same callback code as is used for state alerts. 13 years ago
Victor Julien b8e741de9e Minor optimizations to unified2 and fast.log. 14 years ago
Victor Julien 0a80e362aa Fix some minor clang scan-build warnings. 14 years ago
Victor Julien e237841a8e Fix compilation with profiling enabled. Minor unittest fixes. 14 years ago
Victor Julien cdba2f50d1 Various fixes and improvements based on feedback by Coverity analyzer. 14 years ago
Victor Julien eba3cecc5d Fix unified2 records generated based on reassembled stream data. 14 years ago
Victor Julien 6f0ca120d1 Make sure existing log-pcap and unified2-alert 'limit' settings don't break. 14 years ago
Anoop Saldanha 7c9d1b80fd Update size parsing API with new calls for returing u8, u16, u32 and u64 values. Make updates in the codebase to use these new calls 14 years ago
Anoop Saldanha e0c13434ef bug 333 - support new Size Parsing API. Update various conf params inside the engine to use this API to parse sizes in the format xxx <-just the no represents bytes, xxxkb <- kilobytes, xxxmb <- megabytes, xxxgb <- gigabytes, where xxx is a \d+ 14 years ago
Victor Julien f47f601f09 Fix unified2 setting the wrong eth_type. 14 years ago
Eric Leblond 9422a36851 unified2: avoid to log RAW packet
If the packet datalink is ethernet, we add a fake ethernet
header to stream logging to avoid that barnyard2 create
different files.
14 years ago
Eric Leblond fc56abfcd0 unified2: log an ethernet header for stream alert.
If packet is a of type ethernet, we log the alert reconstructed
payload as an ethernet packet and not a raw packet. This will avoid
to confuse barnyard2 pcap output.
14 years ago
Victor Julien 39edb23ac4 Support stream.inline mode in unified2 tcp segments logging. 14 years ago
Victor Julien 2e2e80b812 Add packet alert flag to indicate a match happened (partly) in the app layer state. Make unified2 use this flag. 14 years ago
Eric Leblond 128261cb97 unified2: Fix event_id computation
This patch fixes event_id computation which was not incremented
for each alert.
14 years ago
Eric Leblond b3023643ec unified2: fix multiple alerts case 14 years ago
Eric Leblond 7fd1e9cacc unified2: synchronize IPv4 and IPv6 code
IPv6 code was missing some points.
14 years ago
Eric Leblond 839b0d9bfe unified2: switch to event->packet->packet mode.
Attach multiple packets to an event instead of using one
event data per packet. This is currrently unsupported by
reporting frontend but at least we don't have multiple
alerts.
14 years ago
Eric Leblond 316f2d7289 unified2: segment callback log raw packet.
As we don't have any trustable information about the ethernet
header, we can simply log RAW packet to avoid to confuse the
analyst.
14 years ago
Eric Leblond a03a402b83 unified2: set datalink to correct value.
The value of datalink could have been modified if the logging
of segment was attempted. This patch restore it to a correct value.
14 years ago
Eric Leblond 50ddd2df43 Restore old barnyard2 support.
Some old version of barnyard2 were needing a workaround in the
packet header building. THis patch introduces a enable-old-barnyard2
configure flag which can be used to restore this behaviour.
14 years ago
Eric Leblond 2f24987f15 unified2: improve packet logging logic.
This patch improves packet logging logic and fix some place
regarding alert generation (event_id, ethernet header).
14 years ago
Eric Leblond 79c329f81b alert-unified2: logging of stream segments.
This patch adds the logging of stream segments. Among other
modifications, it uses a wrapper to fwrite to permit to update
file statistics in an automated manner. Some memcpy have also
been avoided by using pointer to the data.
14 years ago
William Metcalf 3b3f5816bf You spin me right round baby, right round like a rotating packet capture right round. Oh, also log file size counters are now uint64_t 14 years ago
Anoop Saldanha 4307ea2348 Replace all frees with SCFrees 14 years ago
Victor Julien 820b0ded82 Add per packet profiling.
Per packet profiling uses tick based accounting. It has 2 outputs, a summary
and a csv file that contains per packet stats.

Stats per packet include:
 1) total ticks spent
 2) ticks spent per individual thread module
 3) "threading overhead" which is simply calculated by subtracting (2) of (1).

A number of changes were made to integrate the new code in a clean way:
a number of generic enums are now placed in tm-threads-common.h so we can
include them from any part of the engine.

Code depends on --enable-profiling just like the rule profiling code.

New yaml parameters:

profiling:
  # packet profiling
  packets:

    # Profiling can be disabled here, but it will still have a
    # performance impact if compiled in.
    enabled: yes
    filename: packet_stats.log
    append: yes

    # per packet csv output
    csv:

      # Output can be disabled here, but it will still have a
      # performance impact if compiled in.
      enabled: no
      filename: packet_stats.csv

Example output of summary stats:

IP ver   Proto   cnt        min      max          avg
------   -----   ------     ------   ----------   -------
 IPv4       6     19436      11448      5404365     32993
 IPv4     256         4      11511        49968     30575

Per Thread module stats:

Thread Module              IP ver   Proto   cnt        min      max          avg
------------------------   ------   -----   ------     ------   ----------   -------
TMM_DECODEPCAPFILE          IPv4       6     19434       1242        47889      1770
TMM_DETECT                  IPv4       6     19436       1107       137241      1504
TMM_ALERTFASTLOG            IPv4       6     19436         90         1323       155
TMM_ALERTUNIFIED2ALERT      IPv4       6     19436        108         1359       138
TMM_ALERTDEBUGLOG           IPv4       6     19436         90         1134       154
TMM_LOGHTTPLOG              IPv4       6     19436        414      5392089      7944
TMM_STREAMTCP               IPv4       6     19434        828      1299159     19438

The proto 256 is a counter for handling of pseudo/tunnel packets.

Example output of csv:

pcap_cnt,ipver,ipproto,total,TMM_DECODENFQ,TMM_VERDICTNFQ,TMM_RECEIVENFQ,TMM_RECEIVEPCAP,TMM_RECEIVEPCAPFILE,TMM_DECODEPCAP,TMM_DECODEPCAPFILE,TMM_RECEIVEPFRING,TMM_DECODEPFRING,TMM_DETECT,TMM_ALERTFASTLOG,TMM_ALERTFASTLOG4,TMM_ALERTFASTLOG6,TMM_ALERTUNIFIEDLOG,TMM_ALERTUNIFIEDALERT,TMM_ALERTUNIFIED2ALERT,TMM_ALERTPRELUDE,TMM_ALERTDEBUGLOG,TMM_ALERTSYSLOG,TMM_LOGDROPLOG,TMM_ALERTSYSLOG4,TMM_ALERTSYSLOG6,TMM_RESPONDREJECT,TMM_LOGHTTPLOG,TMM_LOGHTTPLOG4,TMM_LOGHTTPLOG6,TMM_PCAPLOG,TMM_STREAMTCP,TMM_DECODEIPFW,TMM_VERDICTIPFW,TMM_RECEIVEIPFW,TMM_RECEIVEERFFILE,TMM_DECODEERFFILE,TMM_RECEIVEERFDAG,TMM_DECODEERFDAG,threading
1,4,6,172008,0,0,0,0,0,0,47889,0,0,48582,1323,0,0,0,0,1359,0,1134,0,0,0,0,0,8028,0,0,0,49356,0,0,0,0,0,0,0,14337

First line of the file contains labels.

2 example gnuplot scripts added to plot the data.
14 years ago
Victor Julien ebe99a2597 Fix unified2 packet length not being set properly for reassembled stream packets. 14 years ago
Victor Julien 4025567a5a Fix a number of unittests not properly initializing a packet causing issues on some archs. 14 years ago
Eric Leblond 64b069369e Unified2: Use local variable for header copy
Due to the chaining of function call, the per-thread buffer was overwritten.
This was causing invalid data to be output.

This patch restores a local variable usage for the writing of the header
which are rather small and thus should not be a performance and security
issue.
14 years ago
Eric Leblond 9d24e3aacc Fix len computation. 14 years ago
Eric Leblond 4b0c8f6567 Use local thread variable buffer in alert unified2. 14 years ago
Victor Julien 75439863ed Shrink PacketAlerts structure so that Packet structure is a lot smaller. Reduce max events per packet from 256 to 15. 14 years ago
Anoop Saldanha a165d45da9 naming changes for runmodes 15 years ago
Victor Julien 5dcaaebf21 Enable logging of stream chunk in IPV6/TCP. Make sure IPV6 events have a ethernet header to work around Barnyard2 not liking DLT_RAW+IPV6. 15 years ago
Victor Julien 93815a1585 Support logging of reassembled stream data in IPv4 unified2. 15 years ago
Victor Julien 5da8bd7c1e Fix unified2 overwriting tag alerts. 15 years ago