Commit Graph

93 Commits (7f2f7cc48de7d842ce6b7dc3dae2a1d5c52aa3f9)

Author SHA1 Message Date
Jason Ish 1601222056 alert-debuglog: trigger rotation on non-decoder events as well
The write handling was missed for non-decoder events that
would trigger rotation after a HUP.
10 years ago
Jason Ish b512580bbe logging: integrate rotation into SCConfLogOpenGeneric.
Addresses issue 1492, and will make it harder to omit
rotation on new outputs.
10 years ago
Victor Julien c1558f5ac4 stream: remove FLOW_NO_APPLAYER_INSPECTION flag
Instead, intruduce StreamTcpDisableAppLayer to disable app layer
tracking and reassembly. StreamTcpAppLayerIsDisabled can be used
to check it.

Replace all uses of FlowSetSessionNoApplayerInspectionFlag and
the FLOW_NO_APPLAYER_INSPECTION.
10 years ago
Duarte Silva 3a18db13dc Simple code fixes
- Removed unnecessary assignment of the data field
- Removed else condition (same function called for IPv4 and IPV6)
- Fixed constants to be a power of two (used in bitwise operations)
11 years ago
Ken Steele 8f1d75039a Enforce function coding standard
Functions should be defined as:

int foo(void)
{
}

Rather than:
int food(void) {
}

All functions where changed by a script to match this standard.
11 years ago
Victor Julien bd490736c2 flow: take flow pkt & byte count out of debug
Until now the flow packet and byte counters were only available in
DEBUG mode. For logging purposes they are now available always.
11 years ago
Victor Julien c66a29b67d flow: track bytes per direction
Track bytes in both flow directions for logging purposes.
11 years ago
Jason Ish fc2014ab40 Unregister for file rotation notification when a context is
de-initialized.  Required for unix-socket mode where
contexts come and go.
11 years ago
Jason Ish e1b97fed70 Add signal based file rotation for:
- alert debug log
- fast log
- stats log
- dns log
- drop log
- file log
- http log
- tls log
- eve/json log
11 years ago
Ken Steele 235cd0211a Alert file formatting clean up.
Put { on new line for function declarations. Remove space after function
name.

Add static to unit tests delcaration.
12 years ago
Victor Julien a3b0577a1f output: add TM_FLAG_LOGAPI_TM thread module flag
The TM_FLAG_LOGAPI_TM flag indicates that a module is run by the log
api, not by the 'regular' thread module call functions.

Set flag in all all Log API users' registration code.

Purpose of this flag is in profiling. In profiling output it will be
used to list log api thread modules separately.
12 years ago
Victor Julien 73377048fd alert-debuglog: minor cleanups
Clean up log functions after packet logger conversion. No more
PacketQueue arguments.
12 years ago
Victor Julien cd4796f3ca alert-debuglog: port to packet logger api
Convert AlertDebugLog to Packet logger API. Convert packet args to
const.
12 years ago
Victor Julien 4b57d0272c alert-debug log cleanups
Make all funcs but registration static.
Remove stale registation prototypes.
Move registation func to the bottom.
12 years ago
Victor Julien 4049c2f74c Packet logging API: convert unified2
Convert unified2 alert to new logging API.
12 years ago
Victor Julien e7df53b136 Display TX id in alert debuglog. 12 years ago
Ken Steele 68d26dcec7 Merge multiple copies of CreateTimeString() to one copy.
There were 8 identical copies of CreateTimeString() in 8 files.
Most used SCLocalTime, to replace localtime_r(), but some did not.
Created one copy in util-time.c.
12 years ago
Victor Julien 4a0050b9ea Print pkt src to alert-debug log 12 years ago
Victor Julien b68d566c44 alert-debuglog: cleanup TCP check 12 years ago
Victor Julien 37c80ea508 If an IP-only pass rule matches, set the no inspect flag for that flow. Bug #718. 13 years ago
Last G 8ae11f73b2 Added parentheses to fix Eclipse static code analysis
Fixed bug in action priority (REJECT_DST had lowest prio)
13 years ago
Victor Julien d68fd54a76 Fix/suppress a couple of harmless compiler warnings. 13 years ago
Eric Leblond e176be6fcc Use unlikely for error treatment.
When handling error case on SCMallog, SCCalloc or SCStrdup
we are in an unlikely case. This patch adds the unlikely()
expression to indicate this to gcc.

This patch has been obtained via coccinelle. The transformation
is the following:

@istested@
identifier x;
statement S1;
identifier func =~ "(SCMalloc|SCStrdup|SCCalloc)";
@@

x = func(...)
... when != x
- if (x == NULL) S1
+ if (unlikely(x == NULL)) S1
13 years ago
Anoop Saldanha 32183faa82 free flowvar entries in flow after live rule swap. Sync flowbits entries into packet struct to be used by alert debuglog when alert debuglog is enabled 13 years ago
Anoop Saldanha f2dd61868d variable names global vars, global no more. Moved to detection engine ctx, a place it belongs 13 years ago
Eric Leblond a0e57f58e5 OpenBSD: introduce SCLocalTime function.
This function is a wrapper to localtime_r. It is needed to avoid
a compilation warning on OpenBSD. I'm forced to type the function
to a non pointer first parameter. If not we will have to use two
differents functions in OpenBSD where tv->tv_sec is a long
(different from time_t).
13 years ago
Victor Julien a3cbe2e1c2 alert-debuglog: add size info for stream chunks and fix a typo. 13 years ago
Anoop Saldanha 5f939412af debuglog now uses the new mem buffer API. Improve file ctx locking to just the file write 13 years ago
Victor Julien ea0d172693 No longer pass StreamMsg to output for alert logging, instead use the same callback code as is used for state alerts. 13 years ago
Victor Julien 19a7e7f395 flow: create a flow lock macro API, implement it for mutex and rwlocks. Mutex remains the default. 14 years ago
Victor Julien 28e15be526 Clean up default output. Use simpler output format for releases. 14 years ago
Victor Julien 2c62b50ed5 Fix 2 compiler warnings. 14 years ago
Mike Pomraning dfec9c0f6a Switch 'fast', 'http-log', 'drop' and 'alert-debug' to SCConfLogOpenGeneric. 14 years ago
Victor Julien e6d8d0443c Unify output functions for alert-debug for IPv4 and IPv6. 14 years ago
Eric Leblond 2073b9db0c debuglog: uses state selection system. 14 years ago
Eric Leblond 1596241687 debuglog: fix segment logging.
StreamSegmentForEach returns the number of segments or < 0 in case
of error. This patch synchronizes debuglog output module with this
behaviour.
14 years ago
Anoop Saldanha 67be07bf15 fix threading issue in debug log. locked mutex isn't freed before returning. fixed 14 years ago
Eric Leblond 2fa837bcec alert-debuglog: Add logging of stream segments.
This patch introduces logging of the stream segments in case of
a signature match on application layer.
14 years ago
Anoop Saldanha 4307ea2348 Replace all frees with SCFrees 14 years ago
Eileen Donlon 89599d3b9b fixed bug 288; corrected config boolean parsing problems 14 years ago
Victor Julien 820b0ded82 Add per packet profiling.
Per packet profiling uses tick based accounting. It has 2 outputs, a summary
and a csv file that contains per packet stats.

Stats per packet include:
 1) total ticks spent
 2) ticks spent per individual thread module
 3) "threading overhead" which is simply calculated by subtracting (2) of (1).

A number of changes were made to integrate the new code in a clean way:
a number of generic enums are now placed in tm-threads-common.h so we can
include them from any part of the engine.

Code depends on --enable-profiling just like the rule profiling code.

New yaml parameters:

profiling:
  # packet profiling
  packets:

    # Profiling can be disabled here, but it will still have a
    # performance impact if compiled in.
    enabled: yes
    filename: packet_stats.log
    append: yes

    # per packet csv output
    csv:

      # Output can be disabled here, but it will still have a
      # performance impact if compiled in.
      enabled: no
      filename: packet_stats.csv

Example output of summary stats:

IP ver   Proto   cnt        min      max          avg
------   -----   ------     ------   ----------   -------
 IPv4       6     19436      11448      5404365     32993
 IPv4     256         4      11511        49968     30575

Per Thread module stats:

Thread Module              IP ver   Proto   cnt        min      max          avg
------------------------   ------   -----   ------     ------   ----------   -------
TMM_DECODEPCAPFILE          IPv4       6     19434       1242        47889      1770
TMM_DETECT                  IPv4       6     19436       1107       137241      1504
TMM_ALERTFASTLOG            IPv4       6     19436         90         1323       155
TMM_ALERTUNIFIED2ALERT      IPv4       6     19436        108         1359       138
TMM_ALERTDEBUGLOG           IPv4       6     19436         90         1134       154
TMM_LOGHTTPLOG              IPv4       6     19436        414      5392089      7944
TMM_STREAMTCP               IPv4       6     19434        828      1299159     19438

The proto 256 is a counter for handling of pseudo/tunnel packets.

Example output of csv:

pcap_cnt,ipver,ipproto,total,TMM_DECODENFQ,TMM_VERDICTNFQ,TMM_RECEIVENFQ,TMM_RECEIVEPCAP,TMM_RECEIVEPCAPFILE,TMM_DECODEPCAP,TMM_DECODEPCAPFILE,TMM_RECEIVEPFRING,TMM_DECODEPFRING,TMM_DETECT,TMM_ALERTFASTLOG,TMM_ALERTFASTLOG4,TMM_ALERTFASTLOG6,TMM_ALERTUNIFIEDLOG,TMM_ALERTUNIFIEDALERT,TMM_ALERTUNIFIED2ALERT,TMM_ALERTPRELUDE,TMM_ALERTDEBUGLOG,TMM_ALERTSYSLOG,TMM_LOGDROPLOG,TMM_ALERTSYSLOG4,TMM_ALERTSYSLOG6,TMM_RESPONDREJECT,TMM_LOGHTTPLOG,TMM_LOGHTTPLOG4,TMM_LOGHTTPLOG6,TMM_PCAPLOG,TMM_STREAMTCP,TMM_DECODEIPFW,TMM_VERDICTIPFW,TMM_RECEIVEIPFW,TMM_RECEIVEERFFILE,TMM_DECODEERFFILE,TMM_RECEIVEERFDAG,TMM_DECODEERFDAG,threading
1,4,6,172008,0,0,0,0,0,0,47889,0,0,48582,1323,0,0,0,0,1359,0,1134,0,0,0,0,0,8028,0,0,0,49356,0,0,0,0,0,0,0,14337

First line of the file contains labels.

2 example gnuplot scripts added to plot the data.
14 years ago
Eric Leblond 6b9d1012ff Transform inet_ntop call into PrintInet one. 14 years ago
Victor Julien 75439863ed Shrink PacketAlerts structure so that Packet structure is a lot smaller. Reduce max events per packet from 256 to 15. 14 years ago
Victor Julien a5d9c86dd3 Shrink Flow structure with 20 bytes (on 32 bit) and reorder it. Clean up init, recycle, destroy macro's. 14 years ago
Anoop Saldanha fe6e41e3ef Removed FLOW_AL_NO_APPLAYER_INSPECTION. Moved it as FLOW_NO_APPLAYER_INSPECTION in Flow->flags. Turned Flow->flags into uint32_t and removed Flow->alflags 14 years ago
Anoop Saldanha ac5584a863 Removed FLOW_AL_PROTO_DETECT_DONE. Replaced it with FLOW_ALPROTO_DETECT_DONE, stored it in Flow->flags 14 years ago
Victor Julien 864c8718e1 Store matching stream msg (ptr) in packets alert structure so it's available to the output plugins. 15 years ago
Victor Julien 6f58ef13c4 Improve error cleanup in output function. Thanks to iswalker. 15 years ago
Eric Leblond dd038c1906 Modify files to avoid direct pckt payload access
This patch implements the needed modification of payload access
in a Packet structure to support the abstraction introduced by
the extended data system.
15 years ago
Victor Julien d48ff8f6aa Extend 'append' option to stats.log as well. Small cleanups. 15 years ago