Commit Graph

32 Commits (be9cd0fd843f1e412bceb3a86b0ad4e5cb7b0782)

Author SHA1 Message Date
cardigliano 57e0bd39e9 pkt acq: introduce break loop API
This patch adds a new callback PktAcqBreakLoop() in TmModule to let
packet acquisition modules define "break-loop" functions to terminate
the capture loop. This is useful in case of blocking functions that
need special actions to take place in order to stop the execution.

Implement this for PF_RING
10 years ago
Victor Julien 14466a803d decode: create util function for basic counter updates 10 years ago
Victor Julien 30cce2bd29 counters: s/SCPerfCounterSetUI64/StatsSetUI64/g 10 years ago
Victor Julien 1c0b4ee0ae counters: s/SCPerfCounterIncr/StatsIncr/g 10 years ago
Victor Julien 8992275b0c counters: s/SCPerfCounterAddUI64/StatsAddUI64/g 10 years ago
Victor Julien b34c6dc93a counters: remove references to SCPerfCounterAddDouble
They were all in comments anyway.
10 years ago
Victor Julien e9b067c1eb counters: make increment call take threadvars
This hides the implementation from the caller.
10 years ago
Victor Julien 9a8bff7d96 counters: threadvars s/sc_perf_pca/perf_private_ctx/g 10 years ago
Victor Julien 98c88d5170 decode: pass ThreadVars to DecodeThreadVarsFree
Flow output thread data deinit function which will be called from
DecodeThreadVarsFree will need it.
11 years ago
Ken Steele 28ccea51d3 Add error checking for pthread_setspecific() and pthread_key_create(). 11 years ago
Ken Steele a38d5a0135 Implement thread specific data option when __thread is not available. 11 years ago
Ken Steele 3c6e01f653 Replace ringbuffer in Packet Pool with a stack for better cache locality
Using a stack for free Packet storage causes recently freed Packets to be
reused quickly, while there is more likelihood of the data still being in
cache.

The new structure has a per-thread private stack for allocating Packets
which does not need any locking. Since Packets can be freed by any thread,
there is a second stack (return stack) for freeing packets by other threads.
The return stack is protected by a mutex. Packets are moved from the return
stack to the private stack when the private stack is empty.

Returning packets back to their "home" stack keeps the stacks from getting out
of balance.

The PacketPoolInit() function is now called by each thread that will be
allocating packets. Each thread allocates max_pending_packets, which is a
change from before, where that was the total number of packets across all
threads.
11 years ago
Victor Julien ab503873ca erf-file: clean up decode thread local storage
Clean up the thread local data the decode portion of ERF users.

Bug #978
11 years ago
Victor Julien f7b1aefaf4 Bug 1107: decoders: bail out on pseudo packets
Flow-timeout code injects pseudo packets into the decoders, leading
to various issues. For a full explanation, see:
  https://redmine.openinfosecfoundation.org/issues/1107

This patch works around the issues with a hack. It adds a check to
each of the decoder entry points to bail out as soon as a pseudo
packet from the flow timeout is encountered.

Ticket #1107.
12 years ago
Victor Julien 5f307acace Pass ThreadVars ptr to various thread init funcs
To be able to register counters from AppLayerGetCtxThread, the
ThreadVars pointer needs to be available in it and thus in it's
callers:

- AppLayerGetCtxThread
- DecodeThreadVarsAlloc
- StreamTcpReassembleInitThreadCtx
12 years ago
Eric Leblond 3088b6ac34 Add invalid pkt counter.
This patch adds and increments a invalid packet counter. It
does this by introducing PacketDecodeFinalize function

This function is incrementing the invalid counter and is also
signalling the packet to CUDA.
12 years ago
Victor Julien 698ff4e4aa Counters: remove all unused parts of the API 12 years ago
Last G 8bb9c3af35 Added return value to non-void function with "forever"-loop to fit
Eclipse static code analysis
13 years ago
Anoop Saldanha b33986c887 Add a packet src for every packet generated inside suricata. 13 years ago
Eric Leblond e176be6fcc Use unlikely for error treatment.
When handling error case on SCMallog, SCCalloc or SCStrdup
we are in an unlikely case. This patch adds the unlikely()
expression to indicate this to gcc.

This patch has been obtained via coccinelle. The transformation
is the following:

@istested@
identifier x;
statement S1;
identifier func =~ "(SCMalloc|SCStrdup|SCCalloc)";
@@

x = func(...)
... when != x
- if (x == NULL) S1
+ if (unlikely(x == NULL)) S1
13 years ago
Jason Ish ea020e2be6 Do not trim the FCS, pcaps converted to ERF will have have an FCS. 13 years ago
Eric Leblond a48d6cb207 erf: fix logical operator usage. 13 years ago
Anoop Saldanha 34581ce902 rx TMs shouldn't return TM_ECODE_FAILED if engine is in shutdown mode + minor cleanup 13 years ago
Anoop Saldanha bc6cf43840 #482 - use decode_flag for all decode TMs. Use the flag as a way to retrieve decode TMs from ThreadVars 13 years ago
Jason Ish 90548837e3 Update the ERF file runmodes to support autofp and single. 13 years ago
Anoop Saldanha 3f1c4efceb Add new flags var to tm module. TMs can now set flags to identify special properties. Also use these to identify receive TMs 14 years ago
Victor Julien 820b0ded82 Add per packet profiling.
Per packet profiling uses tick based accounting. It has 2 outputs, a summary
and a csv file that contains per packet stats.

Stats per packet include:
 1) total ticks spent
 2) ticks spent per individual thread module
 3) "threading overhead" which is simply calculated by subtracting (2) of (1).

A number of changes were made to integrate the new code in a clean way:
a number of generic enums are now placed in tm-threads-common.h so we can
include them from any part of the engine.

Code depends on --enable-profiling just like the rule profiling code.

New yaml parameters:

profiling:
  # packet profiling
  packets:

    # Profiling can be disabled here, but it will still have a
    # performance impact if compiled in.
    enabled: yes
    filename: packet_stats.log
    append: yes

    # per packet csv output
    csv:

      # Output can be disabled here, but it will still have a
      # performance impact if compiled in.
      enabled: no
      filename: packet_stats.csv

Example output of summary stats:

IP ver   Proto   cnt        min      max          avg
------   -----   ------     ------   ----------   -------
 IPv4       6     19436      11448      5404365     32993
 IPv4     256         4      11511        49968     30575

Per Thread module stats:

Thread Module              IP ver   Proto   cnt        min      max          avg
------------------------   ------   -----   ------     ------   ----------   -------
TMM_DECODEPCAPFILE          IPv4       6     19434       1242        47889      1770
TMM_DETECT                  IPv4       6     19436       1107       137241      1504
TMM_ALERTFASTLOG            IPv4       6     19436         90         1323       155
TMM_ALERTUNIFIED2ALERT      IPv4       6     19436        108         1359       138
TMM_ALERTDEBUGLOG           IPv4       6     19436         90         1134       154
TMM_LOGHTTPLOG              IPv4       6     19436        414      5392089      7944
TMM_STREAMTCP               IPv4       6     19434        828      1299159     19438

The proto 256 is a counter for handling of pseudo/tunnel packets.

Example output of csv:

pcap_cnt,ipver,ipproto,total,TMM_DECODENFQ,TMM_VERDICTNFQ,TMM_RECEIVENFQ,TMM_RECEIVEPCAP,TMM_RECEIVEPCAPFILE,TMM_DECODEPCAP,TMM_DECODEPCAPFILE,TMM_RECEIVEPFRING,TMM_DECODEPFRING,TMM_DETECT,TMM_ALERTFASTLOG,TMM_ALERTFASTLOG4,TMM_ALERTFASTLOG6,TMM_ALERTUNIFIEDLOG,TMM_ALERTUNIFIEDALERT,TMM_ALERTUNIFIED2ALERT,TMM_ALERTPRELUDE,TMM_ALERTDEBUGLOG,TMM_ALERTSYSLOG,TMM_LOGDROPLOG,TMM_ALERTSYSLOG4,TMM_ALERTSYSLOG6,TMM_RESPONDREJECT,TMM_LOGHTTPLOG,TMM_LOGHTTPLOG4,TMM_LOGHTTPLOG6,TMM_PCAPLOG,TMM_STREAMTCP,TMM_DECODEIPFW,TMM_VERDICTIPFW,TMM_RECEIVEIPFW,TMM_RECEIVEERFFILE,TMM_DECODEERFFILE,TMM_RECEIVEERFDAG,TMM_DECODEERFDAG,threading
1,4,6,172008,0,0,0,0,0,0,47889,0,0,48582,1323,0,0,0,0,1359,0,1134,0,0,0,0,0,8028,0,0,0,49356,0,0,0,0,0,0,0,14337

First line of the file contains labels.

2 example gnuplot scripts added to plot the data.
14 years ago
Eric Leblond dd038c1906 Modify files to avoid direct pckt payload access
This patch implements the needed modification of payload access
in a Packet structure to support the abstraction introduced by
the extended data system.
15 years ago
Victor Julien 4cacb1e970 Disable adding to unregistered mbit/s counter. 15 years ago
Pablo Rincon 8cc525c939 UDP support at AppLayer message handling 15 years ago
Victor Julien 4e7df60b2f Make pcap file mode read multiple packets per 'read'. Update threading model to deal with this. 15 years ago
Jason Ish a93b2e6b84 Support for reading ERF files. 15 years ago