Commit Graph

227 Commits (ae7aae81dc25271f30d4c26f0588f65ad8f44c09)

Author SHA1 Message Date
Victor Julien d165906397 QA: add --afl-decoder-ppp=<file> 9 years ago
Maurizio Abba 44a7c12ad0 decode-events: counters for decode events errors
We want to add counters in order to track the number of times we hit a
decode event. A decode event is related to an error in the protocol
decoding over a certain packet.

This patch fist modifies the decode-event list, reordering it in order
to separate single packet events from stream-related events and adding
the prefix "decoder" to decode events.
The counters are created during the decode setup and the relative event
counter is increased every time a packet with the flag PKT_IS_INVALID is
finalized in the decode phase
10 years ago
Giuseppe Longo 769722101e decode: add flow memcap counter
This adds a counter indicating how many times
the flow max memcap has been reached

Since there is no always a reference to FlowManagerThreadData,
the counter is put in DecodeThreadVars.

Currently when there is no counter increase in one call of FlowGetNew
because we don't have tv or dtv at the time of the call.

The following is a snippet of the generated EVE entry:
"flow":{"memcap":0,"spare":10000,"emerg_mode_entered":0,"emerg_mode_over":0,"tcp_reuse":0,"memuse":7085248}
10 years ago
Victor Julien f4c9915066 multi-detect: store tenant id in packet
Store tenant id in the packet so that the output API's can log it.
10 years ago
Victor Julien 80615f7d43 ips: move counters in common struct and funcs 10 years ago
Victor Julien 14466a803d decode: create util function for basic counter updates 10 years ago
Victor Julien a083513c49 decode: optimize DecodeThreadVars layout
Put common counters on the first cache line. Please the flow output
pointer last as it's use depends on the flow logging being enabled
and even then it's only called very rarely.
10 years ago
Victor Julien fe5a85aea0 decode: add erspan counter 10 years ago
Victor Julien 928957f0a3 decode: add ERSPANv1 decoder
Only allow v1 to be parsed as thats what is tested.

Take vlan_id from the ERSPAN layer.
10 years ago
Victor Julien aa6b24f814 decode: clean up tunnel decode logic
Don't use mix of existing and custom types to indicate the next
layer.
10 years ago
Victor Julien 04ccfda639 pcap: implement LINKTYPE_NULL
Implement LINKTYPE_NULL for pcap live and pcap file.

From: http://www.tcpdump.org/linktypes.html

"BSD loopback encapsulation; the link layer header is a 4-byte field,
 in host byte order, containing a PF_ value from socket.h for the
 network-layer protocol of the packet.

 Note that ``host byte order'' is the byte order of the machine on
 which the packets are captured, and the PF_ values are for the OS
 of the machine on which the packets are captured; if a live capture
 is being done, ``host byte order'' is the byte order of the machine
 capturing the packets, and the PF_ values are those of the OS of
 the machine capturing the packets, but if a ``savefile'' is being
 read, the byte order and PF_ values are not necessarily those of
 the machine reading the capture file."

Feature ticket #1445
10 years ago
Victor Julien e9857200b3 detect: set action from utility function
Set actions that are set directly from Signatures using the new
utility function DetectSignatureApplyActions. This will apply
the actions and also store info about the 'drop' that first made
the rule drop.
11 years ago
Victor Julien c914f7bcdc detect: cleanup, remove unused order_id
No longer used, so remove.
11 years ago
Eric Leblond e138a2ac1e decode: introduce PacketCallocExtPkt function
In flow timeout handling we need a function that allocate and blank
a place that will be used to put constructed packet data. This new
function has no other goal.
11 years ago
Victor Julien 2fb9611223 flow: add util func to remove packet from flow
Unsets the p::flowflags that were previously set.
11 years ago
gureedo 10104066e1 netmap support 11 years ago
Victor Julien 3499d682c4 flow timeout: cleanups
Rename FlowForceReassemblyForFlowV2 to just FlowForceReassemblyForFlow
as there is no V1.
11 years ago
Jason Ish 3e3ab2dc9f Add MPLS counter.
Check length before decoding each label.
11 years ago
Jason Ish 7642489874 Basic MPLS decoder. 11 years ago
Victor Julien c20bd3bcb2 Optimize Packet Ext data freeing
Move the logic of PacketFreeExtData into a macro 'PACKET_FREE_EXTDATA'.
It was called for each packet.
11 years ago
Eric Leblond 80adc40f68 packet pool: fix ext_pkt cleaning
The field ext_pkt was cleaned before calling the release function.
The result was that IPS mode such as the one of AF_PACKET were not
working anymore because they were not able to send the data which
were initially pointed by ext_pkt.

This patch moves the ext_pkt cleaning to the cleaning macro. This
ensures that the cleaning is done for allocated and pool packets.
11 years ago
Victor Julien 3ee504a3dc packet recycle: remove mutex destroy/init
This was necessary earlier when there was a memset involved.
11 years ago
Victor Julien 231b993f1f packet recycle: split macro
Split PACKET_RECYCLE into 2 parts. One part for cleanup to do before a
packet is returned to the pool, the other after it's retrieved from
the pool.
11 years ago
Victor Julien bbcdb657da ipv6: more robust ipv6 exthdr handling
Skip past Shim6, HIP and Mobility header.

Detect data after 'none' header.
    decode-event:ipv6.data_after_none_header;
11 years ago
Victor Julien 98c88d5170 decode: pass ThreadVars to DecodeThreadVarsFree
Flow output thread data deinit function which will be called from
DecodeThreadVarsFree will need it.
11 years ago
Victor Julien de034f1867 flow: prepare flow forced reuse logging
Most flows are marked for clean up by the flow manager, which then
passes them to the recycler. The recycler logs and cleans up. However,
under resource stress conditions, the packet threads can recycle
existing flow directly. So here the recycler has no role to play, as
the flow is immediately used.

For this reason, the packet threads need to be able to invoke the
flow logger directly.

The flow logging thread ctx will stored in the DecodeThreadVars
stucture. Therefore, this patch makes the DecodeThreadVars an argument
to FlowHandlePacket.
11 years ago
Ken Steele 3c6e01f653 Replace ringbuffer in Packet Pool with a stack for better cache locality
Using a stack for free Packet storage causes recently freed Packets to be
reused quickly, while there is more likelihood of the data still being in
cache.

The new structure has a per-thread private stack for allocating Packets
which does not need any locking. Since Packets can be freed by any thread,
there is a second stack (return stack) for freeing packets by other threads.
The return stack is protected by a mutex. Packets are moved from the return
stack to the private stack when the private stack is empty.

Returning packets back to their "home" stack keeps the stacks from getting out
of balance.

The PacketPoolInit() function is now called by each thread that will be
allocating packets. Each thread allocates max_pending_packets, which is a
change from before, where that was the total number of packets across all
threads.
11 years ago
Giuseppe Longo 4d72911e17 This patch adds the fields into PacketVars struct to setup a packet from a nflog message 11 years ago
Victor Julien d26ceb2356 decode: introduce DecodeThreadVarsFree
As a mirror of DecodeThreadVarsAlloc, DecodeThreadVarsFree is used
to free the memory that DecodeThreadVarsAlloc alloc'd, including
AppLayer storage.
11 years ago
Victor Julien 9eed83c62b profiling: fix memory leak
For packets that were freed, not recycled, profiling memory wasn't
freed:

==15745== 13,312 bytes in 8 blocks are definitely lost in loss record 611 of 615
==15745==    at 0x4C2C494: calloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==15745==    by 0xA190D5: SCProfilePacketStart (util-profiling.c:963)
==15745==    by 0x4E4345: PacketGetFromAlloc (decode.c:134)
==15745==    by 0x83FE75: FlowForceReassemblyPseudoPacketGet (flow-timeout.c:276)
==15745==    by 0x8413BF: FlowForceReassemblyForHash (flow-timeout.c:588)
==15745==    by 0x841897: FlowForceReassembly (flow-timeout.c:716)
==15745==    by 0x9540F6: main (suricata.c:2296)
==15745==
==15745== 14,976 bytes in 9 blocks are definitely lost in loss record 612 of 615
==15745==    at 0x4C2C494: calloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==15745==    by 0xA190D5: SCProfilePacketStart (util-profiling.c:963)
==15745==    by 0x4E4345: PacketGetFromAlloc (decode.c:134)
==15745==    by 0x83FE75: FlowForceReassemblyPseudoPacketGet (flow-timeout.c:276)
==15745==    by 0x841508: FlowForceReassemblyForHash (flow-timeout.c:620)
==15745==    by 0x841897: FlowForceReassembly (flow-timeout.c:716)
==15745==    by 0x9540F6: main (suricata.c:2296)

This patch addresses that.
12 years ago
Victor Julien f791d0f5c5 vlan/QinQ: add vlan_qinq counter
This patch introduces a new counter "decoder.vlan_qinq". It counts
packets that have more than two stacked vlan layers.

Packets with 2 vlan layers will both increment "decoder.vlan" and
"decoder.vlan_qinq".
12 years ago
Victor Julien 2c3a92a1c9 profiling: conditional rule profiling
Add support for conditional rule profiling. Currently only simple
rate limiting is supported, but hardcoded to inspecting rules for
each packet.
12 years ago
Victor Julien 13d491f577 profiling: lower overhead when disabled
Instead of a large (6k+) structure in the Packet, make the profiling
storage dynamic. To do this the Packet->profile is now a pointer.

Initial support for selective sampling, e.g. only profile every
1000th packet.
12 years ago
Victor Julien 347c0df9c4 app-layer-event: refactor
Move app layer event handling into app-layer-event.[ch].
Convert 'Set' macro's to functions.
Get rid of duplication in Set and SetRaw. Set now calls SetRaw.
Fix potentential int overflow condition in the event storage.
Update callers.
12 years ago
Victor Julien 5f307acace Pass ThreadVars ptr to various thread init funcs
To be able to register counters from AppLayerGetCtxThread, the
ThreadVars pointer needs to be available in it and thus in it's
callers:

- AppLayerGetCtxThread
- DecodeThreadVarsAlloc
- StreamTcpReassembleInitThreadCtx
12 years ago
Victor Julien 9634e60e7a app-layer: Use opaque pointers instead of void
For AppLayerThreadCtx, AppLayerParserState, AppLayerParserThreadCtx
and AppLayerProtoDetectThreadCtx, use opaque pointers instead of
void pointers.

AppLayerParserState is declared in flow.h as it's part of the Flow
structure.

AppLayerThreadCtx is declared in decode.h, as it's part of the
DecodeThreadVars structure.
12 years ago
Anoop Saldanha 429c6388f6 App layer API rewritten. The main files in question are:
app-layer.[ch], app-layer-detect-proto.[ch] and app-layer-parser.[ch].

Things addressed in this commit:
- Brings out a proper separation between protocol detection phase and the
  parser phase.
- The dns app layer now is registered such that we don't use "dnstcp" and
  "dnsudp" in the rules.  A user who previously wrote a rule like this -

  "alert dnstcp....." or
  "alert dnsudp....."

  would now have to use,

  alert dns (ipproto:tcp;) or
  alert udp (app-layer-protocol:dns;) or
  alert ip (ipproto:udp; app-layer-protocol:dns;)

  The same rules extend to other another such protocol, dcerpc.
- The app layer parser api now takes in the ipproto while registering
  callbacks.
- The app inspection/detection engine also takes an ipproto.
- All app layer parser functions now take direction as STREAM_TOSERVER or
  STREAM_TOCLIENT, as opposed to 0 or 1, which was taken by some of the
  functions.
- FlowInitialize() and FlowRecycle() now resets proto to 0.  This is
  needed by unittests, which would try to clean the flow, and that would
  call the api, AppLayerParserCleanupParserState(), which would try to
  clean the app state, but the app layer now needs an ipproto to figure
  out which api to internally call to clean the state, and if the ipproto
  is 0, it would return without trying to clean the state.
- A lot of unittests are now updated where if they are using a flow and
  they need to use the app layer, we would set a flow ipproto.
- The "app-layer" section in the yaml conf has also been updated as well.
12 years ago
Ken Steele f9705377ae Remove pkt variable from Packet structure.
The uint8_t *pkt in the Packet structure always points to the memory
immediately following the Packet structure. It is better to simply
calculate that value every time than store the 8 byte pointer.
12 years ago
Eric Leblond f76448c1e6 decode: fix failure in layered tunnel
If we have multiple layer of tunnel, the decoding of initial
Packet will recurse in DecodeTunnel function called in
PacketTunnelPktSetup. If we are not setting the pseudo
packet root before calling DecodeTunnel (as done in previous
code), then the tunnel root will no be correct for the lower
layer packets. This result in an counter problem and a suricata
failure after some time.
12 years ago
Eric Leblond 0460b194b1 decode: clean DecodeThreadVars counter
Speed counters are not compute anymore and can be removed from the
structure definition.
12 years ago
Eric Leblond 3088b6ac34 Add invalid pkt counter.
This patch adds and increments a invalid packet counter. It
does this by introducing PacketDecodeFinalize function

This function is incrementing the invalid counter and is also
signalling the packet to CUDA.
12 years ago
Victor Julien 277fb61c1d defrag: clean up
Rename PacketDefragPktFinishSetup to PacketDefragPktSetupParent to
better refect it's function.
12 years ago
Eric Leblond 3fdf52239d defrag: don't modify packet if defrag fails
If defrag fails dur to an invalid decoding, we are not modifying
the origin packet anymore.
12 years ago
Eric Leblond c611b258a5 decode: PacketTunnelPktSetup replaces PacketPseudoPktSetup
This patch replaces PacketPseudoPktSetup by a better named
PacketTunnelPktSetup function which is also in charge of doing
the decoding of the tunneled packet.
This allow to clean the code. But it also fixes an issue.
Previously, if the DecodeTunnel function was failling (cause of
an invalid packet mainly), the result was that the original packet
to be considered as a tunnel packet (and not inspected by payload
detection).
12 years ago
Eric Leblond d4b7ecfbe3 decode: update API to return error
In some cases, the decoding is not possible and some really invalid
packet can be created. This is in particular the case of tunnel. In
that case, it is more interesting to forget about the tunneled
packet and only consider the original packet.

DecodeTunnel function is maked as warn_unused_result because it is
meaningful for the decoder to know if the underlying data were not
correct. And in this case, only focus detection on the content.
12 years ago
Ken Steele d3c6913e28 Reduce the size of Packet structure
Share memory space for IPV4Vars and (IPV6Vars, IPV6ExtHdrs), since a
packet can only be either IPv4 or IPv6, but not both.

Share memory for TCPVars, UDPVars, ICMPV4Vars and ICMPV6Vars, since a
packet can only be only of these.

Then move other structure members around to remove holes reported by pahole.

This reduces the size of the Packet structure from 2944 bytes (46 cachelines)
down to 1976 (31 cachelines), a 33% reduction.
12 years ago
Ken Steele f16b339fc4 Rename checksums to level3_comp_csum and level4_comp_csum.
This will also sharing even more memory in the Packet_ structure.
12 years ago
Ken Steele c6a8d0ab6b Share Packet checksum values for TCP, UDP, IPv6. ICMPv4 and ICMPv6
Keep a separate checksum for IPV4, since a packet can have both an IPV4
checksum and a TCPV4 checksum, or IPV4 and UDPV4 checksum.

This will allow future sharing of more values.

Use PACKET_RESET_CHECKSUMS() in Unit Tests in place of setting the
individual checksum values.
12 years ago
Eric Leblond eea760de3c decode: fix typo in comment 12 years ago
Victor Julien edeeb7ed44 Store TX id with alerts
When generating an alert and storing it in the packet, store the tx_id
as well. This way the output modules can log the tx_id and access the
proper tx for logging.

Issue #904.
12 years ago