Commit Graph

279 Commits (07df1ce6afffb35c0acd25b3b787ce04643306b1)

Author SHA1 Message Date
Victor Julien 0a809bf577 packet: set unique pkt_src 'flush' packets
Set unique type for capture timeout and for detect reload flush
to assist in debugging.
6 years ago
Victor Julien 6bc7636826 stream: remove unused code
Remove now unused 'pkt_src' type as well.

Remove related unittests.
6 years ago
Victor Julien 579cc9f02b const: constify decoder, app-layer, detect funcs 6 years ago
Victor Julien 399ab35aa1 afl: fix compile warnings for decoder fuzz funcs 6 years ago
Max Fillinger 44bea80d3c decode erspan: Always fill in vlan_id
Fill in the vlan_id fields unconditionally. We can now remove the check
for the vlan.use-for-tracking setting in decode.c. The debug log message
is moved to suricata.c.
6 years ago
Max Fillinger 8d3b04b0e3 decode vlan: Always fill in vlan_id
Since the vlan.use-for-tracking setting is now handled in flow-hash.c,
we can fill in the vlan_id fields unconditionally. This makes the vlanh
fields unnecessary.

Related to https://redmine.openinfosecfoundation.org/issues/3076
6 years ago
Victor Julien 82de6e0659 decoder/vxlan: improvements and cleanups
Implement port config handling. Also check both src port and dest
port for tunnels that only set the destination port to the VXLAN
port. At the point of the check we don't know the packet direction
yet.

Implement as Suricata tunnel similar to Teredo.

Cleanups.
6 years ago
Henrik Lund Kramshoej 3519b011b7 decoder/vxlan: initial implementation of decoder 6 years ago
Victor Julien d307cfad05 packet: add 'swap' function to reverse direction 6 years ago
Victor Julien 6fcd2db043 tile: remove files 7 years ago
Victor Julien 2a3cb32071 stream: no more stream events after known issue
No longer set stream events after a gap or wrong thread. We know
we lost sync and are now in 'lets make the best of it'-mode. No
point in flooding the system with stream events.

Ticket #2484
7 years ago
Victor Julien 932c2a7ec5 eve: fix missing decoder-events in stats
In the eve log the decoder events are added as optional counters. This
behaviour is enabled by default. However, lots of the counters are
missing, as the names colide with other counters.

E.g.

decoder.ipv6 counts ipv6 packets
decoder.ipv6.unknown_next_header counts how often an unknown next
    header is encountered.

In this example 'ipv6' would be both a json integer and a json object.
It appears that jansson favours the first that is generated, so the
event counters are mostly missing.

This patch registers them as 'decoder.events.<event>' instead. As
these names are generated on the fly, a hash table to contain the
allocated strings was added as well.
7 years ago
Victor Julien 11f3659f64 teredo: be stricter on what to consider valid teredo
Invalid Teredo can lead to valid DNS traffic (or other UDP traffic)
being misdetected as Teredo. This leads to false negatives in the
UDP payload inspection.

Make the teredo code only consider a packet teredo if the encapsulated
data was decoded without any 'invalid' events being set.

Bug #2736.
7 years ago
Victor Julien 6c97909a92 stream/events: log as stats 7 years ago
Maurizio Abba 55b14f3721 decode: set pktlen of decode handler to uint32
Change the decode handler signature to increase the size of its decode
handler, from uint16 to uint32. This is necessary to let suricata use
interfaces with mtu > 65535 (ex: lo interface has default size 65536).

It's necessary to change several primitive for Packet manipulation, to
unify the parameter "packet length" whenever we are before IP decoding.

Add tests before calling DecodeIPVX function to avoid a possible
integer overflow over the len parameter.
7 years ago
Jacob Masen-Smith ec77632e84 Adds WinDivert support to Windows builds
Enables IPS functionality on Windows using the open-source
(LGPLv3/GPLv2) WinDivert driver and API.

From https://www.reqrypt.org/windivert-doc.html : "WinDivert is a
user-mode capture/sniffing/modification/blocking/re-injection package
for Windows Vista, Windows Server 2008, Windows 7, and Windows 8.
WinDivert can be used to implement user-mode packet filters, packet
sniffers, firewalls, NAT, VPNs, tunneling applications, etc., without
the need to write kernel-mode code."

- adds `--windivert [filter string]` and `--windivert-forward [filter
    string]` command-line options to enable WinDivert IPS mode.
    `--windivert[-forward] true` will open a filter for all traffic. See
    https://www.reqrypt.org/windivert-doc.html#filter_language for more
    information.

Limitation: currently limited to `autofp` runmode.

Additionally:
- `tmm_modules` now zeroed during `RegisterAllModules`
- fixed Windows Vista+ `inet_ntop` call in `PrintInet`
- fixed `GetRandom` bug (nonexistent keys) on fresh Windows installs
- fixed `RandomGetClock` building on Windows builds
- Added WMI queries for MTU
7 years ago
fooinha f67aa5deaa packet: gre over ip link type 7 years ago
Victor Julien c662383b53 flow: track flow for ICMP
Change packet layout to allow for expected counterpart type.
7 years ago
Victor Julien 7ce77f9351 decode/ipv6: expose addr as 'struct in6_addr' as well 7 years ago
Victor Julien cf2feeecf4 detect/prefilter: redo profiling 8 years ago
Victor Julien 746638b220 cuda: remove
Remove CUDA support as it has been broken for a long time.

Ticket #2382.
8 years ago
Pierre Chifflier 5748df3eed Add support for PCAP LINKTYPE_IPV4 8 years ago
Victor Julien 6ce45bcf38 pfring: various build issues
pfring.h brings a different version of likely/unlikely that gives
warnings. So make sure we include our own before.

Make sure pfring.h isn't included globally due to apparent redefinition
of pthread_rwlock_t.
8 years ago
Alfredo Cardigliano b6baafb3e3 pfring: hw bypass support
This patch adds support for hw bypass by enabling flow offload in the network
card (when supported) and implementing the BypassPacketsFlow callback.
Hw bypass support is disabled by default, and can be enabled by setting
"bypass: yes" in the pfring interface configuration section in suricata.yaml.
8 years ago
Victor Julien d5f7acd860 decoder: implement IEEE802.1AH 8 years ago
Phil Young 17d9616fde napatech: Implementation of packet counters
added util-napatech module which contains implementation threads
for processing statistics.  And modified source-napatech and
runmode-napatech to instantiate the threads.

napatech: Implementation of packet counters

napatech: implementation of statistics counters

napatech: Implementation of packet counters.

napatech: added util-napatech module

napatech: added utils-napatech module.

added include declaration and napatech specific structure when HAVE_NAPATECH
is defined.

Added util-napatech module to project.
8 years ago
Victor Julien 62b6f9fe25 decode: add config option to disable teredo
Ticket #744.
8 years ago
Victor Julien fee0fdc595 pcap: fix linktype raw issues
On OpenBSD 6.0 and 6.1 the following pcap gets a datalink type of
101 instead of our defined DLT_RAW.

    File type:           Wireshark/tcpdump/... - pcap
    File encapsulation:  Raw IP
    File timestamp precision:  microseconds (6)
    Packet size limit:   file hdr: 262144 bytes
    Number of packets:   23
    File size:           11 kB
    Data size:           11 kB
    Capture duration:    7,424945 seconds
    First packet time:   2017-05-25 21:59:31,957953
    Last packet time:    2017-05-25 21:59:39,382898
    Data byte rate:      1536 bytes/s
    Data bit rate:       12 kbps
    Average packet size: 496,00 bytes
    Average packet rate: 3 packets/s
    SHA1:                120cff9878b93ac74b68fb9216027bef3b3c018f
    RIPEMD160:           35fa287bf30d8be8b8654abfe26e8d3883262e8e
    MD5:                 13fe4bc50fe09bdd38f07739bd1ff0f0
    Strict time order:   True
    Number of interfaces in file: 1
    Interface #0 info:
                         Encapsulation = Raw IP (7/101 - rawip)
                         Capture length = 262144
                         Time precision = microseconds (6)
                         Time ticks per second = 1000000
                         Number of stat entries = 0
                         Number of packets = 23

On Linux it is 12.

On the tcpdump/libpcap site the DLT_RAW is defined as 101:
http://www.tcpdump.org/linktypes.html

Strangely, on OpenBSD the DLT_RAW macro is defined as 14 as expected.
So for some reason, libpcap on OpenBSD uses 101 which seems to match
the tcpdump/libpcap documentation.

So this patch adds support for datalink 101 as RAW.
8 years ago
Victor Julien d61fa0c43c tunnel: refactor tunnel verdict handling
Observed:

STARTTLS creates 2 pseudo packets which are tied to a real packet.
TPR (tunnel packet ref) counter increased to 2.

Pseudo 1: goes through 'verdict', increments 'ready to verdict' to 1.
Packet pool return code frees this packet and decrements TPR in root
to 1. RTV counter not changed. So both are now 1.

Pseudo 2: verdict code sees RTV == TPR, so verdict is set based on
pseudo packet. This is too soon. Packet pool return code frees this
packet and decrements TPR in root to 0.

Real packet: TRP is 0 so set verdict on this packet. As verdict was
already set, NFQ reports an issue.

The decrementing of TPR doesn't seem to make sense as RTV is not
updated.

Solution:

This patch refactors the ref count and verdict count logic. The beef
is now handled in the generic function TmqhOutputPacketpool(). NFQ
and IPFW call a utility function VerdictTunnelPacket to see if they
need to verdict a packet.

Remove some unused macro's for managing these counters.
8 years ago
Victor Julien f18c976a8e flow: counters for total number of flows
flow.tcp
flow.udp
flow.icmpv4
flow.icmpv6
8 years ago
Victor Julien d9908216d8 connect/starttls: handle detection corner cases
When switching protocol from http to tls the following corner case
was observed:

 pkt 6, TC "200 connection established"
 pkt 7, TS acks pkt 6 + adds "client hello"
 pkt 8 TC, acks pkt 7
 pkt 8 is where normally the detect on the 200 connection established
       would run however before detection runs the app-layer is called
       and it resets the state

So the issue is missed detection on the last data in the original
protocol before the switch.

Another case was:

TS ->    STARTTLS
TC ->    Ack "STARTTLS data"
         220
TS ->    Ack "220 data"
         Client Hello

In IDS mode, this made a rule that wanted to look at content:"STARTTLS"
in combination with the protocol SMTP 'alert smtp ... content:"STARTTLS";'
impossible. By the time the content would match, the protocol was already
switched.

This patch fixes this case by creating a 'Detect/Log Flush' packet in
both directions. This will force final inspection and logging of the
pre-upgrade protocol (SMTP in this example) before doing the final
switch.
8 years ago
Victor Julien ab1200fbd7 compiler: more strict compiler warnings
Set flags by default:

    -Wmissing-prototypes
    -Wmissing-declarations
    -Wstrict-prototypes
    -Wwrite-strings
    -Wcast-align
    -Wbad-function-cast
    -Wformat-security
    -Wno-format-nonliteral
    -Wmissing-format-attribute
    -funsigned-char

Fix minor compiler warnings for these new flags on gcc and clang.
8 years ago
Victor Julien 971ab18b95 detect / stream: new 'raw' stream inspection
Remove the 'StreamMsg' approach from the engine. In this approach the
stream engine would create a list of chunks for inspection by the
detection engine. There were several issues:

1. the messages had a fixed size, so blocks of data bigger than ~4k
   would be cut into multiple messages

2. it lead to lots of data copying and unnecessary memory use

3. the StreamMsgs used a central pool

The Stream engine switched over to the streaming buffer API, which
means that the reassembled data is always available. This made the
StreamMsg approach even clunkier.

The new approach exposes the streaming buffer data to the detection
engine. It has to pay attention to an important issue though: packet
loss. The data may have gaps. The streaming buffer API tracks the
blocks of continuous data.

To access the data for inspection a callback approach is used. The
'StreamReassembleRaw' function is called with a callback and data.
This way it runs the MPM and individual rule inspection code. At
the end of each detection run the stream engine is notified that it
can move forward it's 'progress'.
8 years ago
Victor Julien 996112edf5 pktvars: same name pktvars, key-value vars 9 years ago
Victor Julien 5e39486399 pkt-var: use id instead of name pointer 9 years ago
Victor Julien e95a0c1344 alert-debug: print flowbit names from VarNameStore 9 years ago
Victor Julien bfd4bc8233 detect: constify Signature/SigMatch use at runtime 9 years ago
Victor Julien fbd69729aa afl: improve packet fuzz testing
Due to the use of AFL_LOOP and initialization/deinit outside of it,
part of the fuzzing relied on the global 'state' in flow and defrag.
Because of this crashes that were found could not be reproduced. The
saved crash input was only the last in the series.

This patch addresses that. It requires a new output directory 'dump'
where the packet fuzzers will store all their input. If the AFL_LOOP
fails the files will not be removed and this 'serie' can be read
again for reproducing the issue.

e.g.: AFL would work with:
--afl-decoder-ppp=@@

and after a crash is found the produced serie can be read with:
--afl-decoder-ppp-serie=1486656919-514163

The series have a timestamp as name and a suffix that controls the
order in which the files will be 'replayed' in Suricata.
9 years ago
Victor Julien a0580d8805 stream: initialize stream segment pool from mtu
If segments section in the yaml is ommitted (default) or when the
pool size is set to 'from_mtu', the size of the pool will be MTU
minus 40. If the MTU couldn't be determined, it's assumed to be
1500, so the segment size for the bool will be 1460.
9 years ago
Victor Julien 1ba15d3721 mtu: track max mtu for capture devices 9 years ago
Victor Julien 8831e5b375 pkt-var: const name 9 years ago
Victor Julien e072e70ea6 alert: fix rate_filter issues
Fix rate_filter issues: if action was modified it wouldn't be logged
in EVE. To address this pass the PacketAlert structure to the threshold
code so it can flag the PacketAlert as modified. Use this in logging.

Update API to use const where possible. Fix a timout issue that this
uncovered.
9 years ago
Jason Ish f81619a13e defrag: set flag on packets reassembled from fragments
Set the PKT_REBUILT_FRAGMENT on packets that are re-assembled
from fragments.
9 years ago
Victor Julien 7d7ec78cc3 app-layer-protocol: improve detection
Add negated matches to match list instead of amatch.

Allow matching on 'failed'.

Introduce per packet flags for proto detection. Flags are used to
only inspect once per direction. Flag packet on PD-failure too.
9 years ago
Victor Julien 8798bf48b2 profiling: support prefilter engines 9 years ago
Giuseppe Longo 616782aa98 packet: add API for bypass 9 years ago
Jason Ish fa27a76462 logging: add profiling back for non-tmm loggers
The loggers moved away from a TMM required a new
profiling support.
9 years ago
Victor Julien 4271d57157 decode: declare IPPROTO_IPIP if OS doesn't have it 9 years ago
Victor Julien 884fddf035 packet: remove empty and unused UDPVars struct 9 years ago
Victor Julien c9756caeef packet: make tcp/udp/icmp vars union non-anonymous
Clean the whole thing after use.
9 years ago