Commit Graph

10626 Commits (a85340b1ab2aa9490a4d4d937ad0f8808b052dfe)

Author SHA1 Message Date
Jason Ish fcbdc30426 dns: create transaction even if z-bit was set
It appears that DNS servers will still process a DNS request even if the
z-bit is set, our parser will fail the transaction. So create the
transaction, but still set the event.

Ticket #4924
5 years ago
Lukas Sismis de53e07559 dpdk/ice: setup RSS for Intel ICE PMD
Set RSS hash function according to Intel ICE PMD available hash functions

Set hash functions according to the support by the ICE PMD, so that no warning
regarding RSS setting is issued.
5 years ago
Lukas Sismis 3f7a50eeb7 dpdk/ixgbe: setup RSS for Intel IXGBE PMD
Set RSS hash function according to Intel IXGBE PMD available hash functions.

During configuration, a warning appeared stating that RSS hash function
has been changed from one value to the other. This has meant that
the supported hash functions did not cover all required hash functions
by the configuration. This commit solves the warning.
5 years ago
Lukas Sismis 639aa04c5f dpdk/i40e: support RSS on Intel i40e PMD driver
Due to peculiar behavior of i40e PMD driver, the RSS is required to be set
via rte_flow rules or a hash filter as compared to other NICs where RSS is
configured through port configuration structure.
RTE_FLOW rules are created on 5-tuples (as opposed to 3-tuple configured
on the other NICs). Fragmented traffic have been tested with this setup
and it has been proven that fragmented packets of the same flow are
received on the same queue. At the same time, setting 3-tuple on rte_flow
rules have not yield in the expected results.

Notes from the experiments:

- Configuration of 5-tuple (as is in the commit):
    fragmented and nonfragmented packets are received by the same workers
    even when I applied seed to alter them via tcpreplay-edit (option --seed)

- Setting only ETH_RSS_FRAG_IPV4 and ETH_RSS_IPV4 (i.e. setting 3-tuple):
    when setting ETH_RSS_IPV4, the PMD driver says that pctype is not
    supported (generally this means that the "type" of traffic is not
    a valid configuration for the i40e)

- Setting only ETH_RSS_FRAG_IPV4 and ETH_RSS_NONFRAG_IPV4_OTHER:
    this doesn't work well, packets of the same flow are received on
    the different workers (my explanation is that the fragmented packets are
    matched with ETH_RSS_FRAG_IPV4 but the other UDP packets are not matched
    with ETH_RSS_NONFRAG_IPV4_OTHER rte_flow rule (they would be matched with
    ETH_RSS_NONFRAG_IPV4_UDP).
5 years ago
Victor Julien f98df5c3fd dpdk: add RSS flags that are set in the NIC 5 years ago
Victor Julien 56dfec48b9 dpdk: add specific error counters 5 years ago
Lukas Sismis a7faed1245 dpdk: initial support with workers runmode
Register a new runmode - DPDK. This enables a new flag on Suricata start
(--dpdk).

With the flag given, DPDK runmode is enabled.

Runmode loads the configuration and then initializes EAL.

If successful, it configures the physical NICs according to the configuration
file. After that, worker threads are initialized and then are in continuous
receive loop.
5 years ago
Jason Ish 92eb14c5ad datasets: initialize after dropping privileges
Move initialization of datasets to a point after privileges
have been dropped.

Ticket 4239
5 years ago
Philippe Antoine dd32238667 ftp: do not set alproto if one was already found
Ticket: 4857

If a pattern such as GET is seen ine the beginning of the
file transferred over ftp-data, this flow will get recognized
as HTTP, and a HTTP state will be created during parsing.

Thus, we cannot override directly alproto's values

This solves the segfault, but not the logical bug that the flow
should be classified as FTP-DATA instead of HTTP
5 years ago
Victor Julien a02f263e56 app-layer/htp: cleanup test 5 years ago
Victor Julien 0a1c3267e6 htp: rename callbacks to make purpose clearer 5 years ago
Victor Julien 258415b23f stream: unify ack'd right edge handling
Use util function in all code needing the ack'd data.
5 years ago
Victor Julien ac11502629 detect/engine: store buffer name in local array
Instead of storing a name and description as a pointer in DetectBufferType
store them in fixed size arrays. This is in preparation of runtime registration
of buffer types, where a constant name/desc is not available.
5 years ago
Victor Julien 6ee818cb3e stream/reassembly: ignore min_inspect_depth on TCP state CLOSED 5 years ago
Victor Julien 55202f826a detect/http: don't set min-inspect-depth higher than setting 5 years ago
Philippe Antoine 27dd0c6b3d eve/ftp-data: log alert metadata in ftp-data object
Ticket: 4860

instead of directly in root
5 years ago
Philippe Antoine 87d9c44ec5 rust: export constants via cbindgen
so that constants are not defined twice in Rust anc C
So that we are sure they have the same value
5 years ago
Philippe Antoine 8feb9c35ae mime: move FindMimeHeaderTokenRestrict to rust
Also fixes the case where the token name is present
in a value
5 years ago
Modupe Falodun 76131c8cff detect-ipopts: convert unittests to FAIL/PASS APIs
Bug: 4047
5 years ago
Victor Julien ecce116117 detect/fast_pattern: allow for rule time registration
In preparation of more dynamic logic in rule loading also doing
some registration, allow for buffers to be registered as fast_patterns
during rule parsing.

Leaves the register time registrations mostly as-is, but copies the
resulting list into the DetectEngineCtx and works with that onwards.
This list can then be extended.
5 years ago
Victor Julien db27244379 detect: add buffer helper functions 5 years ago
Victor Julien 707b75ccda detect: split register time and detect load time buffer funcs 5 years ago
Victor Julien 5bcaae0a01 detect: use hashes for all buffer to id
Instead of a map that is constantly realloc'd, use 2 hash tables for
DetectBufferType entries: one by name (+transforms), the other by
id. Use these everywhere.
5 years ago
Victor Julien 51dcf3d76a detect: increase SigMatch type from u8 to u16 5 years ago
Victor Julien bb3d49d5bf detect: use bool for uint16_t used as bool 5 years ago
Victor Julien 6d7c1519ed common: fix missing ; in header 5 years ago
Philippe Antoine c9d222a483 detect: allows <> syntax for uint ranges 5 years ago
Philippe Antoine 5af4ef4532 detect: use prefilter values for modes 5 years ago
Philippe Antoine 3f15b2492c detect: errors for rule with impossible conditions
Such as >255 for an uint8 field
5 years ago
Philippe Antoine f4449d3fb3 fuzz: restrict flags passed to AppLayerProtoDetectGetProto
Completes commit 05f9b3ffc6
5 years ago
Jason Ish 7732efbec2 app-layer: include decoder events in app-layer tx data
As most parsers use an events structure we can include it in the
tx_data structure to reduce some boilerplate/housekeeping code
in app-layer parsers.
5 years ago
Philippe Antoine 86f5d33f75 enip: fix int warnings
There seems to fix a real bug when an ENIP connection
has more than 65k transactions
5 years ago
Philippe Antoine 86b5c81ea2 dnp3: fix int warnings
There is a hack to know the type of an integer
and do an explicit cast in the python script
generating the C file

Also extends some bounds check against negative values
5 years ago
Philippe Antoine 53fc70a9a7 protodetect: fix int warnings
There is actually a real evasion with AppLayerProtoDetectPMGetProto
using u16 instead of u32 for buflen
5 years ago
Philippe Antoine 46981ccd98 warning: explicit casts to double 5 years ago
Philippe Antoine b88f015bfb source/pcap: remove unused code 5 years ago
Philippe Antoine 05f9b3ffc6 fuzz: restrict flags passed to AppLayerProtoDetectGetProto
So that rust does not panic with an unhandled value
5 years ago
Victor Julien 39bf623fdd af-packet: add send error counter 5 years ago
Victor Julien 373278438d packetpool: ReleasePacket callback check on getter
Any packet coming from the pool should have `PacketPoolReturnPacket`
as its callback. Check that this is the case.
5 years ago
Victor Julien 8a5b945c7b af-packet: only ref mpeer if needed in tpacket v2
We only use it in autofp mode, for reference counting purposes.

Removes 2 atomic operations per packet in the more common workers
runmode.
5 years ago
Victor Julien d272075da0 af-packet: minor output updates 5 years ago
Victor Julien e9c6ad19b3 af-packet: optimize packet setup
Don't set fields we don't use in V3.
5 years ago
Victor Julien cad0ff9ebb af-packet: add counters on how poll() works
Use `capture.afpacket.*` counter name space.
5 years ago
Victor Julien c7ad3f8d30 af-packet: don't check ifstate per send call in IPS
Instead just accept that the socket state leads to `sendto` errors.
So print at most one error per socket.
5 years ago
Victor Julien dab036727f af-packet: simplify AFPWritePacket
Since return code was ignored by all callers, we can just turn it into a
void function and slightly simplify the logic.
5 years ago
Victor Julien 3f79f452ad af-packet: use BUG_ON for 'impossible' condition 5 years ago
Victor Julien 2fab3ff0e8 af-packet: refactor VLAN hdr handling
Update the packet payload after decode, instead of during IPS send.

This means the updates happen in the capture thread, and the VLAN header
is available to logging as well.

Ticket: #4805.
5 years ago
Victor Julien b9189946f9 af-packet: remove tpacket-v1 support
Ticket: #4796.

V2 (for IDS and IPS) and V3 (for IDS) are widely supported. V2 was introduced
in 2008, so we can safely assume that all systems can run V2+.
5 years ago
Jason Ish 1f6a15cdf3 app-layer/template: don't always enable if unittests built
314ec77f88 had the unintended side affect
of enabling the template parser and detection buffer if unittests were
enabled.

Fix this by using the new `Default` method for registering parsers.
However, the buffer still needs an explicit configuration check.

Also convert Notice debug messages to Debug to reduce output when in
unittest mode.  If we feel stronly this should still be Notice in the
template, that is a conversion we can make in the generation script when
generating a new parser.
5 years ago
Jeff Lucovsky 6a470a84e7 tftp: Change references to echo
This commit changes the references to the "echo" protocol to tftp. The
references to echo are part of the template parser code.
5 years ago
Jeff Lucovsky ca7d097225 netmap: V14 API changes
This commit modifies the Netmap packet handling to use API version 14.

@bmeeks8 contributed many changes instrumental to this effort.
5 years ago
Jeff Lucovsky 3496e543af util: Add sys/ioctl.h to common include
This commit adds another system include file based on autoconf to the
common Suricata include file for convenience.
5 years ago
Jeff Lucovsky 84ee01dbe9 util/log: Include device name with thread count 5 years ago
Philippe Antoine c023116857 range: prevents memory leak of file from HTTP2
If a HTTP2 transaction gets freed before the end of the range
request, we need to have the files container which is in
the state, to transfer owernship of this file to the files
container.

Ticket: 4811
5 years ago
Philippe Antoine 15649424a7 conf: avoid quadratic complexity
Ticket: 4812

When adding many sequence nodes
5 years ago
Modupe Falodun c33cfed704 detect-fragoffset: convert unittests to FAIL/PASS APIs
Bug: #4040
5 years ago
Jason Ish 52b9c12f41 smtp: log transaction even if no email present
The SMTP transaction logger was not writing the log if the email
portion of the logger failed, such as in the case of STARTTLS
where this is no email decoded.

Ticket #4817
5 years ago
Sam Muhammed fcf399b02c detect/proto: convert unittests to FAIL/PASS APIs
Task #4027
5 years ago
Modupe Falodun 2a800d572c detect-icode: convert unittests to FAIL/PASS APIs
Bug: #4045
5 years ago
Modupe Falodun 97801c795b detect-id: convert unittests to FAIL/PASS APIs
Bug: #4046
5 years ago
Sam Muhammed 4076c8b762 detect/siggroup: convert unittests to FAIL/PASS APIs
Task #4028
5 years ago
Benjamin Wilkins e21a50fee6 lua: Fix SCRule functions for match scripts
Save Signature structure to lua register so SCRule functions can work
in match scripts, where no PacketAlert is present

Resolves Feature #2450
5 years ago
Jason Ish 9b71f56728 modbus: free eve thread context on deinit
Was triggering ASAN leak detection.
5 years ago
Jason Ish 55ff912ee7 app-layer: remove IsTxEventAware: never used
The function AppLayerParserProtocolIsTxEventAware is not used so
remove.
5 years ago
Jason Ish 1ad71b96da app-layer: remove tx detect state setter and getter
Instead access detect state through AppLayerParserGetTxData.
5 years ago
Jason Ish 9c67c634c1 app-layer: include DetectEngineState in AppLayerTxData
Every transaction has an existing mandatory field, tx_data. As
DetectEngineState is also mandatory, include it in tx_data.

This allows us to remove the boilerplate every app-layer has
for managing detect engine state.
5 years ago
Eric Leblond 6d5f59696d profiling: fix profiling with sample rate
Rules profiling was returning invalid results when used with sample
rate. The problem was that the sample condition was run twice in the
packet flow. As a result, the second pass was not initializing the
variable storing the initial CPU ticks and the resulting performance
counters were reporting invalid values.

Bug: #4836.
5 years ago
Philippe Antoine 16f4e5f31c detect: file_data keyword works on nfs protocol
Ticket: #4839
5 years ago
Philippe Antoine 6cb6225b28 tcp: rejects FIN+SYN packets as invalid
Ticket: #4569

If a FIN+SYN packet is sent, the destination may keep the
connection alive instead of starting to close it.
In this case, a later SYN packet will be ignored by the
destination.

Previously, Suricata considered this a session reuse, and thus
used the sequence number of the last SYN packet, instead of
using the one of the live connection, leading to evasion.

This commit errors on FIN+SYN so that they do not get
processed as regular FIN packets.
5 years ago
Victor Julien 50e2b973ee stream/tcp: handle RST with MD5 or AO header
Special handling for RST packets if they have an TCP MD5 or AO header option.
The options hash can't be validated. The end host might be able to validate
it, as it can have a key/password that was communicated out of band.

The sender could use this to move the TCP state to 'CLOSED', leading to
a desync of the TCP session.

This patch builds on top of
843d0b7a10 ("stream: support RST getting lost/ignored")

It flags the receiver as having received an RST and moves the TCP state
into the CLOSED state. It then reverts this if the sender continues to
send traffic. In this case it sets the following event:

    stream-event:suspected_rst_inject;

Bug: #4710.
5 years ago
Philippe Antoine be617a3c1b protodetect: opposing side cannot change protocol
Ticket: #4562

As the data which triggered the opposing side
was the same protocol and not another one,
that means the protocol change failed.

Prevents a memory leak in later call of AppLayerParserParse
which would allocate a new state and leak the old one
5 years ago
Philippe Antoine f44bbbb9ad smtp: completes RSET transaction on last multiline
Bug: #4561.
5 years ago
Philippe Antoine f211a330dd swf: right input length for decompression
Also when compress_depth reaches buffer_len

Bug: #4536.
5 years ago
Philippe Antoine 689ac97d72 inspect: debug validation to ensure correct argument 5 years ago
Philippe Antoine c3339c853e detect: fixes InspectionBuffer id with transforms
When InspectionBufferGet gets called with base_id
Later InspectionBufferSetup must also be called with base_id

In case there were transforms, we had base_id != list_id

Not calling InspectionBufferSetup with the right id
resulted in leaving a dangling pointer,
because it was not added to det_ctx->inspect.to_clear_queue

Bug: #4681.
5 years ago
Victor Julien 244dd11c34 flow/manager: fix flows not evicted & freed in time
Flows have been shown to linger for a long time w/o giving up their
resources. This would lead to higher memory use and memcaps getting
reached.

Three main causes have been identified:

Slow passes hash passes. By default the flow manager will scan the
flow hash slowly. It is based on the flow timeout settings, and with
the default config it will take 4 minutes for a full scan to be
complete. This leaves a window for flows that are timed out to linger
for minutes longer than expected.

Flow Manager yields under pressure. The per row TryLock causes work
to be delayed more. The Flow manager will use trylock on a hash row
and will yield immediately if the row is busy. This means that it will
take a full pass before the row is revisited again. If the row holds
busy flows, this could happen many times in a row.

Flow Manager favors evicted flows over active flows. The Flow Manager
will only process the evicted flows if they are present. These flows
have been evicted by workers. The active flows on that hash row will
have to wait until the next hash pass. Of course by then there could
be more evicted flows.

Combined these factors could lead to flows not being considered for
freeing and logging for a very long time, potentially even indefinitly.

The patch addresses the latter two flow manager issues by no longer
using TryLock. It will now simply wait for the lock to be released and
then do its work on it. Additionally for each row both the evicted list
and the active flow list will be processed.

Bug: #4650.
5 years ago
Victor Julien ace349d4d9 af-packet: simplify tpacket-v2 setup code
Setup can no longer fail, so make the function void and remove dead
error checking code.
5 years ago
Victor Julien 2cbfcce0ac af-packet: PacketSetData can't fail; remove check
PacketSetData() can't fail unless the input pointer is NULL, which is
impossible from the af-packet paths calling it. Remove error check to
avoid possible branching.
5 years ago
Victor Julien 12252ba751 af-packet: fix if/down issues with tpacket-v2/autofp
The AFPSwitchState function would close the socket and free the
other resources when the interface went down _and_ the ref cnt was
0. However in autofp mode it was common to get to this point while
packets were still processed in the autofp worker threads, meaning
the ref cnt would not be 0. On the interface coming back up the
initialization code would overwrite the socket and rings, leading
to resource leaks.

Socket ref cnt is decremented from the v2 release callback. If the
callback would get to ref cnt 0, the packet would not be released
in the kernel, but it would (possibly) close the socket if the
iface was down, but not free other resources.

This patch changes the logic to first release the packet to the
kernel and then decrement the ref cnt and it makes the main receive
loop the only one responsible for opening and closing sockets. Wait
with closing the socket and rings until the ref count is 0, which can
happen after AFPSwitchState is called due to packets still being
processed by autofp worker threads.

Bug: #4803.
5 years ago
Victor Julien 3f8e15f70c af-packet: packet checks into debug validate check 5 years ago
Victor Julien 5e05fedc90 af-packet: hide all ebpf/bypass logic behind guards
Leave no runtime checks for bypass/ebpf/xdp if not compiled in.
5 years ago
Victor Julien e63db9d1d2 af-packet: minor code cleanups 5 years ago
Victor Julien e3d20acb98 af-packet: simplify socket handling in tpacket v3
Tpacket v3 only supports workers mode, which means the packet that would
reference a socket won't leave the thread. Therefore keeping a ref count
on the socket is not needed.

This patch removes the per packet reference count increment. The decrement
was missing, so this fixes the ref cnt handling so that after a iface up/
down capture can recover.

It should also lead to a minor performance increase as we avoid a round
of atomic operations per packet.

Bug: #4804.
Bug: #4801.
5 years ago
Victor Julien 6eaaafc360 af-packet: minor config parsing cleanup 5 years ago
Victor Julien be122ceb8f af-packet: remove obsolete define 5 years ago
Victor Julien 558930a192 af-packet: remove zero copy flag
Flag was always set for tpacket v2 and v3, which meant the check
for it in the packet setup paths were useless.
5 years ago
Victor Julien ad862fff37 af-packet: avoid flag colision with kernel
Avoid colision of TP_STATUS_USER_BUSY with TP_STATUS_TS_RAW_HARDWARE,
both were using bit 31.

Bug: #4800.
5 years ago
Victor Julien a022648b9e af-packet: fix soft lockup issues
The Suricata AF_PACKET code opens a socket per thread, then after some minor
setup enters a loop where the socket is poll()'d with a timeout. When the
poll() call returns a non zero positive value, the AF_PACKET ring will be
processed.

The ringbuffer processing logic has a pointer into the ring where we last
checked the ring. From this position we will inspect each frame until we
find a frame with tp_status == TP_STATUS_KERNEL (so essentially 0). This
means the frame is currently owned by the kernel.

There is a special case handling for starting the ring processing but
finding a TP_STATUS_KERNEL immediately. This logic then skip to the next
frame, rerun the check, etc until it either finds an initialized frame or
the last frame of the ringbuffer.

The problem was, however, that the initial uninitialized frame was possibly
(likely?) still being initialized by the kernel. A data race between the
notification through the socket (the poll()) and the updating of the
`tp_status` field in the frame could lead to a valid frame getting skipped.

Of note is that for example libpcap does not do frame scanning. Instead it
simply exits it ring processing loop. Also interesting is that libpcap uses
atomic loads and stores on the tp_status field.

This skipping of frames had 2 bad side effects:

1. in most cases, the buffer would be full enough that the frame would
   be processed in the next pass of the ring, but now the frame would
   out of order. This might have lead to packets belong to the same
   flow getting processed in the wrong order.

2. more severe is the soft lockup case. The skipped frame sits at ring
   buffer index 0. The rest of the ring has been cleared, after the
   initial frame was skipped. As our pass of the ring stops at the end
   of the ring (ptv->frame_offset + 1 == ptv->req.v2.tp_frame_nr) the code
   exits the ring processing loop at goes back to poll(). However, poll()
   will not indicate that there is more data, as the stale frame in the
   ring blocks the kernel from populating more frames beyond it. This
   is now a dead lock, as the kernel waits for Suricata and Suricata
   never touches the ring until it hears from the kernel.

   The scan logic will scan the whole ring at most once, so it won't
   reconsider the stale frame either.

This patch addresses the issues in several ways:

1. the startup "discard" logic was fixed to not skip over kernel
   frames. Doing so would get us in a bad state at start up.

2. Instead of scanning the ring, we now enter a busy wait loop
   when encountering a kernel frame where we didn't expect one. This
   means that if we got a > 0 poll() result, we'll busy wait until
   we get at least one frame.

3. Error handling is unified and cleaned up. Any frame error now
   returns the frame to the kernel and progresses the frame pointer.

4. If we find a frame that is owned by us (TP_STATUS_USER_BUSY) we
   yield to poll() immediately, as the next expected status of that
   frame is TP_STATUS_KERNEL.

5. the ring is no longer processed until the "end" of the ring (so
   highest index), but instead we process at most one full ring size
   per run.

6. Work with a copy of `tp_status` instead of accessing original touched
   also by the kernel.

Bug: #4785.
5 years ago
Victor Julien 8b08b0343d af-packet: define all current TP_STATUS_* flags 5 years ago
Modupe Falodun fac33118cc detect-rfb-secresult: convert unittest to FAIL/PASS APIs
Bug: #4055
5 years ago
Modupe Falodun e9779b0fa0 detect-sameip: convert unittests to FAIL/PASS APIs
Bug: #4057
5 years ago
Sam Muhammed 025fd385cd detect/event: convert unittests to FAIL/PASS APIs
Task #4025
5 years ago
Victor Julien 07ce871da4 packetpool: reset PacketRelease on return to pool
Reset PacketRelease callback to make sure its not set to a capture
specific callback.

As an example:

  0x000055e00af09d35 in AFPReleaseDataFromRing (p=0x7f1d884cb830) at source-af-packet.c:653
  0x000055e00af09dd0 in AFPReleasePacket (p=0x7f1d884cb830) at source-af-packet.c:678
  0x000055e00ab53d7e in TmqhOutputPacketpool (t=0x55e00fb79250, p=0x7f1d884cb830) at tmqh-packetpool.c:465
  0x000055e00af08dec in TmThreadsSlotProcessPkt (tv=0x55e00fb79250, s=0x55e012134790, p=0x7f1d884cb830) at tm-threads.h:201
  0x000055e00af08e70 in TmThreadsCaptureInjectPacket (tv=0x55e00fb79250, p=0x7f1d884cb830) at tm-threads.h:221
  0x000055e00af08f2e in TmThreadsCaptureHandleTimeout (tv=0x55e00fb79250, p=0x0) at tm-threads.h:245
  0x000055e00af0ba76 in ReceiveAFPLoop (tv=0x55e00fb79250, data=0x7f1d884ccb60, slot=0x55e01198e4b0) at source-af-packet.c:1321
  0x000055e00ab55257 in TmThreadsSlotPktAcqLoop (td=0x55e00fb79250) at tm-threads.c:312
  0x00007f1dca9d5609 in start_thread (arg=<optimized out>) at pthread_create.c:477
  0x00007f1dca7c6293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Here the packet was a pseudo packet to handle a timeout condition. But
the ReleasePacket callback was still set to AFPReleasePacket from a
previous use of the Packet.

Bug: #4807.
5 years ago
Sam Muhammed 214ea9bea5 detect/payload: convert unittests to FAIL/PASS APIs
Task #4026
5 years ago
Modupe Falodun a59299d128 detect-gid: convert unittests to FAIL/PASS APIs
Bug: #4041
5 years ago
Juliana Fajardini d581fc82b1 util/unittests: delete PASS_IF macro
The logic flow we want to achieve with unittests, where first we have
all FAIL statements and then just one PASS statement could become more
convoluted with the existence of the PASS_IF macro. Besides, what could
be written as a FAIL_IF might in some cases be written in not so clear
ways with the PASS_IF option available.

Also: fix inverted check values in documentation, update copyright year

Optimization: #4795
5 years ago
Juliana Fajardini f328ba527a detect/dsize: convert unittests to FAIL/PASS API
Also: change them to comply with the deletion of PASS_IF macro &
condense checks for invalid dsizes in one test, have all checks on same
valid dsize happen in a single test.

Task: #4021
5 years ago
Juliana Fajardini b3743cf5c0 unittests (assorted): remove PASS_IF macro
Also small documentation clean up and test adjusments where that
was needed.

affected: counters, decode-vntag, detect-mark

Related to #4795
5 years ago
Sam Muhammed 42d4eb6943 detect-engine: convert unittests to FAIL/PASS APIs 5 years ago
Victor Julien 286c510ece flow: immediately evict tcp reused flows
Since we already know we're going to no longer use it, might as well
evict it right away.
5 years ago
Victor Julien 536291054c flow/bypass: clear memory on bypass
Previously the flow would hold on to the app-layer and segment data
until the end of the flow, even though it would never be accessed again.

This patch clears app-layer and stream data, but not stream ssn as its
used in flow logging.

Bug: #4778.
5 years ago
Victor Julien b19d1df69f flow/bypass: add util func to check if flow is bypassed
To hide the ifdefs for capture offload.
5 years ago
Victor Julien ab8f289bb6 flow/worker: run housekeeping for bypassed packets
Run flow eviction and flow inject queues for bypassed packets as well,
to avoid a scenario where these won't get run at all if too much of the
traffic is bypassed.

Bug: #4779.
5 years ago
Victor Julien 41fee41722 flow/manager: remove obsolete code 5 years ago
Victor Julien ec7e0561e8 flow/bypass: use_cnt desync'd on bypassed flows
Locally bypassed flows had unsafe updates to `Flow::use_cnt` leading to a race
issue. For a packet it would do the flow lookup, attach the flow to the packet,
increment the `use_cnt`. Then it would detect that the flow is in the bypass
state, and unlock it while holding a reference (so alos not decrementing the
`use_cnt`). When the packet was then returned to the packet pool, the flow would
be disconnected from the packet, which would decrement `use_cnt` without holding
the flow lock.

This patch addresses this issue by disconnecting the flow from the packet
immediately when the bypassed state is detected. This moves the `use_cnt`
decrement to within the lock.

Bug: #4766.
5 years ago
Philippe Antoine 416575ea02 pcrexform: use substring and not whole match 5 years ago
Philippe Antoine c9d664b0a0 tftp: StringToAppProto case
So, fuzz_applayerparserparse_tftp will fuzz tftp
5 years ago
Jason Ish 07370ed5c0 queue.h: suppress scan-build warnings
If running under scan-build, use our own implementations of all
the macros which include some code to satisfy scan-build
warnings.
5 years ago
Philippe Antoine 77604d86d6 range: move back files ownership in one case
In the case, we receive a range request with expected
overlap then new bytes, but the response does not get to the
new bytes, we are still skipping, but the HttpRangeContainerBlock
had the ownership of the files, and need to give it back
5 years ago
Philippe Antoine bba70607e8 range: checks that end is after start
Otherwise, we end up allocating too much memory
5 years ago
Philippe Antoine 27b4f165b1 loopback: decodes IPv6 from all OSes
As does wireshark
5 years ago
Philippe Antoine b34c025b52 util: avoid calling snprintf in PrintStringsToBuffer
As we print only one character
5 years ago
Philippe Antoine 53ef65d390 http2: enable by default, even if not in config 5 years ago
Philippe Antoine fa4c7626bd http2: null check during upgrade 5 years ago
Philippe Antoine ea4a509a54 app-layer: disable by default if not in configuration
DNP3, ENIP, HTTP2 and Modbus are supposed to be disabled
by default. That means the default configuration does it,
but that also means that, if they are not in suricata.yaml,
the protocol should stay disabled.
5 years ago
Jason Ish 75bc9d9dd8 queue.h: wrap the system sys/queue.h
Instead of using local implementations for the queue.h macro,
wrap the system provided queue.h and then adding missing
features as needed.

The idea is that Suricata when integrated with another library
that includes sys/queue.h can look at the same source of truth
for these macros.

But not all operating systems include a queue.h with the same
features, and some don't include it at all, like Windows. So
on Windows this will be a full implementation of all the queue.h
features Suricata needs.
5 years ago
Philippe Antoine 6fadb97d5d alert: fixes leak in ThresholdHandlePacketRule
ThresholdHandlePacketRule may take ownership of an allocated
DetectThresholdEntry, and places it in a position of the
array th_entry. But it never got released
5 years ago
Philippe Antoine d21a252238 fuzz: target must use the rules it parsed
DetectEngineReloadThreads does not work for the fuzz targets
as there is no_of_detect_tvs = 0 as we did not register
real threads and slots.

So, we force the flow worker module to use the newly detect engine
conetxt with all it needs
5 years ago
Victor Julien fa72a5add8 flow: free spare pool more aggressively
The flows exceeding the spare pools config setting would be freed
per at max 100 flows a second. After a high speed test this would
lead to excessive memory use for a long time.

This patch updates the logic to free 10% of the excess flows per
run, freeing multiple blocks of flows as needed.

Bug: #4731.
5 years ago
Victor Julien ff97d7c15d threading: force break loop on flow inject
Track availability of break loop callback to avoid overhead.
5 years ago
Victor Julien b788d3345c flow: process evicted flows on low/no traffic
In a scenario where there was suddenly no more traffic flowing, flows
in a threads `flow_queue` would not be processed. The easiest way to
see this would be in a traffic replay scenario. After the replay is done
no more packets come in and these evicted flows got stuck.

In workers mode, the capture part handles timeout this was updated to
take the `ThreadVars::flow_queue` into account.

The autofp mode the logic that puts a flow into a threads `flow_queue`
would already wake a thread up, but the `flow_queue` was then ignored.
This has been updated to take the `flow_queue` into account.

In both cases a "capture timeout" packet is pushed through the pipeline
to "flush" the queues.

Bug: #4722.
5 years ago
Victor Julien 31977170a8 threading: minor cleanups 5 years ago
Jeff Lucovsky 314ec77f88 unittests/template: Register template unittests 5 years ago
Jeff Lucovsky 6e149cdec3 unittests/enip: Register ENIP unittests 5 years ago
Philippe Antoine 8a50edbd10 pcre: fixes a memory leak on alloc error 5 years ago
Philippe Antoine 8536048443 http2: do not try to upgrade if http2 is disabled in config 5 years ago
Philippe Antoine 42ba421ca9 http2: flatten code style 5 years ago
Philippe Antoine 527415dba0 protodetect: handle all gaps, even when depth is reached 5 years ago
Philippe Antoine 586522e1e9 pcre: local match data for pcrexform 5 years ago
Philippe Antoine c64a1f6a09 pcre: use thread-storage for matches 5 years ago
Philippe Antoine 3b690e53c8 pcre: using de_ctx in unit tests for free function 5 years ago
Philippe Antoine a049a6c29c pcre: creates a match structure per match run
So that DetectPcrePayloadMatch is thread safe
and does not rewrite a shared parse_regex.match structure
5 years ago
Philippe Antoine 78cf9cfc5f http: range: remove assert that can happen 5 years ago
Philippe Antoine 65a6f61004 http: delete obsolete range log
Commit d776d72711
has been transfering ownership of file container

So, we cannot log it
5 years ago
Philippe Antoine 9b3c355c20 fuzz: adds one target with predefined rules 5 years ago
Jason Ish df0ed6fda4 af-packet: use configured cluster-id when checking for fanout
When testing for fanout support a cluster-id of 1 was always being
used instead of the configured cluster-id. This limited fanout
support to only one Suricata instance.

Instead of hardcoding an ID of 1, use the configured cluster-id.

Also make cluster_id a uint16_t instead of an int in AFPThreadVars.

Redmine issue:
https://redmine.openinfosecfoundation.org/issues/3419
5 years ago
Philippe Antoine 3a230c2208 ipv6: simpler generic overlap condition
This also changes the behavior, as the condition is checked in
every case cf ipv6-malformed-fragments-8
5 years ago
Philippe Antoine 066a3d619e fuzz: updates README
Uses main oss-fuzz repository

More complete flags proposition for direct compilation
5 years ago
Philippe Antoine cb30772372 style: remove latest warnings
about unused variables
5 years ago
Philippe Antoine f6ba3699bb flowint: same analysis warnings as flowbits 5 years ago
Philippe Antoine bce3c46874 pcre2: remove PCRE1 as dependency 5 years ago
Philippe Antoine 56f664af6b pcre2: follow code naming style 5 years ago
Philippe Antoine e10d930660 pcre2: only one DetectParseRegex structure 5 years ago
Philippe Antoine 2a1690e110 pcre2: move header include to suricata-common only 5 years ago
Philippe Antoine 48dd0cf804 pcre2: check for PCRE2_ERROR_UNSET
Needs maybe to be generalized
5 years ago
Philippe Antoine 3de99a214c pcre2: migrate keywords parsing 5 years ago
Philippe Antoine 2dea9a1e37 pcre: use pcre2 to parse detect pcre itself 5 years ago
Philippe Antoine f0f3295ba0 pcre: migrate detect-pcre to pcre2
pcre2 substrings need special free...
5 years ago
Philippe Antoine 3633c48e6e pcre2: migrate utility uses of pcre 5 years ago
Philippe Antoine 95a07c8957 pcre2: migrate transform pcrexform 5 years ago
Philippe Antoine 98f84d5a9b http2: follow range requests
Move the content-range parsing code to rust
5 years ago
Philippe Antoine bded2ec92e range: post process out of order blocks
A block is determined out of order on opening.
But on closing, the gap before it may have been filled.
So, we must post-process it, ie iterate over the red and black
tree so see what blocks we can get.
5 years ago
Philippe Antoine 3b9a4f9f19 range: no final zero in key
for future compatibility with rust
5 years ago
Philippe Antoine 7819f3262f http: more consistent return values for HTPFileOpenWithRange 5 years ago
Philippe Antoine bb7cc6f5e3 http: increase tracked files for ranges as well 5 years ago
Philippe Antoine d776d72711 http: range transfering ownership of file container
To make concurrency reasoning clearer
5 years ago
Philippe Antoine 3ed38d2d5d http: avoid one lock for range append data
Better structure design to ensure that one flow maximum
is owning and appending into the file, adding fileOwning field.

Adds also a gap field in a range buffer, so that we can
feed the gap on closing, when we are protected from concurrency
by a lock, (lock which got removed in the append path)

Fixes memcap when encountering a duplicate while inserting
in red and black tree

Adds many comments
5 years ago
Victor Julien 2ef857e533 thash: add debug validation check for use_cnt 5 years ago
Victor Julien 7a797631e0 http/range: cleanup and simplification
Simplify locking by using the THashData lock instead of a separate
range lock.

Avoid size_t in function arguments.

Clean up file handling functions.

Implement handling of alloc errors.

Rename yaml entry to byterange

Unify public api naming
5 years ago
Victor Julien 5ffc9622b3 http/range: optimize struct layout 5 years ago
Victor Julien fc5632664c http/range: fix memory leak on out of order ranges 5 years ago
Philippe Antoine e82416a415 http/range: reassemble files from different flows with range
adds a container, ie a thread safe hash table whose
key is the filename

keep a tree of unordered ranges, up to a memcap limit

adds HTPFileOpenWithRange to handle like HTPFileOpen
if there is a range : open 2 files, one for the whole reassembled,
and one only for the current range
5 years ago
Philippe Antoine 5cb996efcf util: export Djb2 hash string function 5 years ago
Philippe Antoine f0012f97b2 util: adds util function SCBufferCmp
Compares two buffers with their two sizes
5 years ago
Victor Julien 13a6eb3522 detect: suppress scan-build warning 5 years ago
Jason Ish 102f0c74ff lua: move lua includes to util-lua.h
Moves Lua includes to util-lua.h instead of suricata-common
so plugins (or library users) don't need to be aware of the
Lua that Suricata was built with. Instead only source files
that need to be Lua aware can include util-lua.h.
5 years ago
Philippe Antoine ae4e3e95d5 transform: use generic Free function
to free subelements in the context structure
5 years ago
Philippe Antoine ca054f7264 fuzz: fixes a leak in applayerparse target
If a protocol change was requested, the target did not handle
it as Suricata, as the target is meant to handle only one
app-layer protocol.
5 years ago
Jason Ish 72369fd8ce plugins: check if eve file type is registered
Instead of the 2 registration functions, just use one with
a utility method to check for conflicts against built-in
types.
5 years ago
Jason Ish a8b603e509 plugins: remove internal fields
The internal flag is not really used. This also reverts the behaviour of
the plugin refactor of passing a ConfNode based on the plugin name
instead of the eve ConfNode.
5 years ago
Jason Ish 4ab1950760 eve/syslog: remove "plugin" naming 5 years ago
Jason Ish 4cd99fc266 eve: register internal output file types
Register known eve output file types during eve registration. This
removes the function to load internal plugins as they are not really
plugins and moves the registration of them into their respective
subsystem.
5 years ago
Jason Ish 784a080201 plugins: rename SCPLuginFileType to SCEveFileType
With internal code using the plugin API to register an Eve
filetype, the name plugin no longer makes sense. This is
part of my idea that internal plugins aren't plugins at all,
and the plugin interface should be an abstraction over
internal APIs.

Along that idea, this is the start of a refactor of the plugin
file types to be internal, where the plugin API is just an external
interface to that internal API.
5 years ago
Jeff Lucovsky f221c344ed output/syslog: Register syslog plugin
This commit completes the syslog conversion to an internal output plugin
with registration to make it available as an output file type.
5 years ago
Jeff Lucovsky b4da070c59 output/plugin: Refactor eve registration function 5 years ago
Jeff Lucovsky 98d4f9eaa4 output/syslog: Refactor syslog definitions 5 years ago
Jeff Lucovsky be9fbe3230 output/syslog: Convert syslog eve output to plugin
This commit converts the "built-in" syslog eve output handler into an
internal plugin.
5 years ago
Jeff Lucovsky 0e3773ddef output/json: Refactor internal routines 5 years ago
Jeff Lucovsky 5ba08dfef5 plugin: Refactor and create registration function
This commit refactors the plugin registration code and creates an API
for plugin registration
5 years ago
Jeff Lucovsky 18cc0fefab output/plugin: Load internal plugins
This commit adds an interface for loading plugins that are internal to
Suricata. These are always loaded and are in a modified format from
external pliguns.
5 years ago
Jeff Lucovsky 0ee5532036 plugin: Add "not set" plugin value type
This value is used as a sentinel to determine if the configured
eve.json filetype was located.

First, the built-in and internal output plugins are checked. If the
sentinel value remains set, the external plugins are searched for the
filetype.
5 years ago
Juliana Fajardini b3f447a0df util/lua-common: use lua_pushnumber for SCFileInfo 5 years ago
Juliana Fajardini 1315cb793b util/lua-common: fix SCFileInfo bug & doc comment
The callback for FileInfo was returning the wrong value, resulting
in loss of some tuple values for one calling SCFileInfo in a script.

The documentation comment wasn't mentioning the sha items that are
pushed.
5 years ago
Juliana Fajardini 8b53468d32 util/lua-common: use pushinteger w/ byte & pkt cnt
LuaCallbackStatsPushToStackFromFlow tuple is composed of integer values
not all of them had been converted to lua_pushinteger yet.
5 years ago
Juliana Fajardini 82cd125c62 util/lua-common: update copyright year 5 years ago
Juliana Fajardini 9b6ce27487 util-lua-common: use lua_pushinteger w/ int values
replace lua_pushnumber with lua_pushinteger for SCFlowStats and
SCRuleIds.
5 years ago
Juliana Fajardini 00d7a152eb lua/output: fix typo 5 years ago
Juliana Fajardini 7592a9be43 lua: use pushinteger for int in flow/packet tuples 5 years ago
Shivani Bhardwaj d166acbdab applayer: error if probes are null but port is not
If the default port is set via the Rust registration table but the probe
fns to server and to client are set to None, the port is never used.
Setting port in such a case is useless so error out.
5 years ago
Jason Ish 54be743c48 prelude: remove the prelude output
It was broken in 6 and that didn't cause much issue. Just remove
it for 7.
5 years ago
Victor Julien e1035fd3ae detect/prefilter: bail early if possible 5 years ago
Victor Julien 88bb23b7cf detect/prefilter: update tx_min_progress to uint8_t
Now that our make progress value is 47, we don't need an int.
5 years ago
Victor Julien ed87784907 detect: enforce max app-layer progress
Allow progress values in the range 0-47 so we have 48 bits to track
prefilter engines.

Mark bits 48-62 as reserved explicitly.

Add debug validation checks to make sure the reserved space isn't used.
5 years ago
Victor Julien 932cf0b6a6 detect: track prefilter by progress, not engine
Fix FNs in case of too many prefilter engines. A transaction was tracking
which engines have run using a u64 bit array. The engines 'local_id' was
used to set and check this bit. However the bit checking code didn't
handle int types correctly, leading to an incorrect left shift result of
a u32 to a u64 bit value.

This commit addresses that by fixing the int handling, but also by
changing how the engines are tracked.

To avoid wasting prefilter engine tracking bit space, track what
ran by the progress they are registered at, instead of the individual
engine id's. While we can have many engines, the protocols use far
fewer unique progress values. So instead of tracking for dozens of
prefilter id's, we track for the handful of progress values.

To allow for this the engine array is sorted by tx_min_progress, then
app_proto and finally local_id. A new field is added to "know" when
the last relevant engine for a progress value is reached, so that we
can set the prefilter bit then.

A consquence is that the progress values have a ceiling now that
needs to fit in a 64 bit bitarray. The values used by parsers currently
does not exceed 5, so that seems to be ok.

Bug: #4685.
5 years ago
Victor Julien 9a09fe454b flow: log action applied to all packets
Log if action applied to whole flow is drop or pass.
5 years ago
Victor Julien 3874d08015 tests: fix drop test; cleanup
SigTestDropFlow04 was incorrectly expecting an alert in the packet
following a "drop" packet. The first drop is applied to the flow, so
it should lead to the 2nd packet being dropped before inspection is
run.

Clean up the test as well.
5 years ago
Victor Julien e36b9b89a1 detect/tests: improve detection entry
Lots of tests still use SigMatchSignatures as their main detection
entry function, which bypassed some logic. Make it match main logic
more closely.
5 years ago
Victor Julien 3f4110af32 tests: clean up drop test 5 years ago
Victor Julien 802c1ffee3 detect: enforce flow drops earlier
Enforcing flow drops is now done earlier in the detection engine and
moved out of the IP-only engine where it didn't belong.
5 years ago
Victor Julien aa93984b7e detect: unify alert handling; fix bugs
Unify handling of signature matches between various rule types and
between noalert and regular rules.

"noalert" sigs are added to the alert queue initially, but removed
from it after handling their actions. This way all actions are applied
from a single place.

Make sure flow drop and pass are mutually exclusive.

The above addresses issue with pass and drops not getting applied
correctly in various cases.

Bug: #4663
Bug: #4670
5 years ago
Victor Julien ae89874b06 detect: remove dead code 5 years ago
Victor Julien 33c8fda795 detect/lua: use BIT_U32 for flags 5 years ago
Victor Julien dc6755bf8e detect/lua: minor cleanup 5 years ago
Victor Julien 093ed6f9bc output/tx: check flags using BIT_U32 5 years ago
Victor Julien 29d5eb969e packet: use BIT_U32 for flags 5 years ago
Victor Julien ce18f4b8e2 detect/mpm: micro optimization for initialization
Do less expensive check first.
5 years ago
Victor Julien dfe71bb773 detect: remove ticker
Last consumer of it has been converted.
5 years ago
Victor Julien 9a5c666b26 detect/http: clean up header buffer logic
Simplify and clean up header buffer management. The code was designed
to track buffers for several transactions in parallel, from when the
detection engine wasn't aware of transactions.

For http.start and http.header_names use generic mpm and inspect
functions.
5 years ago
Philippe Antoine ca760e305c ipv6: decoder event on invalid length
From RFC 2460, section 4.5,
each fragment, except the last one, must have a length
which is a multiple of 8
5 years ago
Philippe Antoine 0b0649d98e http2: http.header keyword now works for HTTP2
As well as http.header.raw
5 years ago
Philippe Antoine 9b9f909d7d http2: http.header_names keyword now works for HTTP2 5 years ago
Philippe Antoine 547e9f4ab4 http2: http.host normalized keyword now works for HTTP2 5 years ago
Victor Julien 04ba6dc138 ftp: support per-tx file accounting 5 years ago
Victor Julien 0867b0dbcd smtp: support per-tx file accounting 5 years ago
Victor Julien d74c18ee28 http: support per-tx file accounting 5 years ago
Victor Julien 56d3e28a3a filestore: track files getting stored per tx
Avoid evicting a tx before the filedata logger has decided it is
done.
5 years ago
Victor Julien ca124b033e filestore: store chunks in packet direction
Storing too early can lead to files being considered TRUNCATED if the
TCP state is not yet CLOSED when logging is triggered. This has been
observed with FTP-DATA and might also be an issue with simple HTTP.
5 years ago
Victor Julien c78f5ac316 app-layer/transactions: track files opens and logs
To make sure a transaction is not evicted before all file logging is complete.
5 years ago
Victor Julien 45dc4cdeec eve/files: log in packet direction only
Bug: #3703.

Don't log files too soon.
5 years ago
Vladimir Ivchenko e89e563eb4 GRE: Handling pptp without payload
If one of the ppp peers sends a packet with an acknowledge flag,
the ppp payload will be empty and DecodePPP will return TM_ECODE_FAILED.
To handle this case, the packet_length field in the GRE extended header (https://tools.ietf.org/html/rfc2637#section-4.1) is used.
DecodeGRE no longer tries to parse PPP payload if packet_length is zero.
5 years ago
Joshua Lumb cf9b2b5fd1 detect-dsize: Add ! operator for dsize matching 5 years ago
Victor Julien cd40fcdea7 macset: adjust test to pass after fix 5 years ago
Eric Leblond 328bdf2c61 macset: fix memory size check 5 years ago
Eric Leblond d7468c55ca flow: be sure to check hash till the end 5 years ago
Eric Leblond e531530a67 flow: add comment on flow handling 5 years ago
Eric Leblond c1bffa9545 stream: increase memcap on memory errors 5 years ago
Eric Leblond 0e70958e67 util/streaming: improve error handling
It differentiates memory error than regular ones.
5 years ago
Eric Leblond cce7e4f4cb flow: fix a debug assert
As the FlowBypassedTimeout function is interacting with the capture
method it is possible that the return changes between the call that
did trigger the timeout and the actual state (ie if packets arrive
in between the two calls). So we should not use the call to
FlowBypassedTimeout in the assert.
5 years ago
Eric Leblond 9c89bc80d0 flow: document FlowBypassedTimeout
Main point is to document it is interacting with the capture
layer.
5 years ago
Eric Leblond 9a4ef6b8fc flow: more accurate flow counters
Don't increment the flow timeout counter for flows that are not
really timeout (as use_cnt is non zero). And also don't take into
account bypassed flows in the counter for flow timeout in use.
5 years ago
Victor Julien c51042e093 flow/worker: handle timeout edge case
In the flow worker timeout path it is assumed that we can hand off
flows to the recycler after processing, implying that `Flow::use_cnt` is 0.
However, there was a case where this assumption was incorrect.

When during flow timeout handling the last processed data would trigger a
protocol upgrade, two additional pseudo packets would be created that were
then pushed all the way through the engine packet paths. This would mean
they both took a flow reference and would hold that until after the flow
was handed off to the recycler. Thread safety implementation would make
sure this didn't lead to crashes.

This patch handles this case returning these packets to the pool from
the timeout handling.
5 years ago
Victor Julien c5556b5dd9 flow/worker: set proper end flag 5 years ago
Victor Julien 61f6fe037d flow/manager: set proper end flag 5 years ago
Philippe Antoine bbbb816ed6 detect: debug validation for list ids overflows 5 years ago
Victor Julien 86681c9d7c detect: move init only array to init data 5 years ago
Victor Julien 22dfcc928c detect/analyzer: use rule style pretty print for patterns 5 years ago
Victor Julien 5703aec44e detect/content: generalize pattern pretty printing 5 years ago
Victor Julien e7a74348d7 detect/profile: add support for tx inspection
Add 'inspect_type' "packet" and "tx" for the two record types. Add more metadata
when available.
5 years ago
Victor Julien a2e37522bb detect/analyze: dump patterns facility
Dump all patterns to `patterns.json`, with the pattern, a total count (`cnt`),
count of how many times this pattern is the mpm (`mpm`) and some of the flags.

Patterns are listed per buffer. So payload, http_uri, etc.
5 years ago
Victor Julien 84872ecc54 detect/content: add some more dsize tests 5 years ago
Victor Julien af104dd223 detect/dsize: set depth flag when applying dsize as depth 5 years ago
Victor Julien 36d3c3cb8e detect/analyzer: count mpm with depth, endswith 5 years ago
Victor Julien de4addbc48 detect/analyzer: show payload separately in group dumping 5 years ago
Victor Julien ef89643107 detect/analyzer: add icmp to rule group output 5 years ago
Victor Julien f49c181ceb detect/analyzer: display per rule prefilter details 5 years ago
Victor Julien 16ea200846 detect/analyzer: count prefilter per rule group 5 years ago
Victor Julien 1c5842df12 detect/analyzer: add per rule mpm block to rules.json 5 years ago
Victor Julien 3660b8f829 detect/analyzer: support buffer names in sgh dump 5 years ago
Victor Julien 0ee7159d1d flow: determine packet direction once per packet 5 years ago
Victor Julien 4c7eb64411 decode: convert 'action' macros to inline funcs
Make sure most common branch is handled first to assist branch
prediction.

Macros still play a small role to please our 'action' cocci check.
5 years ago
Victor Julien 2d1580233e detect/mpm: turn factory array into list 5 years ago
Victor Julien b48ccb8d1f detect/stream: don't run mpm on packet if stream is available 5 years ago
Michael Smith a64783b3e2 unix-socket: Avoid spurious logs on close
Avoid spurious logs when suricatasc closes connection.

Use SCLogDebug for control connection EOF, and SCLogError for an error.

As Chandan Chowdhury described in redmine 3685. This makes the logging
consistent with the older `if (client->version <= UNIX_PROTO_V1)` block
about 20 lines above, and avoids polluting the logs with
`Unix socket: lost connection with client`.
5 years ago
Philippe Antoine 44bd3169eb dnp3: regenerate object decoding code
Ticket: #4558
So as to avoid intra-structure overflow
5 years ago
Victor Julien 9551cd0535 threading: don't pass locked flow between threads
Previously the flow manager would share evicted flows with the workers
while keeping the flows mutex locked. This reduced the number of unlock/
lock cycles while there was guaranteed to be no contention.

This turns out to be undefined behavior. A lock is supposed to be locked
and unlocked from the same thread. It appears that FreeBSD is stricter on
this than Linux.

This patch addresses the issue by unlocking before handing a flow off
to another thread, and locking again from the new thread.

Issue was reported and largely analyzed by Bill Meeks.

Bug: #4478
5 years ago
KevinWang cbd03c7ea4 output/redis: Fix possible segv 5 years ago
Juliana Fajardini ff976df704 stream-tcp-reassemble: fix typo, updt copyright yr 5 years ago
Juliana Fajardini 613f9b2f5a stream-tcp-reassemble: fix ConfGetBool unc'kd call 5 years ago
Juliana Fajardini 2e0d76e6e7 stream-tcp: fix typos, update copyright year 5 years ago
Juliana Fajardini 4839088359 stream-tcp: fix ConfGetBool unchecked call 5 years ago
Juliana Fajardini 7198355324 util-napatech: fix typos, update copyright year 5 years ago
Juliana Fajardini fbade25848 util-napatech: fix ConfGetBool unchecked call 5 years ago
Juliana Fajardini 09ea412614 util-debug: fix unchecked ConfGetBool call 5 years ago
Shivani Bhardwaj 8fd47cb84c smtp: fix clang fmt 5 years ago
Shivani Bhardwaj 58ac9b0f38 nfs: Add rust registration function
Get rid of the C glue code and move registration completely to Rust.
5 years ago
Shivani Bhardwaj de50ac631e nfs: Change fn sign as per rust registration requirement
Registering parsers in Rust requires signatures to be a certain way and
compatible with C. Change signatures of all the functions.
Probe fn has also been changed to return AppProto as required by the new
fn signature.
5 years ago
Shivani Bhardwaj e5c948df87 smb: Add rust registration function
Get rid of the C glue code and move registration completely to Rust.
5 years ago
Shivani Bhardwaj 6420df84b7 smb: Change fn sign as per rust registration requirement
Registering parsers in Rust requires signatures to be a certain way and
compatible with C. Change signatures of all the functions.
5 years ago
Jason Ish 222e55847c flow: provide flags accessor function
Add an accessor function for flow flags. To be used by Rust where
the flow struct is an opaque data type.
5 years ago
Victor Julien 843c4b20da stream: check if ACK packet is outdated
Outdated packets are ACK packets w/o data that have an ACK value
lower than our last_ack and also don't have an SACK records that
are new.

This can happen when some packets come in later than others (possibly
due to different paths taken).
5 years ago
Victor Julien b7a79978ac stream/sack: clean up includes 5 years ago
Victor Julien 8eccd02c94 stream/sack: minor debug improvements 5 years ago
Victor Julien 35c2a02eb8 stream: minor debug additions 5 years ago
Victor Julien b08a7b9a66 stream: update memcaps in code to match config 5 years ago
Philippe Antoine f77b027ada app-layer/pd: review bailout conditions
To take TCP window into account
And to actually bail out if we received too much data
where the limit is configured by stream.reassembly.depth
5 years ago
Victor Julien 7a114e506a app-layer/pd: only consider actual available data
For size limit checks consider only available data at the stream start
and before any GAPS.

The old check would consider too much data if there were temporary gaps,
like when a data packet was in-window but (far) ahead of the expected
segment.
5 years ago
Victor Julien be1baa8cab streaming/buffer: account sbb data size
When tracking data track the size of the blocks so that in case
of gaps we can still know how much data we hold.
5 years ago
Juliana Fajardini b8499de498 detect/iprep: convert to FAIL/PASS API 5 years ago
Shivani Bhardwaj a17da8374a counters: only print alerts if stats are enabled 5 years ago
Juliana Fajardini b24fb5781b detect: fix typos and update copyright year 5 years ago
Juliana Fajardini a15fada727 detect: fix bug where rule without sid is accepted
Before, if Suricata parsed a rule without a 'sid' option, instead of
failing that rule, the rule was parsed and attributed a sid 0.
Changes to:
detect-parse:
- add logic to filter out rules without sid;
- change unittest which didn't have a sid, but used to pass.
detect-sid: add unittest for rules without sid or with sid: 0
5 years ago
Philippe Antoine 0eefd90a93 fuzz: only build fuzz_sigpcap_aware if asked
With the other fuzz targets, and do not build it if fuzzpcap
is available but we did not want to build the fuzz targets
5 years ago
Eric Leblond 2c8c043185 stream/tcp: limit ACK validation
Only limit ACK value validation for packet where the ACK bit is
set.
5 years ago
Eric Leblond 556570f7dd stream/tcp: don't reject on bad ack
Not using a packet for the streaming analysis when a non zero
ACK value and ACK bit was unset was leading to evasion as it was
possible to start a session with a SYN packet with a non zero ACK
value to see the full TCP stream to escape all stream and application
layer detection.

This addresses CVE-2021-35063.

Fixes: fa692df37 ("stream: reject broken ACK packets")

Ticket: #4504.
5 years ago
Eric Leblond 0d81173d6e stream/tcp: update ack handling logic
Only update the ack value of a session for regular packets when
the ACK bit is set.
5 years ago
Victor Julien d8d1fbe443 detect/files: fix buffer tracking with multiple files 5 years ago
Victor Julien 3c1cc1e345 mqtt: move sub/unsub limits into app-layer config 5 years ago
Sascha Steinbiss 4c0ef73bf2 detect/mqtt: add topic inspection limit
We add a new 'mqtt.(un)subscribe-topic-match-limit' option
to allow a user to specify the maximum number of topics in
a MQTT SUBSCRIBE or UNSUBSCRIBE message to be evaluated
in detection.
5 years ago
Philippe Antoine 33fa7ab596 smtp: null terminate before calling strtoul
by copying in a temporary buffer
as is done in ByteExtractString
5 years ago
Philippe Antoine 4d2f9cc8a0 swf: right input length for decompression 5 years ago
Philippe Antoine 7d0a39412b detect: use u32 for InspectionBufferMultipleForList
So that we do not have an endless loop casting index to
u16 and having more than 65536 buffers in one transaction

Changes for all protocols, even ones where it is impossible
to have such a pattern, so as to avoid bad pattern copy/paste
in the future
5 years ago
Victor Julien e611adf3dc detect: set event if max inspect buffers exceeded
If a parser exceeds 1024 buffers we stop processing them and
set a detect event instead. This is to avoid parser bugs as well as
crafted bad traffic leading to resources starvation due to excessive
loops.
5 years ago
Victor Julien 3dc50322db detect: fix multi inspect buffer issue; clean up
Fix multi inspect buffer API causing cleanup logic in the single
inspect buffer paths. This could lead to a buffer overrun in the
"to clear" logic.

Multi buffers now use InspectionBufferSetupMulti instead of
InspectionBuffer. This is enforced by a check in debug validation.

Simplify the multi inspect buffer setup code and update the callers.
5 years ago
Victor Julien 23d7beb458 detect: reformat events table 5 years ago
Philippe Antoine 0c948142b9 enip: improve probing parser
Strict length for register sessions
NOP command must have options=0
5 years ago
Philippe Antoine 8bf6530540 config: fix null dereference in MacSetRegisterFlowStorage
Crash happens with
--set outputs.eve-json.types.files.force-magic=yes
5 years ago
Victor Julien 86e600dab8 unittests: optimize RunmodeIsUnittests() 5 years ago
Victor Julien 4ecde6efb0 stream: packet to stream flags macro 5 years ago
Victor Julien beb6b1e0d1 packets: more detailed entry debug for detect/stream 5 years ago
Jeff Lucovsky 61fa748e9d decode/vxlan: Delay var init until needed
This commit modifies the var initialization slightly until after
integrity checks have been performed.
5 years ago