Commit Graph

3693 Commits (6674f4892ca3ebfc993c315f2d73f9e3d3b72be1)

Author SHA1 Message Date
Victor Julien 6e18ed0489 luajit flowvar support
This patch adds flowvar support to luajit. It does so by exposing two special
C functions to the luajit scripts: ScFlowvarGet and ScFlowvarSet.
14 years ago
Ken Steele 9d677ea006 Clear the PKT_ALLOC flag when storing Packets into the Packet pool.
The PKT_ALLOC flag is set by PacketGetFromAlloc(), which needs to be
cleared for Packets in the Packet Pool, so clear the flag here.
14 years ago
Ken Steele 9c7b411a5d More PacketGetFromMalloc() to allocate packets. 14 years ago
Ken Steele 394f99e32c Use PacketGetfromAlloc() for packet allocation instead of SCMalloc.
Only changed in one file for testing.
14 years ago
Anoop Saldanha ab1f8afbc3 Removed Signature->order_id and replaced it with Signature->num. 14 years ago
Anoop Saldanha 43d1229dfa 1. Fix assignment of signums, which affected how we used read
sigs(priority wise) inside staging.

   Previously we would assign signums before sig ordering, and hence the
   order didn't actually reflect the order of the sig in the
   sig_list(assuming sig reordering changed the sig_list).  Staging would
   use the old sig_nums to decide the priority of sigs.
2. Fix sig ordering for flowvar, flowbits, flowint, pktvar sigs.   We have
   introduced a new priority to treat sigs with set + read as lower
   priority compared to set only sigs.
3. Previously we treated sigs with a "priority(keyword)" > another sig's
   priority, as a sig with greater priority than the later.  We have
   reversed it.  Now the sig priority ordering is 1,2,.etc.  Updated
   sigordering unittests to reflect the same.
14 years ago
Anoop Saldanha 9219079e1a Allow protocols to have both app layer keywords, as well as transaction
based ones.

Our general logic and assumption is protocols either support one of the
above and not have both.
14 years ago
Anoop Saldanha a490176c8a More lock fixes for the transaction update. Issues reported by Coverity. 14 years ago
Anoop Saldanha 7cf4042337 Fix luajit compilation failure introduced by the transaction update.
Fix coverity lock issues reported by transaction update as well.
14 years ago
Anoop Saldanha d4d18e3136 Transaction engine redesigned.
Improved accuracy, improved performance.  Performance improvement
noticeable with http heavy traffic and ruleset.

A lot of other cosmetic changes carried out as well.  Wrappers introduced
for a lot of app layer functions.

Failing dce unittests disabled.  Will be reintroduced in the updated dce
engine.

Cross transaction matching taken care of.  FPs emanating from these
matches have now disappeared.  Double inspection of transactions taken
care of as well.
14 years ago
Anoop Saldanha 6dcde9d7e9 hsbd mpm and packet mpm share same mpm ctx id.
This is a bug emanating from we having a var reference for hsbd mpm,
but failing to initialize it, and we default to using the packet mpm.
14 years ago
Ken Steele 93e7304117 Preserve PKT_ALLOC flag inside PACKET_RECYCLE().
The PKT_ALLOC flag was being cleared by PACKET_RECYCLE(), which could
then result in a packet being pushed back to the Packet ring buffer
incorrectly.
14 years ago
Ken Steele 699d9e01f1 Move memset() out of PACKET_INITIALIZE()
The memset() inside PACKET_INITIALIZE() is redundant in some cases and
it is cleaner to do as part of the memory allocation. This simplifies
changes for integrating Tilera mPIPE support because the size of memory
cleared in that case is different from SIZE_OF_PACKET.

For the cases where Packets are directly allocated and then call
PACKET_INITIALIZE() without memset() first, this patch adds memset() calls.

A further change would use GetPacketFromAlloc() directly.
14 years ago
Victor Julien 724ad9e8e7 Detect L1 cache line size at build time. Fall back to 64 bytes if detection failed. 14 years ago
Victor Julien 53fe756798 NFQ: convert batchcount related yaml errors to warnings. 14 years ago
Eric Leblond 703e5848e4 nfq: add errno display when verdict fail
In case of error, errno is set by sendmsg which is called by
nfnetlink and which is called by libnetfilter_queue. This patch
displays the string expression of errno if verdict has failed.
14 years ago
Florian Westphal 8da02115c9 nfq: add support for batch verdicts
Normally, there is one verdict per packet, i.e., we receive a packet,
process it, and then tell the kernel what to do with that packet (eg.
DROP or ACCEPT).

recv(), packet id x
send verdict v, packet id x
recv(), packet id x+1
send verdict v, packet id x+1
[..]
recv(), packet id x+n
send verdict v, packet id x+n

An alternative is to process several packets from the queue, and then send
a batch-verdict.

recv(), packet id x
recv(), packet id x+1
[..]
recv(), packet id x+n
send batch verdict v, packet id x+n

A batch verdict affects all previous packets (packet_id <= x+n),
we thus only need to remember the last packet_id seen.

Caveats:
- can't modify payload
- verdict is applied to all packets
- nfmark (if set) will be set for all packets
- increases latency (packets remain queued by the kernel
  until batch verdict is sent).

To solve this, we only defer verdict for up to 20 packets and
send pending batch-verdict immediately if:
- no packets are currently queue
- current packet should be dropped
- current packet has different nfmark
- payload of packet was modified

This patch adds a configurable batch verdict support for workers runmode.
The batch verdicts are turned off by default.

Problem is that batch verdicts only work with kernels >= 3.1, i.e.
using newer libnetfilter_queue with an old kernel means non-working
suricata. So the functionnality has to be disabled by default.
14 years ago
Florian Westphal 6678c9feb9 nfq: avoid extra copy when running in workers mode
currently, the packet payload recv()d from the nfqueue netlink
socket is copied into a new packet buffer.

This is required because the recv-buffer space used is tied
to the current thread, but a packet may be handed off to other
threads, and the recv-buffer can be re-used while the packet
is handled by another thread.

However, in worker runmode, the packet will always be handled
by the current thread, and the recv-buffer will only be reused
after the entire packet processing stack is done with the packet.

Thus, in worker runmode, we can avoid the copy and assign
the packet data area directly.
14 years ago
Victor Julien b68d566c44 alert-debuglog: cleanup TCP check 14 years ago
Victor Julien 4b3166b193 unified2: more udp fixes 14 years ago
Victor Julien bc3f941acb profiling: enabled app layer profiling for UDP app layer modules 14 years ago
Victor Julien 782aa5adae prelude: only call stream callback for TCP 14 years ago
Victor Julien b54a19937f unified2: only call stream callback for TCP 14 years ago
Victor Julien 00948882e7 Suppress warnings when StreamSegmentForEach is called for UDP or SCTP, unless debug is compiled in. 14 years ago
Victor Julien 3b68a9d1c6 UDP: inspection app layer state as soon as we have it. 14 years ago
Victor Julien f15d97b916 Bug 780 unittests, showing no problem. 14 years ago
Victor Julien b6995f7664 Bug 794: stream SACK list needs to respect memcap 14 years ago
Victor Julien a4fca88ba7 stream: default 'random' setting when running unittests is disabled, so that test results are predictable. 14 years ago
Eric Leblond 9b235b3d9e streaming: randomize chunk size
By randomizing chunk size around the choosen value, it is possible
to escape some evasion technics that are using the fact they know
chunk size to split the attack at the correct place.
This patch activates randomization by default and set the random
interval to chunk size value +- 10%.
14 years ago
Victor Julien 6ba52230ed Update DetectContentDataParse to reflect the actual data types content uses. 14 years ago
Victor Julien 3ad497e74f Remove filemagic debug statement 14 years ago
Victor Julien 19511cda97 Remove obsolete DetectParseContentString function, it has been replaced by DetectContentDataParse 14 years ago
Victor Julien 4d4f8fd358 file: make fileext, filename and filemagic use the same rule parsing function as others. This has as a side effect that we enforce doubly qouted values now. 14 years ago
Victor Julien 8023007fbd flowvar: cleanup keyword argument parsing. Should also address Coverity 400655. 14 years ago
Victor Julien 07b751b0df Coverity 1005134: fix minor memory leak on flowvar rule setup errors. 14 years ago
Victor Julien e45f683c19 Coverity 1005133: fix unlikely case where malformed pcre statement in rule would lead to null-deref. 14 years ago
Victor Julien 4c6463f378 stream: handle extra different SYN/ACK
Until now, when processing the TCP 3 way handshake (3whs), retransmissions
of SYN/ACKs are silently accepted, unless they are different somehow. If
the SEQ or ACK values are different they are considered wrong and events
are set. The stream events rules will match on this.

In some cases, this is wrong. If the client missed the SYN/ACK, the server
may send a different one with a different SEQ. This commit deals with this.

As it is impossible to predict which one the client will accept, each is
added to a list. Then on receiving the final ACK from the 3whs, the list
is checked and the state is updated according to the queued SYN/ACK.
14 years ago
Victor Julien 00a691fc1b flowvar: clean up properly on signature clean up. 14 years ago
Victor Julien 70e2adeb01 flowvar: add unittests for #802. 14 years ago
Victor Julien 4cd736fcc9 flowvar: fix deadlock with http buffers
Bug #802

Flowvars are set from pcre, and lock the flow when being set. However
when HTTP buffers were inspected, flow was already locked: deadlock.

This patch introduces a post-match list in the detection engine thread
ctx, where store candidates are kept. Then a post-match function is used
to finalize the storing if the rule matches.

Solves the deadlock and brings the handling of flowvars more in line
with flowbits and flowints.
14 years ago
Victor Julien 4c2e6a8402 flowvars: update funcs to accept u16 id
All id's are u16, but flowvar functions would only accept u8.

Minor cleanups.
14 years ago
Victor Julien ffffe6c10e profiling: add formatted totals, percents to packet stats 14 years ago
Victor Julien 4165de4771 Minor SigValidate cleanup 14 years ago
Anoop Saldanha 0d7305dfc7 Update the way we handle http_host keywords.
Previously we would have forced all users to use nocase with http_host
keywords(since the hostname buffer is lowercase).

We now error out on sigs that has nocase set with http_host set.  Also if
the http_host pattern or http_host pcre has an uppercase character set, we
invalidate such sigs.  Unittests also updated to reflect the above change.
14 years ago
Victor Julien 9ea4d36f7a Minor reshuffling of Signature struct. 14 years ago
Victor Julien eb11280888 Use define instead of magic number for pmq's per detect thread 14 years ago
Victor Julien 0fa38c13d1 detection engine: consolidate thread setup
DetectEngineThreadCtxInit and DetectEngineThreadCtxInitForLiveRuleSwap did
pretty much the same thing, except for a counters registration. As can be
predicted with code duplication like this, things got out of sync. To make
sure this doesn't happen again, I created a helper function that does the
heavy lifting in this function.
14 years ago
Victor Julien 73158fea33 Fix PmqSetup calls in Liveswap thread init. Func was out of sync with normal thread init. 14 years ago
Victor Julien b8078742c3 stream: intro function for SYN/ACK state update
As the TCP SSN state can be updated from several points in the state
machine on accepting a SYN/ACK, move the update logic into a separate
function.
14 years ago
Victor Julien 28ea129d9b stream: remove unused 'pause' feature 14 years ago
Victor Julien ea8b6078d8 stream: zero ts is a per stream flag
Ssn flag STREAMTCP_FLAG_ZERO_TIMESTAMP was used in stream only. Due to
it's value it did not conflict with a real stream flag. Renamed it to
STREAMTCP_STREAM_FLAG_ZERO_TIMESTAMP.
14 years ago
Victor Julien 374187bf65 stream: don't use ssn timestamp flag in stream
The STREAMTCP_FLAG_TIMESTAMP flag is a ssn flag, however it was used in
the stream flag field. As it has the same value as
STREAMTCP_STREAM_FLAG_DEPTH_REACHED it's possible that stream reassembly
got confused by the timestamp.
14 years ago
Victor Julien 40a5ce8f5f Change logic of SCErrorToString causing any missing entries to result in a compiler warning. 14 years ago
Anoop Saldanha 71ffed5128 Handle the case of pcre combined with a relative content, where pcre has the
set to match from start of line and we discontinue matching on not finding
match.
14 years ago
Anoop Saldanha aa363a8144 unittest to display #784. 14 years ago
Eric Leblond 26b7af1483 Don't try to sniff 'default' interface
Whan running suricata via 'suricata --af-packet', the list of interfaces
was containing the 'default' interface and sniffing it was attempted.
This was not wanted.
14 years ago
Eric Leblond 539de3f5ea bpf filter: use SCLogError instead of fprintf 14 years ago
Eric Leblond b7e78d33b1 af-packet: warn about BPF filter consequence in IPS mode
This patch add a message to warn user about the impact of using a
BPF filter in IPS mode.
14 years ago
Eric Leblond dfbb31df8a Exit if bpf is used in IPS mode 14 years ago
Victor Julien ce99a07582 After some discussion we decided that var declarations inside a for statement are not in line with our coding style. So removing a bunch. Decision was not unanimous ^^. 14 years ago
Anoop Saldanha 8bf034e8c4 Live rule swap logs added to report SigLoadSignatures() failure. Also set
thread_closed flag on exit for live swap thread.
14 years ago
Anoop Saldanha a3212f6a0f Minor fixes against the last set of patches for #564, 565, 581 + fp automation.
Rename struct DetectFigureFPAndId_t_ to DetectFPAndItsId_ and move it's
definition from inside the function where it's used to the global namespace,
as requested on #suricata.

Rename DetectEngineContentModifiedBufferSetup to DetectEngineContentModifierBufferSetup.

Also rename DetectFigureFPAndId() to DetectSetFastPatternAndItsId().

Updated DetectSetFastPatternAndItsId() to not exit on failure and return error.
14 years ago
Anoop Saldanha 6de8b1ed53 fix for #564.
Get rid of the hash table, and use a single-one_time_alloc'ed array for
pattern id assignment.
14 years ago
Anoop Saldanha f58c6589b4 We now print content flags in engine fp analyzer. 14 years ago
Anoop Saldanha e77fd1c883 We now assign ids to fp patterns only. Rest of them don't need one. 14 years ago
Anoop Saldanha 4c6efa2d40 Update content id assignment.
All fp id assignment now happens in one go.
Also noticing a slight perf increase, probably emanating from improved cache
perf.
Removed irrelevant unittests as well.
14 years ago
Anoop Saldanha 60be1751d5 Figure out sig fp during validation stage, instead of staging stage. 14 years ago
Anoop Saldanha 45ff67a2e0 Enable a conf option to enable/disable legacy keywords.
Currently, uricontent is declared a legacy keyword, and is enabled by default.
14 years ago
Anoop Saldanha 601836d831 Fast pattern setup now configurable in our code.
You can either enable/disable fp for a particular type + set priority.
14 years ago
Anoop Saldanha c63317d02e Detect sm_list rearranged for performance reasons. 14 years ago
Anoop Saldanha f8ae53ac02 Further customize content modifier buffer registration.
Allow modifier setups functions to have CustomCallbacks to enable their
internal conditions.
14 years ago
Anoop Saldanha a304a98d1d http_* setup unified. 14 years ago
Anoop Saldanha 434bdca9e2 uricontent simplified to use the existing content + http_uri infrastructure. 14 years ago
Anoop Saldanha 0b5d277254 code cleanup for all content based keywords. 14 years ago
Anoop Saldanha 51dcf19817 turn dce_stub_data into a sticky buffer. 14 years ago
Anoop Saldanha a308d718ae Allow the use of relative without the presence of a related previous keyword. 14 years ago
Victor Julien 4845631335 tcp stream: don't move to LAST_ACK on toserver resent of FIN 14 years ago
Victor Julien 3163243a55 Coverity 989710 and 989711: small recourse leaks in filemd5 parsing code. 14 years ago
Anoop Saldanha 12e4105dc3 fix for #770.
Invalidate sigs with negative depth.
14 years ago
Anoop Saldanha d041b98d95 fix for #771.
Fix /etc/protocols parsing.  Remove trailing newspace stored under some cases.
14 years ago
Victor Julien 37c80ea508 If an IP-only pass rule matches, set the no inspect flag for that flow. Bug #718. 14 years ago
Anoop Saldanha 75130f9702 fix for #769.
Packet inserted by live swap flagged as pseudo packet.
14 years ago
Victor Julien 274641abc2 Fix valgrind error/warning in ip reputation parsing code 14 years ago
Anoop Saldanha c6ec23ca87 fix for #758. Add redmine wiki link and desc for icmp-id keyword. 14 years ago
Victor Julien eeb439c1a3 Open 2.0 dev branch 14 years ago
Victor Julien b66af2c2ed nfq: add missing error string 14 years ago
Eric Leblond 7ec820d3ab Fix potential Null deref. 14 years ago
Victor Julien 8924d7598d Fix potential iprep file parsing issue (2). 14 years ago
Victor Julien 754ae8a1be Fix potential iprep file parsing issue. 14 years ago
Victor Julien 1b363ecb1d Fix test AddressTestParse36 on Big Endian systems 14 years ago
Anoop Saldanha 0febe5a410 fix for #760.
If udpv4 csum isn't calculated, udpv4-csum detection shouldn't run on the
csum.
14 years ago
Anoop Saldanha ce7d78dd69 fix for #725.
Update trec_len, trec_pos to 32 bits from 16 bits.
Handle handshakes that are fragmented across records.
14 years ago
Anoop Saldanha c6d50764e5 temporarily patched smb + dcerpc parsers for direction demaraction. 14 years ago
Eric Leblond 5b067e1abb pcap-file: treat the case of unsupported pcap link
In unix socket mode, Suricata was stopping processing pcap files
when a pcap file with an unsupported datalink was treated. This
patch updates error handling to allow Suricata to treat other
pcap files.
14 years ago
Eric Leblond 350d761961 af-packet: leave reading loop at each turn
The idea of this patch is to be sure to leave the ring reading loop
enough to be able to sync counters. This should fix #706.
14 years ago
Eric Leblond df0e7af8f2 unix-manager: fix thread killing function
The name of the thread was not searched in the correct family.

Reported-by: iswalker <mail2cissp@gmail.com>
14 years ago
Eric Leblond 31c03d38b9 unix socket: add 'dump-counters' command
This patch adds a 'dump-counters' command which answer an output of
all performance counter.
14 years ago
Eric Leblond 5722d8846a unix socket: add 'help' as alias to 'command-list' 14 years ago
Eric Leblond 84322fa556 unix socket: add 'conf-get' command
This patch adds a 'conf-get' command which get the configuration
value from suricata. Argument of the command is the name of the
variable to fetch.
The command syntax is the following:
{
 "command": "conf-get",
 "arguments": { "variable":value}
}
14 years ago
Eric Leblond c961056ed8 unix socket: add 'capture-mode' command
This patch displays what capture mode is used.
14 years ago