Commit Graph

4090 Commits (5330b1cae1b1761e2911e9ddc0a9a94a2a55c034)

Author SHA1 Message Date
Victor Julien bb8298ffa2 Move header thread_affinity declaration to extern to avoid duplicate declarations. 13 years ago
Victor Julien 3470b07ea5 Fix several compile and runtime warnings found by clang 3.2 with the -fsanitize=address option. 13 years ago
Victor Julien c82ecf553a Tag: document in the code that 'tag' is compatible with ip only 13 years ago
Victor Julien d12761233c Don't set tag on pseudo packets 13 years ago
Victor Julien 02cbbd0b89 unified2: fix tags not being logged. Bug #968 13 years ago
Anoop Saldanha 3749fc98fd Modify handling of negated content.
The old behaviour of returning a failure if we found a pattern while
matching on negated content is now changed to continuing searching
for other combinations where we don't find the pattern for the
negated content.

Thanks to Will Metcalf for reporting this.
13 years ago
Victor Julien 8539791c7e Coverity 1038102: remove dead code from host hash 13 years ago
Victor Julien 8237bbf18a Coverity 1038101: remove dead code from host hash timeout code 13 years ago
Victor Julien 440124a4b9 Coverity 1038100: remove dead code from flow hash timeout code(2) 13 years ago
Victor Julien 243060a6b7 Coverity 1038099: remove dead code from flow hash timeout code 13 years ago
Victor Julien 2e82772a0a Coverity 1038098: remove dead code from flow hash 13 years ago
Victor Julien aecefd00bd Coverity 1038095: remove dead code from defrag hash timeout code 13 years ago
Victor Julien 16056d51f2 Coverity 1038094: remove dead code from defrag hash 13 years ago
Victor Julien 32503bafaa Coverity 1038089: error check fseek call 13 years ago
Victor Julien 4827a4dcef Coverity 400477: pcre_get_substring retval
Add missing return code check to pcre_get_substring call.
13 years ago
Victor Julien 790866656b Coverity 1038129 fix
Don't leak memory on malloc error in b2gm mpm implementation.
13 years ago
Victor Julien 33919559d0 Fix memory leak on invalid luajit signature. Coverity 1038520. 13 years ago
Victor Julien 51c6a333d9 geoip: never try to store more locations than possible (Coverity 1038517) 13 years ago
Victor Julien 3cf3b485f2 Coverity 1038138 fix
Clean up parsing code to suppress Coverity:
Dereference before null check (REVERSE_INULL)

Proper checking was already done.
13 years ago
Victor Julien 27ea4232fe Coverity 1038134 fix
Cleaned up error check. "ipdup" can only be non-NULL there, so remove check
that confused coverity.
13 years ago
Victor Julien ecd5c7573b Coverity 1038135 fix
Small cleanup in the error handling. The extra null check confused
Coverity.
13 years ago
Victor Julien 38b6103ff5 Coverity 1038133 fix
Clean up parsing code to suppress Coverity:
Dereference before null check (REVERSE_INULL)

Proper checking was already done.
13 years ago
Ken Steele 50f859e9f2 Move SIMD implementations out of detect.c
Move SIMD the implementations of SigMatchSignaturesBuildMatchArray()
for SSE3 and Tile out of detect.c to reduce the size of the file.

Also moved SIMD unit tests to detect-simd.c
13 years ago
Victor Julien 7f140f6726 Coverity 1038111: fix local overrun of a string in app layer proto detect setup code. 13 years ago
Ken Steele eb4f0da97f Change one more atomic size in detect.h
Change uint16_t to int for better tile atomic performance. Checked with
pahole that it doesn't increase the size of the structure.
13 years ago
Ken Steele b08ddfa7f1 Support for Tile Gx atomic instructions
Tilera's GCC supports the GCC __sync_ intrinsics.

Increase the size of some atomic variables for better performance on
Tile.  The Tile-Gx architecture has native support for 32-bit and
64-bit atomic operations, but not 8-bit and 16-bit, which are emulated
using 32-bit atomics, so changing some 16-bit and 8-bit atomic into
ints improves performance.

Increasing the size of the atomic variables modified in this change
does not increase the total size of the structures in which they
reside because of existing padding requirements. The one case that
would increase the size of the structure (Flow_) was confitionalized
to only change the size on Tile.
13 years ago
Anoop Saldanha 54847e396f unittests for gzip, deflate http compression, multiple stacked
compressions, cunning compression that's not what it says it is, etc.

These unittests are tweaked to pass.  When libhtp fixes these issues
we will have to reenable them.
13 years ago
Anoop Saldanha 94e2527606 Introduce a saner way to validate the completion of request and
response bodies.

Also don't change app state for http from inside inspection.
13 years ago
Anoop Saldanha dcdcbd9721 Fix creating a backup of htp config. This is used by unittests that
changed htp config.
13 years ago
Ken Steele 62540eff3e Align some structures to cacheline
Align strucutres with pthread mutex locks to start on cachelines to keep
the lock within one cacheline.
13 years ago
Ken Steele d84079ba7d Move FlowIncrUsecnt to header file to allow for inlining.
Move FlowIncrUsecnt() and FlowDecrUsecnt() from flow.c to flow.h to
allow for inlining.
13 years ago
Ken Steele e05034f5dd New Multi-pattern matcher, ac-tile, optimized for Tile architecture.
Aho-Corasick mpm optimized for Tilera Tile-Gx architecture. Based on the
util-mpm-ac.c code base. The primary optimizations are:
1) Matching function used Tilera specific instructions.
2) Alphabet compression to reduce delta table size to increase cache
   utilization  and performance.

The basic observation is that not all 256 ASCII characters are used by
the set of multiple patterns in a group for which a DFA is
created. The first reason is that Suricata's pattern matching is
case-insensitive, so all uppercase characters are converted to
lowercase, leaving a hole of 26 characters in the
alphabet. Previously, this hole was simply left in the middle of the
alphabet and thus in the generated Next State (delta) tables.

A new, smaller, alphabet is created using a translation table of 256
bytes per mpm group. Previously, there was one global translation
table for converting upper case to lowercase.

Additional, unused characters are found by creating a histogram of all
the characters in all the patterns. Then all the characters with zero
counts are mapped to one character (0) in the new alphabet. Since
These characters appear in no pattern, they can all be mapped to a
single character and still result in the same matches being
found. Zero was chosen for the value in the new alphabet since this
"character" is more likely to appear in the input. The unused
character always results in the next state being state zero, but that
fact is not currently used by the code, since special casing takes
additional instructions.

The characters that do appear in some pattern are mapped to
consecutive characters in the new alphabet, starting at 1. This
results in a dense packing of next state values in the delta tables
and additionally can allow for a smaller number of columns in that
table, thus using less memory and better packing into the cache. The
size of the new alphabet is the number of used characters plus 1 for
the unused catch-all character.

The alphabet size is rounded up to the next larger power-of-2 so that
multiplication by the alphabet size can be done with a shift.  It
might be possible to use a multiply instruction, so that the exact
alphabet size could be used, which would further reduce the size of
the delta tables, increase cache density and not require the
specialized search functions. The multiply would likely add 1 cycle to
the inner search loop.

Since the multiply by alphabet-size is cleverly merged with a mask
instruction (in the SINDEX macro), specialized versions of the
SCACSearch function are generated for alphabet sizes 256, 128, 64, 32
and 16.  This is done by including the file util-mpm-ac-small.c
multiple times with a redefined SINDEX macro. A function pointer is
then stored in the mpm context for the search function. For alpha bit
sizes of 8 or smaller, the number of states usually small, so the DFA
is already very small, so there is little difference using the 16
state search function.

The SCACSearch function is also specialized by the size of the value
stored in the next state (delta) tables, either 16-bits or 32-bits.
This removes a conditional inside the Search function. That
conditional is only called once, but doesn't hurt to remove
it. 16-bits are used for up to 32K states, with the sign bit set for
states with matches.

Future optimization:

The state-has-match values is only needed per state, not per next
state, so checking the next-state sign bit could be replaced with
reading a different value, at the cost of an additional load, but
increasing the 16-bit next state span to 64K.

Since the order of the characters in the new alphabet doesn't matter,
the new alphabet could be sorted by the frequency of the characters in
the expected input stream for that multi-pattern matcher. This would
group more frequent characters into the same cache lines, thus
increasing the probability of reusing a cache-line.

All the next state values for each state live in their own set of
cache-lines. With power-of-two sizes alphabets, these don't overlap.
So either 32 or 16 character's next states are loaded in each cache
line load. If the alphabet size is not an exact power-of-2, then the
last cache-line is not completely full and up to 31*2 bytes of that
line could be wasted per state.

The next state table could be transposed, so that all the next states
for a specific character are stored sequentially, this could be better
if some characters, for example the unused character, are much more
frequent.
13 years ago
Victor Julien 77b429c402 xff: fix unittest crashes 13 years ago
Victor Julien 05d68ce394 xff: don't do xff check if there are no alerts anyway. 13 years ago
Duarte Silva 7dbb305255 Adds X-Forwarded-For support to the Unified2 output format
- Added the Unified2 file format related constants
 - Added IPv6 support
 - Two modes of operation with a fall-back to "extra-data" mode if
   "overwrite" mode is not applicable
 - Changed the configuration loading code to handle the new
   configuration structure
 - When creating the packet that fakes the one that generated the alert
   the flow direction wasn't taken into account in overwrite mode
 - Fixed BUG_ON condition
13 years ago
Victor Julien 900918a5d1 Bug #948: detect thread local storage support 13 years ago
Ken Steele 0861d3a2a3 Minor optimization in time caching code.
Reduced the size of the cached string buffer from 128 to 32, which is
still larger than the largest possible time string, which is 26
characters.

Added a check for the user passing in an output buffer that is smaller
than the cached string. Previously, the code would have copied past
the end of the users buffer.
13 years ago
Anoop Saldanha 49dcb0ca84 fix for #925.
Log sensible error message when the user doesn't supply a value for
stream.prealloc-sessions or when the values supplied in invalid and
the engine resorts to using a default.
13 years ago
Anoop Saldanha db6ef81fb0 fix for #926.
Supply meaningful error message when user supplies invalid value for
host.prealloc.
13 years ago
Anoop Saldanha b90a56b626 fix for #927.
Print an error message when the user supplies an invalid value for
detect-thread-ratio in the conf file.
13 years ago
Anoop Saldanha bed3f605fa Fix for #922.
Add more relevant error message when we supply invalid value for
defrag.trackers and defrag.hash-size
13 years ago
Anoop Saldanha 6608e7f523 Introduce generic utility API to log message on invalid config entry. 13 years ago
Victor Julien 6d34834623 Runmode fixes and cleanups
Bug #939: thread name buffers are sized inconsistently
These buffers are now all fixed at 16 bytes.

Bug #914: Having a high number of pickup queues (216+) makes suricata crash
Fixed so that we can now have 256 pickup queues, which is the current built-in
maximum. Improved the error reporting.

Bug #928: Max number of threads
Error reporting improved. Issue was the same as #914.
13 years ago
Anoop Saldanha 56143131da Fix unittests that use chunked encoding. 13 years ago
Nelson Escobar ef4d11aeb5 Use the Async versions of SCCudaMemcpy* to improve gpu performance. 13 years ago
Eric Leblond 77f2b9968e autotools: use builddir instead of srcdir
srcdir is supposed to be read-only when running distcheck so it is better to
create the log directory in builddir.
13 years ago
Ignacio Sanchez 1b2f251866 Various custom http logging improvements
Cookie is parsed now using uint8_t pointers (inliniac PR comments)
Changed buffer size to a power of 2 (8192) and cookie value extraction function to static (inliniac PR comments)
Added %b for request size (vinfang patch)
Writing "-" if an unknown % directive is used (vinfang patch)
Fixed bug in cookie parser
Fixed format string issue logging literal values
Improve error handling (Victor Julien comments)

(patchset rebased and reworded by Victor Julien)
13 years ago
Ignacio Sanchez 8051dc8a6a Added modifications suggested by Charles Smutz (https://redmine.openinfosecfoundation.org/issues/602) 13 years ago
Ignacio Sanchez 796bfab231 Added support for %{cookiename}C
Added support for the definition of maximun length. ie: %[50]{user-agent}i
Some small bugfixes
13 years ago
Eric Leblond 3dbf6c6fee solaris: fix compilation failure
This patch fixes a compilation failure on Solaris. Compiler does
not support when a function returning void is used in return of
an other function returning void.
13 years ago
Ken Steele a2b502a30c Formatting change for function call.
Put open brace { for function on a new line to match coding standard.

Changed:

int foo(int x) {
}

to:

int foo(int x)
{
}
13 years ago
Ken Steele d4dd18eb85 Clean up SCLocalTime() usage
Remove cast of return type from SCLocalTime() as it is not needed.
Replace last use of localtime_r() with SCLocalTime().
13 years ago
Ken Steele 77fae5313d On Open BSD systems don't cache time.
Open BSD doesn't support __thread, which is used for time caching, so
don't do time chaching for BSD systems.
13 years ago
Ken Steele 2feb37c155 Cache time conversions for localtime() and CreateTimeString()
When converting a time in seconds (64-bit seconds since 1970) to
Month/Day/Year hours minutes, Suricata calls localtime_r(), which
always aquires a lock and then does complex comutation based on the
current time zone. The time zone can be specified in the TZ
environment variable, which is only parsed the first time it is used,
or from a file. The default file is /etc/localtime. The file is
checked each time to see if it might have changed and is reparsed if
it has changed.

The GLIBC library has a lock inside localtime_r(), which limits
parallelism, which is a problem when the rate of generating alerts is
high, since Suricata generates a new ascii time string for each alert
into fast.log.

This change caches the value returned by localtime_t() and then sets
the seconds within the minute based on the cached start-of-minute
time. All of the values return, expect for the seconds, is constant
within the same minute. Switching to a new seconds could change all
the other values, year, month, day, hour. The cache stores the current
and previous minute values.

The same trick is used in CreateTimeString() for generated time
string. The string, up to the minutes, is cached and then copied into
the result string, followed by printing the new seconds into the
result string.

The seconds within a minute are calculated as the difference in
seconds from the start of the current minute.
13 years ago
Ken Steele 68d26dcec7 Merge multiple copies of CreateTimeString() to one copy.
There were 8 identical copies of CreateTimeString() in 8 files.
Most used SCLocalTime, to replace localtime_r(), but some did not.
Created one copy in util-time.c.
13 years ago
Ken Steele 5532af4621 Create SCMUTEX_INITIALIZER to abstract out PTHREAD_MUTEX_INITIALIZER
This allows replacing pthread mutexes with other types of mutex.
13 years ago
Ken Steele 784843b146 Use Tilera SIMD for Signature matching ala SSE3
Makes use of 8-wide byte compare instructions in signature matching.

For allocating aligned memory, _mm_malloc() is SSE only, so added
check for __tile__ to use memalign() instead.

Shows a 13% speed up.
13 years ago
Ken Steele 22225a7e99 Tile SIMD implementation of SCMemcmp and SCMemcmpLowercase
Based on the SSE3 implementation, it checks 8 bytes at a time.
13 years ago
Anoop Saldanha e68d44b051 fix for #932.
ipv6 tunnel decoder wrongly treats the tunneled ipv6 packets as an ipv4
packet.
13 years ago
Anoop Saldanha e2f4144d99 fix for #920.
Cull the space before the address specified in address var variables.
13 years ago
Duarte Silva ab215c72f6 Now using the common functions 13 years ago
Duarte Silva 0a5c798729 Now using the common functions
- Removed some non printable ANSI characters
- Removed unecessary include
13 years ago
Duarte Silva 8ce95af09c Added the new files containing the repeated functions
- Renamed the functions to something more generic
- Added the source and include files to the Makefile
13 years ago
Anoop Saldanha a44d42b124 Fixes segv inside rule swap under low mem conditions.
We now gracefully exit rule swap on any allocation or other failures.
13 years ago
Anoop Saldanha 8516ba24c9 Rearrange ac state.
Notice a minor speed bump of around 2% on runs.  More updates to follow.
13 years ago
Ken Steele 4b8bb11454 Enable using Tile cycle counter.
The Tile processors all have a cycle counter with a simple interface. Use
that for UtilCpuGetTicks.
13 years ago
Victor Julien 38aaae1fd7 IsRuleReloadSet() shouldn't return an uninitialized value 13 years ago
Eric Leblond 189327981a unittests: fix stream-tcp.c
Lock and recycle fixes for stream-tcp.c
13 years ago
Eric Leblond cd3e32ce19 unittests: some functions needs a flow lock.
In debug validation mode, it is required to call application layer
parsing and other functions with a lock on flow. This patch updates
the code to do so.
13 years ago
Eric Leblond c5bd04f102 unittest: recycle packet before exit
To avoid an issue with flow validation, we need to recycle the packet
before cleaning the flow.
13 years ago
Anoop Saldanha d292f1a529 fix for #915. Fix segv when we send NULL to snprintf. 13 years ago
Eric Leblond c6e8c5bf1f pf_ring: avoid to ask for extended header.
This patch update pf_ring capture to avoid to ask for extended
header. They are only needed when rxonly checksum checks is used
and this is only possible when interface is not a DNA interface.
13 years ago
Victor Julien ff668c2030 Fix Tile compile 13 years ago
Eric Leblond 20ca270dc3 fix pf_ring build 13 years ago
Eric Leblond 2a46f0dae4 suricata: rename SuriInstance to SCInstance. 13 years ago
Eric Leblond 9b422f3a8c suricata: suppress Suri prefix
Suppress Suri prefix in internal function name.
13 years ago
Eric Leblond 18ced653c3 Use a typedef for SuriInstance. 13 years ago
Eric Leblond 2d77e53f2c Add offline flag to SuriInstance and some refactoring 13 years ago
Eric Leblond 34abd818dd Prefix util-conf function with Config 13 years ago
Eric Leblond 7242cb30e7 Move CreateLowercaseTable to GLobalInits 13 years ago
Eric Leblond 02e9851315 Generic code don't need ifdef 13 years ago
Eric Leblond 8c00a963aa Use function for delayed detect setup. 13 years ago
Eric Leblond 4296e5f29e Add functions for elapsed time computation. 13 years ago
Eric Leblond 9d1d08c7a4 Factorize Signature loading 13 years ago
Eric Leblond 20c5683b60 Use function for daemonification and signal handler 13 years ago
Eric Leblond 90aaf55201 set rule_reload as part of SuriInstance 13 years ago
Eric Leblond bb19ce1847 SetBPfString is part of command line parsing 13 years ago
Eric Leblond 1a6983ee19 suricata: use function to print version 13 years ago
Eric Leblond 4f789dbe84 Add function for internal running mode 13 years ago
Eric Leblond d3cb043001 suricata: windows specific in one function 13 years ago
Eric Leblond 4401c048ba Running mode is set earlier so out earlier 13 years ago
Eric Leblond 40a25112a0 kill remaining run_mode usage 13 years ago
Eric Leblond 75fa1e20d7 engine analysis is a running mode 13 years ago
Eric Leblond c0d5ee77f9 get (almost) rid of run_mode variable. 13 years ago
Eric Leblond 80542816cd add internal running mode 13 years ago
Eric Leblond e07fdb20a8 Add SuriInstance structure
To be able to split code in functions in main, we need to pass
information about the current running Suricata to functions.
For that we create a structure to store suricata run parameters.

In this patch it allows to separate command line parsing and to
treat internal running mode in a switch just after command line
parsing.
13 years ago
Eric Leblond 325462d396 Export IsRuleReloadSet and use it. 13 years ago
Eric Leblond 6d9a66d522 unittest: make check use a qa/log dir for logging
This patch is using the qa/log directory to store the output
of the check. In case of success, the directory is deleted.
In case of failure, the directory remains in place.

This should fixes #910.
13 years ago
Eric Leblond 4424f5a231 af-packet: add sanity check in free function 13 years ago
Eric Leblond 8e68b357c7 Suppress Suri prefix. 13 years ago
Eric Leblond 42011e2d32 suricata: function for lowercase table creation 13 years ago
Eric Leblond 132bebb2b2 Simplify code by removing comment 13 years ago
Eric Leblond 07ef1f9837 suricata: add wrapper for interface listing 13 years ago
Eric Leblond 54006de40c Use new function GetLogDirectory() 13 years ago
Eric Leblond 2be7c8aea8 Add util-conf for config util 13 years ago
Eric Leblond 27752818c2 suricata: add some wrapper for config file handling 13 years ago
Eric Leblond b2fa4edd36 move unittest out of suricata.c 13 years ago
Eric Leblond 9a0bf0956b suricata: list cuda cards in separate function 13 years ago
Eric Leblond bed48e3a54 suricata: separate keyword and app layer listing code
The list-keyword and app-layer listing code was spread over all the
init code. This patch introduces a separate file to store non standard
running mode like these ones.
13 years ago
Eric Leblond 135ef0186b runmodes: fix comment 13 years ago
Victor Julien 5a7bf53a6b Storage: rename Init to Alloc to reflect actual functioning. Comment updates. 13 years ago
Victor Julien f06694d0c1 Storage API: add safety check for cases when there is no storage used. 13 years ago
Eric Leblond caf730d988 engine-tag: rename var and add sanity check 13 years ago
Eric Leblond fb55931c30 flow tag: conversion to flow storage API
This patch is updating the flow tag system to use the flow
storage API. The tag_list member of Flow structure is suppressed
and its cleaning operation are suppressed too as this is handled
transparently by the flow storage API.
13 years ago
Eric Leblond 4db2fc2cbb Add per-flow generic storage
This patch adds a per-flow storage that can be created via the functions
available in flow-storage.c.
13 years ago
Eric Leblond 6d08807b2d Host: use global free storage function
This patch is here to avoid that all modules using a local storage
have to update host code to add their free function. It modifies
previous behavior by calling HostFreeStorage in any case.
13 years ago
Victor Julien 27023872de Use Host Storage API for per host thresholding 13 years ago
Victor Julien c08b395c2c Init storage api at start up 13 years ago
Victor Julien 5919901675 Storage API: add registration check closed test in debug mode. 13 years ago
Victor Julien 3447324c36 Move Host Tag storage to Host Storage API. 13 years ago
Victor Julien 0d2a6e515e Add Host specific wrapper to StorageRegister() 13 years ago
Victor Julien b5ccf0b9c7 storage: allow preallocated storage 13 years ago
Victor Julien e2b006f523 host: use storage api 13 years ago
Victor Julien 022c0e466e Initial storage api work 13 years ago
Victor Julien 1c06d52208 Misc fixes after make check feedback 13 years ago
Ken Steele 316190c6b9 Add TILE-Gx mPIPE packet processing support.
The TILE-Gx processor includes a packet processing engine, called
mPIPE, that can deliver packets directly into user space memory. It
handles buffer allocation and load balancing (either static 5-tuple
hashing, or dynamic flow affinity hashing are used here). The new
packet source code is in source-mpipe.c and source-mpipe.h

A new Tile runmode is added that configures the Suricata pipelines in
worker mode, where each thread does the entire packet processing
pipeline.  It scales across all the Gx chips sizes of 9, 16, 36 or 72
cores. The new runmode is in runmode-tile.c and runmode-tile.h

The configure script detects the TILE-Gx architecture and defines
HAVE_MPIPE, which is then used to conditionally enable the code to
support mPIPE packet processing. Suricata runs on TILE-Gx even without
mPIPE support enabled.

The Suricata Packet structures are allocated by the mPIPE hardware by
allocating the Suricata Packet structure immediatley before the mPIPE
packet buffer and then pushing the mPIPE packet buffer pointer onto
the mPIPE buffer stack.  This way, mPIPE writes the packet data into
the buffer, returns the mPIPE packet buffer pointer, which is then
converted into a Suricata Packet pointer for processing inside
Suricata. When the Packet is freed, the buffer is returned to mPIPE's
buffer stack, by setting ReleasePacket to an mPIPE release specific
function.

The code checks for the largest Huge page available in Linux when
Suricata is started. TILE-Gx supports Huge pages sizes of 16MB, 64MB,
256MB, 1GB and 4GB. Suricata then divides one of those page into
packet buffers for mPIPE.

The code is not yet optimized for high performance. Performance
improvements will follow shortly.

The code was originally written by Tom Decanio and then further
modified by Tilera.

This code has been tested with Tilera's Multicore Developement
Environment (MDE) version 4.1.5. The TILEncore-Gx36 (PCIe card) and
TILEmpower-Gx (1U Rack mount).
13 years ago
Victor Julien 04f3f14541 ipv6: fix parsing of malformed ext hdr. Bug #908. 13 years ago
Victor Julien 4b4111e9e2 icmpv6: fix icmp_id and icmp_seq keywords
Bug #907
13 years ago
Victor Julien d82ce3f50c Fix compiler warning due to missing include
decode.c: In function 'DecodeThreadVarsAlloc':
decode.c:437: error: implicit declaration of function 'ConfGetBool'
13 years ago
Victor Julien 16c3487444 Add yaml option to disable vlan ids hashing
In some cases using the vlan id(s) in flow hashing is problematic. Cases
of broken routers have been reported. So this option allows for disabling
the use of vlan id(s) while calculating the flow hash, and in the future
other hashes.

Vlan tracking for flow is enabled by default.
13 years ago
Victor Julien 58ed1f2411 flow: take vlan_id's into account in the flow hash
In VLAN we can have 2 layers of encapsulation. In this patch both
layers are used in the flow hash to distinguish between encapsulated
traffic.
13 years ago
Victor Julien 055b422c28 Remove obsolete code: flow alert sid storage 13 years ago
Victor Julien 9faa4b740d Add --unittests-coverage option to list how many code modules have tests 13 years ago
Victor Julien fc7879322e Rename GetIfaceMaxPayloadSize to GetIfaceMaxPacketSize to reflect the actual function. 13 years ago
Victor Julien bd21b5ed9c Pcap: fix snaplen autodetection, GetIfaceMTU doesn't include link layer length 13 years ago
Anoop Saldanha ee0b21652b fix bug where we were not printing http hostname(printing <unknown>
previously) in httplog, filestore meta and file log.
13 years ago
Victor Julien 7edcc13514 NFQ: fix packets not getting freed 13 years ago
Anoop Saldanha cdaa13012a fix for #882.
Refactor the code that initializes the cuda mpm environment.
13 years ago
Victor Julien 9f3e2f7a92 NFQ: adapt to ReleasePacket API 13 years ago
Ken Steele b076a26cdc Replace ReleaseData function on Packet Structure with ReleasePacket.
This commit allows handling Packets allocated by different methods.
The ReleaseData function pointer in the Packet structure is replaced
with ReleasePacket function pointer, which is then always called to
release the memory associated with a Packet.

Currently, the only usage of ReleaseData is in AF Packet. Previously
ReleaseData was only called when it was not NULL. To implement the
same functionality as before in AF Packet, a new function is defined
in AF Packet to first call the AFP specific ReleaseData function and
then releases the Packet structure.

Three new general functions are defined for releasing packets in the
default case:
    1) PacketFree() - To release a packet alloced with SCMalloc()
    2) PacketPoolReturnPacket() - For packets allocated from the Packet Pool.
                                  Calls RECYCLE_PACKET(p)
    3) PacketFreeOrRelease() - Calls PacketFree() or PacketPoolReturnPacket()
                                 based on the PKT_ALLOC flag.

Having these functions removes the need to check the PKT_ALLOC flag
when releasing a packet in most cases, since the ReleasePacket
function encodes how the Packet was allocated. The PKT_ALLOC flag is
still set and is needed when AF Packet releases a packet, since it
replaces the ReleasePacket function pointer with its own function and
then calls PacketFreeOfRelease(), which uses the PKT_ALLOC flag.
13 years ago
Anoop Saldanha f85a2dc84b fix for #875.
Update configure.ac to check for either 0.5.5 and 0.5.x version of libhtp.
13 years ago
Anoop Saldanha 9698a5d78c Code to enable cuda support for pfring live mode. 13 years ago
Victor Julien 91fb47475b DNS: break out of DNSResponseGetNameByOffset if we're in there too long. Can happen on bad data. 13 years ago
Victor Julien aa26dae5a1 Stream: don't inject stream end pseudo pkt on FinWait2 state. Bug #883. 13 years ago
Victor Julien 2f3f577fb6 DNS: convert info logs to debugs 13 years ago
Victor Julien 97f51c1011 Fix ac-bs and ac-gfbs mpm-algo settings leading to fatal error if CUDA is enabled. Workaround for #882. 13 years ago
Eric Leblond e2334fbfe8 unix socket: fix typo in error message 13 years ago
Eric Leblond c2cbb43776 autotool: INCLUDES usage is deprecated 13 years ago
Eric Leblond 281d2f27f8 Fix compilation warning
A goto could lead to the use de_ctx without declaring it.
13 years ago
Victor Julien f4dcba6de3 In case of fragments, don't consider ports. Bug #847. 13 years ago
Anoop Saldanha e7f09f24c8 Code to enable cuda support for live mode pcap and af-packet. Keep an eye
out on the mailing list and http://planet.suricata-ids.org for performance
and other profiling data.
13 years ago
Victor Julien 51d6c63860 Luajit: fix compilation and tests after libhtp upgrade 13 years ago
Anoop Saldanha 48cf0585fb Suricata upgrade to libhtp 0.5.x.
Remove the support for now unsupported personalities from libhtp -
TOMCAT_6_0, APACHE and APACHE_2_2.  We instead use the APACHE_2
personality.
13 years ago
Victor Julien 080c15b3fc Enable libhtp 0.3.0 compilation and crash free UT run. Still see 5 failed tests. 13 years ago
Victor Julien 538da26812 Fix sgh mpm flags assignment 13 years ago
Eric Leblond 150cd39c6e detect-engine: do a direct update of flag
There is no reason not to update the flag directly. So do it
to avoid to crash the test.
13 years ago
Eric Leblond 2f2916d9ec ccccinelle: add formatted comment for flag test 13 years ago
Victor Julien aafc65c757 Autotools: move libhtp conditionals to configure
In preparation of the libhtp upgrade, move all libhtp related conditionals
to configure. This allows for one set of build scripts that works regardless
of the presence of a local libhtp dir.
13 years ago
Victor Julien 73e27c1fb7 Generate proper errors if sid,gid,rev values are out of range. Bug #779. 13 years ago
Victor Julien 164d60e8cd Yaml: give a more detailed error if the user supplies a directory instead of a yaml file. Bug #803. 13 years ago
Victor Julien a4e838c1d3 TLS: create certs directory during startup if it doesn't exist yet. Bug #710. 13 years ago
Eric Leblond 0e92469222 nfq: be sure to always verdict packets
To be sure to always verdict packets (bug #769), this patch adds
a ReleaseData function to NFQ packets. The release function simply
drop the packet if it has not been verdicted before.
13 years ago
Victor Julien 4a0050b9ea Print pkt src to alert-debug log 13 years ago
Victor Julien 1c371da46d DNS: better handle TX' with lost replies 13 years ago
Victor Julien 0fd9b0c4fa HTP: free TX from transaction free API call 13 years ago
Victor Julien 1367074c75 App layer: clean up TX before lowest active one
Update DNS to handle cleaning up this way.
13 years ago
Victor Julien 0b229ec8b9 DNS: suppress log-dns registration message 13 years ago
Victor Julien f59f90331d Applayer: remove obsolete StateUpdateTransactionId
Also, update StateTransactionFree to take an u64 tx id, so it's
consistant with the rest of the engine.

To reflect these changes, AppLayerRegisterTransactionIdFuncs has
been renamed to AppLayerRegisterTxFreeFunc.

HTP, DNS, SMB, DCERPC parsers updated.
13 years ago
Victor Julien ebab9aee83 DNS: move internal tx id tracking to u64 13 years ago
Victor Julien e8ad876b48 App layer: add 'StateHasEvents' API call
Per TX decoder events resulted in significant overhead to the
detection engine, as it walked all TX' all the time to check
if decoder events were available.

This commit introduces a new API call StateHasEvents, which speeds
up this process, at the expense of keeping a counter in the state.

Implement this for DNS as well.
13 years ago
Anoop Saldanha cd7b4fac40 remove unused pattern id assignment functions. Goodbye 13 years ago
Victor Julien f353fb630c DNS: convert dns_query to sticky buffer 13 years ago
Victor Julien 7292998a58 Content: set up sticky buffers like file_data and dce_stub_data w/o flags, but with a list variable 13 years ago
Victor Julien d476e4e50d Coverity 1040312, 1040313, 1040314 1040315: improve pool thread error handling. 13 years ago
Victor Julien 1373a20e8a Thread: remove thread id 13 years ago
Victor Julien d7aaa9464c Stream: use per thread ssn_pool_id instead of thread id. 13 years ago
Victor Julien 92b7ffad69 Improve memory cleanup in some unittests 13 years ago
Victor Julien fd7899cc8b Stream: fix unittests after ssn pool changes. 13 years ago
Victor Julien aa449d51ca Stream: use per thread ssn pool
Use per thread pools to store and retrieve SSN's from. Uses PoolThread
API.

Remove max-sessions setting. Pools are set to unlimited, but TCP memcap
limits the amount of sessions.

The prealloc_session settings now applies to each thread, so lowered the
default from 32k to 2k.
13 years ago
Victor Julien b6af6cb241 pool: add error msgs and improve memory layout 13 years ago
Victor Julien 5b9ef94f34 pool: add api for per thread pools
This API is a wrapper around the regular pools where the thread pools
are arrays of locks+pools.
13 years ago
Victor Julien 016d03bdaf pool: add error msgs and improve memory layout 13 years ago
Victor Julien 46af6b7e0f Add a per threadvars thread local thread id, that starts at 0 and increments for each thread. 13 years ago
Victor Julien b3b554c269 Coverity 1038959: DNS mpm might use initialized variable 13 years ago
Anoop Saldanha fba95e9125 Remove mpm ctxs in the wrong direction.
A lot of http mpm ctxs have now been removed as a result of this.
13 years ago
Anoop Saldanha 3c2ddf04c1 Update mpm init ctx to not accept the final cuda_rc_module argument.
It was a part of our older architecture and is no longer used.
13 years ago
Victor Julien 33818c0272 DNS: fix CUDA build 13 years ago
Victor Julien be7e6cdd7a DNS: fix warning when debug is not enabled 13 years ago
Victor Julien 571b8ac186 DNS: add support for per TX decoder events. 13 years ago
Victor Julien 9dc04d9fab app layer: add support for per TX decoder events 13 years ago
Victor Julien 72e35efbc6 Reset app layer events when we start inspecting a new TX 13 years ago
Victor Julien 28a6c1d9f8 DNS: add test for app layer event match 13 years ago
Victor Julien 6645620c03 Merge SIG_FLAG_MPM_HTTP and SIG_FLAG_MPM_DNS into SIG_FLAG_MPM_APPLAYER, do the same for the _NEG variant. 13 years ago
Victor Julien 43ba5a677e DNS: enable mpm/fast_pattern support for dns_query 13 years ago
Victor Julien 4817e1305f DNS: add /F modifier to pcre to inspect DNS query name 13 years ago
Victor Julien e567e12230 DNS: add unittests for UDP and TCP for dns_query keyword 13 years ago
Victor Julien f10dd603ff DNS: adding dns_request content modifier 13 years ago
Victor Julien 6674f4892c DNS: add per tx internal id
Add per TX id. Rename transaction_cnt to transaction_max (id) and increment it on tx creation.
13 years ago
Victor Julien 59780ca770 Hacks to enable alert dns even though we have dnstcp and dnsudp parsers. Needs proper solution later. 13 years ago
Victor Julien 8e01cba85d DNS TCP and UDP parser and DNS response logger 13 years ago
Eric Leblond 4521de2dfd Use PACKET_* macro instead of UPDATE
Setting the ACTION_DROP flag can be done via PACKET_DROP instead
of using PACKET_UPDATE_ACTION.
13 years ago
Eric Leblond c0c59fbd17 decode: factorize macro code
PACKET_* are now wrapper to the newly introduced PACKET_SET_ACTION
macro.
13 years ago
Eric Leblond 3f107fa130 decode: Packet action start with PACKET
Rename all Packet action macro to have them prefixed by PACKET.
13 years ago
Anoop Saldanha 3304c91c91 Don't let geoip match on pseudo packets. 13 years ago
Anoop Saldanha 1fb4aae993 Coverity 1038523: Fix using cuda buffer slice that has been returned to the pool. 13 years ago
Victor Julien 51cdd464a6 stream: detect keep-alive and keep-alive ACK 13 years ago
Victor Julien 03c3ff5632 stream: fix typo in function name 13 years ago
Eric Leblond 4c6595f437 Coverity 1038106: fix FP out-of-bond access
A cast during the reading of a configuration variable was invalid
because a 16 bit integer was cast to a 32 bit integer. The called
function is only setting the pointer value to 1 or 0 so there is
no real issue there.
13 years ago
Eric Leblond 7df156ef50 Coverity: 1038139 suppress sanity check
The sanity check was really useless as the NULL value is checked in
the code flow.
13 years ago
Eric Leblond 75cd1f6096 Coverity 1038515: check function return
This is more cosmetic than useful but it is cleaner anyway.
13 years ago
Anoop Saldanha b4e750068f Cuda make distcheck fix for cuda-ptxdump.h 13 years ago
Anoop Saldanha 7d46d59cdd Coverity 1038522: fix memset inside cuda code. Wrong size specified to memset. 13 years ago
Victor Julien d6fcd07a31 Coverity 1038085: remove 'default' statement in SCErrorToString. This way a warning will be given if an error is defined w/o updating this function. 13 years ago
Victor Julien e2444f0ed5 Coverity 1038092 & 1038093: remove dead code 13 years ago
Victor Julien 0902c7f3aa Coverity 1038518: fix wrong error check 13 years ago
Victor Julien db1dad8cc6 Coverity 1038124: memory leak on 'seq' keyword parsing failure 13 years ago
Victor Julien b2e962da03 Coverity 1038123: memory leak on 'flowint' keyword parsing failure 13 years ago
Victor Julien 5c5b2f98dd Coverity 1038116 & 1038117: memory leaks on 'app-layer-event' keyword parsing failure 13 years ago
Victor Julien 778851626c Coverity 1038115: memory leak on 'ack' keyword parsing failure 13 years ago
Victor Julien 98dbf3e62c Coverity 1038113: possibly out of bounds read 13 years ago
Anoop Saldanha 602c91ed41 Minor cosmetic changes to the cuda code.
Moved a couple of functions to more cuda relevant files;
Re-structured some data types.
13 years ago
Anoop Saldanha c9f076def3 Modified CudaBufferCullCompletedSlices.
Allow readers specify max size of data they want to read.
13 years ago
Anoop Saldanha 70cb4d30eb Add a usleep to CudaBuffer culling process. Would lead to a situation where the thread wouldn't care to yield to others." 13 years ago
Anoop Saldanha 17c763f855 Version 1 of AC Cuda. 13 years ago
Anoop Saldanha 2de59fc235 Version 1 of CudaBuffer API. Introduced to buffer data to the gpu.
This version allows async writes to a buffer by threads.  Allows only
sequential reads though.
13 years ago
Anoop Saldanha 557cab3dc9 We call packet and stream mpm as late as possible now. Won't affect the working of the engine.
The rationale behind this is, if we have pkt buffered to the gpu, we'd want
to delay processing the pkt as much as possible.
13 years ago
Anoop Saldanha d2063d98ad pool now uses a queue kinda behaviour when getting/inserting data through poolbuckets. 13 years ago
Anoop Saldanha f4c719b83a code refactoring. Call mpmprefilter slightly later than where it's called atm 13 years ago
Anoop Saldanha b787da5643 Remove all cuda related code in the engine except for the cuda api wrappers 13 years ago
Anoop Saldanha e2a6cfb6a6 update cuda API wrappers 13 years ago
Eric Leblond d8ce2b1ca4 unix-socket: fix OSX build
MSG_NOSIGNAL is not defined on macOSX and SO_NOSIGPIPE is used
instead.
13 years ago
Eric Leblond a35c367942 action handling: use macro for test.
Use test macro instead of direct access to action field.

This patch has been obtained by using the following
spatch file:

  @@
  Packet *p;
  expression E;
  @@

  - p->action & E
  + TEST_PACKET_ACTION(p, E)
13 years ago
Eric Leblond efaa9a7302 action handling: define and use macros
The action field in Packet structure should not be accessed
directly as the tunneled packet needs to update the root packet
and not the initial packet.

This patch is fixing issue #819 where suricata was not able to
drop fragmented packets in AF_PACKET IPS mode. It also fixes
drop capability for tunneled packets.
13 years ago
Anoop Saldanha 429b5cec10 Fix magic unittets.
Fix segv, when magic_load() fails due to the non-availability of default
magic files.
13 years ago
Anoop Saldanha 058e9278c5 Fix wrong casting of htp pointer. Fixed it back to (HTPState *) inside
htp utility functions.
13 years ago
Anoop Saldanha 21f9cc3a39 discontinue matching on buffer if urilen returns a match failure. 13 years ago
Victor Julien 56c6dd9bb2 bytetest: add unittest showing missed detection
Tests recursive and relative negative byte_test matching.
13 years ago
Anoop Saldanha c3d98f9640 Fix the bug specified in the previous commit.
Bug emanates from byte_test, byte_jump and byte_extract keyword being
unable to handle negative offsets when the inspection pointer is at the
end of the buffer.
13 years ago
Anoop Saldanha bd6896bee1 Unit-tests exposing a bug in byte_test, byte_jump and byte_extract.
Bug emanates from all the keywords being unable to handle negative offsets
when the inspection pointer is at the end of the buffer.
13 years ago
Victor Julien 0fbfaadd53 bytetest: fix debug messages not printing negative offset correctly 13 years ago
Anoop Saldanha ff222b51e7 Http trailer headers unittests added. 13 years ago
Anoop Saldanha ab4b15c2e7 fix for #788.
Now depth is kept in mind when we inspect chunks in client/server body.
This takes care of FPs originating from inspecting subsequent chunks that
match with depth, but shouldn't.
13 years ago
Victor Julien f29e5459e6 luajit/flowint: add ScFlowintIncr & ScFlowintDecr
Add flowint lua functions for incrementing and decrementing flowints.

First use creates the var and inits to 0. So a call:

    a = ScFlowintIncr(0)

Results in a == 1.

If the var reached UINT_MAX (2^32), it's not further incremented. If the
var reaches 0 it's not decremented further.

Calling ScFlowintDecr on a uninitialized var will init it to 0.

Example script:

    function init (args)
        local needs = {}
        needs["http.request_headers"] = tostring(true)
        needs["flowint"] = {"cnt_incr"}
        return needs
    end

    function match(args)
        a = ScFlowintIncr(0);
        if a == 23 then
            return 1
        end

        return 0
    end
    return 0

This script matches the 23rd time it's invoked on a flow.
13 years ago
Victor Julien f312486c6e flowvar/luajit: make 'sets' real time. Needed for cross HTTP-header matching. 13 years ago
Victor Julien 72f6bc2aed luajit: add flowint support
Expose ScFlowintGet and ScFlowintSet functions to luajit. These set
flowints in real time, regardless of rule and/or script match.

Example:

function init (args)
    local needs = {}
    needs["http.request_headers"] = tostring(true)
    needs["flowint"] = {"cnt"}
    return needs
end

function match(args)
    a = ScFlowintGet(0);
    if a then
        ScFlowintSet(0, a + 1)
    else
        ScFlowintSet(0, 1)
    end

    a = ScFlowintGet(0);
    if a == 23 then
        return 1
    end

    return 0
end

return 0

Script's init call first registers "cnt" at id 0, then 0 is used to use
this var.
13 years ago
Victor Julien c3c3cd76e5 flowvar/flowint: split set functions into normal and NoLock version, where the latter won't lock the flow. 13 years ago
Victor Julien 57d3cd97f3 flowvar/flowint: make local function static 13 years ago
Victor Julien 6e18ed0489 luajit flowvar support
This patch adds flowvar support to luajit. It does so by exposing two special
C functions to the luajit scripts: ScFlowvarGet and ScFlowvarSet.
13 years ago
Ken Steele 9d677ea006 Clear the PKT_ALLOC flag when storing Packets into the Packet pool.
The PKT_ALLOC flag is set by PacketGetFromAlloc(), which needs to be
cleared for Packets in the Packet Pool, so clear the flag here.
13 years ago
Ken Steele 9c7b411a5d More PacketGetFromMalloc() to allocate packets. 13 years ago