Commit Graph

2624 Commits (dec34afa4039aee40cc765a4699787ebce2da16f)

Author SHA1 Message Date
Victor Julien 004b5dde88 Support libhtp's different handling of CONNECT requests. 14 years ago
Victor Julien 117d51c965 Fix a compile warning when debug is enabled. 14 years ago
Victor Julien 1df3304655 Clean up for unittests code: only compile unittest api code when unittests are enabled. Fix unittest code that wasn't wrapped in the proper UNITTESTS ifdefs. 14 years ago
Victor Julien a138b32533 flow manager: timing change
Set default timeout for the flow manager to wake up to 1 second. The 0.4 sec
performed best on a Xeon, but in kvm vm's it was horrible:

32 bit vm: 60% cpu for flowmgr when idle.
64 bit vm: 30% cpu for flowmgr when idle.

With the 1 second timeout both are at 0.3% cpu.
14 years ago
Victor Julien 786148319c Lower flow manager wake up timer to 0.4 seconds as that performs 2% better in my tests. 14 years ago
Anoop Saldanha 776bf633e3 flow manager code cleanup. Remove unused code + fix indentation. Remove unused vars 14 years ago
Anoop Saldanha 5133098bd6 Accomodate pcap-file mode to signal flow mgr to wakeup when it exceeds a certain time interval. This let's the flow mgr keep in sync with pcap timestamp changes 14 years ago
Anoop Saldanha 9917744707 separate timers for flow mgr thread for normal and emerg mode. Signal flow mgr thread when in emerg mode 14 years ago
Eric Leblond 5a63662766 Flow: use condition system instead of short sleep
Short sleep can lead to some really annoying performance issue in
some environnement like virtual systems. This technic was used in
the flow manager. This patch uses an alternate approach based on
a timed condition which is triggered each time a new flow has to
be created. This avoid to run out of flow. A counter is also done
to be able not to run the cleaning code at each new flow.
14 years ago
Victor Julien 34450b9b57 Don't parse layers / ext headers above ipv6 frag header. This is taken care of by defrag. 14 years ago
Victor Julien 938e9b3db0 Fix filestore related segv. 14 years ago
Victor Julien e6d8d0443c Unify output functions for alert-debug for IPv4 and IPv6. 14 years ago
Victor Julien 3c7f09d1ea Add debug output to engine event. 14 years ago
Victor Julien e6af837b25 Convert StreamTcpSetEvent function into macro. Eases debug. 14 years ago
Victor Julien 58011554b0 Don't consider payload len in ACK value validation check. 14 years ago
Victor Julien 9878eca086 file handling: expand filestore keyword
Filestore keyword by default (... filestore; ... ) marks only the file in the
same direction as the rule match for storing. This makes sense when inspecting
individual files (filemagic, filename, etc) but not so much when looking at
suspicious file requests, where the actual file is in the response.

The filestore keyword now takes 2 optional options:

filestore:<direction>,<scope>;

By default the direction is "same as rule match", and scope is "currently
inspected file".

For direction the following values are possible: "request" and "to_server",
"response" and "to_client", "both".

For scope the following values are possible: "tx" for all files in the current
HTTP/1.1 transation, "ssn" and "flow" for all files in the session/flow.

For the above case, where a suspious request should lead to a response file
download, this would work:

alert http ... content:"/suspicious/"; http_uri; filestore:response; ...
14 years ago
Victor Julien ddfa5c49c6 Stream engine: gap handling
Set a stream event for stream gaps.
Add a (disabled by default) signature to the stream-event.rules.
14 years ago
Victor Julien 45d86ff58a Stream reassembly / app layer: disable gap errors
Gap errors on the app layer are now silently handled. No longer printed
to the screen.
14 years ago
Victor Julien 425294f912 stream reassembly: account stream gaps
Add counter to the stream reassembly engine to count stream gaps. Stream gaps
are the result of missing packets (usually due to packet loss). This missing
data stops the reassembly for the app layer.
14 years ago
Victor Julien d8d8fdd9f5 Improve handling of packets when stream is in the fin_wait1 or fin_wait2 state. 14 years ago
Victor Julien b74c73309b file handling: improve filestore keyword handling
In stateful detection only inspect the file portion of the rule after all
other conditions matched. This to prevent "filestore" from tagging files
for storage during a partial match.

Add a couple of unittests to test the behaviour change.
14 years ago
Victor Julien 4cbe7519fa Add missing file util code. 14 years ago
Victor Julien 56b96363b8 Fix merge artefact. 14 years ago
Victor Julien 63c9a3ab85 Remove duplicate include. 14 years ago
Victor Julien 042fd850fc Make sure we check the sgh for no magic and no store once per flow direction. 14 years ago
Victor Julien f3fbc1a44c file handling: filemagic matching improvement
Magic buffer is a null terminated string. Allow matching on the final
\0 using filemagic:"somevalue|00|"; so we can anchor to the end of the
buffer.
14 years ago
Victor Julien 2ccd35c6e4 Fix code after rebase. 14 years ago
Victor Julien 33848124d1 Fix a multipart body parsing issue. 14 years ago
Victor Julien 96d20098b0 file inspect: stateful inspection split
Split stateful detection of the files in a HTTP state between toserver
and toclient inspection.
14 years ago
Victor Julien d59ca75e46 file extract: split toserver and toclient tracking
Split toserver and toclient file tracking for the http state.
14 years ago
Victor Julien 04ea70ccf7 file extract: pruning
Add pruning of files in memory so we keep only memory what we really need.
Fix magic logic.
Reset file part of the de_state on receiving another file in the same tx.
14 years ago
Victor Julien 1c934acc85 Don't store fd per file (too many fd's). Enable IPv6 storing. Close file on receiving stream end flag. 14 years ago
Victor Julien b402d97179 File carving -- enable reponse file extraction
- Enable response body tracking
- Enable file extraction for responses
- File store meta file includes magic, close reason.
- Option to force magic lookup for all stored files.
- Fix libmagic calls thead safety.
14 years ago
Victor Julien 66a3cd96a8 Prepare HTTP response body tracking. 14 years ago
Victor Julien 417495e542 file-extraction: remove no longer used files. 14 years ago
Victor Julien e1022ee5ae file-extraction: Disconnect file handling from flow and move into the app layer state. 14 years ago
Victor Julien 27645f64c6 Remove unused util-filetype.[ch] from Makefile.am. 14 years ago
Victor Julien 9b62ec65ab Make sure filemagic works properly regardless of filestore being in use for a flow. 14 years ago
Victor Julien 5945e652d6 Initial implementation of filemagic keyword. 14 years ago
Victor Julien f4a6f4b293 Add libmagic detection, linking and a basic API. 14 years ago
Victor Julien 23e01d23d3 Implement filestore keyword, including a way for the stateful detection engine to conclude that a file will never have to be stored. 14 years ago
Victor Julien 3e7baa6810 Fix improper error handling in http body chunk function. 14 years ago
Victor Julien 403b2788d6 Add support for extracting PUT files. 14 years ago
Victor Julien 59cda9a358 Fix not using new htp callback when using the bundled htp. Add indication to --build-info. Fix valgrind warning in test and further improve test. 14 years ago
Victor Julien ef0536794c Adding comments, some cleanups. 14 years ago
Victor Julien 21acd72adf Cleanups to the Multipart parsing code. Fixes to negation in filename and fileext. 14 years ago
Victor Julien 70f0d3d2e7 Add negation to filename and fileext, use same syntax as with content. 14 years ago
Victor Julien 32fb9f375d log-file log-dir option added, meta file created, fixes. 14 years ago
Victor Julien a6b7a560f1 Fix a bug in the HTTP file closing. 14 years ago
Victor Julien 7e3d537338 Fix setting libhtp personality. 14 years ago
Victor Julien 1eef36b011 Initial checkin of a log-file module, that can write files extracted from flows to disk. 14 years ago
Victor Julien 3c1edf3763 Add a file descriptor to the flow file structure. 14 years ago
Victor Julien cd618e48df Allow for 0 (unlimited) HTTP request_body_limit, fix option parsing. 14 years ago
Victor Julien 4723f07254 Improve testing and fix some bugs. 14 years ago
Victor Julien 9d5d46c4bb Implement flow file storage API, create HTP wrappers for it, use it in HTTP parsing. 14 years ago
Victor Julien a0ee6ade3e Improve HTTP multipart parsing, add streaming parsing for files. 14 years ago
Victor Julien 4537f889ef Handle all strings as raw strings in HTTP content-type and content-disposition header parsing. 14 years ago
System Administrator 222bc6e935 Flow files 14 years ago
Pablo Rincon 6d60b3a747 filename and fileext keywords 14 years ago
Victor Julien 06b1d71032 Small optimizations to IPV4 and TCP header parsing. 14 years ago
Eric Leblond 0256ca2422 af-packet: fix compilation on new systems.
Inclusion of if_packet.h was missing when the support of new options
related to packet fanout is present in the file.
14 years ago
Anoop Saldanha bf24272c28 changes to accomodate master rebase 14 years ago
Anoop Saldanha 997eaf42a8 add thread local storage support for smtp + remove pmq that was init/freed as part of smtp_state alloc to use the thread local data passed by the app layer engine 14 years ago
Anoop Saldanha 9a6aef459e modify all relevant app layer API calls to accomodate passing parser local storage argument 14 years ago
Anoop Saldanha d3468d88b0 app layer udp cleanup + update dcerpc udp todo 14 years ago
Anoop Saldanha 01a35bb604 introduce app layer local storage api support 14 years ago
Anoop Saldanha 87599bc78d minor changes in smtp parser decoder wrt direction check loop + add missing ifdef unittests 14 years ago
Anoop Saldanha 3a856fed12 update detection engine to compare flow alproto with sig_alproto, rather than sm alproto. 14 years ago
Anoop Saldanha 4d38a571cc smtp reply code mpm phase support added 14 years ago
Anoop Saldanha 4a6908d3e9 fix smtp parser handling fragmented lines + add new unittests to check the same 14 years ago
Anoop Saldanha 2b356dadff Support for tos keyword added 14 years ago
deltay 211193b0af Get pidfile from config file if not available in command options 14 years ago
Victor Julien 262a7300d7 flow: shrink Flow datatype
Introduce a separate FlowAddress structure for holding the ipv4 or ipv6 address
that doesn't have the family in it like the Address structure. Instead, the
family is stored in the flow as a flag: FLOW_IPV4 and FLOW_IPV6.

Add macro's to check the family, copy the address, etc.

Update many unittests to reflect these changes. Introduce unittest helper
functions for creating and initializing a flow and freeing it again.

On 64 bit this shrinks the flow with 8 bytes.
14 years ago
Victor Julien 06904c9024 App Layer cleanup
Removal of per flow 'aldata' array. It contained a ptr for each ALPROTO. Instead now we have 2 ptrs in the flow: alparser and alstate.
Various cleanups and dead code removal from the app layer API.
Should safe 100+ bytes memory per flow on 64 bit.
Updated lots of unittests to reflect these changes.
14 years ago
Victor Julien a0b532dc45 stream reassembly: simplify base_seq tracking for protocol detection. Shrinks TcpStream structure. 14 years ago
Victor Julien 7e3c15e54a stream: improve TCP ssn reuse cleanup. 14 years ago
Victor Julien 9769510ba3 flow: support requeue of flows from closed to new list for TCP ssn reuse. 14 years ago
Anoop Saldanha 4130c5e2b8 if flow has disabled app layer inspection, disable buffering the segments unnecessarily in inline reassembly 14 years ago
Anoop Saldanha 43cbed8c92 enable toclient alproto detection for inline reassembly 14 years ago
Anoop Saldanha f684b60127 if flow has disabled app layer inspection, disable buffering the segments unnecessarily 14 years ago
Anoop Saldanha 08bd8ec4e2 on failed alproto detection on both sides, only disable app layer inspection. No reassembly disabling for any direction 14 years ago
Victor Julien c9960473bb Fix stream reassembly engine rejecting valid packet for reassembly. 14 years ago
Anoop Saldanha 55ed6c2a55 disable session reassembly for either/both the directions, only when we have established failed proto detection in both the directions 14 years ago
Anoop Saldanha 4650bf7170 minor code cleanup. remove commented out code 14 years ago
Anoop Saldanha de9ad02b59 Remove leftover imap and msn toclient alproto PM contents 14 years ago
Anoop Saldanha caf26c2618 More updates to FFR code. Handle cases where we actually need to force stream reassembly and just have smsgs to be processsed by detection engine separately 14 years ago
Anoop Saldanha bc216a3396 fix/updates to app layer proto detection 14 years ago
Anoop Saldanha 78e6a7f713 enable toclient alproto detection. Detection all current alproto toclient PMP patterns 14 years ago
Anoop Saldanha 9c8d404db1 FFR update-fix. Fix check where we decide whether we need to send pseudo pkt or not 14 years ago
Anoop Saldanha b08b390bcd fix for bug 375 - update radix test that wrongly uses memset and sizeof 14 years ago
Victor Julien 3d845b6c77 Consider Windows new line chars as well when parsing rule files. Bug #374. 14 years ago
Eileen Donlon a92d15ed37 Fixed duplicate signature check 14 years ago
Anoop Saldanha 99baf18c8d updates to ac-gfbs search. Remove unnecessary casting of pointers 14 years ago
Anoop Saldanha 11e7dda59a updates to ac-gfbs search. Introduce handling cases where state_count is < 32k 14 years ago
Anoop Saldanha 708c4ad055 updates to ac-gfbs search. Combine output presence with mod goto table 14 years ago
Anoop Saldanha a4ea7e6197 updates to ac-gfbs search. Combine failure table along with mod goto table for better cache perf 14 years ago
Anoop Saldanha b69ac9514f updates to ac-gfbs search. Disable handling < 65k states separately. Now any state count would be given same treatment 14 years ago
Anoop Saldanha efb4c27b1f updates to ac-gfbs search. Add new unittests + fix cases where we have 2 patterns that are same but one is CS and other CI + Use SCMemcmp for state < 65k instead of custom memcmp 14 years ago
Anoop Saldanha 0920296aaa updates to ac-gfbs search. Remove unnecessary casting of pointers 14 years ago
Anoop Saldanha d149a5e806 updates to ac-gfbs search. Use SCMemcmp instead of the custom pattern searching used 14 years ago
Anoop Saldanha 47f2d6e07b updates to ac-gfbs search. Optimize pointer de-referencing for pid_pat_list 14 years ago
Anoop Saldanha 991f6d2d83 updates to ac-gfbs search. Optimize pointer de-referencing for frequently used pointers 14 years ago
Anoop Saldanha ffb925e3b3 indentation fixes for ac-gfbs 14 years ago
Anoop Saldanha e9eb0e502c updates to ac-gfbs search. Handle cases where we have a single entry for a state goto transition, just like how we handle for no entry for a particular state 14 years ago
Eric Leblond 9b75de3339 pfring: fix compilation when pfring is desactivated. 14 years ago
Eric Leblond 0ac1cabf2a autotools: fix problem of pfring configuration. 14 years ago
deltay d5e254d504 Add pfring bpf filter, require pfring >= 5.1 14 years ago
Eric Leblond 9f73503daa capability: rework capability assignement
THis patch rework the capability code to use a switch
instead of a if. It also "reduces" PF_RING and NFQ capabilities.
14 years ago
Anoop Saldanha d034b10180 remove debug prints added to ac code 14 years ago
Anoop Saldanha 781e7c776f fix indentation in ac code 14 years ago
Anoop Saldanha 5c56053a33 Reintroduced optimized support for < 32k states for ac 14 years ago
Victor Julien fb76561b09 Set version to 1.2dev to reflect we're in the 1.2 branch. 14 years ago
Victor Julien 8cc82c7241 Add -S commandline option that loads a rule file exclusively. Issue #338. 14 years ago
Victor Julien c484b7a59e Bump version to 1.1 (final) 14 years ago
Eric Leblond 62e63e3fe9 af-packet: fix reconnection on netdown error.
AFPRead can fail following a NETDOWN error. This patch treat errors
of AFPRead by forcing a reconnection (instead of exiting thread
with error).
14 years ago
Eric Leblond 361bf22121 af-packet: suppress annoying debug message.
This message was firing multiple per second when a monitored
interface disappear.
14 years ago
Victor Julien 0fadd93011 Fix an invalid free in bpf code. 14 years ago
Victor Julien ea53f72f7d Fix CUDA build. 14 years ago
Eric Leblond 9f7ee03deb log: read output filter from config file.
The output filter was not read from configuration file and thus
not used in this case.
14 years ago
Eric Leblond 866d681ff2 pfring: fix stupid enum usage.
pfring code is not using standard notation for the cluster_type enum
and this leads to a horrific code in pfring acquisition code.
14 years ago
Eric Leblond a6a0d4eae6 pfring: use deinit function.
This fixes #368.
14 years ago
Eric Leblond a54afe7052 Fix printing of sizeof. 14 years ago
Victor Julien 2d16abcf8b Minor code cleanups fixing all GCC 4.6 compiler warnings for default, debug and unittests mode. 14 years ago
Eric Leblond 2387c6b0e8 pcap: Fix setting of buffer size from command line. 14 years ago
Victor Julien 1be65e7b68 Fixes for building in Cygwin. 14 years ago
Victor Julien 85033f5afe Fix windows adapter id being truncated for pcap mode. 14 years ago
Eric Leblond 2bc0be6e65 af-packet: fix compilation problem on windows. 14 years ago
Victor Julien 404868c28b Get rid of strcasestr call as win32 doesn't have it. 14 years ago
Victor Julien 561630d864 Fix SMTP unittest. 14 years ago
Victor Julien 47abd0ef19 Fix compiler warning. 14 years ago
Anoop Saldanha 0acfcc206c fix unittests. fix replace unittests that allow alproto keywords with replace 14 years ago
Anoop Saldanha a0eec3d846 fix detection code that handles cases when we use recursion(from recursive keyword) 14 years ago
Anoop Saldanha 7433d92dd2 undo this commit -
commit eff08f93d8
Author: Anoop Saldanha <poonaatsoc@gmail.com>
Date:   Thu Nov 3 14:31:24 2011 +0530

    update failing unittest to reflect the mpm design update

Fixed a bug in the mpm code that would make all the changes in the commit just undone wrong.
14 years ago
Anoop Saldanha 1b1332fff0 fix mpm bug on running stream mpm for packets not added to stream mpm 14 years ago
Victor Julien 9f0e3f7c85 Bump version to 1.1rc1. 14 years ago
Victor Julien 55da9787a4 Win32 compile fixes. 14 years ago
Victor Julien d070869c48 Reinstate replace validation check. 14 years ago
Anoop Saldanha eff08f93d8 update failing unittest to reflect the mpm design update 14 years ago
Victor Julien af51493da2 Mpm update: Toss out signatures that mix pkt and stream/state. Update profiling code to track new mpm. 14 years ago
Anoop Saldanha 539ce13695 fix broken unittests 14 years ago
Anoop Saldanha 17f3f36d38 packet keywords only added for packet mpm. Rest in stream mpm. Update detection engine to handle the same 14 years ago
Anoop Saldanha e0c36f7aff fix dsize sigs handling. We can't use more than 2 dsizes in the same sig 14 years ago
Anoop Saldanha c7b9d3fecb Remove broken dsize_sm in SigMatch used by dsize in detection engine 14 years ago
Anoop Saldanha d3ca65de03 support bdat smtp keyword - bug #347 14 years ago
Victor Julien 91957d70a8 Fix unittest compiler warning. 14 years ago
Victor Julien b5e17ec1d8 Rewrite SetupLogging to not leak the fd. Thanks to Steve Grubb for advice on this. 14 years ago
Eric Leblond ba9fb53461 threshold: fix handling of multiple threshold.
This patch fixes the unittest and may fix the real work.
14 years ago
Eric Leblond 142fe6e4b6 threshold: fix recently introduced function. 14 years ago
Victor Julien e0d7f64a14 Fix thresholding code suppressing an alert if no threshold/suppress rules needed to be checked. 14 years ago
Eric Leblond 86f9759427 threshold: fix thresholding on signature with multiple threshold.
This patch uses the newly introduced SigGetThresholdTypeIter
function to try all threshold for a signature. This should fix
issue #366.
14 years ago