Commit Graph

518 Commits (a6adb5dbbf9cf4ac634f9f85d551a4c777676813)

Author SHA1 Message Date
Victor Julien 1c8775b340 QA: --afl-rules for faster rule fuzzing 9 years ago
Victor Julien 9b08cdae74 capture: only check for faster methods on -i
Also, since we now default to AF_PACKET for -i if available, only check
for PF_RING and NETMAP.
9 years ago
Victor Julien 053b96458f commandline: add -i arg check 9 years ago
Victor Julien f8852f4415 commandline: use afpacket for -i if available 9 years ago
Victor Julien a3a7d9b299 pcap: unify -i and --pcap parsing 9 years ago
Victor Julien b50111a5a7 commandline: move afpacket parsing into util func 9 years ago
Victor Julien 1fe09a38e0 commandline: move pcap parsing into util func 9 years ago
Victor Julien 7ac7f9cd55 instance: add progname as ptr to argv[0] 9 years ago
Mats Klepsland 8111eb934f QA: add --afl-der=<file>
Expose SSL/TLS certificate decoding (DER) to commandline
using --afl-der=<file>.
9 years ago
Victor Julien d165906397 QA: add --afl-decoder-ppp=<file> 9 years ago
Victor Julien bdaba1d815 QA: expose Mime decoding API to commandline using --afl-mime=<file> 9 years ago
Victor Julien 077ac81688 QA: direct access from commandline to AppLayer API
This patch introduces a new set of commandline options meant for
assisting in fuzz testing the app layer implementations.

Per protocol, 2 commandline options are added:

--afl-http-request=<filename>
--afl-http=<filename>

In the former case, the contents of the file are passed directly to
the HTTP parser as request data.

In the latter case, the data is devided between request and responses.
First 64 bytes are request, then next 64 are response, next 64 are
request, etc, etc.
9 years ago
Victor Julien d461837511 afl: add --afl-parse-rules to return 0 on any rule
When fuzzing, AFL will create lots of malformed rules. We don't want
to error out on those. As we're fuzzing the parser any non-crash
should return 0. Crashes (ASAN or not) will return a non-0 code.
9 years ago
Victor Julien 4c1c13d110 detect reload: improve signal logic 9 years ago
Victor Julien c0294521dd startup: move more into PostConfLoadedSetup 9 years ago
Victor Julien 0ab83288ac startup: move RunUnittests to StartInternalRunMode 9 years ago
Victor Julien 4a2e816bea detect parser: add parse regex util function
Add regex setup and free util functions. Keywords often use a regex
to parse rule input. Introduce a common function to do this setup.

Also create a list of registered regexes to free at engine shutdown.
9 years ago
Jason Ish 13cf2f829e testing: new test macros, new testing documentation group.
Unit testing support macros for failing on expressions,
as well as passing tests on expressions.

If fatal unittests are enabled BUG_ON will be triggered for
an assertion providing the line number of the failure, otherwise
the test will simply fail.

Moved the fatal flag to a global var instead of a configuration
parameter for ease of access from a macro.
9 years ago
Victor Julien 725d6c3739 yaml: convert detect-engine to just detect
Instead of detect-engine which used a list for no good reason, use a
simple map now.

detect:
  profile: medium
  custom-values:
    toclient-groups: 3
    toserver-groups: 25
  sgh-mpm-context: auto
  inspection-recursion-limit: 3000
  # If set to yes, the loading of signatures will be made after the capture
  # is started. This will limit the downtime in IPS mode.
  #delayed-detect: yes
9 years ago
Victor Julien 722e2dbf7c profiling: initial rulegroup tracking
Per rule group tracking of checks, use of lists, mpm matches,
post filter counts.

Logs SGH id so it can be compared with the rule_group.json output.

Implemented both in a human readable text format and a JSON format.
9 years ago
Justin Viiret 13b87f5aff mpm: add Hyperscan integration
This adds an MPM implementation that uses the Hyperscan regex engine
library from Intel, accessible as the "hs" mpm-algo.
9 years ago
Victor Julien 30410e6900 capture: warn -i user if faster options are available
If af-packet, netmap or pfring are available, users should use those
for best performance.
9 years ago
Victor Julien dd98bc353e signals: cleanup signal handling
Simplify handling of USR2 signal. The SCLogInfo usage could lead to
dead locks as the SCLog API can do many complicated things including
memory allocations, syslog calls, libjansson message construction.

If an existing malloc call was interupted, it could lead to the
following dead lock:

 0  __lll_lock_wait_private () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:97
 1  0x0000003140c7d2df in _L_lock_10176 () from /lib64/libc.so.6
 2  0x0000003140c7ab83 in __libc_malloc (bytes=211543457408) at malloc.c:3655
 3  0x0000003140c80ec2 in __strdup (s=0x259ca40 "[%i] %t - (%f:%l) <%d> (%n) -- ") at strdup.c:43
 4  0x000000000059dd4a in SCLogMessageGetBuffer (tval=0x7fff52b47360, color=1, type=SC_LOG_OP_TYPE_REGULAR, buffer=0x7fff52b47370 "", buffer_size=2048,
    log_format=0x259ca40 "[%i] %t - (%f:%l) <%d> (%n) -- ", log_level=SC_LOG_INFO, file=0x63dd00 "suricata.c", line=287, function=0x640f50 "SignalHandlerSigusr2StartingUp", error_code=SC_OK,
    message=0x7fff52b47bb0 "Live rule reload only possible after engine completely started.") at util-debug.c:307
 5  0x000000000059e940 in SCLogMessage (log_level=SC_LOG_INFO, file=0x63dd00 "suricata.c", line=287, function=0x640f50 "SignalHandlerSigusr2StartingUp", error_code=SC_OK,
    message=0x7fff52b47bb0 "Live rule reload only possible after engine completely started.") at util-debug.c:549
 6  0x000000000057e374 in SignalHandlerSigusr2StartingUp (sig=12) at suricata.c:287
 7  <signal handler called>
 8  _int_malloc (av=0x3140f8fe80, bytes=<value optimized out>) at malloc.c:4751
 9  0x0000003140c7ab1c in __libc_malloc (bytes=296) at malloc.c:3657
 10 0x0000000000504d55 in FlowAlloc () at flow-util.c:60
 11 0x00000000004fd909 in FlowInitConfig (quiet=0 '\000') at flow.c:454
 12 0x0000000000584c8e in main (argc=6, argv=0x7fff52b4a3b8) at suricata.c:2300

This patch simply sets a variable and lets the main loop act on that.
9 years ago
Victor Julien f78e990915 signals: handle INT/TERM signals in the main loop 9 years ago
Eric Leblond 994bc15c2a af-packet: disable most code when not built in
When af-packet is not build we can avoid to build a large amount
of code.
10 years ago
Andreas Herz c467c39b61 build-info: workaround special _FORTIFY_SOURCE defines
On systems like Gentoo where _FORTIFY_SOURCE is already defined like
FORTIFY_SOURCE=((defined __OPTIMIZE && OPTIMIZE > 0) ? 2 : 0) the use
within the printf function (%d) won't result in the correct value and
we end up with 'defined' undeclared compile error. This workaround makes
sure that just the resolved value is checked and then printed.
10 years ago
Victor Julien 893b01ed0d bpf: fix memleak on unusual error (CID 1197757) 10 years ago
Victor Julien 3a9bcd6a53 qa: add --simulate-ips option
This option forces the engine in 'IPS' mode. This is useful for testing
some IPS code paths based on pcaps.
10 years ago
Alexander Gozman 368d7e913a In configuration test mode, check signatures if 'delayed-detect' is enabled
When 'detection-engine.delayed-detect' option was set to 'yes',
suricata didn't check signatures validity in configuration test mode.
10 years ago
Eric Leblond 93642a0d1d suricata: can't use -l and unix socket runmode
It is not possible to use simultaneously -l and unix socket
runmode because setting the log directory make it final so
not modifable by other call.

It is a implementation limitation but it does not make sense
to set logging directory to have it overwritten by the first
directory specified 'by pcap-file'. So it seems correct to
only trigger an error if this both options are used at the same
time.
10 years ago
Victor Julien 07d8617b3e multi-detect: improve error handling 10 years ago
Victor Julien d7d76e7b27 multi-detect: use default tenant
The default detect engine can be used as 'default tenant'.
10 years ago
Eric Leblond a4089873c7 rules-reload: fix reload with -s or -S
When using the -S or -s option, the reload was causing the specified
rules file to be forgotten and the default rules to be loaded at
reload time.
10 years ago
Eric Leblond c199fdcbb9 handle MTU discovery in multi iface case
This patch adds handling of multiple interface by using as
default-packet-size the maximum MTU + header length among the
live iface.
10 years ago
Eric Leblond 9672e16323 suricata: clean dump-config output
When user asks for a configuration dump, it is useless to display
the version and CPU info. Also initializing the log system conduct
to overwrite the some log files and in particular suricata.log and
this is annoying as a command should not interfere with a running
daemon.
10 years ago
Victor Julien 7f8795c756 threading: avoid autofp deadlock
When there are many threads and/or the packet pool (max-pending-packets) is
small, a potential dead lock exists between the packet pool return pool
logic and the capture threads. The autofp workers together can have all the
packets in their return pools, while the capture thread(s) are waiting at an
empty pool. A race between the worker threads and the capture thread, where
the latter signals the former, is lost by the capture thread. Now everyone
is waiting.

To avoid this scenario, this patch makes the previously hardcoded 'return
pool' threshold dynamic based on the number of threads and the packet pool
size.

It sets the threshold to the max pending packets value, divided by the number
of lister threads. The max value hasn't changed. Normally, in the autofp
runmode these are the stream/detect/log worker threads.

The max_pending_return_packets value needs to stay below the packet pool size
of the 'producers' (normally pkt capture threads but also flow timeout
injection) to avoid the deadlock.

As it's quite impossible at this time to learn how many threads will be
created before starting the runmodes, and thus spawning the threads and
already initializing the packet pools, this code sets a global variable
after runmode setup, but before the threads are 'unpaused'.
10 years ago
Jason Ish bcda92134d app-layer: template for application layer tx logger 10 years ago
Victor Julien b653479815 detect: make multi tenancy a global switch
At start up we will set this flag based on "multi-detect.enabled".
10 years ago
Eric Leblond 4db0a35f25 tls-store: now a separate module
An design error was made when doing the TLS storage module which
has been made dependant of the TLS logging. At the time there was
only one TLS logging module but there is now two different ones.

By putting the TLS store module in a separate module, we can now
use EVE output and TLS store at the same time.
10 years ago
Victor Julien b51c4e608f logging: optional colors output
Construct message per output method.
10 years ago
Jason Ish ae23144b67 --set - handle spaces on either side of '='
Discard spaces when provided as part of --set around the '='. For
example, "val=key", "val = key", "val= key" and "val =key" are
all equivalent now.
10 years ago
Victor Julien cb5aa8f8d5 counters: work around unix-socket init issues 10 years ago
Victor Julien 752f03e7a4 counters: remaining s/SCPerf/Stats/g 10 years ago
Tom DeCanio 117eed0385 eve-log: add JSON stats logging
Support for counters in stats.log in eve output JSON stream.
10 years ago
Victor Julien 33756abd87 counters: split API init
Split into early ctx initialization and post-config setup.
10 years ago
Victor Julien df5e9d44ca unix-manager: convert to thread module
Sync command thread for unix manager with other managers and make
it a full thread module.
10 years ago
Victor Julien 34f2ff067b reference: update pcre globals use
Don't update globals each time we parse, but instead do it once at
startup.
10 years ago
Victor Julien 46d401e3bb classification: update pcre globals use
Don't update globals each time we parse, but instead do it once at
startup.
10 years ago
Victor Julien 9764a35604 stream: fix --disable-detection reassembly issue
Due to an error at initialization, the stream engine would not disable
'raw' reassembly automatically when --disable-detection was used.

This lead to segments not getting cleared from the segment lists.
10 years ago
Victor Julien 94321b8a2f packet pool: fix memleaks
Don't kill flow manager and recyclers before the rest of the threads. The
packet threads may still have packets from their pools. As the flow threads
would destroy their pools the packets would be lost.

This patch doesn't kill the threads, it just pulls them out of their run
loop and into a wait loop. The packet pools won't be cleared until all
threads are killed.

Wait for flow management threads to close before moving on to the
next steps in the shutdown process.

Don't destroy flow force reassembly packet pool too early. Worker
threads may still want to return packets to it.
10 years ago