Commit Graph

9182 Commits (1d9f37a60e7fde720768f41e5680ee2e02b78ffc)

Author SHA1 Message Date
Philippe Antoine d2b3668a68 init: InitGlobal function
To be reused by fuzz targets
6 years ago
Philippe Antoine f5190da67e util: UTHmemsearch to use memmem if defined 6 years ago
Philippe Antoine ce55d06569 check: Using const keyword for some arguments
For ConfigSetLogDirectory and PrintRawLineHexBuf
6 years ago
Philippe Antoine 12a3a24906 log: can use a file set from env variable
Enables the redirection of log to a file set by an environment
variable SC_LOG_FILE when SC_LOG_OP_IFACE=file
6 years ago
vanlink 2456f27d08 stream/reassembly: fix data overlap check
Fix function CheckOverlap bug.
6 years ago
Victor Julien aeefc82eb9 tls: fix missing extern logic for cert_id tracking 6 years ago
Victor Julien 62c0f3d2b4 stats: fix missing extern keyword 6 years ago
Victor Julien 903291f88a defrag: fix use of globals 6 years ago
Victor Julien 5c3c6c609c threading: fix queue handlers globals use 6 years ago
Victor Julien 3ae1854d2f htp: fix globals use for flags 6 years ago
Victor Julien 85289f3283 proto: fix globals use 6 years ago
Victor Julien 5e583f3a12 flow: fix global variable use 6 years ago
Victor Julien 29f54a34ae stream: fix global declaration of the config 6 years ago
Victor Julien 2436daccd9 threading/modules: fix global declarations 6 years ago
Victor Julien c5f4b41881 ippair: fix global declarations 6 years ago
Victor Julien 0a006d2258 host: fix global declarations 6 years ago
Victor Julien 29780d6164 mpm: fix global declarations 6 years ago
Victor Julien b89059bda7 detect: fix global declaration of sigmatch_table 6 years ago
Victor Julien 0118e07d57 spm: fix global declaration of spm_table 6 years ago
Victor Julien a12c0b499d threading: fix global declaration of threading_set_cpu_affinity 6 years ago
Victor Julien 45955d2e58 unix-socket: avoid using global variable w/o extern 6 years ago
Victor Julien a9a522fac3 decode: fix default-packet-size global variable 6 years ago
Victor Julien 7709b90c16 detect/file-data: remove debug abort that wasn't reachable 6 years ago
Victor Julien ac8ceae9bf detect/file-data: fix function doc 6 years ago
Victor Julien 500e8da63a files: tracking flag update
Improve flow file flags and file flags updates. Introduce a mask
that is set at start up to avoid lots of runtime checks.

Disable cocci flags check as it doesn't support the more dynamic
nature of the flag updates.
6 years ago
Victor Julien a4a4d17ad0 app-layer/files: optimize GetFiles calls
Remove FlowGetProtoMapping calls from the GetFiles wrapper and
get the alstate from the flow directly.
6 years ago
Victor Julien d369e54f1d app-layer: all protocols are tx aware now
So remove the runtime check for it.
6 years ago
Timo Sigurdsson 1262ecbde0 init: Fix dropping privileges in nflog runmode
Using the run-as configuration option with the nflog capture method
results in the following error during the startup of suricata:
[ERRCODE: SC_ERR_NFLOG_BIND(248)] - nflog_bind_pf() for AF_INET failed

This is because SCDropMainThreadCaps does not have any capabilities
defined for the nflog runmode (unlike other runmodes). Therefore, apply
the same capabilities to the nflog runmode that are already defined for
the nfqueue runmode. This has been confirmed to allow suricata start
and drop its privileges in the nflog runmode.

Fixes redmine issue #3265.

Signed-off-by: Timo Sigurdsson <public_timo.s@silentcreek.de>
6 years ago
Victor Julien 7810f22413 decode: remove pseudo packet checks
Bug 1107 checks/hacks should not longer be needed, so remove them.
6 years ago
Victor Julien 272a5f526b threading/queues: simplify error handling 6 years ago
Victor Julien dce227ec88 threading/queues: remove 256 queue limit
Convert fixed size array to a dynamic TAILQ so we can
grow as needed.
6 years ago
Victor Julien 0e3f27a87e threading: remove 'trans_q' array of packet queues
Let the queues code set up PacketQueues on demand.
6 years ago
Victor Julien efa8a69923 packet-queue: create alloc and free functions 6 years ago
Victor Julien 550cfdd98d threading: hide 'trans_q' from queue handlers 6 years ago
Victor Julien 45e5e19e6e threading/threadvars: optimize layout
Make sure StatsPublicThreadContext is on its own cache line.
6 years ago
Victor Julien e3fbdf1948 flowworker/stream: use no-lock packet queue
Use smaller structure for temporary packet queues.
6 years ago
Victor Julien f8aed4ce2d threading: change local packet queue logic
Previously each 'TmSlot' had it's own packet queue that was passed
to the registered SlotFunc as an argument. This was used mostly for
tunnel packets by the decoders and by defrag.

This patch removes that in favor of a single queue in the ThreadVars:
decode_pq. This is the non-locked version of the queue as this is
only a temporary store for handling packets within a thread.

This patch removes the PacketQueue pointer argument from the API.
The new queue can be accessed directly through the ThreadVars
pointer.
6 years ago
Victor Julien b8c2b66d33 packet-queue: introduce a non-locked version
Works exactly like PacketQueue, just does not contain a mutex
and cond var, leading to much reduced memory size.
6 years ago
Victor Julien 9ed260c489 threading: more efficient TmSlot layout 6 years ago
Victor Julien 18e652309f threading: remove 'id' field from TmSlot
Field was now unused.
6 years ago
Victor Julien d7cb0774dd detect: cleanup reload thread handling 6 years ago
Victor Julien 786e697590 threading: simplify flow timeout loop 6 years ago
Victor Julien 261b77742e threading: shrink and reorganize TmSlot 6 years ago
Victor Julien 87c9b11d8c threading/threadvars: rearrange for better cache behavior 6 years ago
Victor Julien 071b753e84 threading/threadvars: remove unused 'prev' field 6 years ago
Victor Julien f53f004917 threading: remove unused 'TmThreadRemove' function 6 years ago
Victor Julien 569a5d985b threading: remove handler names to shink struct
Shrink ThreadVars by removing the queue handler names that are only
used at shutdown. Since this is not performance critical, we can use
the id's to look up the queue handler.
6 years ago
Victor Julien 74a6f8d4dd threading/queues: add way to lookup by ID
In preparation of doing runtime operations by ID instead of by name,
add functions to look up by ID and to convert name to ID.
6 years ago
Victor Julien d0218696ba threading: shrink threadvars struct size 6 years ago
Victor Julien c029599515 threading: remove unused threadvars field 6 years ago
Victor Julien f1ee176111 threading: clarify threadvars fields 6 years ago
Victor Julien d50492cb20 threading: cleanup packet thread shutdown loop 6 years ago
Victor Julien 8e762f5190 source-pcap: remove unused function 6 years ago
Victor Julien 3a703c84ad threading/modules: declare prototypes static
Declare registered threading API funcs static where appropriate.
6 years ago
Victor Julien 7c83cb585e sources: fix pipeline failure handling
When TmThreadsSlotProcessPkt fails it will return the packet that was
passed to it to the packetpool.

Some of the packet sources were doing this manually as well. This patch
fixes those sources.
6 years ago
Victor Julien 49599dfe89 threading: use tm_flowworker for pseudo packets
Pseudo packets don't need to be processed by the decoding layer.
6 years ago
Victor Julien 9df8e1c984 threading: add shortcut to flowworker 6 years ago
Victor Julien 02004fa547 threading: remove per slot post_pq
Use a single packet queue per thread for flow timeout packet
injection. The per slot queue was unused except for this use
case. Having a single queue makes the logic and implementation
simpler.

In case of 'autofp', the per thread packet queue will actually
use the threads input queue. For workers/single a dedicated
queue will be set up.

Rename TmThreadsSlotHandlePostPQs to TmThreadsHandleInjectedPackets
to reflect the changed logic.
6 years ago
Victor Julien 15e3bdb7b8 af-packet: prototypes cleanup
Remove unused prototype.

Declare other prototypes static.
6 years ago
Victor Julien 44d7f636f2 threading: remove post_pq argument from 'SlotFunc'
This was not in use anywhere.
6 years ago
Victor Julien f5045af3e3 runmodes: code cleanups 6 years ago
Victor Julien 1a8562b3c6 detect: clean up threads handling
Clean up reload and break loop thread handling.
6 years ago
Victor Julien e5010d7704 detect: inject packet cleanup 6 years ago
Victor Julien abea227cfc flow-manager: code cleanups 6 years ago
Victor Julien 6fd35fb786 flow-manager: avoid doubly signaling threads
Don't try to wake up the threads we just flagged and validated that
they changed their state.
6 years ago
Victor Julien 539c1a275f threading: remove commented out function 6 years ago
Victor Julien b55f617c2f threading: optimize and unify post_pq checks
TmThreadsSlotProcessPkt did not need to look all 'slots' as only the first
slots post_pq can have been used.

Unify post_pq cleanup handling.
6 years ago
Victor Julien 2a1ed3ba1b threading: remove wrong unlikely statement; minor cleanups 6 years ago
Victor Julien e5192ae20a threading: TmSlot::SlotFunc does not need to be atomic 6 years ago
Victor Julien 89048d71ad threading: fix flags handling by using uint32_t everywhere 6 years ago
Victor Julien 85cf341189 threading: optimize error handling in main packet loop 6 years ago
Victor Julien 9d8ea3b4fe threading: minor code style cleanups 6 years ago
Victor Julien b1056b3836 threading: simplify packetpool checks 6 years ago
Victor Julien 603b2ced47 threading/queues: add shortcut for packetpool check
Allows code simplifycation in the threading loops.
6 years ago
Victor Julien 29cb9d1d52 threading/queues: minor code cleanups 6 years ago
Victor Julien f05c12b70f afl: fix compilation 6 years ago
Victor Julien 01862eae78 afl/decoder: make file dumps optional 6 years ago
Victor Julien 42d112e7b6 detect/address: dead code removal and style cleanups 6 years ago
Jason Ish 80cafb2979 flow: expose last time as a function
This function returns the individual components
of the timeval in output pointers making it suitable
for use over Rust FFI.
6 years ago
Jason Ish d1eab5aa46 defrag: set livedev on the reassembled packet (issue-3380)
Set the livedev on reassembled packets to that of the parent
packet. Fixes issues with multidetect, specifically a segfault
as reported in issue 3380.

Bug #3380.
6 years ago
Victor Julien 9d0976ea8a output/tx: split list of loggers per alproto
This patch splits the list of loggers the tx logging walks into lists per
alproto. The list was getting longer with each eve addition. The result
was that for each tx we would have to loop through multiple loggers that
did not apply to this tx as it was for the wrong protocol.
6 years ago
Victor Julien 5b7aa506c1 output: micro optimization
LogFunc is always set, so don't check for it at runtime.
6 years ago
Victor Julien 4f9e4d41e0 output: optimize root logging loop
Instead of unconditionally looping all the 'root' loggers, loop only
those that are in use.

Root loggers are: packet, tx, file, filedata, streaming.
6 years ago
Victor Julien 07df1ce6af output: clarify registration 6 years ago
Victor Julien 284c3cf68a output/tx: bail early if no flow 6 years ago
Victor Julien ed99e9204f output: fatal error if root logger alloc fails 6 years ago
Danny Browning b573c16dd5 build: cbindgen
Rust headers are now generated using cbindgen. If cbindgen is present, they can
be generated during dist, otherwise they will be available for builds.
6 years ago
Victor Julien 2c050187a3 streaming/api: fix overlap check
In some cases a SBB could be seen as overlapping with the requested
offset, when it was in fact precisely before it. In some special cases
this could lead to the stream engine not progressing the 'raw' progress.
6 years ago
Victor Julien 0f41cf3d74 debug/validation: check tcp/app-layer data lengths 6 years ago
Victor Julien a742c86741 stream: improve app-layer data retrieval with GAPs
Don't assume that the next block after the sbb head is after the
requested offset.

If the next block was before the offset, the returned data_len
would underflow and return a nonsense value to the app-layer.

Bug #2993.
6 years ago
Jeff Lucovsky ed2f6ac64b modbus: Correct typo 6 years ago
Jeff Lucovsky d4428d94de modbus: Update correct TX flags 6 years ago
Jeff Lucovsky 6c2cdbb5f0 analysis: exit if table entries are stale
This commit causes Suricata to exit when a buffer from the analyzer
table is not recognized.

Since the table must match what's registered, exiting will bring noticed
to the condition.
6 years ago
Victor Julien 627cc23769 detect/asn1: fix offset bounds checking 6 years ago
Jason Ish 8609939e60 ipv4: continue parsing options after invalid option
As long as an option has a valid length, we can continue
parsing the options after an invalid one.
6 years ago
Jason Ish df8db1ddb0 ipv4: fail packet decoding on bad ipv4 option length
Currently all failures in IPv4 option decode are ignore with
respect to continuing to handle the packet.

Change this to fail, and abort handling the packet if the
option length is invalid.

Ticket 3328:
https://redmine.openinfosecfoundation.org/issues/3328
6 years ago
Victor Julien fa692df37a stream: reject broken ACK packets
Fix evasion posibility by rejecting packets with a broken ACK field.
These packets have a non-0 ACK field, but do not have a ACK flag set.

Bug #3324.

Reported-by: Nicolas Adba
6 years ago
Victor Julien 9f0294fadc stream: fix SYN_SENT RST/FIN injection
RST injection during the SYN_SENT state could trick Suricata into marking
a session as CLOSED. The way this was done is: using invalid TSECR value
in RST+ACK packet. The ACK was needed to force Linux into considering the
TSECR value and compare it to the TSVAL from the SYN packet.

The second works only against Windows. The client would not use a TSVAL
but the RST packet would. Windows will reject this, but Suricata considered
the RST valid and triggered the CLOSED logic.

This patch addresses both. When the SYN packet used timestamp support
the timestamp of incoming packet is validated. Otherwise, packet responding
should not have a timestamp.

Bug #3286

Reported-by: Nicolas Adba
6 years ago
Victor Julien df74f34a62 decode/tcp: accept TCP fast open cookie request 6 years ago
Victor Julien 040aff5197 htp: close request only from request side
This allows the response side to keep going for just
a bit longer.
6 years ago
Victor Julien 77539e08fc stream: in IDS mode, call app-layer at EOF
On stream end call app-layer with empty message in IDS mode.
6 years ago
Victor Julien eceb7dcba4 eve: support pcap_filename for unix socket mode
Bug #3390.
6 years ago
Philippe Antoine 4a2918e6b5 yaml: clarify comment about dump-all-headers
Logs a warning if the value is unknown
Fixes #2810
6 years ago
Victor Julien 007a461d69 detect/parse: track negation during address parsing
Fix address negation detection not resolving variables when
looking for the negation.

This patch makes use of the actual parsing routines to relay this
information to the signature parser.

Bug #3389.

Fixes: 92f08d85aa ("detect/iponly: improve negation handling in parsing")
6 years ago
Victor Julien 34b7035a0d detect/iponly: debug output improvements 6 years ago
Victor Julien 618ad0d92f app-layer: optimize inspection id tracking
Increase the inspect id for a completely inspected tx in any case.
This avoids re-evaluating transactions.

Reported-by: Ilya Bakhtin
6 years ago
Victor Julien f302f3543f files: add call for setting inspect sizes
The inspect sizes are currently only used during file prune
house keeping for SMTP.
6 years ago
Victor Julien f9f958d66e smtp: fix and clean up new file handling
Set tx id on files that were just opened.

Move logic to a small util func.
6 years ago
Victor Julien 683b22d114 smtp: use FILE_USE_DETECT for raw-extract 6 years ago
Victor Julien 21760bfc76 files: change pruning behavior
If file prune is called inspect has already run. So if file is closed
we can just prune. No need to consider a window anymore.

When still in progress, fix the left_edge calculation.
6 years ago
Victor Julien 682014619f files: fix FILE_USE_DETECT with --disable-detection
Don't set FILE_USE_DETECT flag if detect is disabled.
6 years ago
Victor Julien 4ac9cd2c70 files: move smtp prune logic to main
Now that we call the file prune loop very regularly, we can move the
SMTP specific inspection pruning logic into this loop. Helps with
cases there we don't (often) update a files inspection trackers.
6 years ago
Victor Julien 4b7599af90 http/file: modernize unittests
Part of ticket #2975.
6 years ago
Victor Julien 1cdb2182e4 fastlog: apply icmp type logic to icmpv6 too 6 years ago
Victor Julien 5ef05ffad1 http/multipart: small cleanup 6 years ago
Victor Julien aae00df4df http/multipart: use wider type for boundary lengths
Use uint32_t for a local type instead of uint8_t to avoid casts.

Length should always stay under this regardless.
6 years ago
Victor Julien 4d0db9cb4a http/multipart: optimize form end search
If we already know that the boundary exists, we can start looking
there. Otherwise, we can skip trying as the boundary is a subset
of the form end marker.
6 years ago
Victor Julien 54d93e1eb9 http/multipart: process incomplete file data
Start processing multipart data as soon as it is available to
allow inspection sooner.
6 years ago
Jeff Lucovsky fcfb679893 detect/analyzer: Suppress direction warnings
This commit ensures direction warnings for ICMP v4 and v6
are suppressed and corrects check so that both protocols
are checked (instead of the same protocol being checked twice).
6 years ago
Eric Leblond 1b9009ea0e suricata: fix computing of default packet size
Update the default packet size computation to use LiveDeviceName
instead of LiveDevice as the LiveDevice list is not built when
the default packet size is built.
6 years ago
Victor Julien c010f092e0 detect/replace: fix debug print issue
Don't print field that will likely not be 0 terminated.
6 years ago
Victor Julien c3ea5e71e5 detect/file.data: fix buffer reusing id 0 6 years ago
Victor Julien cd66c37711 http/multipart: use proper progress value to test eof 6 years ago
Philippe Antoine 08b84e060b fastlog: use icmp type and code instead of port
Fixes #3266
6 years ago
Philippe Antoine 75a7d9641c fastlog: move code to reduce variable scope 6 years ago
Philippe Antoine c2fdd7c969 transform: fixes comment about compress_whitespace 6 years ago
Philippe Antoine 9126fc25c1 transform: updates doc about compress_whitespace
And removes duplicate test from strip_whitespace
6 years ago
Eric Leblond 3ded7f1170 qa/coccinelle: fix false positive in setter getter
Coccinelle test was doing a false positive on the function
AppLayerParserStateSetFlag and AppLayerParserStateIssetFlag.
To address that, this patch adds a new coccinelle markup:

 /* coccinelle: AppLayerParserStateSetFlag():2,2:APP_LAYER_PARSER_ */

It indicates that AppLayerParserStateSetFlag is a setter and getter
and that the checks should be disabled inside the function.

Currently this markup is only used for that but following patch will
add some checks on option value.
6 years ago
Jeff Lucovsky 8f4f1cb633 detect/analyzer: Improved fast pattern display
When transforms are part of a rule, improve information displayed with
fast patterns to include the original buffer name and whether any
transform(s) are applied.
6 years ago
Jeff Lucovsky c88c1f1e14 detect/analyzer: Suppress direction warnings
This commit suppresses direction warnings by the rules analyzer for ICMP
and ICMPV6 since it's not actionable.
6 years ago
Victor Julien 83bbe287e7 stats: fix stats not always syncing in flow timeout 6 years ago
Jason Ish ba3a2c31bf app-layer: validate TX detect flag callbacks
Check that both are set or unset.
6 years ago
Jason Ish 706558d4d5 enip: add tx detect flags 6 years ago
Jason Ish cb62c8dacf dcerpc: add tx detect flags 6 years ago
Jason Ish 21f014f5c3 modbus: add tx detect flags 6 years ago
Jason Ish 20bc08a722 app-layer: add tx detect functions to register struct 6 years ago
Jason Ish fdb587d2fc detect-engine: check for tx detect flag support
When registing a detection engine, check that the app-layer
protocol supports tx detect flags. Exit with a fatal
error if it does not as this is a code implementation
error that should be resolved during development.
6 years ago
Jason Ish b1beb76fd7 ftpdata: add tx detect flags 6 years ago
Jason Ish 62e4211f04 debug: add SCReturnBool function exit macro 6 years ago
Jason Ish 739df21e2d app-layer: method to see if parser supports tx detect flags
Add method to check if a parser for an app-layer protocol
supports tx detect flags.

This is a bit of a hack for now as where we need to run
this check from we do not have the IP protocol.
6 years ago
Jeff Lucovsky 218a5c4345 mpm: Fix typos and spelling errors 6 years ago
Jeff Lucovsky aef24bee96 detect: Fix spelling errors 6 years ago
Jeff Lucovsky f318a46d34 detect: Improve handling of variable values
When one of offset/depth/distance is from a variable, adjust the depth
by the offset as is done with scalar values at parse time.
6 years ago
Jeff Lucovsky db8527e7b3 detect/mpm: Improved handling of variable values
This commit removes the offset and depth if either of these values are
dependent upon a byte-extract operation.
6 years ago
Victor Julien 94982ae690 http: split request/response tx id handling
When HTTP pipelining was in use, the transaction id used for events
and files could be off. If the request side was several requests ahead
of the responses, it would use the HtpState::transaction_cnt for events
and files, even though that is only incremented on complete requests.

Split request and response tx id tracking. The response is still handled
by the HtpState::transaction_cnt, but the request side is now handled by
its own logic.
6 years ago
Victor Julien b82e71b95e files: remove FILE_USE_TRACKID flag
Once it was optional but as it no longer is it is no longer useful.

Remove it.
6 years ago
Victor Julien f9155aa121 files: simplify pruning logic
Since ebcc4db84a the flow worker runs
file pruning after parsing, detection and loging. This means we can
simplify the pruning logic. If a file is in state >= CLOSED, we can
prune it. Detection and outputs will have had a final chance to
process it.

Remove the calls to the pruning code from Rust. They are no longer
needed.
6 years ago
Victor Julien ab471c3054 app-layer: don't consider tx flags if not registered
If a protocol does not support TxDetectFlags, don't try to use them.

The consequence of trying to use them was that a TX would never be
considered done, and it would never be freed. This would lead to excessive
memory use and performance problems due to walking an ever increasing
list.
6 years ago
Eric Leblond 54d3620662 source-pcap-file: honor bpf filter on command line
When a BPF filter is given on the command line when reading a
pcap file, the BPF filter is not honored.

The regression has been introduced in:

commit 3ab9120821
Author: Dana Helwig <dana.helwig@protectwise.com>
Date:   Thu Apr 27 11:17:16 2017 -0600

    source-pcap-file: Pcap Directory Mode (Feature #2222)

Reported-By: Tim Colin <tcolin@et.esiea.fr>
6 years ago
Eric Leblond 860f43753c source-pcap-file: fix memory leak on pcap filter 6 years ago