8762 Commits (0bb07b550c1f00a95e6803aba7cdeb22d637fbc2)

Author	SHA1	Message	Date
Bill Meeks	a291209e47	detect/geoip: migrate to GeoIP2 database format ... Issue #2765	7 years ago
Victor Julien	d6323ae33d	detect/mpm: improve stats reporting	7 years ago
Victor Julien	24f0092b72	detect: add ipv6.hdr sticky buffer ... Inspects IPv6 header and extension headers.	7 years ago
Victor Julien	3c9a557810	decode/ipv6: track length of ext hdrs	7 years ago
Victor Julien	9252400f68	decoder/ipv6: minor cleanups ... Remove unused field and macros. Minor code style cleanups.	7 years ago
Victor Julien	4ac327f5b5	detect/ipv4: add ipv4.hdr sticky buffer	7 years ago
Victor Julien	367e3e1895	detect/tcp/udp: minor cleanups	7 years ago
Victor Julien	4dff903b35	detect: introduce pkt mpm engines ... Instead of the hardcode L4 matching in MPM that was recently introduced, add an API similar to the AppLayer MPM and inspect engines. Share part of the registration code with the AppLayer. Implement for the tcp.hdr and udp.hdr keywords.	7 years ago
Victor Julien	14896365ef	detect: remove Threadvars argument from API calls ... Remove it as it's (almost) never used. If it is really needed it can be accessed through DetectEngineThreadCtx::tv as well.	7 years ago
Victor Julien	c1dd4534d9	detect/bsize: support transforms in case w/o content	7 years ago
Victor Julien	82de6e0659	decoder/vxlan: improvements and cleanups ... Implement port config handling. Also check both src port and dest port for tunnels that only set the destination port to the VXLAN port. At the point of the check we don't know the packet direction yet. Implement as Suricata tunnel similar to Teredo. Cleanups.	7 years ago
Henrik Lund Kramshoej	3519b011b7	decoder/vxlan: initial implementation of decoder	7 years ago
Victor Julien	35b88991c3	mem: fix shadow declaration warning ... Avoid clash by adding a leading underscore to the declaration in the macro. These temporary vars should never clash with valid variables from the code where they are called from.	7 years ago
Philippe Antoine	e30a77c5a1	warnings : Fixes integer sizes in format strings	7 years ago
Victor Julien	ba5679de0f	profiling: fix undefined profiling code use	7 years ago
Victor Julien	ac694b089a	detect: add udp.hdr sticky buffer	7 years ago
Victor Julien	bdf53f449c	detect/tcp: rename tcp keyword files	7 years ago
Victor Julien	47919fd0e4	detect/tcp: add tcp.<field> notation variants	7 years ago
Victor Julien	0cecb1c3b2	detect/prefilter: minor error handling cleanup	7 years ago
Victor Julien	35be8385eb	detect: tcp.hdr sticky buffer ... Sticky buffer to inspect the TCP header.	7 years ago
Victor Julien	47ef8f5822	detect/mpm: add l4-header support ... Prepare MPM part of the detection engine for a new type of per packet matching, where the L4 header will be inspected. Preparation for TCP header inspection keyword.	7 years ago
Victor Julien	0965afd661	detect: pkt inspect engines ... Instead of hard coded calls to the inspection logic for payload inspection and 'MATCH'-list inspection use a callback approach. This will register a callback per 'sm_list' much like how app-layer inspect engines are registered. This will allow for adding more types later without adding runtime overhead. Implement the callback for the PMATCH and MATCH logic.	7 years ago
Victor Julien	2a778e439a	detect/build: minor code cleanups	7 years ago
Victor Julien	66648df099	detect: add tcp.mss keyword ... Allows matching on TCP option MSS. Syntax: tcp.mss:<value>; tcp.mss:<value1>-<value2>; tcp.mss:<op><value>; Operator can be: >, <.	7 years ago
Jeff Lucovsky	d2fdbc7d6f	output/json: Avoid use of uninitialized value ... In JsonPacket, a Base64Decode error could cause an uninitialized variable to be used because its return value is ignored.	7 years ago
Jeff Lucovsky	a5d9d37c34	rust/parser: Extend Rust parser for event-by-id ... Extend the Rust parsing infrastructure with the "get event info by id" calls. This changeset extends the parser structure, the C-based registration handlers and the template parser.	7 years ago
Jeff Lucovsky	294d0e8cba	snmp: Fix type	7 years ago
Jeff Lucovsky	9ccc28baeb	rust/smb: Implement get event by id	7 years ago
Victor Julien	28ed0d3a18	nfs: implement get_event_info_by_id callback	7 years ago
Victor Julien	429ca858dc	rust/gen: turn *mut*const T into const T **	7 years ago
Jeff Lucovsky	d568e7fadd	eve/logging: 2991 Optimize logging by TX ... This changeset makes changes to the TX logging path. Since the txn is passed to the TX logger, the TX can be used directly instead of through the TX id.	7 years ago
Jeff Lucovsky	488446cf37	logging/anomaly: Track event log progress ... This changeset adds a mechanism to track when individual events are logged. Transactions can be provided more than once; track events to prevent event re-logging.	7 years ago
Jeff Lucovsky	4a39d7a1ee	app-layer: Extend event container with progress	7 years ago
Jeff Lucovsky	07c05f7dd8	logging/anomaly: TX based logging addition	7 years ago
Jeff Lucovsky	36644907f5	app-layer/logging Add TX packet logging support ... Add transactional logging support for packet based loggers, like the anomaly logger.	7 years ago
Jeff Lucovsky	f7b934f83f	app-layer/logging: protocol parser updates	7 years ago
Jeff Lucovsky	49438569a2	logging: anomaly log updates	7 years ago
Jeff Lucovsky	50e23ba93a	app-layer: Initial app layer logging	7 years ago
Victor Julien	4e8d38348f	flow: no bypass manager for Windows	7 years ago
Victor Julien	287b87b81b	bypass: switch to gettimeofday	7 years ago
Victor Julien	d8614a15c6	mingw: fix compile error ... Declare _POSIX_C_SOURCE before sys/time.h to avoid: util-time.c: In function 'SCUtcTime': util-time.c:222:12: error: implicit declaration of function 'gmtime_r'; did you mean 'gmtime_s'? [-Werror=implicit-function-declaration] 222 | return gmtime_r(&timep, result); | ^~~~~~~~ | gmtime_s util-time.c:222:12: warning: returning 'int' from a function with return type 'struct tm *' makes pointer from integer without a cast [-Wint-conversion] 222 | return gmtime_r(&timep, result); | ^~~~~~~~~~~~~~~~~~~~~~~~ util-time.c: In function 'SCLocalTime': util-time.c:305:9: error: implicit declaration of function 'localtime_r'; did you mean 'localtime_s'? [-Werror=implicit-function-declaration] 305 | localtime_r(&timep, &cached_local_tm[lru]); | ^~~~~~~~~~~ | localtime_s util-time.c:321:56: warning: comparison between pointer and integer 321 | if (localtime_r(&timep, &cached_local_tm[lru]) == NULL) | ^~ cc1.exe: some warnings being treated as errors Tickets: #2994 #3051	7 years ago
Eric Leblond	6126f105ea	util-ebpf: fix creation of flow from pinned maps	7 years ago
Eric Leblond	7df3007066	af-packet: xdp pinned maps need to be read	7 years ago
Eric Leblond	4ba02830b6	flow-bypass: registration of non periodic check ... This patch adds the capability to register a set of functions without providing a periodic check function. This permit to run a task only at init.	7 years ago
Eric Leblond	3e8fd26ee3	flow-hash: real hash computation for FlowKey	7 years ago
Eric Leblond	0963fea390	util-ebpf: log bypassed flow maps count	7 years ago
Eric Leblond	7e5439bc15	bypass: add counter for local captured bypass ... Packets from captured bypassed flows that are received by Suricata before the capture method start to bypass them can represent an important part due to various buffer and insertion delay. This patch adds a two counters to know the number of packets and bytes in this case.	7 years ago
Eric Leblond	de52e78e65	bypass: flow bypass is not ebpf only	7 years ago
Eric Leblond	c938dbde27	util-ebpf: early exit if no map	7 years ago
Eric Leblond	e5d325a6ca	util-ebpf: reindex	7 years ago
Eric Leblond	aa989a7445	af-packet: some conditional fields	7 years ago
Eric Leblond	abbb066a96	bypass: bytes and pkts counters for local bypass	7 years ago
Eric Leblond	9206b30fe1	af-packet: better accounting and error handling ... This patch improves the bypass error handling add adds more counters to the interface so it is possible to get a view on success and failure of insertion in the eBPF maps via the `iface-bypassed-stat` command.	7 years ago
Eric Leblond	aeb2bd3aa1	util-ebpf: optimization on flow storage queries	7 years ago
Eric Leblond	288f335aa5	util-ebpf: simplify free function ... First key can't be null.	7 years ago
Eric Leblond	f4abe2f9c0	util-ebpf: set livedev in flow ... This will fix the accounting for pinned maps as the livedev field of Flow is used to do the accounting of bypass flows.	7 years ago
Eric Leblond	98e7d9d1c0	util-device: introduce bypassed stats sub function	7 years ago
Eric Leblond	f29a4b8bee	flow-manager: move bypass timeout to a inline func	7 years ago
Eric Leblond	89e8cb50ed	util-ebpf: case is not possible so remove warning	7 years ago
Eric Leblond	8c97998cb9	bypass: implement iface-bypassed-stat for callback	7 years ago
Eric Leblond	51ab06256a	bypass: account callback method in stats	7 years ago
Eric Leblond	f78e5ba1e1	bypass: restore interface counter	7 years ago
Eric Leblond	d119845d98	bypass: compress flow keys structure	7 years ago
Eric Leblond	69d2c8eb75	ebpf: get rid of hash in map value	7 years ago
Eric Leblond	b07bda7a7b	bypass: new callback stragegy ... This patch introduces and uses a new bypass strategy based on a callback. EBPF bypass implementation is updated to use this new strategy. Once the flow manager detect that a flow should be timeouted, it asks the capture method if it has seen packets in the interval. If it is the case the lastts of the flow is updated and the timeout is postponed.	7 years ago
Eric Leblond	fcae1c18de	af-packet: improve error handling for some hw ... Some cards seems to return EAGAIN when there is no more place in the hash table.	7 years ago
Eric Leblond	0bfbb4a889	bypass: fix accounting ... The flow bypass stats is computed at every pass so the accounting needs to be done at each pass. This patch fixes the accounting in the flow_bypassed counters.	7 years ago
Eric Leblond	44566e5a24	ebpf: only display that file is loaded if we do it	7 years ago
Eric Leblond	5e62ae6d28	af-packet: avoid error flooding when bypass fails	7 years ago
Eric Leblond	af6daceeda	util-ebpf: more useful error message ... At the time of the writing, libbpf output useful error message on strdout only and errno is not really interesting. So let's tell user to look at stdout.	7 years ago
Eric Leblond	0f64c25b73	util-ebpf: improve code readability ... As pointed by Victor Julien, the pkts_cnt usage was quite confusing so functions are now returning a bool.	7 years ago
Eric Leblond	a8f35cc30e	util-ebpf: discard flow if no Flow storage	7 years ago
Eric Leblond	efb648aa24	util-ebpf: fix ebpf bypass ... Fix endian order in eBPF bypass. It has to be updated after the bypassed flows handling change.	7 years ago
Eric Leblond	f8aa9ee986	bypass: fix wait time at exit ... The loop on bypassed flow maps can take a few seconds on heavily loaded system causing Suricata to not honor a stop before a few seconds. This patch adds the code needed to detect the need to exit from the check loop.	7 years ago
Eric Leblond	a277f2eb0c	af-packet: fix use after free on config ... ASAN did find that afp config was used after free. This was in fact done in the Flow bypass manager hence this patch.	7 years ago
Eric Leblond	fc2f2fa7d3	bypass: allow bypass for packet without flow ... For capture method that have their own flow structure (not maintained by Suricata), it can make sense to bypass a packet even if there is no Flow in Suricata. For AF_PACKET it does not make sense as the eBPF map entry will be destroyed as soon as it will be checked by the flow bypass manager. Thus we shortcut the bypass function if ever no Flow is attached to the packet. This path also removes reference to Flow in the bypass functions for AF_PACKET. It was not necessary and we possibly could benefit of it if ever we change the bypass algorithm.	7 years ago
Eric Leblond	6ab1cbcb8e	bypass: use flow storage for bypass counter ... There is a synchronization issue occuring when a flow is added to the eBPF bypass maps. The flow can have packets in the ring buffer that have already passed the eBPF stage. By consequences, they are not accounted in the eBPF counter but are accounted by Suricata flow engine. This was causing counters to be completely wrong. This code fixes the issue by avoiding the counter change in invalid case. To avoid adding 4 64bits integers to the Flow structure for the bypass accounting, we use instead a FlowStorage. This limits the memory usage to the size of a pointer.	7 years ago
Eric Leblond	4e6add7faa	bypass: generalize iface bypass stats ... Introduce functions in util-device.c to be able to manage the flow bypassed count stats.	7 years ago
Eric Leblond	258e90be76	util-ebpf: change flow accounting logic ... Update the flow counters during the life of a bypassed flow instead of just accounting at the end of it.	7 years ago
Eric Leblond	3026e9a80d	util-ebpf: better error handling	7 years ago
Eric Leblond	2ffd3ad2b7	util-ebpf: better error handling of map unlink	7 years ago
Eric Leblond	b952b32a26	util-ebpf: rename field 'unlink' to avoid confusion	7 years ago
Eric Leblond	edf2db4e30	af-packet: improve warning message	7 years ago
Eric Leblond	567b5ee1bc	af-packet: rename option 'no-percpu-hash'	7 years ago
Eric Leblond	0f6b1297a9	af-packet: warn when XDP is not supported	7 years ago
Eric Leblond	b1c9b39435	af-packet: remove question from code	7 years ago
Eric Leblond	1992093c88	flow-bypass: rename variables	7 years ago
Eric Leblond	d239e0f2d5	flow-hash: doc and code cleaning	7 years ago
Eric Leblond	b736344975	flow-bypass: clock_gettime error handling ... Only reason clock_gettime could fail is a permission so let's error and leave the flow bypass manager if it is the case. Also let's suppress the error message if ever the error appear in the middle of a run (which is unlikely).	7 years ago
Eric Leblond	4129938c21	util-ebpf: log level fixes and code cleaning	7 years ago
Eric Leblond	140269a6be	util-ebpf: init code optimization	7 years ago
Eric Leblond	c5e2af0545	util-ebpf: fix error reported by coccinelle check ... Some allocation errors were not checked during init phase.	7 years ago
Eric Leblond	c1fd0da550	af-packet: add vlan_id in bypass key ... Bypassing on vlan was not supported due to the missing key.	7 years ago
Eric Leblond	651a27e4fb	ebpf: fix percpu hash handling ... An alignement issue was preventing the code to work properly. We introduce macros taken from Linux source code sample to get something that should work on the long term.	7 years ago
Eric Leblond	142c69e1ef	flow-bypass: increase bypass timeout ... This is needed as we did switch from counter maintained in kernel to internal polling so we need a bigger value.	7 years ago
Eric Leblond	b8e184ceb5	flow-bypass: fix timeout of maps bypassed flows ... The time is taken from the parameter and is checked against real flow entries so we need a standard time.	7 years ago
Eric Leblond	07d0bd3a0f	util-ebpf: fix IPv6 deletion loop	7 years ago
Eric Leblond	b481f290e2	af-packet: fix bypass for IPv6	7 years ago
Eric Leblond	3bd8ba5d00	util-ebpf: add message if key deletion fails	7 years ago
Eric Leblond	269f601f8a	util-ebpf: can't delete in place so update algo	7 years ago
Eric Leblond	5b056c15bf	af-packet: fix default in pinned maps name	7 years ago
Eric Leblond	eff56acca5	af-packet: be sure to nullify option if not set	7 years ago
Eric Leblond	36c6a62954	util-ebpf: simplify function declarations	7 years ago
Eric Leblond	69630d7a17	util-ebpf: micro optimization	7 years ago
Eric Leblond	d21c3a6555	util-ebpf: create flow from bypassed flows	7 years ago
Eric Leblond	04c65a309e	flow-hash: new function to get flow from flowkey	7 years ago
Eric Leblond	885fc992de	ebpf: make table iterator generic ... Also adds a basic skeleton for flow creation loop.	7 years ago
Eric Leblond	880c42f11c	af-packet: bypass with init function	7 years ago
Eric Leblond	522e98d830	util-ebpf: fix iteration in flow timeout ... We were not setting the key using the correct item in map. Result was deletion of wrong flow.	7 years ago
Eric Leblond	0c3e1e8579	af-packet: correctly set up hardware offload	7 years ago
Eric Leblond	bd28f77a1e	af-packet: fix loading of ebpf filter	7 years ago
Eric Leblond	6fdcb127e9	af-packet: fix the start when XDP is pinned	7 years ago
Eric Leblond	7e0ef4cec8	util-ebpf: change return of pinned maps loading ... The calling function needs to be able to see when this is a success and XDP do not need to be reloaded.	7 years ago
Eric Leblond	d950a9f272	util-ebpf: conditional flow table loading	7 years ago
Eric Leblond	d25e8dbfc7	af-packet: implement pinned-maps-name	7 years ago
Eric Leblond	36838017fe	af-packet: fix build when eBPF not built-in	7 years ago
Eric Leblond	b1769d5f8f	util-ebpf: implement pinned maps loading ... Load flow tables at start if asked to.	7 years ago
Eric Leblond	96f1454ebf	util-ebpf: only unlink pinned maps in eBPF filter	7 years ago
Eric Leblond	4cf531008e	af-packet: conditionaly remove XDP filter ... Only remove the XDP filter if we are in XDP mode and not using the pinned maps.	7 years ago
Eric Leblond	4f48c45727	util-ebpf: conditional pinning of maps ... Only pin maps if `pinned-maps` is set in the configuration. This ensure backward compatibility.	7 years ago
Eric Leblond	73a928fa0b	util-ebpf: fix loop on maps ... We were missing the last element of the map by working on previous key instead of current key.	7 years ago
Eric Leblond	3ce69c446b	util-ebpf: suppress spaces at end of line	7 years ago
Eric Leblond	954762a429	ebpf: add filter by maps on example filter	7 years ago
Eric Leblond	5d8ac36a49	util-ebpf: pin the maps ... By pinning the maps we are creating a file in /sys/fs/bpf that can be used by external program to access the map. This has multiple benefits such as handling list from an external program. The pinned maps could be persistent accross Suricata reload but this can be complicated in term of handling everything in the life of Suricata.	7 years ago
Eric Leblond	bf4381b17b	ebpf: document XDP iface redirect	7 years ago
Eric Leblond	a37a7c4a21	ebpf: reduce counter size to allow netronome offload	7 years ago
Eric Leblond	315c29a8e6	ebpf: change the logic to avoid ktime usage ... Kernel time is not available (and/or costly) on NIC such as Netronome so we update the logic to detect dead flows based on a lack of update of packets counters. This way, the XDP filter will be usable by network card. This patch also updates the ebpf code to support per CPU and regular mapping. Netronome is not supporting it and the structure is using atomic for counter so the cost of simultaneous update is really low. This patch also updates the xdp_filter to be able to select if the flow table is per CPU on shared. Second option will be used for hardward offload. To deactivate the per cpu hash, you need to set USE_PERCPU_HASH to 0. This patch also adds an new option to af-packet named no-percpu-hash If this option is set to yes then the Flow bypassed manager thread will use one CPU instead of the number of cores. By doing that we are able to handle the case where USE_PERCPU_HASH is unset (so hardware offload for Netronome). This patch also remove aligment indications in the eBPF filter. This was not really needed and it seems it is causing problem with some recent version of LLVM toolchain.	7 years ago
Eric Leblond	1745408c13	flow-manager: no force reassembly on bypassed flow ... When a bypassed flow is created we are forcing the reassembly so we don't need to do it again when it timeout.	7 years ago
Shivani Bhardwaj	7c7e04487f	detect/content: Message for escaping backslash ... So far, if the rule loaded had a backslash character ("\") in its content field, the rule will fail to load but without giving a descriptive error message. This patch tells the user to escape the troubling character. Before ``` 9409] 7/6/2019 -- 16:12:22 - (detect-engine-loader.c:184) <Error> (DetectLoadSigFile) -- [ERRCODE: SC_ERR_INVALID_SIGNATURE(39)] - error parsing signature "alert tcp $HOME_NET any -> $EXTERNAL_NET any (msg:"Backslash needs escaping msg"; flow:established,to_server; content:"MyBackslash\here"; sid:86; rev:1; )" from file /var/lib/suricata/rules/myrule.rules at line 1 ``` After ``` [9409] 7/6/2019 -- 16:12:22 - (detect-content.c:155) <Error> (DetectContentDataParse) -- [ERRCODE: SC_ERR_INVALID_SIGNATURE(39)] - '\' has to be escaped [9409] 7/6/2019 -- 16:12:22 - (detect-engine-loader.c:184) <Error> (DetectLoadSigFile) -- [ERRCODE: SC_ERR_INVALID_SIGNATURE(39)] - error parsing signature "alert tcp $HOME_NET any -> $EXTERNAL_NET any (msg:"Backslash needs escaping msg"; flow:established,to_server; content:"MyBackslash\here"; sid:86; rev:1; )" from file /var/lib/suricata/rules/myrule.rules at line 1 ``` Closes redmine ticket #2626	7 years ago
Victor Julien	7513f6cae2	threads: minor code cleanups	7 years ago
Victor Julien	92d38683ce	threads: improve flow timeout loop ... Improve thread safety and remove BUG_ON	7 years ago
Victor Julien	e6b633cf95	stats: walk tv_root under lock	7 years ago
Victor Julien	9474b84683	stats: more accurate time handling for wakeup thread	7 years ago
Victor Julien	ab1268cfea	stats: minor code cleanups	7 years ago
Philippe Antoine	b5f3e03209	New app layer event for invalid http request line ... Handles logs from libhtp even if case of error	7 years ago
Philippe Antoine	4a74e9edcc	signature: error for rules with illegal port ... Fixes #2080	7 years ago
Philippe Antoine	248f9c66cf	warnings: fixes integer sizes in format strings ... Fixes #3009	7 years ago
Philippe Antoine	a3e38861b9	http: adds unit tests for HTPParseContentRange	7 years ago
Philippe Antoine	90ab0b0ec2	http: fix overflow in HTPParseContentRange	7 years ago
Victor Julien	39cf5b0c8b	snmp: fix 'defined but not used' warning ... Remove HAVE_RUST guards as well.	8 years ago
Pierre Chifflier	8c61cfdca5	snmp.version: move unittests to tests/	8 years ago
Pierre Chifflier	58d85b1460	snmp.pdu_type: move unittests to tests/	8 years ago
Pierre Chifflier	ff0ff6344f	snmp.community: move unittests to tests/	8 years ago
Pierre Chifflier	509a54281f	detect/snmp: convert snmp.community keyword to v2, and MPM	8 years ago
Pierre Chifflier	c67a53773b	detect/snmp: rename keywords to snmp.<name>, and make rust mandatory	8 years ago
Pierre Chifflier	9dfec7e734	SNMP: add the "snmp.pdu_type" detection keyword	8 years ago
Pierre Chifflier	e1dd19a0eb	SNMP: add the "snmp.community" detection keyword	8 years ago
Pierre Chifflier	aa608e0ca2	SNMP: add the "snmp.version" detection keyword	8 years ago
Pierre Chifflier	ed4823bbe8	SNMP: add missing case for profiling enum	8 years ago
Pierre Chifflier	6fc7fc74cb	SNMP: add logger	8 years ago