You cannot select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
suricata/src/flow.h

670 lines
23 KiB
C

/* Copyright (C) 2007-2025 Open Information Security Foundation
*
* You can copy, redistribute or modify this Program under the terms of
* the GNU General Public License version 2 as published by the Free
* Software Foundation.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* version 2 along with this program; if not, write to the Free Software
* Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA
* 02110-1301, USA.
*/
/**
* \file
*
* \author Victor Julien <victor@inliniac.net>
*/
#ifndef SURICATA_FLOW_H
#define SURICATA_FLOW_H
/* forward declaration for macset include */
typedef struct FlowStorageId FlowStorageId;
#include "decode.h"
#include "util-time.h"
#include "util-exception-policy.h"
#include "util-exception-policy-types.h"
#include "util-var.h"
#include "util-optimize.h"
#include "util-validate.h"
#include "app-layer-protos.h"
/* Part of the flow structure, so we declare it here.
* The actual declaration is in app-layer-parser.c */
typedef struct AppLayerParserState_ AppLayerParserState;
#define FLOW_QUIET true
#define TOSERVER 0
#define TOCLIENT 1
/* per flow flags */
/** At least one packet from the source address was seen */
#define FLOW_TO_SRC_SEEN BIT_U32(0)
/** At least one packet from the destination address was seen */
#define FLOW_TO_DST_SEEN BIT_U32(1)
/** next packet in toclient direction will act on updated app-layer state */
#define FLOW_TC_APP_UPDATE_NEXT BIT_U32(2)
/** Flow is marked an elephant flow */
#define FLOW_IS_ELEPHANT BIT_U32(3)
firewall: start of firewall rules support Config: Firewall rules are like normal rule, with some key differences. They are loaded separate, and first, from: ```yaml firewall-rule-path: /etc/suricata/firewall/ firewall-rule-files: - fw.rules ``` Can also be loaded with --firewall-rules-exclusive: Mostly for QA purposes. Allow -S with --firewall-rules-exclusive, so that firewall and threat detection rules can be tested together. Rules: Differences with regular "threat detection" rules: 1. these rules are evaluated before threat detection rules 2. these rules are evaluated in the order as they appear in the rule file 3. currently only rules specifying an explicit hook at supported a. as a consequence, no rules will be treated as (like) IP-only, PD-only or DE-only Require explicit action scope for firewall rules. Default policy is drop for the firewall tables. Actions: New action "accept" is added to allow traffic in the filter tables. New scope "accept:tx" is added to allow accepting a transaction. Tables: Rulesets are per table. Table processing order: `packet:filter` -> `packet:td` -> `app:*:*` -> `app:td`. Each of the tables has some unique properties: `packet:filter`: - default policy is `drop:packet` - rules are process in order - action scopes are explicit - `drop` or `accept` is immediate - `accept:hook` continues to `packet:td` `packet:td`: - default policy is `accept:hook` - rules are ordered by IDS/IPS ordering logic - action scopes are implicit - actions are queued - continues to `app:*:*` or `alert/action finalize` `app:*:*`: - default policy is `drop:flow` - rules are process in order - action scopes are explicit - `drop` is immediate - `accept` is conditional on possible `drop` from `packet:td` - `accept:hook` continues to `app:td`, `accept:packet` or `accept:flow` continues to `alert/action finalize` `app:td`: - default policy is `accept:hook` - rules are ordered by IDS/IPS ordering logic - action scopes are implicit - actions are queued - continues to `alert/action finalize` Implementation: During sigorder, split into packet:filter, app:*:* and general td. Allow fw rules to work when in pass:flow mode. When firewall mode is enabled, `pass:flow` will not skip the detection engine anymore, but instead process the firewall rules and then apply the pass before inspecting threat detect rules.
5 months ago
/** All packets in this flow should be accepted */
#define FLOW_ACTION_ACCEPT BIT_U32(4)
// vacancy bit 5
/** Packet payloads belonging to this flow should not be inspected */
#define FLOW_NOPAYLOAD_INSPECTION BIT_U32(6)
/** All packets in this flow should be dropped */
#define FLOW_ACTION_DROP BIT_U32(7)
/** Sgh for toserver direction set (even if it's NULL) */
#define FLOW_SGH_TOSERVER BIT_U32(8)
/** Sgh for toclient direction set (even if it's NULL) */
#define FLOW_SGH_TOCLIENT BIT_U32(9)
/** packet to server direction has been logged in drop file (only in IPS mode) */
#define FLOW_TOSERVER_DROP_LOGGED BIT_U32(10)
/** packet to client direction has been logged in drop file (only in IPS mode) */
#define FLOW_TOCLIENT_DROP_LOGGED BIT_U32(11)
/** flow has alerts */
#define FLOW_HAS_ALERTS BIT_U32(12)
/** Pattern matcher alproto detection done */
#define FLOW_TS_PM_ALPROTO_DETECT_DONE BIT_U32(13)
/** Probing parser alproto detection done */
#define FLOW_TS_PP_ALPROTO_DETECT_DONE BIT_U32(14)
/** Expectation alproto detection done */
#define FLOW_TS_PE_ALPROTO_DETECT_DONE BIT_U32(15)
/** Pattern matcher alproto detection done */
#define FLOW_TC_PM_ALPROTO_DETECT_DONE BIT_U32(16)
/** Probing parser alproto detection done */
#define FLOW_TC_PP_ALPROTO_DETECT_DONE BIT_U32(17)
/** Expectation alproto detection done */
#define FLOW_TC_PE_ALPROTO_DETECT_DONE BIT_U32(18)
#define FLOW_TIMEOUT_REASSEMBLY_DONE BIT_U32(19)
/** flow is ipv4 */
#define FLOW_IPV4 BIT_U32(20)
/** flow is ipv6 */
#define FLOW_IPV6 BIT_U32(21)
#define FLOW_PROTO_DETECT_TS_DONE BIT_U32(22)
#define FLOW_PROTO_DETECT_TC_DONE BIT_U32(23)
/** Indicate that alproto detection for flow should be done again */
#define FLOW_CHANGE_PROTO BIT_U32(24)
#define FLOW_WRONG_THREAD BIT_U32(25)
/** Protocol detection told us flow is picked up in wrong direction (midstream) */
#define FLOW_DIR_REVERSED BIT_U32(26)
/** Indicate that the flow did trigger an expectation creation */
#define FLOW_HAS_EXPECTATION BIT_U32(27)
/** All packets in this flow should be passed */
#define FLOW_ACTION_PASS BIT_U32(28)
#define FLOW_TS_APP_UPDATED BIT_U32(29)
#define FLOW_TC_APP_UPDATED BIT_U32(30)
/** next packet in toserver direction will act on updated app-layer state */
#define FLOW_TS_APP_UPDATE_NEXT BIT_U32(31)
/* File flags */
#define FLOWFILE_INIT 0
/** no magic on files in this flow */
#define FLOWFILE_NO_MAGIC_TS BIT_U16(0)
#define FLOWFILE_NO_MAGIC_TC BIT_U16(1)
/** even if the flow has files, don't store 'm */
#define FLOWFILE_NO_STORE_TS BIT_U16(2)
#define FLOWFILE_NO_STORE_TC BIT_U16(3)
/** no md5 on files in this flow */
#define FLOWFILE_NO_MD5_TS BIT_U16(4)
#define FLOWFILE_NO_MD5_TC BIT_U16(5)
/** no sha1 on files in this flow */
#define FLOWFILE_NO_SHA1_TS BIT_U16(6)
#define FLOWFILE_NO_SHA1_TC BIT_U16(7)
/** no sha256 on files in this flow */
#define FLOWFILE_NO_SHA256_TS BIT_U16(8)
#define FLOWFILE_NO_SHA256_TC BIT_U16(9)
/** no size tracking of files in this flow */
#define FLOWFILE_NO_SIZE_TS BIT_U16(10)
#define FLOWFILE_NO_SIZE_TC BIT_U16(11)
/** store files in the flow */
#define FLOWFILE_STORE_TS BIT_U16(12)
#define FLOWFILE_STORE_TC BIT_U16(13)
#define FLOWFILE_NONE_TS (FLOWFILE_NO_MAGIC_TS | \
FLOWFILE_NO_STORE_TS | \
FLOWFILE_NO_MD5_TS | \
FLOWFILE_NO_SHA1_TS | \
FLOWFILE_NO_SHA256_TS| \
FLOWFILE_NO_SIZE_TS)
#define FLOWFILE_NONE_TC (FLOWFILE_NO_MAGIC_TC | \
FLOWFILE_NO_STORE_TC | \
FLOWFILE_NO_MD5_TC | \
FLOWFILE_NO_SHA1_TC | \
FLOWFILE_NO_SHA256_TC| \
FLOWFILE_NO_SIZE_TC)
#define FLOWFILE_NONE (FLOWFILE_NONE_TS|FLOWFILE_NONE_TC)
#define FLOW_IS_IPV4(f) \
(((f)->flags & FLOW_IPV4) == FLOW_IPV4)
#define FLOW_IS_IPV6(f) \
(((f)->flags & FLOW_IPV6) == FLOW_IPV6)
#define FLOW_GET_SP(f) \
((f)->flags & FLOW_DIR_REVERSED) ? (f)->dp : (f)->sp;
#define FLOW_GET_DP(f) \
((f)->flags & FLOW_DIR_REVERSED) ? (f)->sp : (f)->dp;
#define FLOW_COPY_IPV4_ADDR_TO_PACKET(fa, pa) do { \
(pa)->family = AF_INET; \
(pa)->addr_data32[0] = (fa)->addr_data32[0]; \
} while (0)
#define FLOW_COPY_IPV6_ADDR_TO_PACKET(fa, pa) do { \
(pa)->family = AF_INET6; \
(pa)->addr_data32[0] = (fa)->addr_data32[0]; \
(pa)->addr_data32[1] = (fa)->addr_data32[1]; \
(pa)->addr_data32[2] = (fa)->addr_data32[2]; \
(pa)->addr_data32[3] = (fa)->addr_data32[3]; \
} while (0)
/* Set the IPv4 addressesinto the Addrs of the Packet.
* Make sure p->ip4h is initialized and validated.
*
* We set the rest of the struct to 0 so we can
* prevent using memset. */
#define FLOW_SET_IPV4_SRC_ADDR_FROM_PACKET(ip4h, a) \
do { \
(a)->addr_data32[0] = (uint32_t)(ip4h)->s_ip_src.s_addr; \
(a)->addr_data32[1] = 0; \
(a)->addr_data32[2] = 0; \
(a)->addr_data32[3] = 0; \
} while (0)
#define FLOW_SET_IPV4_DST_ADDR_FROM_PACKET(ip4h, a) \
do { \
(a)->addr_data32[0] = (uint32_t)(ip4h)->s_ip_dst.s_addr; \
(a)->addr_data32[1] = 0; \
(a)->addr_data32[2] = 0; \
(a)->addr_data32[3] = 0; \
} while (0)
/* Set the IPv6 addressesinto the Addrs of the Packet.
* Make sure p->ip6h is initialized and validated. */
#define FLOW_SET_IPV6_SRC_ADDR_FROM_PACKET(ip6h, a) \
do { \
(a)->addr_data32[0] = (ip6h)->s_ip6_src[0]; \
(a)->addr_data32[1] = (ip6h)->s_ip6_src[1]; \
(a)->addr_data32[2] = (ip6h)->s_ip6_src[2]; \
(a)->addr_data32[3] = (ip6h)->s_ip6_src[3]; \
} while (0)
#define FLOW_SET_IPV6_DST_ADDR_FROM_PACKET(ip6h, a) \
do { \
(a)->addr_data32[0] = (ip6h)->s_ip6_dst[0]; \
(a)->addr_data32[1] = (ip6h)->s_ip6_dst[1]; \
(a)->addr_data32[2] = (ip6h)->s_ip6_dst[2]; \
(a)->addr_data32[3] = (ip6h)->s_ip6_dst[3]; \
} while (0)
/* pkt flow flags */
#define FLOW_PKT_TOSERVER 0x01
#define FLOW_PKT_TOCLIENT 0x02
#define FLOW_PKT_ESTABLISHED 0x04
#define FLOW_PKT_TOSERVER_FIRST 0x08
#define FLOW_PKT_TOCLIENT_FIRST 0x10
/** last pseudo packet in the flow. Can be used to trigger final clean,
* logging, etc. */
#define FLOW_PKT_LAST_PSEUDO 0x20
#define FLOW_END_FLAG_EMERGENCY 0x01
#define FLOW_END_FLAG_TIMEOUT 0x02
#define FLOW_END_FLAG_FORCED 0x04
#define FLOW_END_FLAG_SHUTDOWN 0x08
#define FLOW_END_FLAG_TCPREUSE 0x10
/** Mutex or RWLocks for the flow. */
//#define FLOWLOCK_RWLOCK
#define FLOWLOCK_MUTEX
#ifdef FLOWLOCK_RWLOCK
#ifdef FLOWLOCK_MUTEX
#error Cannot enable both FLOWLOCK_RWLOCK and FLOWLOCK_MUTEX
#endif
#endif
#ifdef FLOWLOCK_RWLOCK
#define FLOWLOCK_INIT(fb) SCRWLockInit(&(fb)->r, NULL)
#define FLOWLOCK_DESTROY(fb) SCRWLockDestroy(&(fb)->r)
#define FLOWLOCK_RDLOCK(fb) SCRWLockRDLock(&(fb)->r)
#define FLOWLOCK_WRLOCK(fb) SCRWLockWRLock(&(fb)->r)
#define FLOWLOCK_TRYRDLOCK(fb) SCRWLockTryRDLock(&(fb)->r)
#define FLOWLOCK_TRYWRLOCK(fb) SCRWLockTryWRLock(&(fb)->r)
#define FLOWLOCK_UNLOCK(fb) SCRWLockUnlock(&(fb)->r)
#elif defined FLOWLOCK_MUTEX
#define FLOWLOCK_INIT(fb) SCMutexInit(&(fb)->m, NULL)
#define FLOWLOCK_DESTROY(fb) SCMutexDestroy(&(fb)->m)
#define FLOWLOCK_RDLOCK(fb) SCMutexLock(&(fb)->m)
#define FLOWLOCK_WRLOCK(fb) SCMutexLock(&(fb)->m)
#define FLOWLOCK_TRYRDLOCK(fb) SCMutexTrylock(&(fb)->m)
#define FLOWLOCK_TRYWRLOCK(fb) SCMutexTrylock(&(fb)->m)
#define FLOWLOCK_UNLOCK(fb) SCMutexUnlock(&(fb)->m)
#else
#error Enable FLOWLOCK_RWLOCK or FLOWLOCK_MUTEX
#endif
App layer API rewritten. The main files in question are: app-layer.[ch], app-layer-detect-proto.[ch] and app-layer-parser.[ch]. Things addressed in this commit: - Brings out a proper separation between protocol detection phase and the parser phase. - The dns app layer now is registered such that we don't use "dnstcp" and "dnsudp" in the rules. A user who previously wrote a rule like this - "alert dnstcp....." or "alert dnsudp....." would now have to use, alert dns (ipproto:tcp;) or alert udp (app-layer-protocol:dns;) or alert ip (ipproto:udp; app-layer-protocol:dns;) The same rules extend to other another such protocol, dcerpc. - The app layer parser api now takes in the ipproto while registering callbacks. - The app inspection/detection engine also takes an ipproto. - All app layer parser functions now take direction as STREAM_TOSERVER or STREAM_TOCLIENT, as opposed to 0 or 1, which was taken by some of the functions. - FlowInitialize() and FlowRecycle() now resets proto to 0. This is needed by unittests, which would try to clean the flow, and that would call the api, AppLayerParserCleanupParserState(), which would try to clean the app state, but the app layer now needs an ipproto to figure out which api to internally call to clean the state, and if the ipproto is 0, it would return without trying to clean the state. - A lot of unittests are now updated where if they are using a flow and they need to use the app layer, we would set a flow ipproto. - The "app-layer" section in the yaml conf has also been updated as well.
12 years ago
#define FLOW_IS_PM_DONE(f, dir) (((dir) & STREAM_TOSERVER) ? ((f)->flags & FLOW_TS_PM_ALPROTO_DETECT_DONE) : ((f)->flags & FLOW_TC_PM_ALPROTO_DETECT_DONE))
#define FLOW_IS_PP_DONE(f, dir) (((dir) & STREAM_TOSERVER) ? ((f)->flags & FLOW_TS_PP_ALPROTO_DETECT_DONE) : ((f)->flags & FLOW_TC_PP_ALPROTO_DETECT_DONE))
#define FLOW_IS_PE_DONE(f, dir) (((dir) & STREAM_TOSERVER) ? ((f)->flags & FLOW_TS_PE_ALPROTO_DETECT_DONE) : ((f)->flags & FLOW_TC_PE_ALPROTO_DETECT_DONE))
App layer API rewritten. The main files in question are: app-layer.[ch], app-layer-detect-proto.[ch] and app-layer-parser.[ch]. Things addressed in this commit: - Brings out a proper separation between protocol detection phase and the parser phase. - The dns app layer now is registered such that we don't use "dnstcp" and "dnsudp" in the rules. A user who previously wrote a rule like this - "alert dnstcp....." or "alert dnsudp....." would now have to use, alert dns (ipproto:tcp;) or alert udp (app-layer-protocol:dns;) or alert ip (ipproto:udp; app-layer-protocol:dns;) The same rules extend to other another such protocol, dcerpc. - The app layer parser api now takes in the ipproto while registering callbacks. - The app inspection/detection engine also takes an ipproto. - All app layer parser functions now take direction as STREAM_TOSERVER or STREAM_TOCLIENT, as opposed to 0 or 1, which was taken by some of the functions. - FlowInitialize() and FlowRecycle() now resets proto to 0. This is needed by unittests, which would try to clean the flow, and that would call the api, AppLayerParserCleanupParserState(), which would try to clean the app state, but the app layer now needs an ipproto to figure out which api to internally call to clean the state, and if the ipproto is 0, it would return without trying to clean the state. - A lot of unittests are now updated where if they are using a flow and they need to use the app layer, we would set a flow ipproto. - The "app-layer" section in the yaml conf has also been updated as well.
12 years ago
#define FLOW_SET_PM_DONE(f, dir) (((dir) & STREAM_TOSERVER) ? ((f)->flags |= FLOW_TS_PM_ALPROTO_DETECT_DONE) : ((f)->flags |= FLOW_TC_PM_ALPROTO_DETECT_DONE))
#define FLOW_SET_PP_DONE(f, dir) (((dir) & STREAM_TOSERVER) ? ((f)->flags |= FLOW_TS_PP_ALPROTO_DETECT_DONE) : ((f)->flags |= FLOW_TC_PP_ALPROTO_DETECT_DONE))
#define FLOW_SET_PE_DONE(f, dir) (((dir) & STREAM_TOSERVER) ? ((f)->flags |= FLOW_TS_PE_ALPROTO_DETECT_DONE) : ((f)->flags |= FLOW_TC_PE_ALPROTO_DETECT_DONE))
App layer API rewritten. The main files in question are: app-layer.[ch], app-layer-detect-proto.[ch] and app-layer-parser.[ch]. Things addressed in this commit: - Brings out a proper separation between protocol detection phase and the parser phase. - The dns app layer now is registered such that we don't use "dnstcp" and "dnsudp" in the rules. A user who previously wrote a rule like this - "alert dnstcp....." or "alert dnsudp....." would now have to use, alert dns (ipproto:tcp;) or alert udp (app-layer-protocol:dns;) or alert ip (ipproto:udp; app-layer-protocol:dns;) The same rules extend to other another such protocol, dcerpc. - The app layer parser api now takes in the ipproto while registering callbacks. - The app inspection/detection engine also takes an ipproto. - All app layer parser functions now take direction as STREAM_TOSERVER or STREAM_TOCLIENT, as opposed to 0 or 1, which was taken by some of the functions. - FlowInitialize() and FlowRecycle() now resets proto to 0. This is needed by unittests, which would try to clean the flow, and that would call the api, AppLayerParserCleanupParserState(), which would try to clean the app state, but the app layer now needs an ipproto to figure out which api to internally call to clean the state, and if the ipproto is 0, it would return without trying to clean the state. - A lot of unittests are now updated where if they are using a flow and they need to use the app layer, we would set a flow ipproto. - The "app-layer" section in the yaml conf has also been updated as well.
12 years ago
#define FLOW_RESET_PM_DONE(f, dir) (((dir) & STREAM_TOSERVER) ? ((f)->flags &= ~FLOW_TS_PM_ALPROTO_DETECT_DONE) : ((f)->flags &= ~FLOW_TC_PM_ALPROTO_DETECT_DONE))
#define FLOW_RESET_PP_DONE(f, dir) (((dir) & STREAM_TOSERVER) ? ((f)->flags &= ~FLOW_TS_PP_ALPROTO_DETECT_DONE) : ((f)->flags &= ~FLOW_TC_PP_ALPROTO_DETECT_DONE))
#define FLOW_RESET_PE_DONE(f, dir) (((dir) & STREAM_TOSERVER) ? ((f)->flags &= ~FLOW_TS_PE_ALPROTO_DETECT_DONE) : ((f)->flags &= ~FLOW_TC_PE_ALPROTO_DETECT_DONE))
App layer API rewritten. The main files in question are: app-layer.[ch], app-layer-detect-proto.[ch] and app-layer-parser.[ch]. Things addressed in this commit: - Brings out a proper separation between protocol detection phase and the parser phase. - The dns app layer now is registered such that we don't use "dnstcp" and "dnsudp" in the rules. A user who previously wrote a rule like this - "alert dnstcp....." or "alert dnsudp....." would now have to use, alert dns (ipproto:tcp;) or alert udp (app-layer-protocol:dns;) or alert ip (ipproto:udp; app-layer-protocol:dns;) The same rules extend to other another such protocol, dcerpc. - The app layer parser api now takes in the ipproto while registering callbacks. - The app inspection/detection engine also takes an ipproto. - All app layer parser functions now take direction as STREAM_TOSERVER or STREAM_TOCLIENT, as opposed to 0 or 1, which was taken by some of the functions. - FlowInitialize() and FlowRecycle() now resets proto to 0. This is needed by unittests, which would try to clean the flow, and that would call the api, AppLayerParserCleanupParserState(), which would try to clean the app state, but the app layer now needs an ipproto to figure out which api to internally call to clean the state, and if the ipproto is 0, it would return without trying to clean the state. - A lot of unittests are now updated where if they are using a flow and they need to use the app layer, we would set a flow ipproto. - The "app-layer" section in the yaml conf has also been updated as well.
12 years ago
/* global flow config */
typedef struct FlowCnf_
{
uint32_t hash_rand;
uint32_t hash_size;
uint32_t prealloc;
uint32_t timeout_new;
uint32_t timeout_est;
uint32_t emergency_recovery;
enum ExceptionPolicy memcap_policy;
SC_ATOMIC_DECLARE(uint64_t, memcap);
} FlowConfig;
16 years ago
/* Hash key for the flow hash */
typedef struct FlowKey_
{
Address src, dst;
Port sp, dp;
uint8_t proto;
uint8_t recursion_level;
uint16_t livedev_id;
uint16_t vlan_id[VLAN_MAX_LAYERS];
} FlowKey;
typedef struct FlowAddress_ {
union {
uint32_t address_un_data32[4]; /* type-specific field */
uint16_t address_un_data16[8]; /* type-specific field */
uint8_t address_un_data8[16]; /* type-specific field */
} address;
} FlowAddress;
#define addr_data32 address.address_un_data32
#define addr_data16 address.address_un_data16
#define addr_data8 address.address_un_data8
typedef unsigned short FlowStateType;
/** Local Thread ID */
typedef uint16_t FlowThreadId;
#include "util-storage.h"
/**
* \brief Flow data structure.
*
* The flow is a global data structure that is created for new packets of a
* flow and then looked up for the following packets of a flow.
*
* Locking
*
* The flow is updated/used by multiple packets at the same time. This is why
* there is a flow-mutex. It's a mutex and not a spinlock because some
* operations on the flow can be quite expensive, thus spinning would be
* too expensive.
*
* The flow "header" (addresses, ports, proto, recursion level) are static
* after the initialization and remain read-only throughout the entire live
* of a flow. This is why we can access those without protection of the lock.
*/
typedef struct Flow_
{
/* flow "header", used for hashing and flow lookup. Static after init,
* so safe to look at without lock */
FlowAddress src, dst;
union {
Port sp; /**< tcp/udp source port */
struct {
uint8_t type; /**< icmp type */
uint8_t code; /**< icmp code */
} icmp_s;
struct {
uint32_t spi; /**< esp spi */
} esp;
};
union {
Port dp; /**< tcp/udp destination port */
struct {
uint8_t type; /**< icmp type */
uint8_t code; /**< icmp code */
} icmp_d;
};
uint8_t proto;
uint8_t recursion_level;
uint16_t vlan_id[VLAN_MAX_LAYERS];
flow: redesign of flow timeout handling Goals: - reduce locking - take advantage of 'hot' caches - better locality Locking reduction New flow spare pool. The global pool is implmented as a list of blocks, where each block has a 100 spare flows. Worker threads fetch a block at a time, storing the block in the local thread storage. Flow Recycler now returns flows to the pool is blocks as well. Flow Recycler fetches all flows to be processed in one step instead of one at a time. Cache 'hot'ness Worker threads now check the timeout of flows they evaluate during lookup. The worker will have to read the flow into cache anyway, so the added overhead of checking the timeout value is minimal. When a flow is considered timed out, one of 2 things happens: - if the flow is 'owned' by the thread it is handled locally. Handling means checking if the flow needs 'timeout' work. - otherwise, the flow is added to a special 'evicted' list in the flow bucket where it will be picked up by the flow manager. Flow Manager timing By default the flow manager now tries to do passes of the flow hash in smaller steps, where the goal is to do full pass in 8 x the lowest timeout value it has to enforce. So if the lowest timeout value is 30s, a full pass will take 4 minutes. The goal here is to reduce locking overhead and not get in the way of the workers. In emergency mode each pass is full, and lower timeouts are used. Timing of the flow manager is also no longer relying on pthread condition variables, as these generally cause waking up much quicker than the desired timout. Instead a simple (u)sleep loop is used. Both changes reduce the number of hash passes a lot. Emergency behavior In emergency mode there a number of changes to the workers. In this scenario the flow memcap is fully used up and it is unavoidable that some flows won't be tracked. 1. flow spare pool fetches are reduced to once a second. This avoids locking overhead, while the chance of success was very low. 2. getting an active flow directly from the hash skips flows that had very recent activity to avoid the scenario where all flows get only into the NEW state before getting reused. Rather allow some to have a chance of completing. 3. TCP packets that are not SYN packets will not get a used flow, unless stream.midstream is enabled. The goal here is again to avoid evicting active flows unnecessarily. Better Localily Flow Manager injects flows into the worker threads now, instead of one or two packets. Advantage of this is that the worker threads can get packets from their local packet pools, avoiding constant overhead of packets returning to 'foreign' pools. Counters A lot of flow counters have been added and some have been renamed. Overall the worker threads increment 'flow.wrk.*' counters, while the flow manager increments 'flow.mgr.*'. Additionally, none of the counters are snapshots anymore, they all increment over time. The flow.memuse and flow.spare counters are exceptions. Misc FlowQueue has been split into a FlowQueuePrivate (unlocked) and FlowQueue. Flow no longer has 'prev' pointers and used a unified 'next' pointer for both hash and queue use.
6 years ago
uint8_t vlan_idx;
flow: redesign of flow timeout handling Goals: - reduce locking - take advantage of 'hot' caches - better locality Locking reduction New flow spare pool. The global pool is implmented as a list of blocks, where each block has a 100 spare flows. Worker threads fetch a block at a time, storing the block in the local thread storage. Flow Recycler now returns flows to the pool is blocks as well. Flow Recycler fetches all flows to be processed in one step instead of one at a time. Cache 'hot'ness Worker threads now check the timeout of flows they evaluate during lookup. The worker will have to read the flow into cache anyway, so the added overhead of checking the timeout value is minimal. When a flow is considered timed out, one of 2 things happens: - if the flow is 'owned' by the thread it is handled locally. Handling means checking if the flow needs 'timeout' work. - otherwise, the flow is added to a special 'evicted' list in the flow bucket where it will be picked up by the flow manager. Flow Manager timing By default the flow manager now tries to do passes of the flow hash in smaller steps, where the goal is to do full pass in 8 x the lowest timeout value it has to enforce. So if the lowest timeout value is 30s, a full pass will take 4 minutes. The goal here is to reduce locking overhead and not get in the way of the workers. In emergency mode each pass is full, and lower timeouts are used. Timing of the flow manager is also no longer relying on pthread condition variables, as these generally cause waking up much quicker than the desired timout. Instead a simple (u)sleep loop is used. Both changes reduce the number of hash passes a lot. Emergency behavior In emergency mode there a number of changes to the workers. In this scenario the flow memcap is fully used up and it is unavoidable that some flows won't be tracked. 1. flow spare pool fetches are reduced to once a second. This avoids locking overhead, while the chance of success was very low. 2. getting an active flow directly from the hash skips flows that had very recent activity to avoid the scenario where all flows get only into the NEW state before getting reused. Rather allow some to have a chance of completing. 3. TCP packets that are not SYN packets will not get a used flow, unless stream.midstream is enabled. The goal here is again to avoid evicting active flows unnecessarily. Better Localily Flow Manager injects flows into the worker threads now, instead of one or two packets. Advantage of this is that the worker threads can get packets from their local packet pools, avoiding constant overhead of packets returning to 'foreign' pools. Counters A lot of flow counters have been added and some have been renamed. Overall the worker threads increment 'flow.wrk.*' counters, while the flow manager increments 'flow.mgr.*'. Additionally, none of the counters are snapshots anymore, they all increment over time. The flow.memuse and flow.spare counters are exceptions. Misc FlowQueue has been split into a FlowQueuePrivate (unlocked) and FlowQueue. Flow no longer has 'prev' pointers and used a unified 'next' pointer for both hash and queue use.
6 years ago
/* track toserver/toclient flow timeout needs */
union {
struct {
uint8_t ffr_ts:4;
uint8_t ffr_tc:4;
};
uint8_t ffr;
};
/** Thread ID for the stream/detect portion of this flow */
FlowThreadId thread_id[2];
struct Flow_ *next; /* (hash) list next */
/** Incoming interface */
struct LiveDevice_ *livedev;
/** flow hash - the flow hash before hash table size mod. */
uint32_t flow_hash;
/** timeout in seconds by policy, add to Flow::lastts to get actual time this times out.
* Ignored in emergency mode. */
uint32_t timeout_policy;
/* time stamp of last update (last packet). Set/updated under the
* flow and flow hash row locks, safe to read under either the
* flow lock or flow hash row lock. */
SCTime_t lastts;
flow: redesign of flow timeout handling Goals: - reduce locking - take advantage of 'hot' caches - better locality Locking reduction New flow spare pool. The global pool is implmented as a list of blocks, where each block has a 100 spare flows. Worker threads fetch a block at a time, storing the block in the local thread storage. Flow Recycler now returns flows to the pool is blocks as well. Flow Recycler fetches all flows to be processed in one step instead of one at a time. Cache 'hot'ness Worker threads now check the timeout of flows they evaluate during lookup. The worker will have to read the flow into cache anyway, so the added overhead of checking the timeout value is minimal. When a flow is considered timed out, one of 2 things happens: - if the flow is 'owned' by the thread it is handled locally. Handling means checking if the flow needs 'timeout' work. - otherwise, the flow is added to a special 'evicted' list in the flow bucket where it will be picked up by the flow manager. Flow Manager timing By default the flow manager now tries to do passes of the flow hash in smaller steps, where the goal is to do full pass in 8 x the lowest timeout value it has to enforce. So if the lowest timeout value is 30s, a full pass will take 4 minutes. The goal here is to reduce locking overhead and not get in the way of the workers. In emergency mode each pass is full, and lower timeouts are used. Timing of the flow manager is also no longer relying on pthread condition variables, as these generally cause waking up much quicker than the desired timout. Instead a simple (u)sleep loop is used. Both changes reduce the number of hash passes a lot. Emergency behavior In emergency mode there a number of changes to the workers. In this scenario the flow memcap is fully used up and it is unavoidable that some flows won't be tracked. 1. flow spare pool fetches are reduced to once a second. This avoids locking overhead, while the chance of success was very low. 2. getting an active flow directly from the hash skips flows that had very recent activity to avoid the scenario where all flows get only into the NEW state before getting reused. Rather allow some to have a chance of completing. 3. TCP packets that are not SYN packets will not get a used flow, unless stream.midstream is enabled. The goal here is again to avoid evicting active flows unnecessarily. Better Localily Flow Manager injects flows into the worker threads now, instead of one or two packets. Advantage of this is that the worker threads can get packets from their local packet pools, avoiding constant overhead of packets returning to 'foreign' pools. Counters A lot of flow counters have been added and some have been renamed. Overall the worker threads increment 'flow.wrk.*' counters, while the flow manager increments 'flow.mgr.*'. Additionally, none of the counters are snapshots anymore, they all increment over time. The flow.memuse and flow.spare counters are exceptions. Misc FlowQueue has been split into a FlowQueuePrivate (unlocked) and FlowQueue. Flow no longer has 'prev' pointers and used a unified 'next' pointer for both hash and queue use.
6 years ago
FlowStateType flow_state;
/** flow tenant id, used to setup flow timeout and stream pseudo
* packets with the correct tenant id set */
uint32_t tenant_id;
App layer API rewritten. The main files in question are: app-layer.[ch], app-layer-detect-proto.[ch] and app-layer-parser.[ch]. Things addressed in this commit: - Brings out a proper separation between protocol detection phase and the parser phase. - The dns app layer now is registered such that we don't use "dnstcp" and "dnsudp" in the rules. A user who previously wrote a rule like this - "alert dnstcp....." or "alert dnsudp....." would now have to use, alert dns (ipproto:tcp;) or alert udp (app-layer-protocol:dns;) or alert ip (ipproto:udp; app-layer-protocol:dns;) The same rules extend to other another such protocol, dcerpc. - The app layer parser api now takes in the ipproto while registering callbacks. - The app inspection/detection engine also takes an ipproto. - All app layer parser functions now take direction as STREAM_TOSERVER or STREAM_TOCLIENT, as opposed to 0 or 1, which was taken by some of the functions. - FlowInitialize() and FlowRecycle() now resets proto to 0. This is needed by unittests, which would try to clean the flow, and that would call the api, AppLayerParserCleanupParserState(), which would try to clean the app state, but the app layer now needs an ipproto to figure out which api to internally call to clean the state, and if the ipproto is 0, it would return without trying to clean the state. - A lot of unittests are now updated where if they are using a flow and they need to use the app layer, we would set a flow ipproto. - The "app-layer" section in the yaml conf has also been updated as well.
12 years ago
uint32_t probing_parser_toserver_alproto_masks;
uint32_t probing_parser_toclient_alproto_masks;
uint32_t flags; /**< generic flags */
uint16_t file_flags; /**< file tracking/extraction flags */
/** destination port to be used in protocol detection. This is meant
* for use with STARTTLS and HTTP CONNECT detection */
uint16_t protodetect_dp; /**< 0 if not used */
/* Parent flow id for protocol like ftp */
int64_t parent_id;
#ifdef FLOWLOCK_RWLOCK
SCRWLock r;
#elif defined FLOWLOCK_MUTEX
SCMutex m;
#else
#error Enable FLOWLOCK_RWLOCK or FLOWLOCK_MUTEX
#endif
/** protocol specific data pointer, e.g. for TcpSession */
void *protoctx;
/** mapping to Flow's protocol specific protocols for timeouts
and state and free functions. */
uint8_t protomap;
uint8_t flow_end_flags;
/* coccinelle: Flow:flow_end_flags:FLOW_END_FLAG_ */
AppProto alproto; /**< \brief application level protocol */
AppProto alproto_ts;
AppProto alproto_tc;
/** original application level protocol. Used to indicate the previous
protocol when changing to another protocol , e.g. with STARTTLS. */
AppProto alproto_orig;
/** expected app protocol: used in protocol change/upgrade like in
* STARTTLS. */
AppProto alproto_expect;
/** detection engine ctx version used to inspect this flow. Set at initial
* inspection. If it doesn't match the currently in use de_ctx, the
* stored sgh ptrs are reset. */
uint32_t de_ctx_version;
/** ttl tracking */
uint8_t min_ttl_toserver;
uint8_t max_ttl_toserver;
uint8_t min_ttl_toclient;
uint8_t max_ttl_toclient;
/** which exception policies were applied, if any */
uint8_t applied_exception_policy;
/** application level storage ptrs.
*
*/
AppLayerParserState *alparser; /**< parser internal state */
void *alstate; /**< application layer state */
/** toclient sgh for this flow. Only use when FLOW_SGH_TOCLIENT flow flag
* has been set. */
const struct SigGroupHead_ *sgh_toclient;
/** toserver sgh for this flow. Only use when FLOW_SGH_TOSERVER flow flag
* has been set. */
const struct SigGroupHead_ *sgh_toserver;
/* pointer to the var list */
GenericVar *flowvar;
struct FlowBucket_ *fb;
SCTime_t startts;
uint32_t todstpktcnt;
uint32_t tosrcpktcnt;
uint64_t todstbytecnt;
uint64_t tosrcbytecnt;
Storage storage[];
} Flow;
flow-manager: optimize hash walking Until now the flow manager would walk the entire flow hash table on an interval. It would thus touch all flows, leading to a lot of memory and cache pressure. In scenario's where the number of tracked flows run into the hundreds on thousands, and the memory used can run into many hundreds of megabytes or even gigabytes, this would lead to serious performance degradation. This patch introduces a new approach. A timestamp per flow bucket (hash row) is maintained by the flow manager. It holds the timestamp of the earliest possible timeout of a flow in the list. The hash walk skips rows with timestamps beyond the current time. As the timestamp depends on the flows in the hash row's list, and on the 'state' of each flow in the list, any addition of a flow or changing of a flow's state invalidates the timestamp. The flow manager then has to walk the list again to set a new timestamp. A utility function FlowUpdateState is introduced to change Flow states, taking care of the bucket timestamp invalidation while at it. Empty flow buckets use a special value so that we don't have to take the flow bucket lock to find out the bucket is empty. This patch also adds more performance counters: flow_mgr.flows_checked | Total | 929 flow_mgr.flows_notimeout | Total | 391 flow_mgr.flows_timeout | Total | 538 flow_mgr.flows_removed | Total | 277 flow_mgr.flows_timeout_inuse | Total | 261 flow_mgr.rows_checked | Total | 1000000 flow_mgr.rows_skipped | Total | 998835 flow_mgr.rows_empty | Total | 290 flow_mgr.rows_maxlen | Total | 2 flow_mgr.flows_checked: number of flows checked for timeout in the last pass flow_mgr.flows_notimeout: number of flows out of flow_mgr.flows_checked that didn't time out flow_mgr.flows_timeout: number of out of flow_mgr.flows_checked that did reach the time out flow_mgr.flows_removed: number of flows out of flow_mgr.flows_timeout that were really removed flow_mgr.flows_timeout_inuse: number of flows out of flow_mgr.flows_timeout that were still in use or needed work flow_mgr.rows_checked: hash table rows checked flow_mgr.rows_skipped: hash table rows skipped because non of the flows would time out anyway The counters below are only relating to rows that were not skipped. flow_mgr.rows_empty: empty hash rows flow_mgr.rows_maxlen: max number of flows per hash row. Best to keep low, so increase hash-size if needed. flow_mgr.rows_busy: row skipped because it was locked by another thread
9 years ago
enum FlowState {
FLOW_STATE_NEW = 0,
FLOW_STATE_ESTABLISHED,
FLOW_STATE_CLOSED,
FLOW_STATE_LOCAL_BYPASSED,
#ifdef CAPTURE_OFFLOAD
FLOW_STATE_CAPTURE_BYPASSED,
#endif
};
#ifdef CAPTURE_OFFLOAD
#define FLOW_STATE_SIZE 5
#else
#define FLOW_STATE_SIZE 4
#endif
typedef struct FlowProtoTimeout_ {
uint32_t new_timeout;
uint32_t est_timeout;
uint32_t closed_timeout;
uint32_t bypassed_timeout;
} FlowProtoTimeout;
typedef struct FlowProtoFreeFunc_ {
void (*Freefunc)(void *);
} FlowProtoFreeFunc;
typedef struct FlowBypassInfo_ {
bool (* BypassUpdate)(Flow *f, void *data, time_t tsec);
void (* BypassFree)(void *data);
void *bypass_data;
uint64_t tosrcpktcnt;
uint64_t tosrcbytecnt;
uint64_t todstpktcnt;
uint64_t todstbytecnt;
} FlowBypassInfo;
flow: redesign of flow timeout handling Goals: - reduce locking - take advantage of 'hot' caches - better locality Locking reduction New flow spare pool. The global pool is implmented as a list of blocks, where each block has a 100 spare flows. Worker threads fetch a block at a time, storing the block in the local thread storage. Flow Recycler now returns flows to the pool is blocks as well. Flow Recycler fetches all flows to be processed in one step instead of one at a time. Cache 'hot'ness Worker threads now check the timeout of flows they evaluate during lookup. The worker will have to read the flow into cache anyway, so the added overhead of checking the timeout value is minimal. When a flow is considered timed out, one of 2 things happens: - if the flow is 'owned' by the thread it is handled locally. Handling means checking if the flow needs 'timeout' work. - otherwise, the flow is added to a special 'evicted' list in the flow bucket where it will be picked up by the flow manager. Flow Manager timing By default the flow manager now tries to do passes of the flow hash in smaller steps, where the goal is to do full pass in 8 x the lowest timeout value it has to enforce. So if the lowest timeout value is 30s, a full pass will take 4 minutes. The goal here is to reduce locking overhead and not get in the way of the workers. In emergency mode each pass is full, and lower timeouts are used. Timing of the flow manager is also no longer relying on pthread condition variables, as these generally cause waking up much quicker than the desired timout. Instead a simple (u)sleep loop is used. Both changes reduce the number of hash passes a lot. Emergency behavior In emergency mode there a number of changes to the workers. In this scenario the flow memcap is fully used up and it is unavoidable that some flows won't be tracked. 1. flow spare pool fetches are reduced to once a second. This avoids locking overhead, while the chance of success was very low. 2. getting an active flow directly from the hash skips flows that had very recent activity to avoid the scenario where all flows get only into the NEW state before getting reused. Rather allow some to have a chance of completing. 3. TCP packets that are not SYN packets will not get a used flow, unless stream.midstream is enabled. The goal here is again to avoid evicting active flows unnecessarily. Better Localily Flow Manager injects flows into the worker threads now, instead of one or two packets. Advantage of this is that the worker threads can get packets from their local packet pools, avoiding constant overhead of packets returning to 'foreign' pools. Counters A lot of flow counters have been added and some have been renamed. Overall the worker threads increment 'flow.wrk.*' counters, while the flow manager increments 'flow.mgr.*'. Additionally, none of the counters are snapshots anymore, they all increment over time. The flow.memuse and flow.spare counters are exceptions. Misc FlowQueue has been split into a FlowQueuePrivate (unlocked) and FlowQueue. Flow no longer has 'prev' pointers and used a unified 'next' pointer for both hash and queue use.
6 years ago
#include "flow-queue.h"
typedef struct FlowLookupStruct_ // TODO name
{
/** thread store of spare queues */
FlowQueuePrivate spare_queue;
DecodeThreadVars *dtv;
FlowQueuePrivate work_queue;
uint32_t emerg_spare_sync_stamp;
} FlowLookupStruct;
/** \brief prepare packet for a life with flow
2 years ago
* Set PKT_WANTS_FLOW flag to indicate workers should do a flow lookup
* and calc the hash value to be used in the lookup and autofp flow
* balancing. */
void FlowSetupPacket(Packet *p);
flow: redesign of flow timeout handling Goals: - reduce locking - take advantage of 'hot' caches - better locality Locking reduction New flow spare pool. The global pool is implmented as a list of blocks, where each block has a 100 spare flows. Worker threads fetch a block at a time, storing the block in the local thread storage. Flow Recycler now returns flows to the pool is blocks as well. Flow Recycler fetches all flows to be processed in one step instead of one at a time. Cache 'hot'ness Worker threads now check the timeout of flows they evaluate during lookup. The worker will have to read the flow into cache anyway, so the added overhead of checking the timeout value is minimal. When a flow is considered timed out, one of 2 things happens: - if the flow is 'owned' by the thread it is handled locally. Handling means checking if the flow needs 'timeout' work. - otherwise, the flow is added to a special 'evicted' list in the flow bucket where it will be picked up by the flow manager. Flow Manager timing By default the flow manager now tries to do passes of the flow hash in smaller steps, where the goal is to do full pass in 8 x the lowest timeout value it has to enforce. So if the lowest timeout value is 30s, a full pass will take 4 minutes. The goal here is to reduce locking overhead and not get in the way of the workers. In emergency mode each pass is full, and lower timeouts are used. Timing of the flow manager is also no longer relying on pthread condition variables, as these generally cause waking up much quicker than the desired timout. Instead a simple (u)sleep loop is used. Both changes reduce the number of hash passes a lot. Emergency behavior In emergency mode there a number of changes to the workers. In this scenario the flow memcap is fully used up and it is unavoidable that some flows won't be tracked. 1. flow spare pool fetches are reduced to once a second. This avoids locking overhead, while the chance of success was very low. 2. getting an active flow directly from the hash skips flows that had very recent activity to avoid the scenario where all flows get only into the NEW state before getting reused. Rather allow some to have a chance of completing. 3. TCP packets that are not SYN packets will not get a used flow, unless stream.midstream is enabled. The goal here is again to avoid evicting active flows unnecessarily. Better Localily Flow Manager injects flows into the worker threads now, instead of one or two packets. Advantage of this is that the worker threads can get packets from their local packet pools, avoiding constant overhead of packets returning to 'foreign' pools. Counters A lot of flow counters have been added and some have been renamed. Overall the worker threads increment 'flow.wrk.*' counters, while the flow manager increments 'flow.mgr.*'. Additionally, none of the counters are snapshots anymore, they all increment over time. The flow.memuse and flow.spare counters are exceptions. Misc FlowQueue has been split into a FlowQueuePrivate (unlocked) and FlowQueue. Flow no longer has 'prev' pointers and used a unified 'next' pointer for both hash and queue use.
6 years ago
void FlowHandlePacket (ThreadVars *, FlowLookupStruct *, Packet *);
void FlowInitConfig(bool);
void FlowReset(void);
void FlowShutdown(void);
void FlowSetHasAlertsFlag(Flow *);
int FlowHasAlerts(const Flow *);
void FlowSetChangeProtoFlag(Flow *);
void FlowUnsetChangeProtoFlag(Flow *);
int FlowChangeProto(Flow *);
void FlowSwap(Flow *);
void FlowRegisterTests(void);
int FlowSetProtoFreeFunc(uint8_t, void (*Free)(void *));
static inline void FlowSetNoPayloadInspectionFlag(Flow *);
int FlowGetPacketDirection(const Flow *, const Packet *);
void FlowCleanupAppLayer(Flow *);
flow-manager: optimize hash walking Until now the flow manager would walk the entire flow hash table on an interval. It would thus touch all flows, leading to a lot of memory and cache pressure. In scenario's where the number of tracked flows run into the hundreds on thousands, and the memory used can run into many hundreds of megabytes or even gigabytes, this would lead to serious performance degradation. This patch introduces a new approach. A timestamp per flow bucket (hash row) is maintained by the flow manager. It holds the timestamp of the earliest possible timeout of a flow in the list. The hash walk skips rows with timestamps beyond the current time. As the timestamp depends on the flows in the hash row's list, and on the 'state' of each flow in the list, any addition of a flow or changing of a flow's state invalidates the timestamp. The flow manager then has to walk the list again to set a new timestamp. A utility function FlowUpdateState is introduced to change Flow states, taking care of the bucket timestamp invalidation while at it. Empty flow buckets use a special value so that we don't have to take the flow bucket lock to find out the bucket is empty. This patch also adds more performance counters: flow_mgr.flows_checked | Total | 929 flow_mgr.flows_notimeout | Total | 391 flow_mgr.flows_timeout | Total | 538 flow_mgr.flows_removed | Total | 277 flow_mgr.flows_timeout_inuse | Total | 261 flow_mgr.rows_checked | Total | 1000000 flow_mgr.rows_skipped | Total | 998835 flow_mgr.rows_empty | Total | 290 flow_mgr.rows_maxlen | Total | 2 flow_mgr.flows_checked: number of flows checked for timeout in the last pass flow_mgr.flows_notimeout: number of flows out of flow_mgr.flows_checked that didn't time out flow_mgr.flows_timeout: number of out of flow_mgr.flows_checked that did reach the time out flow_mgr.flows_removed: number of flows out of flow_mgr.flows_timeout that were really removed flow_mgr.flows_timeout_inuse: number of flows out of flow_mgr.flows_timeout that were still in use or needed work flow_mgr.rows_checked: hash table rows checked flow_mgr.rows_skipped: hash table rows skipped because non of the flows would time out anyway The counters below are only relating to rows that were not skipped. flow_mgr.rows_empty: empty hash rows flow_mgr.rows_maxlen: max number of flows per hash row. Best to keep low, so increase hash-size if needed. flow_mgr.rows_busy: row skipped because it was locked by another thread
9 years ago
void FlowUpdateState(Flow *f, enum FlowState s);
int FlowSetMemcap(uint64_t size);
uint64_t FlowGetMemcap(void);
uint64_t FlowGetMemuse(void);
enum ExceptionPolicy FlowGetMemcapExceptionPolicy(void);
FlowStorageId GetFlowBypassInfoID(void);
void RegisterFlowBypassInfo(void);
void FlowGetLastTimeAsParts(Flow *flow, uint64_t *secs, uint64_t *usecs);
uint32_t FlowGetFlags(Flow *flow);
uint16_t FlowGetSourcePort(Flow *flow);
uint16_t FlowGetDestinationPort(Flow *flow);
flow-manager: optimize hash walking Until now the flow manager would walk the entire flow hash table on an interval. It would thus touch all flows, leading to a lot of memory and cache pressure. In scenario's where the number of tracked flows run into the hundreds on thousands, and the memory used can run into many hundreds of megabytes or even gigabytes, this would lead to serious performance degradation. This patch introduces a new approach. A timestamp per flow bucket (hash row) is maintained by the flow manager. It holds the timestamp of the earliest possible timeout of a flow in the list. The hash walk skips rows with timestamps beyond the current time. As the timestamp depends on the flows in the hash row's list, and on the 'state' of each flow in the list, any addition of a flow or changing of a flow's state invalidates the timestamp. The flow manager then has to walk the list again to set a new timestamp. A utility function FlowUpdateState is introduced to change Flow states, taking care of the bucket timestamp invalidation while at it. Empty flow buckets use a special value so that we don't have to take the flow bucket lock to find out the bucket is empty. This patch also adds more performance counters: flow_mgr.flows_checked | Total | 929 flow_mgr.flows_notimeout | Total | 391 flow_mgr.flows_timeout | Total | 538 flow_mgr.flows_removed | Total | 277 flow_mgr.flows_timeout_inuse | Total | 261 flow_mgr.rows_checked | Total | 1000000 flow_mgr.rows_skipped | Total | 998835 flow_mgr.rows_empty | Total | 290 flow_mgr.rows_maxlen | Total | 2 flow_mgr.flows_checked: number of flows checked for timeout in the last pass flow_mgr.flows_notimeout: number of flows out of flow_mgr.flows_checked that didn't time out flow_mgr.flows_timeout: number of out of flow_mgr.flows_checked that did reach the time out flow_mgr.flows_removed: number of flows out of flow_mgr.flows_timeout that were really removed flow_mgr.flows_timeout_inuse: number of flows out of flow_mgr.flows_timeout that were still in use or needed work flow_mgr.rows_checked: hash table rows checked flow_mgr.rows_skipped: hash table rows skipped because non of the flows would time out anyway The counters below are only relating to rows that were not skipped. flow_mgr.rows_empty: empty hash rows flow_mgr.rows_maxlen: max number of flows per hash row. Best to keep low, so increase hash-size if needed. flow_mgr.rows_busy: row skipped because it was locked by another thread
9 years ago
/** ----- Inline functions ----- */
static inline AppProto FlowGetAppProtocol(const Flow *f)
{
return f->alproto;
}
static inline void *FlowGetAppState(const Flow *f)
{
return f->alstate;
}
/** \brief Set the No payload inspection Flag without locking the flow.
*
* \param f Flow to set the flag in
*/
static inline void FlowSetNoPayloadInspectionFlag(Flow *f)
{
SCEnter();
SCLogDebug("flow %p", f);
f->flags |= FLOW_NOPAYLOAD_INSPECTION;
SCReturn;
}
/** \brief Reference the flow
* \note This should only be called once for a destination
* pointer */
static inline void FlowReference(Flow **d, Flow *f)
{
if (likely(f != NULL)) {
DEBUG_VALIDATE_BUG_ON(*d == f);
if (*d == f)
return;
*d = f;
}
}
static inline void FlowDeReference(Flow **d)
{
if (likely(*d != NULL)) {
*d = NULL;
}
}
/** \brief create a flow id that is as unique as possible
* \retval flow_id unsigned 64bit id
*/
static inline uint64_t FlowGetId(const Flow *f)
{
uint64_t id = (uint64_t)(SCTIME_SECS(f->startts) & 0xFFFF) << 48 |
(uint64_t)(SCTIME_USECS(f->startts) & 0xFFFF) << 32 | (uint64_t)f->flow_hash;
2 years ago
/* reduce to 51 bits as JavaScript and even JSON often seem to
* max out there. */
id &= 0x7ffffffffffffLL;
return id;
}
static inline bool FlowIsBypassed(const Flow *f)
{
if (
#ifdef CAPTURE_OFFLOAD
f->flow_state == FLOW_STATE_CAPTURE_BYPASSED ||
#endif
f->flow_state == FLOW_STATE_LOCAL_BYPASSED) {
return true;
}
return false;
}
int FlowClearMemory(Flow *,uint8_t );
AppProto FlowGetAppProtocol(const Flow *f);
void *FlowGetAppState(const Flow *f);
uint8_t FlowGetDisruptionFlags(const Flow *f, uint8_t flags);
App layer API rewritten. The main files in question are: app-layer.[ch], app-layer-detect-proto.[ch] and app-layer-parser.[ch]. Things addressed in this commit: - Brings out a proper separation between protocol detection phase and the parser phase. - The dns app layer now is registered such that we don't use "dnstcp" and "dnsudp" in the rules. A user who previously wrote a rule like this - "alert dnstcp....." or "alert dnsudp....." would now have to use, alert dns (ipproto:tcp;) or alert udp (app-layer-protocol:dns;) or alert ip (ipproto:udp; app-layer-protocol:dns;) The same rules extend to other another such protocol, dcerpc. - The app layer parser api now takes in the ipproto while registering callbacks. - The app inspection/detection engine also takes an ipproto. - All app layer parser functions now take direction as STREAM_TOSERVER or STREAM_TOCLIENT, as opposed to 0 or 1, which was taken by some of the functions. - FlowInitialize() and FlowRecycle() now resets proto to 0. This is needed by unittests, which would try to clean the flow, and that would call the api, AppLayerParserCleanupParserState(), which would try to clean the app state, but the app layer now needs an ipproto to figure out which api to internally call to clean the state, and if the ipproto is 0, it would return without trying to clean the state. - A lot of unittests are now updated where if they are using a flow and they need to use the app layer, we would set a flow ipproto. - The "app-layer" section in the yaml conf has also been updated as well.
12 years ago
void FlowHandlePacketUpdate(Flow *f, Packet *p, ThreadVars *tv, DecodeThreadVars *dtv);
App layer API rewritten. The main files in question are: app-layer.[ch], app-layer-detect-proto.[ch] and app-layer-parser.[ch]. Things addressed in this commit: - Brings out a proper separation between protocol detection phase and the parser phase. - The dns app layer now is registered such that we don't use "dnstcp" and "dnsudp" in the rules. A user who previously wrote a rule like this - "alert dnstcp....." or "alert dnsudp....." would now have to use, alert dns (ipproto:tcp;) or alert udp (app-layer-protocol:dns;) or alert ip (ipproto:udp; app-layer-protocol:dns;) The same rules extend to other another such protocol, dcerpc. - The app layer parser api now takes in the ipproto while registering callbacks. - The app inspection/detection engine also takes an ipproto. - All app layer parser functions now take direction as STREAM_TOSERVER or STREAM_TOCLIENT, as opposed to 0 or 1, which was taken by some of the functions. - FlowInitialize() and FlowRecycle() now resets proto to 0. This is needed by unittests, which would try to clean the flow, and that would call the api, AppLayerParserCleanupParserState(), which would try to clean the app state, but the app layer now needs an ipproto to figure out which api to internally call to clean the state, and if the ipproto is 0, it would return without trying to clean the state. - A lot of unittests are now updated where if they are using a flow and they need to use the app layer, we would set a flow ipproto. - The "app-layer" section in the yaml conf has also been updated as well.
12 years ago
#endif /* SURICATA_FLOW_H */