20 Commits (6885b668832d884ebbbedfb036b260259dfe5e74)

Author	SHA1	Message	Date
Victor Julien	b3599507f4	flow: redesign of flow timeout handling ... Goals: - reduce locking - take advantage of 'hot' caches - better locality Locking reduction New flow spare pool. The global pool is implmented as a list of blocks, where each block has a 100 spare flows. Worker threads fetch a block at a time, storing the block in the local thread storage. Flow Recycler now returns flows to the pool is blocks as well. Flow Recycler fetches all flows to be processed in one step instead of one at a time. Cache 'hot'ness Worker threads now check the timeout of flows they evaluate during lookup. The worker will have to read the flow into cache anyway, so the added overhead of checking the timeout value is minimal. When a flow is considered timed out, one of 2 things happens: - if the flow is 'owned' by the thread it is handled locally. Handling means checking if the flow needs 'timeout' work. - otherwise, the flow is added to a special 'evicted' list in the flow bucket where it will be picked up by the flow manager. Flow Manager timing By default the flow manager now tries to do passes of the flow hash in smaller steps, where the goal is to do full pass in 8 x the lowest timeout value it has to enforce. So if the lowest timeout value is 30s, a full pass will take 4 minutes. The goal here is to reduce locking overhead and not get in the way of the workers. In emergency mode each pass is full, and lower timeouts are used. Timing of the flow manager is also no longer relying on pthread condition variables, as these generally cause waking up much quicker than the desired timout. Instead a simple (u)sleep loop is used. Both changes reduce the number of hash passes a lot. Emergency behavior In emergency mode there a number of changes to the workers. In this scenario the flow memcap is fully used up and it is unavoidable that some flows won't be tracked. 1. flow spare pool fetches are reduced to once a second. This avoids locking overhead, while the chance of success was very low. 2. getting an active flow directly from the hash skips flows that had very recent activity to avoid the scenario where all flows get only into the NEW state before getting reused. Rather allow some to have a chance of completing. 3. TCP packets that are not SYN packets will not get a used flow, unless stream.midstream is enabled. The goal here is again to avoid evicting active flows unnecessarily. Better Localily Flow Manager injects flows into the worker threads now, instead of one or two packets. Advantage of this is that the worker threads can get packets from their local packet pools, avoiding constant overhead of packets returning to 'foreign' pools. Counters A lot of flow counters have been added and some have been renamed. Overall the worker threads increment 'flow.wrk.*' counters, while the flow manager increments 'flow.mgr.*'. Additionally, none of the counters are snapshots anymore, they all increment over time. The flow.memuse and flow.spare counters are exceptions. Misc FlowQueue has been split into a FlowQueuePrivate (unlocked) and FlowQueue. Flow no longer has 'prev' pointers and used a unified 'next' pointer for both hash and queue use.	5 years ago
Victor Julien	ab1200fbd7	compiler: more strict compiler warnings ... Set flags by default: -Wmissing-prototypes -Wmissing-declarations -Wstrict-prototypes -Wwrite-strings -Wcast-align -Wbad-function-cast -Wformat-security -Wno-format-nonliteral -Wmissing-format-attribute -funsigned-char Fix minor compiler warnings for these new flags on gcc and clang.	8 years ago
Victor Julien	db24258acf	Undo changes from 88b8f15663. Atomic stack implementation had a-b-a problem.	14 years ago
Victor Julien	88b8f15663	Add atomic stack implementation. Convert flow spare queue to use this stack. Remove now unused flow-queue code.	14 years ago
Victor Julien	0150e66ede	flow engine: improve scalability ... Major redesign of the flow engine. Remove the flow queues that turned out to be major choke points when using many threads. Flow manager now walks the hash table directly. Simplify the way we get a new flow in case of emergency.	14 years ago
Victor Julien	fbbdbb251f	flow engine: remove unneeded 'need_srclock' argument for FlowRequeue	14 years ago
Victor Julien	0331da9773	flow engine: introduce FlowRequeueMoveToSpare ... As part of a clean up of how FlowRequeue is used, introduce FlowRequeueMoveToSpare for moving a flow from a locked queue to the spare queue.	14 years ago
Victor Julien	7fa3df33f2	flow engine: introduce FlowRequeueMoveToBot ... As part of a clean up of how FlowRequeue is used, introduce FlowRequeueMoveToBot for moving a flow to the bottom of it's queue.	14 years ago
Victor Julien	fc248ca7a1	Many small performance updates.	15 years ago
Pablo Rincon	29a6fc2f03	Adding some flow improvements and recovery on emergency mode	15 years ago
William Metcalf	2eef905c07	GPL and Copyright header updates.	15 years ago
William Metcalf	ce01927515	Import of GPLv2 Header 050410	16 years ago
Victor Julien	ae2698c9cc	Properly cleanup used mutexes and condition vars in the flow subsys.	16 years ago
Victor Julien	ecf86f9c23	Rename to Suricata.	16 years ago
Pablo Rincon	e26833be3f	Changing mutex/spinlocks/conditions naming types	16 years ago
Pablo Rincon	769022f4be	Adding support for Mac OS X, FreeBSD, centrailizing mutex/spins/conditions in a macro API, and some unittests	16 years ago
Brian Rectanus	fa5939ca91	64 bit cleanup part2	16 years ago
Victor Julien	689bbfdc45	Rename all structure definitions in the "typedef struct _SomeStruct" format to "typedef struct SomeStruct_" to make the Doxygen output more useful. ... Remove the Trie multi pattern matcher code. It wasn't used anymore.	16 years ago
Victor Julien	8397413942	Comment updates.	16 years ago
Victor Julien	bab4b62376	Initial add of the files.	16 years ago