Switch StreamBufferBlocks implementation to use RBTREE instead of
a list. This makes inserts/removals and lookups a lot cheaper if
the number of data gaps is large.
Use separate compare functions for inserts and regular lookups.
Inserts care about the offset, while lookups care about the blocks
right edge as well.
Add list of 'blocks'. This list contains offsets and lengths to
continuous data blocks. This is useful for TCP tracking where we
can have data gaps.
The blocks don't contain any data themselves, instead they contain
lenght and offsets. This way no extra copying is needed.
On inserting new data, existing blocks are expanded instead of
having multiple neighbouring blocks.
When memory allocations happened in HTTP body and general file
tracking, malloc/realloc errors (most likely in the form of memcap
reached conditions) could lead to an endless loop in the buffer
grow logic.
This patch implements proper error handling for all Append/Insert
functions for the streaming API, and it explicitly enables compiler
warnings if the results are ignored.
Add a new API to store data from streaming sources, like HTTP body
processing or TCP data.
Currently most of the code uses a pattern of list of data chunks
(e.g. TcpSegment) that is reassembled into a large buffer on-demand.
The Streaming Buffer API changes the logic to store the data in
reassembled form from the start, with the segments/chunks pointing
to the reassembled data.
The main buffer storing the data slides forward, automatically or
manually. The *NoTrack calls allows for a segmentless mode of
operation.
This approach has two main advantages:
1. accessing the reassembled data is virtually cost-free
2. reduction of allocations and memory management
Victor Julien