ftp: do not create more than max-tx transactions

Ticket: 8592

Fixes: 5ddd808e9b ("ftp: don't halt the flow when raising
too_many_transactions")

In the case we receive a big chunk of TCP data, we end up
creating much more than max-tx transactions, and have
quadratic complexity on this packet, even if all
these transactions get cleaned up at the end of the processing
of this packet.

doc/userguide/configuration/suricata-yaml.rst

@@ -2037,6 +2037,10 @@ An app-layer event `protocol.too_many_transactions` is triggered when this value
 The point of this parameter is to find a balance between the completeness of analysis
 and the resource consumption.
 
+When this threshold is reached, a new transaction will not be allocated,
+and the flow will be put in error state, as it would become too expensive
+in terms of CPU for Suricata to continue processing it.
+
 For HTTP2, this parameter is named `max-streams` as an HTTP2 stream will get translated
 into one Suricata transaction. This configuration parameter is used whatever the
 value of `SETTINGS_MAX_CONCURRENT_STREAMS` negotiated between a client and a server

src/app-layer-ftp.c

@@ -234,6 +234,7 @@ static FTPTransaction *FTPTransactionCreate(FtpState *state)
                 break;
             }
         }
+        return NULL;
     }
     FTPTransaction *tx = FTPCalloc(1, sizeof(*tx));
     if (tx == NULL) {