app-layer: fix inverted tx inspected flag check

Issue: 8628

AppLayerParserSetTransactionInspectId() had an inverted guard when
tagging completed transactions as inspected: it set the INSPECTED flag
only on txs that already had it, so a finished tx was never newly
marked.

On a normal flow DetectRunTx() sets the same flag and masks the
mistake. On a passed flow -- a `pass` rule or a pass-the-flow exception
policy -- detection is skipped and this function is the only thing that
marks txs inspected. With the inverted guard the transactions are never
flagged, AppLayerParserTransactionsCleanup() never frees them, and the
per-flow transaction list grows without bound while each packet
re-scans it, giving O(n^2) cleanup cost.

This is an unintended regression from 834378ff88.

(cherry picked from commit 006e21c1cc)

src/app-layer-parser.c

@@ -781,7 +781,7 @@ void AppLayerParserSetTransactionInspectId(const Flow *f, AppLayerParserState *p
         if (tag_txs_as_inspected) {
             const uint8_t inspected_flag = (flags & STREAM_TOSERVER) ? APP_LAYER_TX_INSPECTED_TS
                                                                      : APP_LAYER_TX_INSPECTED_TC;
-            if (txd->flags & inspected_flag) {
+            if (!(txd->flags & inspected_flag)) {
                 txd->flags |= inspected_flag;
                 SCLogDebug("%p/%" PRIu64 " in-order tx is done for direction %s. Flags %02x", tx,
                         idx, flags & STREAM_TOSERVER ? "toserver" : "toclient", txd->flags);
@@ -820,7 +820,7 @@ void AppLayerParserSetTransactionInspectId(const Flow *f, AppLayerParserState *p
             AppLayerTxData *txd = AppLayerParserGetTxData(ipproto, alproto, tx);
             const uint8_t inspected_flag = (flags & STREAM_TOSERVER) ? APP_LAYER_TX_INSPECTED_TS
                                                                      : APP_LAYER_TX_INSPECTED_TC;
-            if (txd->flags & inspected_flag) {
+            if (!(txd->flags & inspected_flag)) {
                 txd->flags |= inspected_flag;
                 SCLogDebug("%p/%" PRIu64 " out of order tx is done for direction %s. Flag %02x", tx,
                         idx, flags & STREAM_TOSERVER ? "toserver" : "toclient", txd->flags);