util/base64: don't reset decoded bytes in RFC4648

Old behavior:
With RFC4648, the decoded bytes were reset to 0 in case an unusual
character was encountered in the encoded string. This worked out fine
for small test cases where there weren't many bytes to be decoded.

Problem:
If a big encoded string had a character outside of the base alphabet,
the processing would stop and the number of decoded bytes were set to 0.
However, even though the processing should stop at the invalid
character, the number of decoded bytes should correctly store the bytes
decoded up until the point an invalid characted was encountered.

New behavor:
For any base64 encoded string given to the base64 decoder in RFC4648
mode, we make sure that the number of decoded bytes correctly reflect
the number of bytes processed up until the string was valid. This makes
sure any further calculations/use of the decoded data is done correctly.

Redmine ticket: 5885

src/util-base64.c

@@ -118,10 +118,10 @@ Base64Ecode DecodeBase64(uint8_t *dest, uint32_t dest_size, const uint8_t *src,
             /* Invalid character found, so decoding fails */
             if (src[i] != '=') {
                 valid = false;
-                if (mode != BASE64_MODE_RELAX) {
+                ecode = BASE64_ECODE_ERR;
+                if (mode == BASE64_MODE_STRICT) {
                     *decoded_bytes = 0;
                 }
-                ecode = BASE64_ECODE_ERR;
                 break;
             }
             padding++;