Fix DTLS 1.3 unified header fixed bits mask

DTLS13_FIXED_BITS_MASK used 0x111 (hex 273) instead of 0x7 (decimal 7,
binary 111). Per RFC 9147 Section 4, the top 3 bits of the unified
header flags byte must be 001. The incorrect hex value caused the mask
to only check bit 5 instead of bits 5, 6, and 7, allowing bytes with
bits 6 or 7 set to be misidentified as unified DTLS 1.3 headers.

Author: LinuxJedi

src/dtls13.c

@@ -93,7 +93,7 @@ typedef struct Dtls13RecordPlaintextHeader {
 #define DTLS13_SEQ_8_LEN 1
 
 /* fixed bits mask to detect unified header  */
-#define DTLS13_FIXED_BITS_MASK (0x111 << 5)
+#define DTLS13_FIXED_BITS_MASK (0x7 << 5)
 /* fixed bits value to detect unified header  */
 #define DTLS13_FIXED_BITS (0x1 << 5)
 /* ConnectionID present bit in the unified header flags */