ReM3 - Reverse Engineering Challenge Writeup#

Challenge: ReM3
Category: Reverse Engineering
Flag: KCTF{w3Lc0m3_T0_tHE_r3_w0rLD}

Challenge Description#

A 500MB binary file rem3.ks was provided (heavily padded). The challenge was to reverse engineer the flag validation logic and extract the correct flag.

Initial Analysis#

File Information#

1
$ file rem3.ks
2
rem3.ks: ELF 64-bit LSB executable, x86-64, dynamically linked
3

4
$ ls -lh rem3.ks
5
-rwxr-xr-x 1 kali kali 500M Jan 20 03:23 rem3.ks

Fake Flags in Strings#

1
$ strings rem3.ks | grep KCTF
2
KCTF{fake_flag_for_reversers}
3
KCTF{hash_passes_but_fake!!!}
4
KCTF{str1ngs_lie_dont_trust!}

These are decoy flags designed to mislead analysts using simple strings approach.

Reverse Engineering Analysis#

Program Flow#

Length Check: Input must be exactly 29 characters (0x1d)
FNV-1a Hash Check: Compares hash against 0xe76fa3daba5d6f3a - leads to fake flag
XOR Transform: Complex transformation function at 0x14c0
Multi-part Comparison: Compares transformed input against encrypted target data

Key Functions#

Hash Check (Decoy Path)#

At 0x1162-0x116f:

1
movabs $0xe76fa3daba5d6f3a,%rax
2
cmp    %rax,%rdx
3
je     0x123d  ; Prints fake flag if hash matches

Transform Function at 0x14c0#

This is the real validation - a complex XOR/rotation cipher:

Uses two 64-bit keys: r10 = 0x2f910ed35ca71942, r9 = 0x6a124de908b17733
Applies position-dependent XOR
Applies rotate-left and rotate-right operations
Uses cascading state variables

Target Encrypted Data (from .rodata)#

1
0x2160: dc6bbb4dfd25e47ec326  (bytes 0-9)
2
0x2150: f572ab96fc8d551093c1  (bytes 10-19)
3
0x2140: fd81465b7e33838f2f    (bytes 20-28)

Solution#

Transform Function Implementation#

1
#!/usr/bin/env python3
2
import struct
3
import string
4

5
r10 = 0x2f910ed35ca71942
6
r9 = 0x6a124de908b17733
7

8
def transform(data):
9
    """Replicate the XOR/rotation cipher at 0x14c0"""
10
    if len(data) != 29:
11
        data = data[:29] if len(data) > 29 else data + b'\x00' * (29 - len(data))
12
    data = bytearray(data)
13
    r8 = 0
14
    edi = 0
15
    esi = 0xffffffc3 & 0xffffffff
16

17
    for i in range(29):
18
        ebx = i & 7
19
        shift1 = ebx * 8
20

21
        # Extract byte from r10
22
        rax = (r10 >> shift1) & 0xff
23
        ecx = ((i * 8) + 0x10) & 0x38
24

25
        # XOR with input
26
        al = (rax + edi) & 0xff
27
        al = al ^ data[i]
28
        edi = (edi + 0x1d) & 0xffffffff
29

30
        # Rotate left
31
        r14_val = (r9 >> ecx) & 0xff
32
        r14_low = r14_val & 0x7
33
        al = ((al << r14_low) | (al >> (8 - r14_low))) & 0xff
34

35
        # More XOR operations
36
        r14_val2 = (r9 >> shift1) & 0xff
37
        al = (al + (esi & 0xff)) & 0xff
38
        ecx2 = r14_val2 ^ r8
39
        r8 = (r8 + 0x11) & 0xffffffff
40
        al = al ^ (ecx2 & 0xff)
41

42
        # Rotate right
43
        esi_low = esi & 0x7
44
        al = ((al >> esi_low) | (al << (8 - esi_low))) & 0xff
45

46
        data[i] = al
47

48
        # Update state for next iteration
49
        ecx3 = ((i * 8) + 0x18) & 0x38
50
        rbx_val = (r10 >> ecx3) & 0xff
51
        ecx_final = rbx_val ^ 0xa5
52
        ecx_final = (ecx_final + esi) & 0xff
53
        esi = (al + ecx_final) & 0xffffffff
54

55
    return bytes(data)

Brute Force Solver#

Since the transformation has cascading dependencies, we use a greedy character-by-character approach:

1
target = bytes.fromhex('dc6bbb4dfd25e47ec326f572ab96fc8d551093c1fd81465b7e33838f2f')
2
charset = string.ascii_lowercase + string.ascii_uppercase + string.digits + '_!@#$%'
3

4
# Known format: KCTF{.......................}
5
flag = bytearray(b'KCTF{' + b'A' * 23 + b'}')
6

7
for pos in range(5, 28):
8
    best_char = 'A'
9
    best_match = 0
10

11
    for c in charset:
12
        flag[pos] = ord(c)
13
        out = transform(bytes(flag))
14

15
        # Count cumulative matching bytes
16
        cumulative = sum(1 for i in range(pos+1) if out[i] == target[i])
17
        if cumulative > best_match:
18
            best_match = cumulative
19
            best_char = c
20

21
    flag[pos] = ord(best_char)
22
    print(f'Position {pos}: {best_char}')
23

24
print(f'Flag: {bytes(flag).decode()}')

Output#

1
Pos 5: w
2
Pos 6: 3
3
Pos 7: L
4
Pos 8: c
5
Pos 9: 0
6
Pos 10: m
7
Pos 11: 3
8
Pos 12: _
9
Pos 13: T
10
Pos 14: 0
11
Pos 15: _
12
Pos 16: t
13
Pos 17: H
14
Pos 18: E
15
Pos 19: _
16
Pos 20: r
17
Pos 21: 3
18
Pos 22: _
19
Pos 23: w
20
Pos 24: 0
21
Pos 25: r
22
Pos 26: L
23
Pos 27: D
24

25
Flag: KCTF{w3Lc0m3_T0_tHE_r3_w0rLD}

Verification#

1
$ echo 'KCTF{w3Lc0m3_T0_tHE_r3_w0rLD}' | ./rem3.ks
2
=== KCTF Reverse Challenge ===
3
Enter flag: Success! Real flag accepted.

Key Takeaways#

Don’t Trust Strings: The binary contained multiple fake flags designed to trap lazy reversers
Understand the Flow: The FNV hash check was a red herring - the real validation used a custom cipher
Custom Cipher Analysis: The transform used XOR + rotations with position-dependent keys
Cascading State: Each byte’s transformation depended on previous results, requiring character-by-character solving
Size Padding: The 500MB size was artificial padding to make analysis slower

Tools Used#

file - Binary identification
strings - Initial string extraction (revealed decoys)
objdump -d - Disassembly
Python3 - Cipher reimplementation and brute force

Author: MR. Umair
Date: January 20, 2026
Competition: KnightCTF 2026