Introduction and tools used
Initial Analyis
Debugging and Dynamic Analysis
Reverse Engineering the Code
TL;DR: My first CTF challenge with a group of friends went well !
The challenge consisted of a small Windows executable. Upon receiving the file, I immediately fired up Ghidra, x86dbg, and Cheat Engine, each providing valuable assistance in its own way.
Ghidra features an excellent decompiler that, while not reconstructing the original source code, translates assembly into C-like pseudocode. This significantly simplifies the process of understanding the program’s logic and execution flow.
x86dbg (also known as x64dbg or x96dbg) is a powerful and fast debugger equipped with tools for module and stack analysis. Its intuitive interface displays loaded symbols (DLLs), which is particularly useful for identifying and analyzing function calls during reverse engineering.
Finally, the classic Cheat Engine—a tool that many reversers first encounter when experimenting with memory and instruction modifications. One of its standout features is Structure Dissect, which helps interpret data structures by identifying pointers, integer values, floats, booleans, and more. Additionally, it can automatically detect RTTI (Run-Time Type Information) classes left behind by C++ programs that use polymorphism, making object exploration much more intuitive.
First step was to figure out if the binary was a 32 or 64 bit executable, ghidra immediately figured it out that it was a 32bit one. While Ghidra was occupied doing the code analysis, I’ve attached x32dbg to misc200.exe, my goal was to analyze strings loaded in memory that ghidra might have missed.
And oh my, I love seeing this type of hint in Base64 format.
Once decoded it says: “Reverse the mystery”
Combined with the “Funny message” option in the command prompt window:
“Like, man… If this code was a monster, we’d totally unmask it and find out it’s just a spooky old code!”
Really took me a long-ish time to figure out it was a “Scooby-Doo” reference. The important key was the word unmask which is typically associated with ciphers and xor encryptions.
Since the flag was clearly hidden, and the base64-encoded string was our only viable lead, I started by placing breakpoints on key imported functions that might be involved in processing it. Given that the program was written in C++, my initial candidates were:
std::cout, std::reverse, memcpy and memmove
At this stage, Cheat Engine’s Code Filter tool was particularly useful. In short, it helps identify instructions actively modifying a specific memory address, which is great for tracking transformations applied to the encoded string.
However, none of these breakpoints revealed any noticeable changes to the string in memory—no modifications, and no clear hints in the ESP register. This strongly suggested that the encryption/decryption process was not relying on standard Win32 functions.
With this in mind, I suspected the program might be using a custom XOR-based encryption, as XOR is a common lightweight technique for obfuscation in CTF challenges.
The program provided by the CTF challenge had a very simple menu I could interact with
So I went back in ghidra at the location of the menu handling, copied the decompiled code into a text editor and manually started renaming variables that I could quickly understand their purpose.
One interesting part of the function was Concatenate String which had a secret behavior.
Normally this would happen
Then code would jump to this location
But you know something is coming. The previous screenshot I showed was only a snippet of the full if statement. Don’t be scared.
At first the defined variables with hex data in it seemed interesting, perhaps it was part of a cipher?
What led me to the flag was while (uVar6 < 0x1c); after looking at it for a good 10 seconds, I realized that 1C in hex was 28 and uVar6 was perhaps an index. And there’s only one string memory that is exactly of length 28 “UmV2ZXJzZSB0aGUgbXlzdGVyeQ==”
And oh my, I was happy to see that! That 0xbe string is a xor loop that takes the hex byte of a character, then it does its xor magic. I placed a BP on that instruction
inserted the base64 encoded string as both inputs
and boom, our flag was written in memory!
What a fun challenge !
ヽ(°〇°)ﾉ