Reverse Engineering

Reverse engineering (RE) is the systematic approach to determine the inner working of an application by inspecting the application’s logic flow and responses to specific inputs. One will often encounter CTF challenges supplying a sample application (such as malware) that must be reverse engineered in order to uncover hidden keys or serial numbers within the applications or in order to unlock hidden execution paths or bypass expected execution paths.

In order to achieve this you must first understand the control flow of the application and how one can analyse a compiled application.

Requirements

  • Build a strong foundation in programming languages, such as C, C++, Python, and Assembly. Understanding the underlying logic and structures will make reverse engineering tasks more manageable.
  • Familiarize yourself with different file formats and their structures. Knowing the ins and outs of various formats will aid in identifying hidden data and vulnerabilities.
  • Explore reverse engineering tools like Ghidra, IDA Pro, OllyDbg, and Radare2. Learn their capabilities, strengths, and weaknesses to choose the right tool for each task.

Basic RE Activities

  • Check the File Type: determine the file type by executing the file [filename] command in a terminal.
  • Execute the Binary: interact with the binary in a controlled, sandbox environment. Check for inputs requested.
  • Print Strings: print all the strings present in the binary I used strings -a [filename] command.

Advanced RE Activities

Fuzzing

Test the input response of the application. Test how the application responds when presented with unexpected inputs (submitting an Integer in a String field or visa versa). Determine what controls are in place to prevent unexpected user behaviour. Map out these controls to start identifying control flows within the application. Fuzzing could be a viable technique to identify insecure input validation and possible injection paths.

Decompilation

Another common activity during the reverse engineering process is to decompile the source code of an application. Decompiling an application attempts to convert the binary code into the higher order language it was written in. Higher order languages tend to be more human readable but depending on how the source code was obfuscated, the decompiled code might still be difficult to decipher. It is recommended you step through the code blocks to determine the logical flow of the program. This might be combined with the control flows identified during the previous step.

Decompilers:

For Android applications, common decompiling software include: APKTool, Dex2Jar and Java-Decompiler.

If a program cannot be decompiled a Hex Editor can be used to inspect the compiled code. The “strings” command can also be used on a compiled application to extract all strings found within the application, in some cases this might lead to some information regarding error messages and code control checks.

Disassemble

A disassembler tool breaks down a compiled program into machine code.

Disassemblers:

Debugging

A debugging application allows you to perform dynamic analysis and set break points to step through the execution of a program. Another option would be to perform active analysis and step through the code with a debugger like OllyDbg.

Additional Resources

Practice

The following TryHackMe rooms offer practical exercises to grasp the topics discussed above.