Web Banner
Back to Pre-Conference Training Page

Back to Conference Home Page

Reverse Engineering Proprietary Protocols

Two-Day Interactive (Classroom) Training - OWASP New Zealand Day 2025

Abstract

Protocol Reverse Engineering (PRE) recovers specifications of undocumented protocols. It’s crucial for analyzing malware and enhancing IDS. This training covers PRE principles, writing protocol dissectors, and fuzzing for vulnerabilities, using tools like Scapy and Boofuzz.

Target Audience

  1. DFIR practitioner - to investigation malicious activity in the network
  2. Reverse Engineer - write a custom client that fully replicates the existing client software/game.
  3. Bug Hunter - Write protocol fuzzer for Black Box testing for application processing remote data, for example, lots of IoT Devices use custom protocol for efficient communication.
  4. Malware Analyst - To decode C&C server commands and the data which is exfiltrated
  5. Threat Hunting - write network signatures for new emerging APT threats or it could be an intruder in your network, this course will help you decode network and analyze network traffic.
  6. Developers - who don’t have access to source code or protocol documentation, it usually happens when you are dealing with a legacy system which is too old and the company cannot find any documentation and you intend to migrate the system to new technology.
  7. While debugging software over the network, writing a protocol dissector can help you to get a deeper understanding of network communication done by your software.
  8. Helps you to do network debugging/diagnostics of application layer data.
  9. It helps you understand what is really transmitted over the network.
  10. RED Team - take advantage of what the Security Operation Center (SOC) doesn’t know. Look for data leaks, do attacks like inject, replay and spoofing.
  11. Vulnerability Researcher/Exploit Developer - this will also help exploit developer and vulnerability research to reproduce remote vulnerability and find zero-day bugs.

Course Details

Dates: Tuesday and Wednesday, 2-3 September 2025

Time: 8:45 a.m. to 5:30 p.m. (NZST) each day

Instructor: Munawwar Hussain Shelia (Vulnerability Researcher, Qualcomm)

Course Fee: NZ $1,000.00 (plus GST and ticketing fees)

Registration Site: https://events.humanitix.com/owaspnz2025-training

Maximum Enrolment: 50 attendees

Prerequisites

What Students Should Bring and Do Before Class

What Will Students Be Provided With

Course Description

A network protocol defines the format and semantics of message exchange between applications. In modern times there are a myriad of proprietary application protocols like Skype Protocol, Dropbox Protocol, etc which applications use to achieve various goals like bandwidth efficiency, custom encryption/compression, etc. These protocols could have security vulnerabilities. Protocol Reverse Engineering (PRE) is not only useful for offensive purposes but also used by modern Intrusion Detection Systems(IDS), they use the knowledge of protocol specification to do Deep Packet Inspection(DPI) which can enhance its capabilities, where it earlier relied just based on pattern matching which may produce lots of false positives. Custom protocols are not only used by legitimate applications but also by malware and botnets like Zeus, Emotet, etc. By reversing malware protocol you can connect to malware servers and track their campaigns.

Protocol Reverse Engineering(PRE) is an art and science of recovering the protocol specification of the obscure/proprietary protocol whose documentation is unavailable or poorly documented. There are efforts to develop automated PRE tools but they are largely academic and are not mature enough to be usable, and can’t give the accuracy a human analyst can offer. Automated tools face the challenges of heterogeneous protocol data which is often a mixture of text and binary, and it has different data types and variable-length fields and this is the reason I have created this training, it is to help you understand these challenges and learn to recover protocol specification.

This training is divided into two parts, in the first part we will learn about Protocol Reverse Engineering principles. We will look at some of the common data formats and other protocol structures and with that understanding we will write a protocol dissector using Scapy framework for a target Desktop game Minetest (open source implementation of Minecraft). Minetest is online multiplayer game in which different players can connect to the server and play with other players, there are also many public servers which you can connect and play. Once we have written the decoder we will sniff the connection and look at the communication flow between the client and the server which we will capture and re-analyze the traffic to improve the dissector further, using this newly improved dissector we will implement a custom game client/bot which will connect to the server and play as a Bot player.

In the second part, with a decent understanding of the Minetest Protocol we will move on to the offensive side of the training and try to fuzz the game server to find some security vulnerabilities, we will start with basic Fuzzer and try to do incremental improvement such that we have good code coverage. Leveraging their reverse-engineered understanding of the protocol, participants will employ Generational Fuzzing by defining the protocol specification in the Boofuzz fuzzing framework and subsequently fuzzing the application. The training will also explore Mutation Fuzzing as an alternative approach to identify potential crashes or vulnerabilities.

Why should people attend your course?

Use the knowledge gained in this training to start exploring some Open and Close Protocol to improve your understanding of this topic. That will help you to get a deeper understanding of some underlying issues more closely.

Course Outline

  1. Networking Basics
  2. Capturing Network Traffic
    1. Passive analysis
      1. Network Sniffing
      2. Syscall hooking (strace)
    2. Active analysis
      1. Network Proxies
  3. Protocol Reversing
    1. Protocol Structure
      1. Common data format
      2. Data Encoding
      3. Binary Protocol Structure
      4. Text Protocol Structure
    2. Protocol Flow
  4. Protocol Dissector (targeting Minetest game)
    1. Scapy 101
    2. Implementing protocol dissector in scapy for Minetent game. This section will have Labs on
      1. Protocol decoding TLV format
      2. Packet decompression
      3. Packet Reassembly
  5. Custom Client (Bot Player for Minetest Game)
    1. Brief Understanding of Application
    2. Authenticate the client
    3. Establish a valid session
    4. Some game hacks like making the player fly
    5. Create A Bot Army (if time permits)
  6. Protocol Fuzzing (targeting Minetest game)
    1. What is fuzzing?
    2. Implement Mutation Fuzzer
    3. Implement Dumb Fuzzer
    4. Implement Generation Fuzzing (Protocol Aware Fuzzing)
    5. Createing Harness

Tools of the Trade

Below are some of the tools that you will learn in this training that will make you Protocol Reversing experience more fun.

  1. Protocol Reversing tools
  2. Wireshark
  3. Scapy
  4. strace
  5. scapy
  6. Protocol Fuzzing Tool
  7. Boofuzz (Sulley) fuzzing framework

Your Instructor

Photo of Munawwar Hussain Shelia

Munawwar Hussain Shelia is currently working as a Vulnerability Researcher at Qualcomm. His primary role involves hunting bug and squashing them before product hits the market. He also develops tools to automate the process of bug detection. With a background Reverse engineeering and software development, he possesses a unique perspective on product design which enabling him to effectively identify vulnerabilities. In 2019, he conducted the “Practical IoT Hacking” Training at Nullcon and other conferences. Additionally, he also delivered a talk at the different conferences. His areas of expertise include Reverse Engineering, Binary Analysis, Malware Analysis, and Software Exploitation, topics on which he frequently shares insights through his blog, https://taintedbits.com. He has conducted training for various governmental and private organizations worldwide. Notably, he has discovered and reported vulnerabilities in IoT devices and published a paper on Android Malware.