Eldad Eilam is a consultant in the field of reverse engineering. He assists clients with operating system and in-depth software reverse engineering, and has devoted several years to developing advanced reverse engineering techniques.

Foreword vii

Acknowledgments xi

Introduction xxiii

Part I Reversing 101 1

Chapter 1 Foundations 3

What Is Reverse Engineering? 3

Software Reverse Engineering: Reversing 4

Reversing Applications 4

Security-Related Reversing 5

Malicious Software 5

Reversing Cryptographic Algorithms 6

Digital Rights Management 7

Auditing Program Binaries 7

Reversing in Software Development 8

Achieving Interoperability with Proprietary Software 8

Developing Competing Software 8

Evaluating Software Quality and Robustness 9

Low-Level Software 9

Assembly Language 10

Compilers 11

Virtual Machines and Bytecodes 12

Operating Systems 13

The Reversing Process 13

System-Level Reversing 14

Code-Level Reversing 14

The Tools 14

System-Monitoring Tools 15

Disassemblers 15

Debuggers 15

Decompilers 16

Is Reversing Legal? 17

Interoperability 17

Competition 18

Copyright Law 19

Trade Secrets and Patents 20

The Digital Millenium Copyright Act 20

DMCA Cases 22

License Agreement Considerations 23

Code Samples & Tools 23

Conclusion 23

Chapter 2 Low-Level Software 25

High-Level Perspectives 26

Program Structure 26

Modules 28

Common Code Constructs 28

Data Management 29

Variables 30

User-Defined Data Structures 30

Lists 31

Control Flow 32

High-Level Languages 33

C 34

C++ 35

Java 36

C# 36

Low-Level Perspectives 37

Low-Level Data Management 37

Registers 39

The Stack 40

Heaps 42

Executable Data Sections 43

Control Flow 43

Assembly Language 101 44

Registers 44

Flags 46

Instruction Format 47

Basic Instructions 48

Moving Data 49

Arithmetic 49

Comparing Operands 50

Conditional Branches 51

Function Calls 51

Examples 52

A Primer on Compilers and Compilation 53

Defining a Compiler 54

Compiler Architecture 55

Front End 55

Intermediate Representations 55

Optimizer 56

Back End 57

Listing Files 58

Specific Compilers 59

Execution Environments 60

Software Execution Environments (Virtual Machines) 60

Bytecodes 61

Interpreters 61

Just-in-Time Compilers 62

Reversing Strategies 62

Hardware Execution Environments in Modern Processors 63

Intel NetBurst 65

µops (Micro-Ops) 65

Pipelines 65

Branch Prediction 67

Conclusion 68

Chapter 3 Windows Fundamentals 69

Components and Basic Architecture 70

Brief History 70

Features 70

Supported Hardware 71

Memory Management 71

Virtual Memory and Paging 72

Paging 73

Page Faults 73

Working Sets 74

Kernel Memory and User Memory 74

The Kernel Memory Space 75

Section Objects 77

VAD Trees 78

User-Mode Allocations 78

Memory Management APIs 79

Objects and Handles 80

Named objects 81

Processes and Threads 83

Processes 84

Threads 84

Context Switching 85

Synchronization Objects 86

Process Initialization Sequence 87

Application Programming Interfaces 88

The Win32 API 88

The Native API 90

System Calling Mechanism 91

Executable Formats 93

Basic Concepts 93

Image Sections 95

Section Alignment 95

Dynamically Linked Libraries 96

Headers 97

Imports and Exports 99

Directories 99

Input and Output 103

The I/O System 103

The Win32 Subsystem 104

Object Management 105

Structured Exception Handling 105

Conclusion 107

Chapter 4 Reversing Tools 109

Different Reversing Approaches 110

Offline Code Analysis (Dead-Listing) 110

Live Code Analysis 110

Disassemblers 110

IDA Pro 112

ILDasm 115

Debuggers 116

User-Mode Debuggers 118

OllyDbg 118

User Debugging in WinDbg 119

IDA Pro 121

PEBrowse Professional Interactive 122

Kernel-Mode Debuggers 122

Kernel Debugging in WinDbg 123

Numega SoftICE 124

Kernel Debugging on Virtual Machines 127

Decompilers 129

System-Monitoring Tools 129

Patching Tools 131

Hex Workshop 131

Miscellaneous Reversing Tools 133

Executable-Dumping Tools 133

DUMPBIN 133

PEView 137

PEBrowse Professional 137

Conclusion 138

Part II Applied Reversing 139

Chapter 5 Beyond the Documentation 141

Reversing and Interoperability 142

Laying the Ground Rules 142

Locating Undocumented APIs 143

What Are We Looking For? 144

Case Study: The Generic Table API in [...] 145

RtlInitializeGenericTable 146

RtlNumberGenericTableElements 151

RtlIsGenericTableEmpty 152

RtlGetElementGenericTable 153

Setup and Initialization 155

Logic and Structure 159

Search Loop 1 161

Search Loop 2 163

Search Loop 3 164

Search Loop 4 165

Reconstructing the Source Code 165

RtlInsertElementGenericTable 168

RtlLocateNodeGenericTable 170

RtlRealInsertElementWorker 178

Splay Trees 187

RtlLookupElementGenericTable 188

RtlDeleteElementGenericTable 193

Putting the Pieces Together 194

Conclusion 196

Chapter 6 Deciphering File Formats 199

Cryptex 200

Using Cryptex 201

Reversing Cryptex 202

The Password Verification Process 207

Catching the "Bad Password" Message 207

The Password Transformation Algorithm 210

Hashing the Password 213

The Directory Layout 218

Analyzing the Directory Processing Code 218

Analyzing a File Entry 223

Dumping the Directory Layout 227

The File Extraction Process 228

Scanning the File List 234

Decrypting the File 235

The Floating-Point Sequence 236

The Decryption Loop 238

Verifying the Hash Value 239

The Big Picture 239

Digging Deeper 241

Conclusion 242

Chapter 7 Auditing Program Binaries 243

Defining the Problem 243

Vulnerabilities 245

Stack Overflows 245

A Simple Stack Vulnerability 247

Intrinsic Implementations 249

Stack Checking 250

Nonexecutable Memory 254

Heap Overflows 255

String Filters 256

Integer Overflows 256

Arithmetic Operations on User-Supplied Integers 258

Type Conversion Errors 260

Case-Study: The IIS Indexing Service Vulnerability 262

CVariableSet::AddExtensionControlBlock 263

DecodeURLEscapes 267

Conclusion 271

Chapter 8 Reversing Malware 273

Types of Malware 274

Viruses 274

Worms 274

Trojan Horses 275

Backdoors 276

Mobile Code 276

Adware/Spyware 276

Sticky Software 277

Future Malware 278

Information-Stealing Worms 278

BIOS/Firmware Malware 279

Uses of Malware 280

Malware Vulnerability 281

Polymorphism 282

Metamorphism 283

Establishing a Secure Environment 285

The Backdoor.Hacarmy.D 285

Unpacking the Executable 286

Initial Impressions 290

The Initial Installation 291

Initializing Communications 294

Connecting to the Server 296

Joining the Channel 298

Communicating with the Backdoor 299

Running SOCKS4 Servers 303

Clearing the Crime Scene 303

The Backdoor.Hacarmy.D: A Command Reference 304

Conclusion 306

Part III Cracking 307

Chapter 9 Piracy and Copy Protection 309

Copyrights in the New World 309

The Social Aspect 310

Software Piracy 310

Defining the Problem 311

Class Breaks 312

Requirements 313

The Theoretically Uncrackable Model 314

Types of Protection 314

Media-Based Protections 314

Serial Numbers 315

Challenge Response and Online Activations 315

Hardware-Based Protections 316

Software as a Service 317

Advanced Protection Concepts 318

Crypto-Processors 318

Digital Rights Management 319

DRM Models 320

The Windows Media Rights Manager 321

Secure Audio Path 321

Watermarking 321

Trusted Computing 322

Attacking Copy Protection Technologies 324

Conclusion 324

Chapter 10 Antireversing Techniques 327

Why Antireversing? 327

Basic Approaches to Antireversing 328

Eliminating Symbolic Information 329

Code Encryption 330

Active Antidebugger Techniques 331

Debugger Basics 331

The IsDebuggerPresent API 332

SystemKernelDebuggerInformation 333

Detecting SoftICE Using the Single-Step Interrupt 334

The Trap Flag 335

Code Checksums 335

Confusing Disassemblers 336

Linear Sweep Disassemblers 337

Recursive Traversal Disassemblers 338

Applications 343

Code Obfuscation 344

Control Flow Transformations 346

Opaque Predicates 346

Confusing Decompilers 348

Table Interpretation 348

Inlining and Outlining 353

Interleaving Code 354

Ordering Transformations 355

Data Transformations 355

Modifying Variable Encoding 355

Restructuring Arrays 356

Conclusion 356

Chapter 11 Breaking Protections 357

Patching 358

Keygenning 364

Ripping Key-Generation...