Reverse engineering Apple's OS X
A thunking good time
Mac Secrets A number of folks have asked me what tools and techniques I use to reverse engineer Cocoa executables. I thought it would be worth taking some time out from documenting undocumented APIs to show you how easy it is to do the same thing for yourself.
My number-one favorite tool is class-dump, a command-line utility written originally by Steve Nygard. You can feed an executable to class-dump, and it will print out all the Objective-C class declarations contained within the file. This information alone is often enough to get you started with an undocumented API.
But class-dump is not without its wrinkles. First, you need to be aware that class-dump can have problems with Objective-C 2.0 files. To fix that, there's a variant of class-dump available called class-dump-x. Just search online, and you'll find it. The latest version of class-dump is 3.1.2.
The next problem is that class-dump can get very confused if the executable contains any references to C++ classes, spewing out all sorts of junk to stderr. The workaround is simply to pipe class-dump's output to a file, separating the garbage from the class-declarations you want to see.
Finally, class-dump can get confused by certain "fat" binaries, especially recent ones that contain 64-bit executables. The workaround here is simply to use the lipo or ditto tools to create a single-architecture executable that class-dump will then accept. You can read more about class-dump here.
OK, you've got your class declarations, but you want to look at the code itself, right? Another favorite tool is otx, which you can find here.
You might already be familiar with otool, a command-line utility that's bundled with OS X. otool generates a code disassembly (either PowerPC or Intel) of a specified executable, but the output generated is not very user friendly. otx is effectively an "otool after-burner" that drastically improves the output of otool.
Amongst other things, it annotates the code listing by placing Objective-C style comments alongside method calls. If you're working with C++ code, otx will also try to "unmangle" C++ symbols to their original state. This is very useful when examining certain Apple frameworks such as CoreUI, much of which is implemented using C++. For more on CoreUI, see my earlier article here.