The rise of zero-day patches

Interview Zero-day exploits were once the realm of just underground and elite hackers, but their increased prevalence is bringing a positive new trend: unofficial patches from members of the community, offered for protection before official vendor patches appear.

Federico Biancuzzi interviewed Landon Fuller, who wrote Mac OS X patches for recent Month of Apple Bugs vulnerabilities, and the ZERT team, which has offered patches for critical Microsoft Windows zero-days that were actively exploited.

Part 1: Landon Fuller on patching Mac OS X

Could you introduce yourself?

Landon Fuller: I'm a 24 year old bicycle addict. I'm also the director of infrastructure for Three Rings Design, a small video game company based here in San Francisco.

Why have you chosen to write zero-day patches?

Landon Fuller: Writing these patches provides challenging, discrete problems that are fun to solve. Additionally, I personally appreciate having the option to apply a patch while waiting for a vendor fix - that's an option that I'm happy to provide to others.

How much time do you need to develop a patch?

Landon Fuller: Generally speaking, it takes anywhere from two to eight hours. My steps for creating a patch are:

1. Recreate the test case and reproduce the issue. For example, to reproduce the Java GIF vulnerability from the provided description, I had to read up on the GIF format and adapt ImageMagick to produce specifically broken image files.
2. Find the vulnerable code. This is generally a straight-forward debugging exercise, but not always - in patching the Flip4Mac vulnerability, I found that Telestream had stripped all but a few symbols from their binary. Finding the vulnerable code required working backwards up the soon-to-be mangled stack on my x86 system, locating and labeling functions according to their assumed purpose and finding the bug (unexpected signed integer overflow where an unsigned integer should have been used). I then had to do the same thing on my PowerPC system - considerably simpler once I'd already found the issue. William Carrel also helped debug that one.
3. Determine the safest way to patch an issue and implement the patch. Sometimes this can be tricky. With the CoreGraphics PDF infinite-loop DoS, two functions had to be patched - an entry function, where a per-thread counter was initialised, and an internal function, where the counter was incremented. When the counter function hit the maximum loop count, the patched code returned an error its caller could handle, and CoreGraphics recognised the PDF as corrupt. Additionally, I implemented an idea suggested by William Carrel - to protect against recursion, the outer counter initialiser maintained a reference count that was incremented each time it was called, and decremented directly prior to exiting. The loop counter was not deallocated until the reference count hit 0.
4. Test the patch. Prior to releasing a patch, I would test the fix on both x86 and PowerPC systems. I also implemented a regression test suite, and used that to test the patches for all previous issues as well. Occasionally testing would find an issue, but for the most part, this was the quickest step.

How much additional effort is needed to support localised versions of the software?

Landon Fuller:

Fortunately, Apple's means of localisation does not require implementing multiple versions of a single piece of software. Localised resources are independent of the code, and in nearly all cases the same patch can be used for any localisation.

What type of problems (if any) did you have to handle while dealing with two types of architectures (PowerPC and x86)?

Landon Fuller: Apple (via NeXT) has solved the multiple architecture issue very nicely - the i386 and PowerPC releases of Mac OS X are built, for the most part, from the same (or very similar) code base.

The patch was written in C and cross-compiled for both architectures; after reverse engineering and implementing a patch on x86, I would verify the function signatures on PowerPC and run the regression tests. Only a few instances - such as the Flip4Mac patch - required any architecture-specific wrangling.

Did you use the available source code of Darwin to solve any of the bugs?

Landon Fuller: While none of the non-kernel Apple bugs were in available source, I did do some research on issues using the available Mac OS X source - one example being the CoreFoundation %n format string specifier.

Does the MacOS X license limit your power to develop and/or distribute fixes?

Landon Fuller: I'm not a lawyer, and that's a complex area of law. That said, I believe that this kind of work is protected under existing US law, and even more importantly, is ethically sound practice.

How do you install your patches? Do you need to include some pieces of the file you are going to patch? Do you think this could become a legal problem?

Landon Fuller: The patches are installed as plugins to Application Enhancer.

The code is loaded into vulnerable applications at runtime; it finds and patches the vulnerable functions by name, whether they are in libraries, plugins, or the applications themselves. This means that the same patch can generally be used on both PowerPC and x86 systems, no files are ever modified on disk, and also, the contents of the original file are not required.

Sometimes the symbols are stripped and can't be found by name - this was the case with Flip4Mac. I implemented the patch by finding one of the few public symbols that are exported by the plugin, and then computing the offset to the vulnerable code on both PowerPC and x86 systems. At runtime, the patch does the same thing I did manually - it locates the address of the public symbol, adds the offset I computed, and patches the function at the resultant address.

Each patch is locked to the specific, latest version of the software in question, and refuses to patch newer versions - as new software releases are installed, there's no worry about patch incompatibilities causing crashes, or more importantly, getting in the way of an official vendor fix.

Have you involved Apple during the process of developing or testing a patch, or shared your findings after the patch was released?

Landon Fuller: I've made all of the code to the patches available to the public, including Apple, and I've also privately reported to the vendor any additional issues I've found in the course of producing those patches.

Have you looked into Apple's official patches for these bugs? From your analysis, what can you say?

Landon Fuller: The fixes will generally not be entirely the same, as run-time function patching necessitates wrapping the vulnerable code and preventing failure conditions from occurring. An official patch will simply fix the root issue.

Generally vendors take a long time to release patches. They often claim that this time is required to test all possible configurations and interactions with other software to avoid breaking production systems. How can your approach be faster doing reverse engineering?

Landon Fuller: Vendors typically have much more extensive testing, review, and verification processes in place; they often support multiple released versions of the software, and multiple products that need to be reviewed for the same flaw. Releases need to be tested for regressions and public release needs to be scheduled. These processes are in place to ensure that customers are well-served by stable software releases.

It may be that development/release process changes could reduce turn-around for security issues that's difficult to evaluate from the outside. I do think it's clear, however, that releasing patches for critical issues should take days or weeks, not months.

Ultimately, the turnaround from many vendors during the "Month of Apple Bugs" was exemplary - OmniGroup had a patch out for OmniWeb within hours.