The rise of zero-day patches

Original URL: https://www.theregister.com/2007/03/02/zero-day_patches_interviews/

The experts speak

Posted in Channel, 2nd March 2007 13:01 GMT

Interview Zero-day exploits were once the realm of just underground and elite hackers, but their increased prevalence is bringing a positive new trend: unofficial patches from members of the community, offered for protection before official vendor patches appear.

Federico Biancuzzi interviewed Landon Fuller, who wrote Mac OS X patches for recent Month of Apple Bugs vulnerabilities, and the ZERT team, which has offered patches for critical Microsoft Windows zero-days that were actively exploited.

Part 1: Landon Fuller on patching Mac OS X

Could you introduce yourself?

Landon Fuller: I'm a 24 year old bicycle addict. I'm also the director of infrastructure for Three Rings Design, a small video game company based here in San Francisco.

Why have you chosen to write zero-day patches?

Landon Fuller: Writing these patches provides challenging, discrete problems that are fun to solve. Additionally, I personally appreciate having the option to apply a patch while waiting for a vendor fix - that's an option that I'm happy to provide to others.

How much time do you need to develop a patch?

Landon Fuller: Generally speaking, it takes anywhere from two to eight hours. My steps for creating a patch are:

Recreate the test case and reproduce the issue. For example, to reproduce the Java GIF vulnerability from the provided description, I had to read up on the GIF format and adapt ImageMagick to produce specifically broken image files.
Find the vulnerable code. This is generally a straight-forward debugging exercise, but not always - in patching the Flip4Mac vulnerability, I found that Telestream had stripped all but a few symbols from their binary. Finding the vulnerable code required working backwards up the soon-to-be mangled stack on my x86 system, locating and labeling functions according to their assumed purpose and finding the bug (unexpected signed integer overflow where an unsigned integer should have been used). I then had to do the same thing on my PowerPC system - considerably simpler once I'd already found the issue. William Carrel also helped debug that one.
Determine the safest way to patch an issue and implement the patch. Sometimes this can be tricky. With the CoreGraphics PDF infinite-loop DoS, two functions had to be patched - an entry function, where a per-thread counter was initialised, and an internal function, where the counter was incremented. When the counter function hit the maximum loop count, the patched code returned an error its caller could handle, and CoreGraphics recognised the PDF as corrupt. Additionally, I implemented an idea suggested by William Carrel - to protect against recursion, the outer counter initialiser maintained a reference count that was incremented each time it was called, and decremented directly prior to exiting. The loop counter was not deallocated until the reference count hit 0.
Test the patch. Prior to releasing a patch, I would test the fix on both x86 and PowerPC systems. I also implemented a regression test suite, and used that to test the patches for all previous issues as well. Occasionally testing would find an issue, but for the most part, this was the quickest step.

How much additional effort is needed to support localised versions of the software?

Landon Fuller:

Fortunately, Apple's means of localisation does not require implementing multiple versions of a single piece of software. Localised resources are independent of the code, and in nearly all cases the same patch can be used for any localisation.

What type of problems (if any) did you have to handle while dealing with two types of architectures (PowerPC and x86)?

Landon Fuller: Apple (via NeXT) has solved the multiple architecture issue very nicely - the i386 and PowerPC releases of Mac OS X are built, for the most part, from the same (or very similar) code base.

The patch was written in C and cross-compiled for both architectures; after reverse engineering and implementing a patch on x86, I would verify the function signatures on PowerPC and run the regression tests. Only a few instances - such as the Flip4Mac patch - required any architecture-specific wrangling.

Did you use the available source code of Darwin to solve any of the bugs?

Landon Fuller: While none of the non-kernel Apple bugs were in available source, I did do some research on issues using the available Mac OS X source - one example being the CoreFoundation %n format string specifier.

Does the MacOS X license limit your power to develop and/or distribute fixes?

Landon Fuller: I'm not a lawyer, and that's a complex area of law. That said, I believe that this kind of work is protected under existing US law, and even more importantly, is ethically sound practice.

How do you install your patches? Do you need to include some pieces of the file you are going to patch? Do you think this could become a legal problem?

Landon Fuller: The patches are installed as plugins to Application Enhancer.

The code is loaded into vulnerable applications at runtime; it finds and patches the vulnerable functions by name, whether they are in libraries, plugins, or the applications themselves. This means that the same patch can generally be used on both PowerPC and x86 systems, no files are ever modified on disk, and also, the contents of the original file are not required.

Sometimes the symbols are stripped and can't be found by name - this was the case with Flip4Mac. I implemented the patch by finding one of the few public symbols that are exported by the plugin, and then computing the offset to the vulnerable code on both PowerPC and x86 systems. At runtime, the patch does the same thing I did manually - it locates the address of the public symbol, adds the offset I computed, and patches the function at the resultant address.

Each patch is locked to the specific, latest version of the software in question, and refuses to patch newer versions - as new software releases are installed, there's no worry about patch incompatibilities causing crashes, or more importantly, getting in the way of an official vendor fix.

Have you involved Apple during the process of developing or testing a patch, or shared your findings after the patch was released?

Landon Fuller: I've made all of the code to the patches available to the public, including Apple, and I've also privately reported to the vendor any additional issues I've found in the course of producing those patches.

Have you looked into Apple's official patches for these bugs? From your analysis, what can you say?

Landon Fuller: The fixes will generally not be entirely the same, as run-time function patching necessitates wrapping the vulnerable code and preventing failure conditions from occurring. An official patch will simply fix the root issue.

Generally vendors take a long time to release patches. They often claim that this time is required to test all possible configurations and interactions with other software to avoid breaking production systems. How can your approach be faster doing reverse engineering?

Landon Fuller: Vendors typically have much more extensive testing, review, and verification processes in place; they often support multiple released versions of the software, and multiple products that need to be reviewed for the same flaw. Releases need to be tested for regressions and public release needs to be scheduled. These processes are in place to ensure that customers are well-served by stable software releases.

It may be that development/release process changes could reduce turn-around for security issues that's difficult to evaluate from the outside. I do think it's clear, however, that releasing patches for critical issues should take days or weeks, not months.

Ultimately, the turnaround from many vendors during the "Month of Apple Bugs" was exemplary - OmniGroup had a patch out for OmniWeb within hours.

Part 2: ZERT

What is ZERT?

ZERT: ZERT (the Zero-day Emergency Response Team) is a group of engineers with extensive experience in reverse engineering software, firmware and hardware coupled with liaisons from industry, community and incident response groups. While ZERT works with several Internet security operations and has liaisons to anti-virus and network operations communities, ZERT is not affiliated with a particular vendor. The purpose of ZERT is not to "crack" products, but rather to "uncrack" them by averting security vulnerabilities in them before they can be widely exploited.

Why do we need a Zero-day Emergency Response Team?

ZERT: In recent times there has been significantly more effort by criminal elements to exploit zero-day vulnerabilities for financial gain. In some cases the pervasiveness of vulnerabilities presents a critical risk to a significant number of internet users. At such time it may be desirable to some people to have an option to implement interim safeguards until a manufacturer's patch is available. ZERT members work together as a team to release a non-vendor patch when a so-called "0day" (zero-day) exploit appears in the open which poses a serious risk to the public, to the infrastructure of the internet, or both.

What type of channels do you control to look for zero-day exploits?

ZERT: ZERT members belong to and work with a wide variety of organisations that encounter zero-day exploits or reports thereof.

At the moment you have released some patches for Microsoft software. Do you plan to work on other vendors' software too?

ZERT: When a zero-day is announced members of ZERT will discuss the perceived exploitability of the vulnerability and the anticipated impact. If the vulnerability is deemed to be critical enough to warrant a patch and the members have the time and skills to create a patch that they are comfortable with then the name of the vendor is not really a concern.

How much time do you need to develop a patch? How much additional effort is need to support localised versions of the software?

ZERT: The amount of time to develop, test and release a patch will vary from vulnerability to vulnerability. There really haven't been enough patches released by ZERT to have a meaningful historical reference. To date we have not discussed the localisation of patches.

Generally closed-source vendors take a long time to release patches. They often claim that this time is required to test all possible configurations and interactions with other software to avoid breaking production systems. How can your approach be faster even without access to source code? What type of reliability testing do you make?

ZERT: ZERT always recommends that anyone using a third party patch perform extensive testing in their environment to determine the suitability and compatibility of the patch. ZERT cannot perform the in-depth testing that we would expect of the vendor. If a person is faced with taking one or more servers or workstations offline or trying a third party patch until the vendor has an authorized solution, the choice may be to risk a patch that has undergone less than optimal testing. Generally, if vendor supplied workarounds are viable those would be a first choice.

ZERT developers and beta testers test the patches on a variety of systems, but we do not claim to be able to perform the exhaustive testing that a vendor would. That is why ZERT always recommends caution in the use of third party patches and proper testing in the user's environment.

Did you have any legal problem posting some disassembled code from a Microsoft patch in your paper [PDF]?

ZERT: No, we have not had any legal problems at all.

ZERT has a legal advisor, but we have not had legal issues at all, other than how we wish to license our patches. ZERT does not publish vulnerabilities, ZERT provides patches for published vulnerabilities.

From your analysis of official patches from Microsoft, what can you say? What approach do they use when fixing bugs?

ZERT: ZERT has no interest in the variety of ways that a vendor may choose to approach the problems they solve. Vendors must choose the methods they deem most suitable on a case by case basis. ZERT's focus is generally on interim solutions for critical vulnerabilities.

How do you install your patches? Do you need to include some pieces of the file you are going to patch? Do you think this could become a legal problem?

ZERT: There are a variety of ways to patch vulnerabilities. Care is taken not to redistribute code in a manner that would violate license agreements. We have had no reason to expect any legal problems to date and do not expect any in the future. We license our software as open source under BSD or GPL.

Will new Vista security mechanisms block the use of external patches?

ZERT: This will depend on what needs to be patched. As long as it is not the kernel or a signed driver it is unlikely to have much effect on third party patches, and probably virtually no effect on patching vulnerabilities for other vendor's products.

Does Microsoft EULA limit your power to develop and/or distribute fixes?

ZERT: Microsoft EULAs do not apply to Linux, Apple, Sun, Dlink, Linksys, etc. ZERT is not a vendor centric organisation. We have not had any issues with Microsoft in regards to EULA infringements.

Are you aware of any new clause included in Windows Vista EULA that could affect your work?

ZERT: ZERT enjoys open communications with Microsoft. If a Microsoft EULA is ever an issue with respect to a ZERT patch we are confident that Microsoft will advise us of an concerns. We do not anticipate any issues with any vendor's EULAs however.

Federico Biancuzzi is freelancer. In addition to SecurityFocus he also writes for ONLamp, LinuxDevCenter, and NewsForge.

This article originally appeared in Security Focus.