Original URL: http://www.theregister.co.uk/2007/02/07/stefan_esser_interview/

PHP security from the inside

Stefan Esser gets set for the month of PHP bugs

By Federico Biancuzzi

Posted in Security, 7th February 2007 11:47 GMT

Stefan Esser is the founder of both the Hardened-PHP Project and the PHP Security Response Team (which he recently left). Federico Biancuzzi discussed with him how the PHP Security Response Team works, why he resigned from it, what features he plans to add to his own hardening patch, the interaction between Apache and PHP, the upcoming "Month of PHP bugs" initiative, and common mistakes in the design of well-known applications such as WordPress.

Could you introduce yourself?

Stefan Esser: I am an independent German security consultant and application developer. For about five years I have contributed dozens of advisories about security holes in software like CVS, Subversion, Linux, NetBSD, MySQL, PHP, Samba and even the Microsoft Xbox to the security community. Six years ago I started working with PHP and became one of its core developers. Due to my in-depth knowledge of the PHP source code, I found most of the vulnerabilities in PHP that were reported during the last years. I am the founder of the Hardened-PHP Project and the PHP Security Response Team that I recently left.

Why did you start the Hardened-PHP Project?

Stefan Esser: The Hardened-PHP Project was founded in 2004 in response to a number of security bugs I found in the PHP source code. The idea of a hardened version of PHP was actually much older, but it was never implemented until 2004.

The problem I saw with PHP was that often some new feature was hacked into it in a dirty way without thinking about the consequences for other areas of the code. Additionally, after having fixed a number of remote exploits in CVS versions of PHP before they made it to release versions, I stopped trusting the code at all. Therefore, the low-level protections like canaries, safe_unlink were implemented. A

dditionally it seemed a good idea to stop remote URL includes by completely forbidding them. The HTTP response splitting problem was also killed at the low level by stopping newlines in PHP's header() function. During the time more and more features made it into Hardened-PHP. [That is] until I was ordered by the PHP Group to rename my "fork" because of the PHP license. Because the PHP license actually contains such a paragraph the Patch was renamed into the Hardening-Patch for PHP, although I still don't get why they attack a PHP security extension in such a way. Later that year the Hardened-PHP Project became bigger, when two german security researchers from the PHP community joined the team. We started offering PHP security audits and PHP security courses. The two other team members finished their german PHP Security book and gave several PHP Security talks at various PHP conferences.

What is Suhosin?

Stefan Esser: Suhosin is the successor of the Hardening-Patch for PHP. Suhosin is a South Korean word and means something similar to "guardian angel". The idea behind Suhosin was to split the functionality of the Hardening-Patch into two parts. The Zend Engine protection, that protects the core from buffer overflows and that can only be implemented as a patch, and all the other protections that can be implemented in a PHP extension.

Suhosin contains lots of features that fix the misbehaviour of PHP that could result in security vulnerabilities. Usually these protections are transparent to the code. A lot of code had to be redone, because of the requirement that we did not want to break binary compatibility with PHP anymore and because it is a lot trickier to implement some of Suhosin's features from within an extension.

The goal was to have two independent parts that can be used alone or in combination, so that the user can decide for himself if he wants the core protection or just wants the high level protections. However, even when both parts are used, the binary compatibility with normal PHP is not broken, such that it is now possible to run third-party binary PHP extensions, like the ZendPlatform. Additionally this now allows Linux distributions to easier integrate Suhosin. And for most of them there are already official or unofficial packages available.

In short: Suhosin is now easier to use for the end user and easier for us to develop. Suhosin already contains many features like transparent cookie or session data encryption, experimental heuristic SQL injection detection, URL include blocking, mail() header injection blocking, basic variable filtering and a lot more. Many new features are planed for 2007. However we are currently working on a better documentation that explains all protections in-depth.

What plans do you have for Suhosin?

Stefan Esser: There are a bunch of features on the "to do" list for Suhosin. The development version currently contains some features not yet released to the public, like a simple heuristic to detect SQL injection attempts. It is planned that future versions of Suhosin will be able to learn behaviour patterns, like valid SQL queries, valid request parameters and to warn or block when a violation occurs. There are also plans to add mod_security style filters, with the major difference being that the filtering will be using the same parsing routines as PHP and therefore it will not be vulnerable to all the mod_security bypass tricks.

You founded the PHP Security Response Team. How does it work?

Stefan Esser: Well the real story is that when I first reported security holes in PHP there was no security@php contact, and everything had to go through the PHP group. I considered this a bad process and convinced the PHP group to create a dedicated PHP Security Response Team.

Basically, the PHP Security Response Team is nothing more than a mailinglist for a few core developers, and it receives security reports sent to security@php dot net.

There is no active manual auditing process inside PHP and I do not see how there could be one, because PHP developers are not security experts. While there is the Coverity source code scanner and there are a few people among the PHP developers that search for security holes, they only find minor things from time to time. Bigger security bugs in PHP are usually only found by third-parties or by me. However sometimes after a bug report, someone among the developers searches for similar code areas and finds similar bugs.

The other problem is that among the (maybe) 10 members of the mailing list, there are only two or three that do something like writing back to the reporters or trying to fix the code. Most of the others are just reading and giving comments from time to time. For example, saying that they do not want to have test exploits for fixed bugs in the PHP source code.

Why did you resign from the PHP Security Response Team?

Stefan Esser: First of all I did not suddenly resign, the response team knew I was very unhappy with the way things were dealt with, and I questioned the sense of the Security Response Team several times before I actually left. I resigned from the team for many reasons. First, I dislike the ways bugs are handled:

  1. Fixes get committed to public CVS (sometimes even quickly) => so that everyone can see them.
  2. It usually takes weeks or months until the next release [to fix the bug].
  3. No security updates [are released], only complete new releases (minor PHP revisions quite often break some functionality, usually by mistake).
  4. No test cases for fixed security bugs.
    • Sometimes the fix is not working correctly (for example, when running on 64-bit systems).
    • Bug gets reintroduced later.
  5. Bugfixes are often done sloppily.
    • Sometimes it fixes only the symptoms (the same thing stays exploitable through other paths).
    • Bugfixing code introduces new security holes.
  6. No information policy => if a bug was found by another developer, you will never see more than a single line of problem description anywhere.

Secondly, during the last year I was quite often called an immoral traitor or other things from PHP (core) developers for disclosing security holes in PHP to the public. One does not feel like a team member when the PHP Security Team/PHP Group has nothing more to say about this than: "Well that is your problem, not ours".

And the third point is that I strongly disagree with some of the members of the team about the definition of a security expert. My personal opinion is that there are far too many people out there that call themselves Web Applicaton/PHP security experts and write books/give talks about PHP Security, while they only know basic XSS and SQL Injection stuff (actually when you look at various PHP Security Talks you will often see nothing PHP-specific in them, with the possible exception of remote URL includes).

Browsing the changelog I see fixes for segfaults, memory leaks, and problems with missing checks. Maybe some of these bugs could be exploitable, but they will surely stop some scripts from working reliably. I am wondering how is it possible that a big project such as PHP can't spot functions that stop working correctly, but worked in previous releases. Doesn't PHP use any regression testing?

Stefan Esser: One of the biggest problem with this is that Release Candidates of PHP are not tested by the majority of people. For the PHP developers (and of course, for every vendor) it is not possible to test in all configurations or test them in all code areas. Sometimes the bugs are hidden in obscure extensions that are only used by a handful of people, or only in a code path that is usually not taken. The PHP source code has grown too fast and, although many regression tests exist, they are far from covering all code paths. (However there are currently some people working on getting more code covered by the test cases).

Do you think that security bugs should be fixed with separate patches instead of forcing users to upgrade to the latest minor release of PHP Core that fixes them, but which also include new features and potentially new/different bugs?

Stefan Esser: From my point of view, security bugs should be fixed by releasing minor releases that only contain the security bug fixes. Releasing only patchfiles is too complicated for a large number of users. PHP did both [of these] in the past (releasing new minor releases and a seperate patchfile), but due to the large number of security vulnerabilities that were reported, this was stopped, because that resulted in too many minor releases.

How do PHP5's security features compare with PHP4's?

Stefan Esser: Well, the PHP5 core does not have some of the weaknesses of the PHP4, like small 16-bit reference counters that can easily overflow. Additionally, more people use PHP's Object-Oriented features (although basic Object-Oriented features were present in PHP4). The use of objects makes some code easier to read, and fewer global variables are used which makes register_globals less dangerous.

PHP5 comes with better hashing functions by default (PHP 4 had only SHA1 and MD5). On the other hand some of the old PHP code is now vulnerable, because some things have changed. For example, the magic_quotes_gpc feature no longer includes the _SERVER/_ENV variables. Code that previously thought it does not need to prepare these, for example the USER_AGENT string, is now vulnerable.

In short I think PHP5 brought a few new things that improved security, but on the other hand it changed some internals that could have negative side-effects on old code. And of course the code base is now larger and therefore more bugs are possible.

Apache and PHP are a very common bundle. Is there any problem specific to their interaction?

Stefan Esser: Well, because Apache fails to open several file handles/sockets with the close on exec flag it is, for example, possible to takeover the httpd socket and or read/write the logfiles. Additionally, things like the SSL private key are in the memory of the httpd daemon and can therefore be leaked (this is also a problem for those following the recommendation to store the database credentials inside the httpd.conf, because I am pretty sure they can be read too).

The problem is that mod_php shares the address space with the apache daemon. Therefore any security bug in PHP that allows reading the memory will completely leak information stored by apache or other modules, in this case mod_ssl. This is not PHP specific. It is a general problem. If you find a bug in mod_python, the same should be possible with python code. It is more a problem of modules running in the same address space. The moment you are using a CGI model this is no longer an issue.

From what I've heard Apache 2 runs well with PHP. The only problem with Apache 2 is that some of its MPM modules actually are multithreaded. The problem with PHP is that it links against a lot of third party libraries that are either thread safe or not. Normally the PHP core should be thread safe. But the moment a function is called that is provided by a third-party library, one can never be sure. Therefore Apache 2 is fine as long a non-threaded MPM is used. If you use a threaded MPM it is like russian roulette depending on the extensions you are using.

From a security point of view, what is the difference between running PHP as a web server module and as a CGI binary?

Stefan Esser: Oh, that is really simple. If you are running PHP as web server module, PHP code gets executed with the permissions of the user owning the web server process. Multiple VHOSTs will all run with the same privileges and can influence each other. There are things in PHP like safe_mode or open_basedir but they can be bypassed in many ways. Therefore an exploit against one of those VHOSTs can theoretically affect other VHOSTs. Additionally exploits against the PHP core will allow access to Apache's memory. If you are using mod_ssl this allows for stealing the private key for the SSL cert from Apache's memory from within a PHP script. (Data that is normally only accessible by root - Oh I think I will demonstrate this in the Month of PHP bugs).

If you are using PHP as CGI it is possible to let the PHP scripts of different VHOSTs be executed with different Unix users. Additionally things like chrooting the PHP process are possible...

Please tell us more about your 'Month of PHP bugs' initiative.

Stefan Esser: The Month of PHP bugs will take place in March 2007. Its goal is to make people and especially the PHP developers aware that bugs in PHP exist. While this sounds obvious for everyone on the outside, it is actually required. PHP has a very bad reputation when it comes to security, which is mostly caused by all the advisories about security holes in PHP applications. For some of the reported bug classes like SQL injection and XSS, this is quite unfair, because those can happen in any language. But Remote File Inclusions, vulnerabilities due to register_globals or other problems within the PHP engine (e.g. zend_hash_del_key_or_index bug) are fully to blame on the PHP language. Unfortunately this kind of thinking is not appreciated by the PHP developers and they continue to claim that PHP is not worse than other languages, and that only badly written PHP applications are the problem. The Month of PHP bugs will show however that a lot of bugs in PHP's own source code exist.

We will disclose different types of bugs, mainly buffer overflows or double free(/destruction) vulnerabilities, some only local, but some remotely trigger-able (for example, because they are in functions usually exposed to user input). Additionally there are some trivial bypass vulnerabilities in PHP's own protection features. Only holes within the code shipped with the default distribution of PHP will be disclosed. That means we will not disclose holes in extensions that only exist in PECL, while we are sure that those contain vulnerabilities, too. Most of the holes were previously disclosed to the vendor, but not all.

As a vulnerability reporter you feel kinda puzzled how people among the PHP Security Response Team can claim in public that they do not know about any security vulnerability in PHP, when you disclosed about 20 holes to them in the two weeks before. At this point you stop bothering whether anyone considers the disclosure of unreported vulnerabilities unethical. Additionally a few of the reported bugs have been known for years among the PHP developers and will most probably never be fixed. In total we have more than 31 bugs to disclose, and therefore there will be days when more than one vulnerability will be disclosed.

You said that some PHP contributors are not very security-wise developers. At the same time I can say that a lot of software written in PHP seems to have security problems, often because of bad design. Which bugs do you consider riskier?

Stefan Esser: I am quite sure that most servers are hacked through PHP application vulnerabilities and not through vulnerabilities in the PHP core. This might have been different five years ago when there was the quite easy exploitable fileupload overflow vulnerability. Therefore it is quite logical to say that a bug in an application is more dangerous. Additionally it is usually a lot easier to write an exploit for an include vulnerability or SQL injection than an exploit for a remote code execution vulnerability hidden in the core. Therefore the number of potential attackers is far higher for application vulnerabilities. It is so much easier to take over an admin account with a simple XSS vulnerability than to send carefully crafted requests to achieve a good heap layout and then trigger an overflow. Therefore I consider application vulnerabilities very risky.

On the other hand it should not be forgotten that many vulnerabilities like HTTP response splitting, mail header injection, url includes, and so on are only possible because of bad design decisions (security bugs) of PHP. Additionally there have been a number of core bugs like the zend_hash_del_key issue or the GLOBALS overwrite that introduced remote security holes into securely written applications. There have also been remote code execution vulnerabilities in PHP that allowed attacks against any server using PHP, but luckily they were not many. Knowing that there are tons of local vulnerabilities in PHP, I cannot trust any unknown PHP code because there is no way PHP can stop arbitrary machine code execution.

In short: if PHP script holes exist, an attacker will usually choose them to break into the server because this is much easier. Once he is able to execute arbitrary PHP code he can exploit the local holes to break out of open_basedir/safe_mode/disable_function restrictions and he has the same power any direct attack against the core has. On the other hand he may not need the local exploits because he is only after the SQL data.

How should we protect our SQL data?

Stefan Esser: Well Rule #1 is to only allow access to the data the web application really needs.

As long as PHP scripts are allowed to retrieve data and the Access Checks are done in PHP land there is nothing one can do about this. Stored Procedures would have to do the "security/user rights" checks itself. The moment someone is able to execute PHP code, you have lost. There are lots of functions in PHP that, for example, allow connecting to database servers directly by TCP. And even if those functions are deactivated in the configuration, PHP still has tons of local exploits that can be used to directly execute machine code.

Some people recommend putting the database credentials into the environment variables via the web server configuration, stored in files only readable by root. When the PHP script just uses these environment variables, it is not possible to see the credentials from other PHP code.

Suhosin on the other hand comes with an experimental feature that automatically prefixes a string to the username when it is sent to the database. This allows for using an empty username within the PHP script. Suhosin will prefix it with the username prefix that is configured within the VHOST. This allows two things. First of all, the full username is only known to the httpd.conf file and the database. Secondly, even if the full username leaks, another VHOST cannot use the same credentials because whatever username it supplies, it will always get it's own username-prefix prefixed. However in both cases all the database credentials can be stolen when someone gets access to the apache memory.

Have you ever looked at the WordPress source code? I'm asking this because when I did, a lot of time ago, I saw that they used just one SQL user with complete privileges. So every exploit could have deleted/copied all your data. The crazy thing is that WordPress is now one of the most used blogging platforms on the planet. If you'd like, you could instead comment on the security of some big/known software written in PHP that you had time to test...

Stefan Esser: Well, I released two WordPress advisories recently. From my point of view, WordPress is not well designed. This starts for example with the fact that they are escaping all input for the database in the beginning, and later when issuing the queries they just put variables directly into the query. The bug I released (charset conversion SQL injection) would not have been possible if they had chosen the more common design, to escape everything right before it is put into the query. Others might argue that they should better use prepared statements and variable binding, but WordPress has to be compatible with old MySQL databases and PHP installations that do not support this. Another problem of WordPress is that it is sooo user friendly that it spits out detailed error messages when a SQL query fails, such that a potential attacker can gain information about the query. This for example leaks the database table prefix.

The problem with many of these big PHP applications like WordPress and PHPBB is that they were started in the days when security was not taken so seriously, and from that day they have grown and grown. In many cases it would have been better to just rewrite them from scratch, but that is of course a lot of work and most people don't like the idea.

This article originally appeared in Security Focus.

Copyright © 2007, SecurityFocus

Federico Biancuzzi is freelancer. In addition to SecurityFocus he also writes for ONLamp, LinuxDevCentre, and NewsForge.