UK drone collision study didn't show airliner window penetration

Full report obtained by El Reg casts doubt on public test conclusions

An Airbus A320-series airliner cockpit. Pic: Shutterstock
An Airbus A320 cockpit. A testbed cockpit was used for some of the drone impact trials

Exclusive A British drone collision study used as evidence for the government’s flagship drone pilot registration law found UAVs pose less of a risk to airliners than government officials and trade unions have claimed.

The study, which the government refused to reveal in full despite being asked by industry and news media alike, is the key piece of supposedly scientific evidence backing its proposed Drone Bill.

That Parliamentary bill is due to be published in spring this year. It will create new criminal offences targeted at fliers of drones that weigh more than 250g who fail to register with the government and pass mandatory safety tests before using their craft. The risks caused by drones to manned aircraft were allegedly quantified by the study. The Register has obtained a redacted copy of the study and compared the public statements from its sponsors with what the research actually states.

A press release summary of the drone collision study, first published when the government revealed its registration plans last year, was used by airline pilots’ trade union BALPA as evidence of a “proven drone collision threat”.

Original caption: 'The picture below, taken during the testing, shows a larger hobbyist-class drone penetrating an aircraft windscreen.' This is not true

Original caption: 'The picture below, taken during the testing, shows a larger hobbyist-class drone penetrating an aircraft windscreen.'

However, as is evident from the full study, the only windscreen that was penetrated was penetrated by “components” in pre-testing. The photo appears to be a still from a series - beginning on p52 of the redacted study – showing a lighter drone impacting, and failing to penetrate, the window.

In reality, the full study – carried out on behalf of the government by defence R&D company Qinetiq and testing firm Natural Impacts – found that for airliners, the risk posed was far less alarming than both the union and the Department for Transport had claimed. Instead of penetrating cockpit windows, rigorous tests of drones launched against Airbus A320 windscreen panes, fitted to a testbed cockpit, found that drone-airliner collisions will crack but not penetrate such windows.

The Airbus A320 series, comprising the A318, A319, A320 and A321 aircraft, is one of the most widely-used airliner families in the world.

When it announced the drone registration scheme (PDF, 65 pages) the government said: “For airliners, the test results are more reassuring - only a much heavier drone of above around 2kg in weight would cause critical damage and only when airliners fly at higher speeds.”

BALPA was explicitly involved with the study “to assure safety by influencing requirements such as minimum design and operational standards for [remotely piloted air systems, i.e. drones].”

A trade union spokeswoman told us: “We stand by the characterisation we made at the time of the report’s publication. Drones pose a huge potential risk to aviation safety if not flown sensibly.”

The Department for Transport eventually replied to us after publication. We updated this article with their response at the bottom.

Cracking work

The one test scenario in which drone-like items did penetrate an airliner windscreen involved scrap cockpit window panes obtained by Natural Impacts being “loosely supported at representative angles with laboratory clamps” before having drone-like components fired at them, as part of a “de-risking” process before the actual tests. No complete drones were launched at these windscreens. Yet government officials cited these firings in their press release as if they were part of the contracted study, under the term “Airliner-B”.

An extract showing how the Airliner-B windscreen results were cited as if they were a real test

Extract from the government press release showing how the ‘Airliner-B’ smash-it-’n’-see firings were cited as if they were part of the scientific study

Drone researcher Ian Povey of Clear Vision Security told El Reg that this revealed the unscientific nature of the study.

“Somehow 'redundant' windscreens that were used for 'unplanned' test firing, which were 'loosely supported' at guesswork angles made it into the official summary report. The presentation of the Airliner-B results was on a par with all of the other windscreen results despite the fact that [they] weren't subject to the same scientific standards,” he said.

An outline of the Airliner-B "testing" can be found on page 23 of the full study (see link at bottom).

Povey described the study as “a damning indictment on the UK government's approach to drones, particularly hobbyist users,” adding that in his view, “it's based on flawed methods, bad science, inaccurate data, and with the intention of causing public hysteria to push through legislation.”

Drone manufacturer DJI’s corporate comms veep for EMEA, Barbara Stelzner, told us:

It is now clear that the DfT study did not accurately model how drones perform in mid-air collisions. Instead of testing drones, it relied on javelin-shaped devices built of stronger and less flexible material, arrayed in a straight line instead of a typical mass distribution. These choices ensured the collision testing would generate more damage and made its conclusions unreliable. This information was not disclosed to the public in DfT’s initial statements, to the detriment of legitimate research and the ultimate goal of public safety.

She added: “DJI calls on the DfT to release a full and unredacted version of the study’s data, so independent researchers can scrutinise it and evaluate whether European authorities can use it confidently as a basis for future regulation.”

Speeds and smashes

The study found that the only way to achieve penetration after tests of a 1.2kg drone had failed to break through an airliner cockpit window was to assemble a 4kg “projectile” and launch it at speeds of up to 350 knots (400mph/650kph). Precise speeds and other useful test methodology details were redacted from the full report seen by The Register, though its authors noted that the speed range was chosen to represent the difference between indicated airspeed (IAS) at sea level and at altitude, plus the closing speed of the drone and the airliner.

The drones were assumed to be travelling at speeds between 20kts and 50kts, the study’s authors added. On the face of it, this detail gives this single part of the study greater validity than a similar US report from late last year, which only modelled collisions at 250kts. That speed is significant because, in the vast majority of countries, 250kts is generally regarded as the aviation speed limit below 10,000ft, with exceptions.

The 4kg “projectile” (the term is the one used by the study’s authors) was made up of a number of loosely drone-related components, including arms from quadcopters – and, strangely, a full-sized SLR camera, something not found on commercially available camera drones. The projectile also incorporated balsa wood bracing to strengthen it and prevent it from disintegrating in the air cannon used to launch it at the windscreen.

Crucially, that wood bracing was not included in computer modelling of the impacts. The full study’s executive summary acknowledged this, with its author writing that “overly-conservative” results from computer modelling calibrated against practical tests may have come about because of “known simplification to the RPAS [drone] material models and suspected differences in the construction of the windshields from the supplied data. This is an area that is worthy of further exploration.”

No such caveat was in the government press release summarising the study.

Clear Security's Povey commented: "The smaller drone (1.2kg), typical of a DJI Phantom class, could be fired whole and produced no critical damage to the airliner windscreens. The 'drone' that supposedly caused serious damage to the windscreen isn't a drone at all, it's a projectile that was designed to be able to include some of the components of a drone that could be fired like an arrow from the cannon. All of the modelling and impact testing of this object have no correlation to an actual drone in flight."

The study also looked at what happens when drones collide with helicopter windscreens, covering types that are certified to withstand birdstrikes and types that are not. It found that both birdstrike-resistant windscreens and non-birdstrike-certified windows in helicopters (and, by extension, light aircraft) can be penetrated by drones, though precise details of the failure modes involved and so on are redacted from the study.

It is without question that helicopters are at great risk from drones colliding with their windscreens at speed, potentially causing life-changing injuries or worse to their pilots.

Although researchers set out to examine what happens when a drone flies into a helicopter’s tail rotor, for reasons that were redacted from the study report, no tests on this were carried out.

Comment: Tell us about it

The redacted study as obtained by The Register can be downloaded from our website as a 6MB PDF file (140 pages).

Readers will see that it contains such a heavy level of redaction that it is very difficult to draw any meaningful conclusions from it. James Bell, the Department for Transport’s “head of drones – programme and future strategy” told us in a letter that this is because terrorists could use the full study to help them crash an airliner.

Yet that risk, while very real, is not relevant here. If a terrorist wants to fly a drone into an airliner with the intent of causing a crash, the parts of it to target are obvious to anyone with a GCSE-level understanding of physics. The real issue is that, egged on by a trade union that wants to see the drone industry strangled, the British government wants to set the threshold for drone registration at an absurdly low level that covers a number of children’s toys. The 250g lower weight limit for registration was, as far as can be divined from the study report, simply not tested; the lowest weight category picked was 1.2kg, and while that drone smashed through helicopter windscreens, it did not penetrate an A320 windscreen.

This is not to downplay the concerns of airline pilots. Drones are, and will remain, a hazard to manned aviation unless flown responsibly. Mandatory safety testing and pilot registration will help keep the skies safe for all, as unpopular as those moves are. Few genuinely believe that registration and safety testing are bad things.

With that said, regulating toys off the back of a study whose methodology was suspect from the moment its summary was published, and the full version of which is being kept under tight wraps by officials, is concerning and does not build public confidence in the testing, the Department for Transport or the proposed new laws. The public has a right to expect that when public money is spent on justifying new legislation, the basis for doing so is sound and peer-reviewable.

Neither of those things are true with the drone collision study. The government must do the right thing and release the full study. ®

Update

The Department for Transport responded to our enquiries only after publication. A spokeswoman told us by email: “The safety of the public and manned aviation is our primary consideration in the use of drones. The materials and methods used in these tests were carefully selected and expert engineers wholly agreed that the results were reliable."

“The study findings will be used to improve our knowledge of the potential risks caused by drones and implement mitigations to ensure they are used in a safe manner.”

However, she denied that the government is using this study as a basis for legislating, insisting that the decision to legislate had already been made on the basis of a consultation issued in early 2017.

She also responded to our specific questions about the study. We reproduce those questions and the DfT's belated answers in full:

El Reg: Why was this additional, non-scoped, unscientific testing carried out outside the scope of the original contract?

DfT: It’s incorrect to imply that that the windscreens used or methods applied were sub-standard. The screens themselves were carefully selected to ensure that they were structurally sound and supported using clamps.

While the results from Airliner-B use a different methodology from the Airliner-A test results provided, they should not be ignored and were included in the interest of transparency. This test was part of the originally scoped work and was specifically designed to de-risk the test methodology.

El Reg: What was the purpose of including this wholly unsound testing in the press release as if it had been carried out in the same rigorous way as the Airliner-A tests? Did DfT consider the very real risk of misleading the public by doing so?

DfT: We firmly rebut the suggestion that any results we publish would be misleading. Several expert engineers from industry were consulted on the results and wholly agreed that they would have been the same if the windscreen was supported in a cockpit frame. In fact hiding these results would have been inappropriate.

El Reg: Why did the July press release not include the vital information that the Airliner-B windscreens were totally unsupported?

DfT: As described above, Airliner-B windscreens were not totally unsupported and given the expert views of the results it was agreed that it was not necessary to include this information in the summary report.

El Reg: The projectiles intended to represent 4kg-class drones were not only very different from actual drones, as the study reveals, but the actual testing also included wood and other packing which was not modelled, leaving the modelled collision effects open to challenge and criticism. Even though the executive summary of the study acknowledges this at page 3, the July press release does not include this caveat at all.

Was this because DfT did not want to acknowledge that the study failed to produce strong enough evidence justifying its plan to legislate?

DfT: The press notice accurately reflects the conclusions of the report, the results of which are supported by several industry experts and show a clear justification in legislating to improve safety in this area.




Biting the hand that feeds IT © 1998–2018