More like this

Software

Arrow

Developer

'No regrets' says chap who felled JavaScript's Jenga tower – as devs ask: Have we forgotten how to code?

NPM republishes unpublishing rules

Analysis The programmer who sparked a brief meltdown in the JavaScript world last week says he has no regrets – and that it should be a learning experience for the community.

Indeed it has been: NPM, the popular package manager at the heart of last Tuesday's kerfuffle, has changed its systems to prevent another collapse of JavaScript's Jenga tower of dependencies.

Announced today, NPM will now no longer allow developers to automatically unpublish an open-source module if the package is older than 24 hours. If a coder wishes to withdraw their module, he or she must ask NPM's support team for approval, and the module will be removed if doing so does not break other installations.

"If a publisher unpublishes a package that others depend upon, this breaks every downstream project that depends upon it, possibly thousands of projects," said NPM's Ashley Williams.

"Last Tuesday’s events revealed that this danger isn’t just hypothetical, and it’s one for which we already should have been prepared. It’s our mission to help the community succeed, and by failing to protect the community, we didn’t uphold that mission. We’re sorry."

To recap: last week, programmer Azer Koçulu withdrew all of his 273 open-source JavaScript modules from the NPM public registry after one of his packages, called kik, drew the attention of lawyers representing Kik, the instant-messaging app. Kik wanted Koçulu to rename his module, he refused, so Kik complained to NPM. The registry sided with Kik and transferred the module to the app's owners.

Koçulu was furious and unpublished his NPM modules in protest. This open-source haul included left-pad, an 11-line JavaScript function that pads out the lefthand side of strings with spaces or zeroes.

This snippet was relied upon by widely used frameworks including Node, React, Babel, and thousands of projects worldwide: it is downloaded about 16 times a minute or 23,000 times a day. When the code disappeared from the NPM registry, software depending on it couldn't find it and download it, and thus broke. This left thousands of developers unable to install, build or deploy their web applications for a few hours.

The left-pad module was quickly taken over by a new maintainer, who republished the source code. Due to NPM's rules at the time, the code emerged with a new version number; projects and frameworks specifically requesting the previous version, 0.0.3, were still unable to fetch the dependency, and thus remained broken.

In the end, NPM made the unprecedented decision to resurrect v0.0.3 to end the chaos. Now it's decided that in future, heavily depended-upon modules cannot be automatically removed by developers. Programmers must petition NPM, which will try to talk them out of doing anything rash.

Here's an example scenario given by NPM:

Brenna is a maintainer of a popular package named “supertools”. Supertools has 3 published versions: 0.0.1, 0.3.0, and 0.3.1. Many packages depend on all the versions of supertools, and, across all versions, supertools gets around 2 million downloads a month.

Brenna does a huge refactor and publishes 1.0.0. An hour later, she realizes that there is a huge vulnerability in the project and needs to unpublish. Version 1.0.0 is less than 24 hours old. Brenna is able to unpublish version 1.0.0.

Embarrassed, Brenna wants to unpublish the whole package. However, because the other versions of supertools are older than 24 hours Brenna has to contact support@npmjs.com to continue to unpublish. After discussing the matter, Brenna opts instead to transfer ownership of the package to Sarah.

"This policy is a first step towards balancing the rights of individual publishers with npm’s responsibility to maintain the social cohesion of the open source community," added Williams.

Koçulu – a self-taught software engineer from Turkey now living in Oakland, California – told The Register: "Regrets? Of course not. In the long term, the community will benefit from this experience a lot."

The vulnerable Jenga tower of dependencies revealed by the left-pad incident provoked a lot of soul searching in the JS developer community. As far as Koçulu is concerned, though, the blame lies in NPM's handling of the matter.

"I think the level of dependencies doesn't matter," he said. "The problem is with a widely used open-source tool, NPM, that suddenly becomes someone's private company, and they want to make it a business. When a company like Kik approaches, they wanna do their best to serve them. They forgot their whole business depends on people's unpaid efforts."

Kik has since apologized for the way it approached Koçulu (its legal eagle emailed him: "We don’t mean to be a dick about it ... our trademark lawyers are going to be banging on your door and taking down your accounts and stuff like that"). The instant-messaging biz said it wanted to publish its own open-source Kik module, which would collide with Koçulu's kik code, which simplifies the creation of new projects.

When code reuse turns ugly

According to Perl granddaddy Larry Wall, there are three great virtues of a programmer: impatience, hubris, and laziness. Impatience makes people write code that just gets stuff done quickly. Hubris makes people write high-quality code that they can be proud of.

Laziness is the virtue key to this affair: "Write labor-saving programs that other people will find useful, and document what you wrote so you don't have to answer so many questions about it."

Laziness encourages developers to reuse other developers' code, especially if it is good quality software. It's the reason why no one sane reimplements their own printf() or encryption algorithm or 3D engine in every one of their programs; there are libraries full of useful code to do all that. It saves time and it prevents the introduction of new bugs. If someone improves a library, then everyone wins by pulling in this shared code.

It's easy to see how JavaScript's Jenga tower was constructed. Lots and lots of concise and time-saving open-source modules, all easily located and installed by NPM's free tools, depended upon by larger modules, which are depended upon by even larger projects. As we saw, removing one of these building blocks caused the structure to disintegrate.

But at what point does code reuse stop being a good idea and become a dysfunction? Maybe it's when a software community's dependencies are fractured into tiny shards, spread over the carpet and furniture, like a martini glass after it is hurled at a wall in a cliched row between warring parents. Wait, too much info?

Is it possible that we've become too lazy? Rather than write one-line functions, folks are pulling in outside code, and thus overly relying on dependencies. Here's some widely used open-source NPM modules that caught our eye:

isArray

This returns true if a variable is an array. It is four lines long, and is copyright Julian Gruber.

var toString = {}.toString;

module.exports = Array.isArray || function (arr) {
  return toString.call(arr) == '[object Array]';
};

isArray was downloaded 18,040,936 times in the past month, or seven times a second in the past day.

Slash

This converts Windows-style file paths to Unix-style file paths. It is nine lines long and is copyright Sindre Sorhus.

module.exports = function (str) {
        var isExtendedLengthPath = /^\\\\\?\\/.test(str);
        var hasNonAscii = /[^\x00-\x80]+/.test(str);

        if (isExtendedLengthPath || hasNonAscii) {
                return str;
        }

        return str.replace(/\\/g, '/');
};

Slash was downloaded 2,469,841 times in the past month, or nearly once a second over the past day.

Pretty-bytes

This is a 21-line snippet that converts an integer representing a number of bytes into a human-friendly string, eg: prettyBytes(1337) produces "1.34 kB". It is, again, copyright Sindre Sorhus.

var numberIsNan = require('number-is-nan');

module.exports = function (num) {
        if (typeof num !== 'number' || numberIsNan(num)) {
                throw new TypeError('Expected a number, got ' + typeof num);
        }

        var exponent;
        var unit;
        var neg = num < 0;
        var units = ['B', 'kB', 'MB', 'GB', 'TB', 'PB', 'EB', 'ZB', 'YB'];

        if (neg) {
                num = -num;
        }

        if (num < 1) {
                return (neg ? '-' : '') + num + ' B';
        }

        exponent = Math.min(Math.floor(Math.log(num) / Math.log(1000)), units.length - 1);
        num = Number((num / Math.pow(1000, exponent)).toFixed(2));
        unit = units[exponent];

        return (neg ? '-' : '') + num + ' ' + unit;
};

Pretty-bytes was downloaded 1,582,973 times in the past month, or 34 times a minute in the past day.

The isArray module was highlighted by David Haney, who last week pointedly asked: have we forgotten how to program? Ironically, Haney is an engineering manager at Stack Overflow, a website that developers routinely scour for code to copy when they get stuck on a problem.

"I get the impression that the NPM ecosystem participants have created a fetish for micro-packages," Haney noted. "Rather than write any functions or code, it seems that they prefer to depend on something that someone else has written ... In my opinion, if you cannot write a left-pad, is-positive-integer, or isArray function in five minutes flat (including the time you spend Googling), then you don’t actually know how to code.

"Take on a dependency for any complex functionality that would take a lot of time, money, and/or debugging to write yourself. But, for the love of all that is programming, write your own bloody basic programming functions. Taking on dependencies for these one-liners is just nuts."

Haney has a point. The stability of many projects unnecessarily relies on lots of teeny-tiny functions masquerading as packages. Pretty-bytes is pretty useful, but it can be written in five minutes. The same goes for Slash and isArray. Yes, code reuse is a virtue, but dependency addiction is a sin.

If the real problem is a hyper-fractured, super-lazy ecosystem, then perhaps NPM's new hardline policy on unpublishing modules will bake in some glue. In a roundabout way, it will form a standard library the JS world needs.

We just hope its support team can cope with the extra handholding it will need to do. And perhaps some programmers will take an extra couple of minutes to write a simple function rather than import yet another module. ®

PS: NPM's tools suffered from a security flaw that was fixed over the weekend. It allowed a malicious package to execute a script during installation to include itself in a new package that it then published to the registry, and to other packages owned by that user – a worm-like module was possible, in other words.

Sponsored: 2016 Cyberthreat defense report