Node’s social pariahs

I learned a new expression on the node mailing list this week: social pariahs. The node.js police is after them, and it looks like I’m on the black list. I should probably take it easy and just “roll in my grave”, like Marcel did :-).

But Mikeal followed up with a blog article and I’d like to respond. Unfortunately, comments are turned off on his blog so I’m responding here (BTW, I wonder how comments work with horizontal scrolling).

Compatibility

I’ll be quick on this one. Yes, compatibility is very important and you need some rules if you want to build a lively ecosystem. The module system and the calling conventions are key. I learned this 25 years ago, when designing APIs on VAX/VMS. VMS had this great concept of common calling conventions which made it possible to link together modules written in different languages. Nothing new under the skies here.

Promise libraries are problematic in this respect because they promote a different API style for asynchronous functions. The standard callback(err, result) pattern is replaced by a pair of callback and errback, plus an optional progress callback, with different signatures. So you need wrappers to convert between the two API styles. Not a problem today as the wide majority of node.js libraries stick to node’s callback style but it could cause fragmentation if promises were to gain momentum.

Streamline.js is a good node.js citizen

Mikeal is quite vocal against streamline.js but I doubt that he has even read the README file. He is missing some very important points:

Streamline is not a library, it is a language tool, a precompiler.

Streamline is fully aligned on node’s callback convention.

Streamline is not trying to disrupt the ecosystem, it is trying to help people consume compliant code, and also produce compliant code.

To illustrate this, let me go back to the example that revived the debate this week on the mailing list. As I wrote in my post, streamline lets you chain the 3 asynchronous calls in a single line of code:

function computeAsyncExpression(_) {
  return (Object1.retrieveNum1(_) + Object2.retrieveNum2(_)) /  Object3.retrieveNum3(_);
}

The streamline preprocessor transforms this into (*):

function computeAsyncExpression(cb) {
  if (cb == null) return __future(computeAsyncExpression, 0);
  Object1.retrieveNum1(function(err, v1) {
    if (err) return cb(err);
    Object2.retrieveNum2(function(err, v2) {
      if (err) return cb(err);
      Object3.retrieveNum3(function(err, v3) {
        if (err) return cb(err);
        cb(null, (v1 + v2) / v3);
      });
    });
  });
}

(*) actual code is a bit different but the differences are irrelevant here.

So the computeAsyncExpression function generated by streamline is more or less what the OP posted on the mailing list. It is a regular node.js function with a callback. You can call it like any other node.js API that you would have implemented directly in JavaScript with callbacks.

Streamline.js does not try to enforce a new API style; it just helps you write functions that conform to node’s callback conventions. And for lazy people like me, writing one line instead of 10 is a big win.

I did not talk about the first line in the generated function:

  if (cb == null) return __future(computeAsyncExpression, 0);

This is not a standard node.js pattern! What does it do?

If you pass a null or undefined callback to a standard node API, you usually get an exception. This is considered to be a bug and you have to fix your code and pass a valid callback.

Streamline handles this case differently, by returning a future instead of throwing an exception. The returned future works very much like a promise but it does not come with a new API pattern. Instead, a streamline future is a function that takes a regular node.js callback as parameter. You typically use it as:

  var future = computeAsyncExpression(null);
  // code that executes in parallel with computeAsyncExpression
  ...
  // now, get a result from the future
  future(function(err, result) {
    // async computation is over, handle the result
  });

Streamline is not introducing a disruptive API pattern here. It is leveraging the existing callback pattern.

So far so good but streamline also supports a fibers mode, and experimental support for generators. Is this still aligned on node’s callback convention?

The answer may seem surprising but it is a YES. If you precompile the computeAsyncExpression(_) function with the --fibers option, what you get is still a regular asynchronous node.js function that you can call with a regular callback. This function uses fibers under the hood but it retains the standard callback signature. I won’t explain the technical details here because this would drive us too far but this is how it is.

And when generators land into V8 and node, it will be the same: streamline will give you the option to use them under the hood but still produce and consume standard callback APIs!

Libraries and Applications

The second point I wanted to discuss in this response is the distinction between libraries and applications. The node.js ecosystem is not just about people who publish modules to NPM. There are also lots of people who are building applications and services with node.js. Maybe they do not directly contribute to the ecosystem because they do not share their code but they contribute to the success and visibility of the platform.

I did not write streamline.js because I wanted to flood NPM with idiosyncratic modules. I wrote it because I was developing an application with a team of developers and I wanted application code that is robust, easy to write and easy to maintain. I wrote it because we had started to develop our application in callback style and we had produced code that was too convoluted and too fragile. Also we had reached the end of our prototyping phase and were about to move to a more industrial phase, and the learning curve of callbacks was just too high.

If I were in the business of writing NPM modules, I would probably think twice before writing them with streamline: there is a slight overhead because of some of the comfort features that streamline gives you (robust exception handling, TLS-like context, long stack trace); it is a different language, like CoffeeScript, which may rebuke people who want to fork, etc. I would probably use it to write drivers for complex legacy protocols (we are doing this internally in our project) but I would probably stick to raw callbacks for creative, very lightweight modules.

But I’m not writing NPM modules; I’m writing applications and services. And if I post a link to streamline to the mailing list it is because I think that this tool may help other people who are trying to write applications and who are running into the problems that we ran into more than 2 years ago: robustness, maintainability, learning curve, etc. To plagiarize Mikeal:

I feel really bad for people that ask incredibly simple questions on this list and get these incredibly complex answers!

I may have been too vocal on the mailing list at some point but I’m trying to be much more discreet these days. The streamline ecosystem is not very large but the feedback that I get is very positive. People like the tool and it solves their problem. So I don’t feel bad posting a link when the async topic comes back to the mailing list. Maybe a tool like streamline can help the OP solve his problem. And even if it does not, it won’t hurt the OP to take a look, discover that there is not one way of dealing with async code. He’ll learn along the way and make up his own mind.