TJ leaving node.js

Posted on July 5, 2014 by Bruno Jouhier

I just saw the news. TJ Holowaychuk, one of node’s important and respected contributors is leaving node.js for Go. Of course, this is not good news, especially for people like me who have invested a lot into node.js and have bet an industrial project on it.

Why is TJ leaving? Part of it has to do with the intrinsic attractiveness of Go. But a large part is related to deficiencies on the node side. Usabililty and the lack of robust error handling come first:

Error-handling in Go is superior in my opinion. Node is great in the sense that you have to think about every error, and decide what to do. Node fails however because:

  • you may get duplicate callbacks
  • you may not get a callback at all (lost in limbo)
  • you may get out-of-band errors
  • emitters may get multiple “error” events
  • missing “error” events sends everything to hell
  • often unsure what requires “error” handlers
  • “error” handlers are very verbose
  • callbacks suck

TJ also complains about APIs, tooling, lack of conventions:

Streams are broken, callbacks are not great to work with, errors are vague, tooling is not great, community convention is sort of there, but lacking compared to Go. That being said there are certain tasks which I would probably still use Node for, building web sites, maybe the odd API or prototype. If Node can fix some of its fundamental problems then it has good chance at remaining relevant, but the performance over usability argument doesn’t fly when another solution is both more performant and more user-friendly.

I have been supervising a large node.js project at Sage. We started 4 years ago and we have faced the issues that TJ mentions very early in our project. After 6 months of experimentation in 2010, I was seriously questioning the viability of node.js for our project and I was contemplating a backtrack. The reasons were precisely the ones that TJ gives today: usability, maintainability, robustness.

Yet, we went ahead with node.js; we put more and more people on the project and we successfully released a new version of our product last month, with a new web stack based on node.js. Our developers are very productive and generally happy to work with node.js.

Why?

Simply because the problems that TJ is mentioning don’t apply to us, and to others who have chosen the same approach:

  • Error handling and robustness are not issues for us. We are writing all our code with streamline.js. This lets us use good old structured exception handling. IMO this is even better than Go because you don’t have to check error codes after every call.
  • We never get duplicate callbacks; callbacks don’t get lost; errors are always reported in context, … All these problems are simply gone!
  • Debugging works and exceptions have understandable stacktraces.
  • We use an alternate streams library, based on callbacks rather than events, which keeps our code simple, robust and easy to understand.

So let us not throw the baby with the bath water. The problems that TJ puts forwards are very real but they are not insurmountable. You can write robust, elegant and maintainable node.js code today!

Maybe it is time to reconsider a few things:

  • Stop being stubborn about callbacks and push one of the alternatives: generators, fibers, preprocessors (a la streamline) (*). Probably not for the core code itself because of the performance overhead, but as an option for userland code.
  • Investigate alternatives for the streams API. Libraries like Tim Caswell’s min-stream or my own ez-streams should be considered. My experience with ez-streams is that a simple callback-based API makes a huge difference on usability and robustness (**).

(*) I left promises out of the list. IMO, they don’t cut it on usability.
(**) ez-streams is partly implemented with streamline.js, which will probably be a showstopper for some but its API is a pure callback API and it could easily be re-implemented in pure callbacks.

As I said in my intro above, I have a very strong investment in node.js and I really want the platform to continue to grow and succeed. Three years ago, node.js was the coolest platform because of the unique blend of JavaScript and asynchronous I/O. But there are alternatives today and people are asking for more, especially on usability and robustness.

The problems raised by TJ cannot be ignored.

This entry was posted in Uncategorized and tagged . Bookmark the permalink.

7 Responses to TJ leaving node.js

  1. jstuartmill says:
    July 5, 2014 at 21:25

    Thank you for this. One of the most coherent pieces I’ve seen on TJ’s departure.

    Reply

  2. Pingback: Crisis in the House of Node.js? | Rambling Mind Blasts

  3. Pingback: Should I Node or Should I Go? » Stories from a Software Tester

  4. Dean Chen (@DeanChen) says:
    December 20, 2014 at 07:14

    Can you please elaborate on the usability problems with promises?

    Reply
    • Bruno Jouhier says:
      December 20, 2014 at 12:24

      Let’s take the benchmark from http://strongloop.com/strongblog/node-js-callback-hell-promises-generators/.

      Here is the Sync version:

      var fs = require('fs');
var path = require('path');

module.exports = function(dir) {
  var files = fs.readdirSync(dir);
  var stats = files.map(function(file) {
    return fs.statSync(path.join(dir, file));
  });
  var largest = stats.filter(function(stat) {
    return stat.isFile();
  }).reduce(function(prev, next) {
    if (prev.size > next.size) return prev;
    return next;
  });
  return files[stats.indexOf(largest)];
};

      The streamline.js version:

      var fs = require('fs');
var path = require('path');

module.exports = function(dir, _) {
  var files = fs.readdir(dir, _);
  var stats = files.map_(_, function(_, file) {
    return fs.stat(path.join(dir, file), _);
  });
  var largest = stats.filter(function(stat) {
    return stat.isFile();
  }).reduce(function(prev, next) {
    if (prev.size > next.size) return prev;
    return next;
  });
  return files[stats.indexOf(largest)];
};

      And now the promise version:

      var fs = require('fs');
var path = require('path');
var Q = require('q');
var fs_readdir = Q.denodeify(fs.readdir);
var fs_stat = Q.denodeify(fs.stat);

module.exports = function(dir) {
  return fs_readdir(dir).then(function(files) {
    var promises = files.map(function(file) {
      return fs_stat(path.join(dir, file));
    });
    return Q.all(promises).then(function(stats) {
      return [files, stats];
    });
  }).then(function(data) {
    var files = data[0];
    var stats = data[1];
    var largest = stats.filter(function(stat) {
      return stat.isFile();
    }).reduce(function(prev, next) {
      if (prev.size > next.size) return prev;
      return next;
    });
    return files[stats.indexOf(largest)];
  });
};

      The Sync and streamline versions are both 16 lines of code. The developer uses the JS language to express his control flow: the steps are statements separated by semicolons; values returned by functions that do I/O are just “returned”.

      The promises version is 26 lines of code. The sequence of steps is not expressed with basic language constructs (statements separated by semicolons) but with special APIs (then, Q.all). The callback hell pyramid is avoided but there is still a fair amount of callback noise. As values cannot be “returned” directly, we have 3 extra callbacks just to pass values around.

      It may not seem like a huge difference on this simple example but when you have a large project, it makes a big difference. Put yourself in the shoes of someone who is reading lots of code like the above: how long will it take him/her to understand the streamline.js version? How long to understand the Q version? Do the extra 10 lines and callbacks help him/her understand what the code does? It’s not just about writing the code, it’s also about being able to quickly grasp what a piece of code does. And, IMO, the less noise the better.

      Now I’d like to soften this: promises alone don’t cut it on usability. But the combination of promises and async/await, which is being considered for ES7, will cut it because it will allow values to be “returned” directly from async calls into an existing scope.

      Reply
  5. kiliAgigorm says:
    July 11, 2020 at 03:34

    hi there!

    Reply

Leave a comment