Node JS scalable programming

I have been doing research on how to utilize many machines to complete a task faster using node js. It is a rendering project that I’m doing where I need to render several hundred images in good quality. So obviously the way to do it is to divide up the task into smaller pieces, so that’s exactly what I decided to do.

For all kinds of parallel programming I prefer using Node JS, because of it’s async model and the ease with which you can run code remotely. JavaScript can be easily stringified like this: var foo = funciton (){}; foo.toString(). And then sent over the network. So this way you can easily send cpu intensive work tasks to other machines and have them execute remotely and then return the result.

As long as you have a machine that has SSH installed, it is fairly easy to use node js ssh2 module and ssh to each of the worker machines, run a script remotely and then combine the results into the final work. In this case it was blender files that had to be rendered. And since blender is scriptable in python, it is also fairly easy to automate the whole render process.

So the way to distribute a render task to a cluster of computers is as follows:

• write python script that takes render parameters such as frame to render etc. this script will be invoked from javascript
• write a node js program that ssh:s into each of the machines and fires up a blender process rendering a fragment of the work
• when the blender process completes, grab the result and store it on the master node.
• repeat.