Automating Complex Workflows with Grunt Custom Tasks

Grunt Custom Task header

Grunt.js has come onto our web development scene pretty quickly. It’s only at 0.4.1 at the time I write this, but it seems to be a pretty standard part of many of our toolboxes already. It’s fairly common to see it in a web project wherein the developer may need to pull in JavaScript dependencies with Bower, minify some CSS and JS, build static HTML from templates, and even deploy the final project to the server.

You can find plenty of write-ups on that type of workflow online. I wrote something on it back in July, and even Flippin’ Awesome featured a great intro to Grunt at around the same time by author Mária Jurčovičová. My semi-tongue-in-cheek write-up was inspired by the fact that all the cool kids were using Grunt but the conversation around it wasn’t focused on what it could do, just that we should use it. My spidey sense didn’t tingle so much about the larger picture of what we could do with it, and I really only got a narrow view of a typical web workflow using it.

In this article I will go through how my view of Grunt came to change and how building a custom task for Grunt made me see all kinds of possibilities for the tool.

Background

Before we go on, let’s rewind about a year before any of us heard of Grunt. My pet project at the time was a music aggregation service that now runs my site Blastanova and compiles a playlist and zip file of songs I use to run my radio show on Codebass Radio (where Geek and Music Combine!). All of that was done in Node.js. In terms of tasks, the project involved scraping websites and RSS feeds, downloading music, transcoding YouTube to MP3, mapping the songs to Spotify links, Tweeting out newly discovered songs, and, well, those are very nontraditional tasks if you are in the Grunt world now. At the time, I wrote my own little taskrunner that worked fairly well.

Fast-forward to this month, and it seemed worthwhile to get my Node.js service for Blastanova caught up to speed with the newly crowned and beloved taskrunner, Grunt.js. It’s a relief to offload the task-running glue of my project to a massively used and iterated open-source project. I can use a bunch of convenience features that already exist like copying files, FTPing, reading JSON configurations, et cetera…and I can also build on them and create custom tasks unique to my project.

Now I have a new project for which I’m using Grunt.js. I’ll take this opportunity to share a task Grunt.js can perform that I find pretty darn awe-inspiring because of the awesomeness of the most well-known, cross-platform, free video utility available today. FFMPEG is used all over the place to convert a piece of audio or video from one format to another and/or inject metadata into them. Some folks will use it to transform video on the fly for you to watch if your platform doesn’t support a specific format. Me? I’m running a radio show, so my Grunt task needs to be the sort of thing where I send video in and get audio out.

So there it is, my current project problem set-up. Let’s talk about how to get it done!

Anatomy of a Custom Grunt Task

You can find and install a variety of handy Grunt tasks with npm, like file copying, minifying, et cetera. Each of us has specific needs, so we might need to crank out our own tasks.

Let’s refresh our memories and look at how to run an existing Grunt task we’ve grabbed from npm or GitHub. In a Grunt.js file, we will typically

  1. Load the task. For example: 
    grunt.loadNpmTasks('grunt-contrib-copy');
  2. Register the task: 
    grunt.registerTask('name_of_thing_I_call_from_command_line', [‘my_task_name’, ‘another_custom_task_to_call’, ‘another_task:a_subtask’]);
  3. Configure the task: 
    grunt.initConfig({
‘mytaskname’: {
            ‘asubtaskinmytask’: {
                config: ‘someconfig’,
        config2: ‘someotherconfig’
            },
}
});

Now that we remember how to call a task, let’s break these steps down in terms of writing our own task.

Anatomy of a Custom Grunt Task: Load the Task

Our first step was loading the task. The method we used was loadNpmTasks. This method looks up a folder named “grunt-contrib-copy” in our “node_modules” folder. The assumption here is that if we use npm install to bring in the “grunt-contrib-copy” task, it will end up in our “node_modules” folder, and that’s where Grunt should look to find any npm-related tasks.

We don’t want to go through all the hoopla of pushing our task onto Github and then registering it on npm. So we’re going to call grunt.loadTasks('allmytasks');.

When we do this, instead of looking in our “node_modules” folder, the method will look in the “allmytasks” folder I have in my project root. So now, what do we put in the “allmytasks” folder?

Anatomy of a Custom Grunt Task: Register the Task

This is where we really start cranking out our Node.js code. But don’t worry, we have a little bit of a template to follow:

module.exports = function(grunt) {
    grunt.registerTask('my_task_name', 'a nice, wordy description of my task', function() {
    grunt.log.writeln(‘The only thing my task does is write something to the log’);
    });
};

Keep in mind, we are writing this JavaScript inside the “allmytasks” folder in the root of our project. The name of the file we create doesn’t matter, just as long as it’s a JS file.

By virtue of loading our “allmytasks” folder and writing a custom JS file registering the “my_task_name”, our task is now available to run from our main GruntFile.js like so:

grunt.registerTask('name_of_thing_I_call_from_command_line', [‘my_task_name’]);

Now, this is great and all, but I tend to make quite a few tasks that all live in the same file and have a similar namespace. That’s why I like to use the multitask!

module.exports = function(grunt) {
    grunt.registerMultiTask('my_task_name', 'A nice task with two subtasks', function() {
        switch (this.target) {
            case ‘a_sub_task’:
                grunt.log.writeln(‘The only thing my subtask does is write something to the log’);
        break;

    case ‘another_sub_task’:
                grunt.log.writeln(‘This subtask writes something ELSE to the log’);
        break;      
        }
    });
};

With the multitask, we’re basically taking the target property of the incoming task and doing something different based on the contents of the string. We can call it in our Grunt.js file like this:

grunt.registerTask('name_of_thing_I_call_from_command_line', [‘my_task_name:a_sub_task’, ‘my_task_name:another_sub_task’ ]);

So far so good, right? We’ve successfully loaded and registered a custom task with two custom subtasks. The last piece of the puzzle is to configure our tasks. It’s not exactly necessary if we don’t require any configuration, but to make our tasks flexible and reusable, we should pop in some configuration.

Anatomy of a Custom Grunt Task: Configure the Task

In the interest of getting to a simple, real-world example, let’s include a simple multitask I have in my project. All it does is makes some directories if they aren’t already created. That way, if it’s a first run of the project, we can be sure the directories are there and ready to be used. My music aggregator and radio show is called “SharkAttack”, hence the “SA” initials.

module.exports = function(grunt) {
    grunt.registerMultiTask('sa', 'Misc Tasks', function() {
        switch (this.target) {
            case "init":
                grunt.log.write("SA - Init Task");
                for (var c in this.data.dirs) {
                    grunt.file.mkdir(this.data.dirs[c]);
                }
                break;
        }
    });
};

You might notice a couple of new things here. First, grunt.file.mkdir(). Yep! That’s a nice little convenience function in Grunt to make a directory.

Second, you’ll notice this.data. Previously, we talked about using this.target to know what the subtask was. Contained in the this variable are a bunch of other things, but here we use this.data to get our configuration values.

Meanwhile, in our Grunt.js file, our configuration looks like this:

grunt.initConfig({
"sa": {
init: {
                dirs: [ “/data/temp", “/media”, “/voiceovers”, “/showoutput”]
            }
            }
)};

Here, you’ll see that we’re passing our dirs (or directories) array right into our task…quite easily! And it’s all referenced from this.data.myconfigvariable within the registered custom task.

Asynchronous Tasks

One more thing before we get to the video! As you might imagine, it can take a significant amount of time to transcode, or convert, video. How long it takes depends on the file size, length, and other factors, but the point is that it’s definitely not instantaneous. Making directories, as Grunt does it, is synchronous. This means that when it hits that line of code to create those directories, the task pauses while it completes that line of code and then continues running when it’s done.

Asynchronous tasks, on the other hand, will keep running our code, even if something we kicked off takes a while. If we have an asychronous method in our project, Grunt really doesn’t like it unless we do a little extra work. For example, while we wait for our video transcoding to complete, Grunt will finish all the other stuff and just think everything is done. That one thing we’re waiting for in the background will just be killed!

So the goal here is to make Grunt wait for our custom task to finish if we have something asynchronous inside. To do this, we simply call the following when we start the task:

var done = this.async();

Next, we use that variable to signal the task is over when it’s time:

done.apply();

So for example:

module.exports = function(grunt) {
    grunt.registerTask('task', 'A Task Description’, function() {
            var done = this.async();
            var async = setInterval( function() {
done.apply();
    }, 1000);
        }
    });
}

In the above example, we paused our task for one second. But the great effect is that Grunt will wait for us to call it back and won’t shut down on us in the meantime!

Using FFMPEG to Convert a Video to MP3

Let’s step away from Grunt for a bit and do some vanilla JavaScript and Node.js. There are a number of FFMPEG wrappers out there that work with Node.js. I chose “ffmpeg-node”. At the time I was researching this project, a year ago, it was the most straightforward to use for me and functioned most similarly to FFMPEG on the command line using its exec function. The “ffmpeg-node” project has a few other convenience functions, but the exec call just spawns FFMPEG and sends the arguments directly over to the utility. When it’s done, the callback function passed in is fired.

Before we get to the exact syntax, I should tell you we have to install something and deal with one annoying legal snafu. We’re looking to output MP3. Unfortunately, the legal rights to decode and encode MP3 aren’t free! Any software or hardware used to play MP3 today likely pays a licensing fee for its ability to decode or encode the media. FFMPEG is free software and its creators don’t  want to deal with royalties. As a result, they allow us to download and use the LAME encoder to get around this snafu.

On my Ubuntu machine, I’ve installed LAME via the “libavcodec” plugin and FFMPEG by using:

sudo apt-get install ffmpeg libavcodec-extra-53

A little Googling reveals similar “brew” install instructions for Mac and other install instructions for Windows. Once that is all installed we should be able to use FFMPEG on our command line as well as in Node. In my Node project I use it like this,

var ffmpeg = require('ffmpeg-node');
ffmpeg.exec(["-i", "myvideo.mp4", "myaudio.mp3"], callback);

function callback(error, info) {
console.log(“Transcoding is done!”)
}

It’s a pretty straightforward argument set here. The -i signals that we’d like to supply an input file, and the last argument is the output file. FFMPEG is smart enough to know we want to convert the audio from the video because we supplied those file extensions.

Way too easy, right? Want to do something a little more complex? Let’s try injecting some metadata into our new MP3 file. Here’s a snippet to do that:

var ffmpeg = require('ffmpeg-node'),
ffmpeg.exec(["-i", "myfile.mp3", "-y", "-acodec", "copy", "-metadata", "title=mytitle", "myfile.mp3"], callback);

function callback(error, info) {
console.log(“Transcoding is done!”
}

Like I said, this one is a bit more complex. We still have our input file: “myfile.mp3”.  We still have our output file at the end: “myfile.mp3”. The -metadata flag signals that we have some metadata in the next argument. Here we inject “mytitle” into the “title” field of our MP3.

The surprises here are the -y, -acodec, and copy. First off, if we simply transcode one MP3 to another MP3, FFMPEG assumes the bitrate is 64kbps by default. This is bad! What if our source was 128kbps? Our output is suddenly half the quality!

That’s where we say we want to take the audio codec (acodec) and copy the data completely rather than processing it. So not only will we not suffer a quality loss and have to look up what the bitrate is but we also won’t spend extra CPU cycles processing the audio.

Next, the -y flag simply indicates “yes”. During the process of outputting to the same file we used as our source, FFMPEG will ask us via the command line if we’d like to overwrite the file. Since we are in the middle of a Grunt or Node.js task, this question is never posed to the user and the process appears to lock up. Passing a -y flag will simply force the answer to this prompt as a “yes”.

Making the Grunt Video Conversion Task

At this point, we’ve made our own Grunt task (and subtasks), and we’ve learned how to use FFMPEG through Node.js. All that’s left is to tie it together into a Grunt conversion task!

module.exports = function(grunt) {
var ffmpeg = require('ffmpeg-node');
        grunt.registerMultiTask('sa', 'Convert video and inject metadata', function() {
        var done = this.async();

            switch (this.target) {
                case "transcode":
                        grunt.log.write("SA - Transcode Task");
                        ffmpeg.exec(["-i", this.data.infile, this.data.outfile], function() {
                done.apply();
});
                        break;

                case "metadata-inject":
ffmpeg.exec(["-i", this.data.infile, "-y", "-acodec", "copy", "-metadata", "title=" +  this.data.title, this.data.outfile], 
function() {
                done.apply();
});
                        break;
            }
    });
};

Let’s also mock up a quick configuration to drive our task:

grunt.initConfig({
"sa": {
transcode: {
                    infile: “myfile.mp4”,
        outfile: “myfile.mp3”
            },
metadata-inject: {
                    infile: “myfile.mp3”,
        outfile: “myfile.mp3”,
        title: “my song title”
            }
            }
)};

Lastly, we can run/define this task in the GruntFile.js along with our above config:

grunt.registerTask('default', [‘sa:transcode', ‘sa:metadata-inject']);

All in all, my final code is a little more complicated because it runs through a list of files versus just one and does many more tasks and operations on my media library, but what I have here is the basic idea. And that’s what Grunt is: a pretty simple idea executed well. We don’t have to use it to run tasks like this, but it does save a lot of time and effort, and gives us a nice command line interface. Our final task could be simply run on the command line by typing

grunt

Where to Go From Here

I’ve gone ahead and added a curl and clean task to make this fully functional over on GitHub. To run it, first make sure you have FFMPEG and LAME installed like I mentioned above. Then, simply clone the repo, run npm install in the directory to grab the dependencies, and finally run grunt. It will download the first minute of the open source animation “Big Buck Bunny”, transcode it to an MP3 file, then inject a title into the metadata.

That in itself makes a nice little packaged script to be run by continuous integration tools like Jenkins or Bamboo. So, next time you consider just using Grunt on your boring old HTML/CSS application, do it and something cool!

Image courtesy of https://commons.wikimedia.org/wiki/File:Rfel_vsesmer_front.png

Previous

Easy API Scaffolding with Simple-API and Node.js

Rethinking JavaScript’s Try/Catch

Next