Tag Archives: JavaScript

Coaxing Sound out of the Browser

Leave a reply

“Get someone else to blow your horn and the sound will carry twice as far.” – Will Rogers

Having the utmost respect for Will Rogers and his timeless wisdom, I must apologize to his memory for using this quote for purposes other than originally intended.  The quote struck me as somehow appropriate,  because I want to write a few words on the use of sound in browsers and I want to recommend the adoption of a particular JavaScript library to make this as painless as possible.  Rather than creating your own solution from scratch, in the final analysis I am going to suggest letting Howler.js “blow your horn” for you.

This article touches on developing pages and browser applications using sound  – past, present and future.  However, most of the article will be devoted to alternatives for using sound in today’s development environment and I’ll show you two different ways to do it.  A link to a working demo will be included at the bottom of the article and we will walk through the heart of the code that drives this demo, leaving you with two alternative modules for playing sound that you may freely incorporate into your own work.

In the beginning, there was silence.  The early days of browsers were devoid of any method to play any sound at all. Over time, some limited proprietary sound playing capability began to appear in different browsers, but this was essentially unusable.   Eventually, plug-ins appeared that made the playing of sound in the browser a reality.  One of these plug-ins, Flash, still enjoys some popularity even today, particularly as a fallback mechanism for out of date browsers that don’t support more modern methodologies.  However, most developers would rather not have to use a plug-in, which requires client-side installation and maintenance by users.  So a more modern built-in solution is typically preferable.

Enter the HTML5 Audio element.  HTML5 audio is the dominant form of sound expression in today’s browsers.  In fact, this mechanism is present in all modern browsers (see Can I use the HTML5 audio element).  More importantly, it works reasonably well for playing basic sounds in simple scenarios – although the implementation does differ significantly across different platforms and caveats abound.   Here I am tempted to begin describing the features of the HTML5 Audio element and contrasting their implementation in major browsers – however, that is not really the thrust of this article (though basics will be discussed) and that information can be found elsewhere on the web.  I’ll provide a link at the bottom of the article to detailed information on the HTML5 Audio specification.  I’ll also provide a link that discusses implementation limitations on some platforms – notably iOS with mobile Safari.  The topic of mobile Safari does figure prominently in one of the code examples that follow.

The wave of the future for playing sound in the browser appears to be an API known as Web Audio.  This is an open source, WC3 supported, Chrome-originated specification.  This specification is much more extensive and capable than the HTML5 Audio specification.  Unfortunately, though it currently enjoys support from modern web-kit browsers,  there is insufficient penetration in the browser ecosphere to be able to rely upon it as yet, unless you are targeting specific browser platforms (see Can I use the Web Audio API).  Earlier, I foreshadowed a conclusion made in this article by suggesting that Howler.js is the best way to implement browser sounds at this time.  One significant reason for this is that by default, Howler.js will use the Web Audio API, to play your sounds if support is present.  If it is not present, it will fall back to an HTML5 Audio solution.

Let’s focus on the present now and take a look at solutions that work and are broadly applicable today.  If you want to use sound in the browser today, it probably means that you will have at least some degree of reliance on HTML5 Audio – either directly using “native” JavaScript or else indirectly through some library such as Howler.js (since Howler.js will automatically fall back to using HTML5 Audio if it must).  It should be noted that there are other JavaScript sound libraries, but these tend to start with HTML5 Audio support and fall back to Flash support if HTML5 Audio is not supported.  While a Flash fallback is useful in a small number of scenarios, this has become increasingly unnecessary since HTML5 Audio support in modern browsers is nearly ubiquitous.  Personally I’d rather go with a forward-looking solution like Howler.js that tries to support that latest technologies and used HTML5 Audio as the fallback.

Here’s a quick review of the core ideas behind playing sound in the browser and using HTML5 Audio.   At the most basic level, the process of causing a browser to emit sound consists of two steps:

  1. Loading / pre-loading your sound resource(s).
  2. Playing your sound resource(s).

Regardless of which browser is involved, when employing HTML5 Audio it is the audio element that handles both of these tasks.  It will even provide a sound playing UI if you enable it.  For our purposes however, we would prefer to dynamically create an audio element with no visible UI, and use it from within JavaScript code to both load and play sounds.

Whether a static or a dynamic audio element is employed, or instead a dynamic audio element is used, the manner in which it works is configured by setting it’s attributes.  The following are the “most important” attributes and are enough to get started with – if your sound playing needs are fairly basic, it may be that you will need to set no other attributes.

  • The controls attribute – Add this to the element by itself (with no associated value) and it will enable the build-in sound playing API.  This is normally used when a static element is employed but useful to know about, even though neither code example in this article uses it.
  • The src attribute – This is used to define the sound resource (i.e., some file path or URL) that the element will act upon.
  • The preload attribute  – This is supposed to allow control of if and when the element will preload the sound (buffer it in memory) to minimize or eliminate lag between the request to play the sound and the actual playing of the sound.  If you do not specify a preload attribute, the default value of “auto” will be applied, which is pretty much what you’d always want anyway.  Unfortunately, some browsers – notably mobile Safari – ignore this setting and will refuse to preload, only loading sound after a user interaction is detected.

Aside from the above-listed attributes, there are only perhaps two more things more you must know about the audio element – how to know when a sound resource is ready to play and how to actually play it.  Some sound resources can be very large – for example, music files.  A significant amount of time can pass between initiation of preload and the ability to actually play a sound.  The canplaythrough event is useful in detecting when a sound has been loaded completely enough to be able to play it without problems.  Which leads us to the other thing you should know – the play method of the HTML5 audio element must be called from JavaScript code to actually play the sound if the built-in sound UI is not employed.

In short, using the HTML5 audio element is pretty easy - unfortunately the lack of a single standardized audio format and the uneven implementation of the specification across browsers presents difficulties to the web developer who would like to write once and run everywhere.   We’ll look at the worst of these difficulties and then examine work-arounds and approaches to deal with them.  Once this is out of the way, we’ll dive into some code, the implementation of which has been designed to avoid or minimize these difficulties.

Probably the first thing to consider and possibly the most significant, is to decide what audio format to use for your sound.  There are several formats available but here is a short-list of some of the most popular and summary comments on each:

  • The wav format -  On the plus side, this format provides good sound quality and is usable nearly everywhere – the most notable exception being Internet Explorer, which does not support wav except in the mobile version.  On the negative side, files of this format are quite large, and as mentioned already, it is not supported by desktop I.E.
  • The mp3 format – On the plus side, this format provides reasonably good sound quality (although it is lossy storage format) with small file sizes (it is a compressed).  Additionally, this format is supported by every modern browser except Opera.  On the negative side, there is typically a small lag when decoding the compression on this file - even if pre-loaded – and again as already mentioned, no Opera support.  Additionally, there are potential legal issues with using this format which although quite standard in the music industry, is proprietary in nature.  The good news for consumers is that several of the patents related to mp3 usage have already gone through one patent extension and the last of these remaining patents are due to expire sometime in 2015.  Although one or two patents related to mp3 are still enforced by two companies, this enforcement is ebbing and might not even be an issue except for high volume usages (and in these cases the financial demands for usage are not so unreasonable).  Caveat: Let it be understood that I am not a lawyer and not offering legal advice – use mp3 at your own risk.
  • The m4a aka ACC format – This is a variation on MPEG4 for audio files, a descendant of mp3.  There are several plusses compared to mp3: files are smaller, the sound is higher quality and usage is completely royalty-free.  The downsides are that support for this format is probably not quite as widespread as it is for mp3.  Also, I believe there is still a decode lag on this format.
  • The ogg format – This is a proprietary but license-free format that offers good quality and reasonable size files.  The downside is that it is only available on certain platforms: Chrome, Firefox and Opera.  Neither Internet Explorer nor Safari support it at this time.

If you have to choose one format, wav is a pretty good choice for small sounds and m4a (ACC) is a pretty good choice for large sounds and music files.  The mp3 format may also be a possibility (just an observation – follow it at your own risk).  However, be aware of the aforementioned caveats when choosing these or any formats.  The good news is, you can specify multiple formats.  This is easy and reliable if you put static tags in your html, and more difficult if using JavaScript.  The canPlayType method is used for this, but instead of returning true or false, it returns “probably”, “maybe” or empty string – go figure.  Another of the nice features of Howler.js is that you can specify an array of files in different formats for the same sound, and Howler.js will figure out which to play.  Of course, one problem in going with the multiple format approach is that you have to provide multiple resources, which is more hassle and not ideal on mobile devices with their limited storage space.

We’ll look at just one more concern – the issues with mobile Safari – and then finally move on to some actual code.  Mobile Safari has the following issues with playing sounds (this list is derived from a somewhat famous article by Remy Sharp on his blog – see link at bottom of page):

  • iPhones do not like playing too much audio at once, it gets very choppy.
  • iPads do not play more than one audio stream at once.
  • iOS will not preload the audio unless the user initiates the action.
  • There is about a 1/2 second delay before iOS is able to play the audio.  This is because the audio object (in iOS, not HTML5) is being created.

The list above can be condensed into two major problems:

  • Apple iOS / mobile Safari uses a singleton audio object to play sounds and switching between sound resources always incurs a lag while the backing audio object is being created and initialized.
  • A user must take an action in the UI before each sound can be loaded – this effectively means that there is no preload and you must suffer an additional lag to load each sound the first time it is played.

There is an approach that  helps to address these two problems: the use of a single sound file containing sound sprites (aka audio sprites).  The term “sound sprite” or if you prefer, “audio sprite”, is a derivative of the css or image sprite idea employed in game development.  Image are often packed together in single files known as sprite sheets.  These sprite sheets consist of multiple embedded images, which are in turn used as textures to give game sprites their unique individual appearance.  Placing sprites all in one sprite sheet results in better organization, better performance and better memory usage.  Similarly, a sound sprite is an embedded sound in a single larger file containing other embedded sounds.  You get some of the same benefits using this technique that are gained through the use of image sprites, but that’s not the best reason to use them.  the biggest reason that sound sprites are worth using is that they help surmount some difficulties experienced in playing sound in Apple’s mobile Safari browser.  Creating JavaScript code to use a sound sprite is no trivial matter (also see the above-mentioned Remy Sharp article for details on how to do this).  Fortunately, once again Howler.js can come to your rescue.  It provides an API for using a sound sprite in addition to providing APIs for using stand alone sound resource files – and don’t forget, it will use Web Audio if it is supported, which gives better results than HTML5 Audio.

Now, at last, we get to look at some code -  the first example is a Sounds object that relies on plain HTML5 code and the Audio element.  This Sounds object is in its own module, contained within an IIFE (Immediately Invoked Function Expression), and uses the revealing module pattern to expose functionality.  The object is stored in a single global app variable, making it ready for use anywhere within the app with no pollution of the global name space.  Take a look at the code below.

The gist of the code above is that five different sound files are loaded and methods are provided to play these sounds individually.  Each sound is assigned to its own dynamically created Audio element, with the src attribute of that element referencing the actual file containing the sound.  The preload attribute is not used here, since the default for the preload attribute is “auto”, and that is what we want.  This is the best shot at getting our sounds preloaded, but in reality, a preload of “auto” is only considered a request, and browsers may honor it or ignore it (e.g. mobile Safari ignores it).  I should note here that canplaythrough event that is used here as a means of knowing when a sound has been loaded is not perfect – at least not on some browsers.  In Chrome, if I specified a path to a file that did not exist, the canplaythrough event  still fired - though it fired almost immediately – I suppose because there was nothing to load.  To complete the explanation of this Sounds object, a reference to each audio element, and thus each sound, is stored in the sounds object as a property using the associative array syntax, with the property name being a short descriptive name for the sound.  This name is the argument value that is passed to the play method, denoting which sound to play.  I have used objects very much like this in small projects and apps and they work well for providing basic sounds.

Now we’ll look at a Howler.js version of this same object.  This implementation uses a sound sprite file, although as previously mentioned, Howler.js also supports the playing of stand alone files as the “hand-made” object above does.  However, Howler provides much more built in capability than the object above and it does so in only about 9K of code.  Take a look at the Howler version of a Sound controlling object below.

In the above example, there are four sounds embedded in one file – the loading and playing details all handled by Howler.js.  This file is borrowed from a JSFiddle example created by Aaron Gloege (see link at bottom of page). Note that for each sound, a starting point with the file and duration a duration (both in milliseconds) are specified.  When Howler.js plays one of these sounds, it seeks to the proper point in the file and plays it for the specified duration.  It is also worth noting that each embedded sound is surrounded by a small amount of silence (generally a second or two) which allows for some leeway in the precision of the audio playing mechanism.  Despite major internal differences, the API for the HSounds object is pretty much the same as the Sounds object API.  So I’ll not repeat a description of that.

This article is drawing to a close – I hope you found the explanations, the demos and the stealable code to be useful.  As promised at the start of the article, a small demo of a page that uses both of these objects can be found on Uberiquity.com, here.  In the sections below I provide other links pertinent to this article and also attribution for the remaining sound files (those other than the file I borrowed from Aaron Gloege’s JSFiddle).

Resources cited in this article:

Remy Sharp’s Audio Sprites article

Aaron Gloege’s JSFiddle Audio Sprites example

Howler.js Home Page

Using HTML5 audio and video (MDN)

 

Attributions (for sound files other than those used by Mr. Gloege):

  • The “1″, “2″ and “3″ sounds are being used under the  Creative Commons Attribution License (3.0) and were created by Margo Heston.  They can be found in a pack on freesound.org at http://www.freesound.org/people/margo_heston/packs/12534/
  • The shore sound and the ding sound are being used under the Creative Commons 0 License.

 

 

 

Getting Closure on JavaScript Closures

Leave a reply

“I’m not interested in closure.” – Larry David

Larry wasn’t talking about JavaScript when he uttered these words, but if he had been, I would have to disapprove.  Understanding closures is important and is expected of professional-level JavaScript programmers.  The proper use of closures adds useful variable and function hiding techniques to a programmer’s repertoire – on the other hand, an inadvertent creation of a closure can in some scenarios lead to a memory leak.  In this article, closures will be clearly explained and the useful effects as well as the potential unwanted effects of closures will be examined.

Closure can be a difficult concept to grasp.  In my early JavaScript days, I became aware of the concept but struggled to understand it.  Eventually, I stumbled upon some decent examples (there are many poor ones out there) and forced myself to study them until closure finally made sense to me.  So, I understand both “not getting it” and “getting it”, which in my opinion puts me in a good position to explain closure.  If it has been fuzzy or incomprehensible to you until now, this article should put an end to that.

We’ll be looking at four examples of closure. The first two are very similar to one another and both of these are stripped of any extraneous code. The third example is somewhat more complex because I am trying to use closure to demonstrate a real-world analog of something that is like closure. I tried to keep this example as simple as possible, but there is some risk that this additional complexity might dirty the waters and obscure the closure within. If you find Example 3 confusing, ignore it and consider only on Examples 1 and 2, which illustrate the same basic principle as Example 3.  Example 4, while important, can also be temporarily ignored if you are just trying to come to grips with closure – perhaps you can come back to Example 4 at another time if it does not make sense at first.  Any or all of these examples can be run from the following links: Example 1, Example 2, Example 3, Example 4.  It may or may not be helpful to see these small demos before walking through the code.   The same links will be also be provided at the end of the article.

So, what is a closure?  Closure is automatically enabled when one function is defined inside another.  In fact, when you see the word “closure”, it might be best to think of the word “enclosure” instead.  Why?  Because “enclosure” is a more appropriate word for what closure is.  If a function is defined inside of an enclosing function, then that inner function enjoys not only it’s own scope and the global scope, but also the scope of its enclosing function.  These are standard JavaScript scoping rules, but there is one more point to understand in order to grasp the idea of closure.  During run time, even if the outer enclosing function has executed and is no longer in context, if you have a reference to the inner function on hand, the inner function still knows about the scope of the enclosing function that invoked it. The inner function retains access to the variables and any other functions of the enclosing function.  It is important to remember that it is not the state of those variables at the time of the call that is preserved, it is the reference to those variables.  This is an important distinction that will be explored in Example 4, but for now we should focus on the most basic explanation of closure.

The example below should clarify the basic explanation of closure given above.  The actual JavaScript code that forms the closure consists of only a handful of lines.  However, the entire example is not very large and I think it will make the explanation easier to understand if both the closure code and the supporting HTML with other code is shown. (Note: Normally, good practice is to separate JavaScript from HTML, but for an example like this it is better to include them together.)

In Example 1 above, turn your attention to lines 8 through 16 – you will find there the hallmark of a closure: a function within a function.  The enclosing function, “makeGreeting”, has one argument named “who”, one internal variable named “greeting”, and one internal function named “GreetWhomever”.  Notice that the GreetWhomever function takes advantage of the scope of it’s enclosing function to utilize both the variable named greeting and the argument named who.  These variables don’t belong to the inner GreetWhomever function, but the scoping rules of JavaScript allow the inner function to see the variables and arguments of it’s enclosing function.  What we have here are the makings for a closure, but not the closure in action.  To see a closure at work, you can run the demo for Example 1.  Each button gives a separate greeting: one is “Hello World!”, the other is “Hello John”.  Let’s examine the code a little more closely to see why this is so.

Looking at the last line of the makeGreeting function, it can be seen that the definition of the GreetWhomever function is being returned.  Note that it is not the result of the function, but a reference to the function itself that is being returned. So, when the global variables “greetWorld” and “greetJohn” are being initialized, they are actually receiving a reference to the GreetWhomever function.  This is borne out by the fact that in the onclick handlers of the two buttons, it can be seen that parenthesis has been used in conjunction with the variable names in order to invoke them as functions.  Ok, so that is understandable, but what of the closure magic?  Look again at the initializion of the greetWorld and greetJohn variables.  In each case, the enclosing makeGreeting function is being called, but a different greeting recipient is being passed in each case.  The value of the greeting recipient is therefore present in the “who” argument of the makeGreeting function.  Upon examining the inner GreetWhomever function, it can be seen that this “who” argument as well as the value of the “greeting” variable of the makeGreeting function are being used by greetWhomever. We already knew that it could do so, because an inner function enjoys the scope of it’s enclosing function.  The really interesting thing is, once the greetWhomever function has been returned and stored in the two global variables named greetWorld and greetJohn, the makeGreeting function has already been executed and is “gone”.  You’d think it has gone out of scope – but it hasn’t, because it is a closure for the inner function.  The inner function still has access to the variables of the outer enclosing function, even though the outer function has already executed.  This is closure at work.

Now let’s take a look at Example 2.   I will only show the code for the JavaScript of this lesson.  The HTML for this example is the same as in Example 1.  In fact, the JavaScript is almost the same as well, though not quite.  Example 2 exists for two reasons.  First, it repeats the lesson that was exposed in Example 1, but in a slightly different way – hopefully driving the lesson home.  The other reason is that the syntax employed in Example 2 is a lot more common than the syntax employed in the previous example.

In Example 2 above, the inner function is an anonymous function – it has no name.  This is a syntax that will be much more frequently encountered in actual JavaScript code than the syntax employed in Example 1.  Nonetheless, the code in this example works exactly the same as the code does in that example.  The inner function still utilizes the enclosing function’s variables (i.e. argument “who” and variable “greeting”) after the enclosing function has executed and is “gone”. If you run the demo for this example, you will see that it behaves exactly the same as the demo for Example 1.

My goal for Example 3 was to come up with a real-world analog of closure, and use JavaScript closure to make the example work.  I wanted to come up with a real-world analog into the article to help cement what JavaScript closure is.  The example I chose was that of a letter from friends or family vacationing in Hawaii (snail mail is not quite dead, though we are getting close).  The idea is that the letter itself is enclosed in an envelope.  We open the envelope and read the letter, and the letter informs us that there is a also a small picture inside the envelope that we missed when we pulled out the letter.  Because of the reference to the picture in the letter, we fish the picture out of the envelope to have a look at it.  The parallels to closure may or may not be obvious, but I’ll point them out anyway.  The envelope is like the enclosing outer function.  The letter inside is like the inner function.  The letter refers to a picture that we also discover inside the enclosing envelop.  The picture is like the data that belongs to the enclosing function and yet can still be referred to by the letter. For this example, I’ll present the entire HTML and JavaScript that make up the example.

In the rather long example above, take a look at lines 9 through 27.  It can be seen that this is pretty much more of the same – a function within a function.  The enclosing function here is named “envelope”.  The inner function is actually a constructor function for an object called “Letter” – but it is still a function and the closure trick still works.  In this example, the only variable that the inner function uses from the outer enclosing function is the Image variable, “picHawaiiCoast”. Nevertheless, a closure is a closure and this example functions very similarly to the first two examples.

In one way, this real-world analog of an envelop with a letter and a picture enclosed does not match the reality of JavaScript closures.  With the physical objects we can open the envelope and pull the picture out – this is precisely what closures can prevent – direct access to the variables belonging to the outer function.  I tried to gloss over this and make the analogy seem close to the way closures work by saying that “we had missed the picture until the reference to it within the letter made it available”.  However, it is important to remember that in JavaScript, access to an outer function that is “no longer present”, is granted only through the reference to the inner function at hand.  The bottom-most button in the example demonstrates this. Now that you have been exposed to the code for Example 3, it might make sense to run it (see links at top or bottom of article) if you have not already done so.

Earlier in this article, in the paragraph in which closure was defined, I made this comment: “It is important to understand that it is not the state of those variables at the time of the call that is preserved, it is the reference to those variables”.  The purpose of the fourth and last example is to help explain what that means.  Example 4 provides a UI with two buttons – each button causes console output (accessible in Chrome via Ctrl-Shift-J, in Firefox via Ctrl-Shift-K and in IE via F12 – apologies to Safari and Opera users, I am unfamiliar with those).  One button is backed by code using a closure that makes a naive attempt to count from 1 to 3.  The other button is backed by code using a closure and an additional level of closure to fix the problem with the naive attempt.  If you were to run the examples and press each button in succession (naive, then non-naive) you would see the following output in the console:

Examine the code behind each of these buttons below.  (The minimal HTML needed to execute the example has been omitted).

Code behind the “Naive Count 1 to 3″ button:

The “naive” code above fails, outputting three consecutive lines of “Count = 4″, instead of the expected “Count = 1″, “Count = 2″ and “Count = 3″ on three consecutive lines.  Why?  Because the closure from which each function is referencing variables is the same closure.  Furthermore, the state of the count variable in that closure is such that count is equal to 4 when the functions are finally executed.  It makes no difference at all that as the individual functions were stuffed into the array, the value of count was 0, 1 and 2 respectively.

Code behind the “Count 1 to 3″ button:

The “non-naive” code above succeeds, outputting a count of 1, 2 and 3 as expected.  Compare the two sets of code above.  The “non-naive” successful code wraps the original inner function in a new function, passing in the value of the outer function’s count variable via an argument.  This works, because we have created a new, intermediate closure around each innermost function.  Study it and think about it until it makes sense.  If it doesn’t click for the moment, that is alright, you will still have made the most important mental leap in understanding closures if you comprehend Examples 1 and 2.  You can come back to this later when you stumble upon a closure bug like this and then this example will help you figure it out.  Similar problems can crop up when setting up event handlers, so keep that in mind.

So what are closures good for?  At the top of the article I promised that we’d delve into some useful features of closures.  I’ll cover two reasons here.

  • The first reason is that they are a great way of hiding variables and functions – i.e. making them private.  Only the inner function has access to this data once the outer function has executed.  This mimics the idea of private members in object oriented languages such as C++ and Java, allowing the construction of objects with both private data and private function members.  Example 3 is a case in point,  although a more useful pattern for creating a reusable object with private members would result if the enclosing function returned the inner “constructor” function instead of using the inner function to create a new object.
  • A second reason I will point out, though not discuss in detail, is that when using closures to define member variables for objects, no “this” reference is involved.  That is different than the syntax necessary when defining a constructor function with instance variables and methods.  That difference is important when using object methods in conjunction with things like event handlers, which reset the “this pointer” and can cause bugs in code when a method of an object is passes as the callback for an event handler.  That particular problem is non-existent when a closure was used to create member variables and functions for an object.

Are there any other problems that can occur if closures are not understood or go unrecognized?  In addition to the type of bug that Example 4 embodies, in some browsers and some scenarios, the formation of a closure can result in a memory leak.  This problem can often occur in conjunction with the establishment of an inadvertent circular reference due to the formation of a closure during event callback creation.  I am not going to attempt to explain this in detail here – it is a topic for an entire other article.  I will however, point you to this excellent article on JavaScript memory leaks, located on the IBM developerWorks site: Memory leak patterns in JavaScript.

Well, it’s time to bring this article to a close.  I hope that if you were “closure-challenged” (which is not uncommon), this article helped you get some closure on closures.  As promised above, here again are the links to the four examples discussed in this article: Example 1, Example 2, Example 3, Example 4.

 

 

 

 

 

 

 

Examining Underscore.js Search and Filter Collection Functions

Leave a reply

“Price is what you pay. Value is what you get.” – Warren Buffett

By any measure, Underscore.js is a great deal. Underscore.js is a tiny open source JavaScript utility library that packs about 80 useful and eye-opening functions into only about 5kb (minified and gzipped).  A significant number of these functions are applicable to collections.  In Underscore.js, the term “collections” can refer to arrays, objects, and array-like objects such as arguments and NodeList.  Follow this link to examine the Underscore.js documentation for these functions.  In this article we will be looking at actual usage for some of these functions.  You can find a demo/test page that runs these examples here.

A number of these collection functions are ideal for search-related applications. It’s interesting how many of the Underscore collection functions are reminiscent of SQL. If one were to look at the full list, one would find that about half of them have names and/or do things that are related to SQL select functionality, e.g.:

  • each
  • find
  • filter
  • where
  • findWhere
  • reject
  • every
  • contains
  • max
  • min
  • sortBy
  • groupBy
  • countBy

It is worth noting that the popular JavaScript MV* library, Backbone.js, proxies and uses many Underscore.js functions.  It makes particularly good use of many of these SQL-evocative functions for assisting programmers in the management of collections of Backbone Models.  The search functionality that this article examines, or analogs thereof, could just as easily be implemented in Backbone.js as it could via the more low-level Underscore.js.

There are small number of usage patterns shared by these Underscore.js collection functions.  We’ll be looking at four collection functions, chosen as being representatives of both the functionality and pattern usage differences that pertain to the functions in the above list.  We are going to begin by examining the _.where function, which has great functionality even though it follows a simple usage pattern.  Take a look at the code below.  (Note for programmers unfamiliar with Underscore.js:  As befits the name, all Underscore functions begin with an underscore (_) character, similar to how jQuery functions begin with dollar sign ($) character.)

The code above produces the following output:

In the test function shown above, note first the signature for the _.where function.  It takes two arguments, “list” and “properties”.  The first argument, list, is one that is found in all of the Underscore.js collection functions.  Recall that collection functions work on arrays, objects, and array-like objects such as arguments and NodeList.  The list argument can represent any of these things.  It is important to understand that the nature of the reference passed via the list argument will slightly alter the usage pattern for these collection functions.  In this particular test we are passing an array of objects named “testArr” as the list.  The other argument required by _.where is the properties argument, which must be an object containing one or more properties that are used as the where criteria, i.e., what to search for in the given list.  In the test case above we are passing an object containing two search properties – we are looking for things that are green and have a tail.  In the input list we have four objects, all of them containing color properties and all of them containing tail properties.  Of the four, only one of them – an iguana – is both green and has a tail.  As is to be expected, when this test is run, only one object is returned – the iguana object.

The next function we will be looking at is the _.each function.  I chose this function as one of the four because of it’s flexibility and because it is a good one to use in illustrating the full range of usage patterns that may be encountered when using the Underscore.js collection functions.  Take some time to really look at the two code examples for this function – if you can get through this part the rest of the article is down-hill.  Whereas the _.where function is fairly simple – taking only a list and a set of search criteria as its arguments – the _.each function is inherently more capable, but with less specified functionality at the outset.  In addition to the list argument that we have already touched upon in the previous example, the _.each function has an iterator argument and an optional context argument instead of a “properties to search for” argument.  The iterator argument represents a function to be passed and the optional context argument represents a “this pointer” context to be bound to the iterator function. The ability to reassign the this pointer for an object or function is a commonly used JavaScript technique.  An in-depth explanation of this technique is outside the scope of this particular article, however we will see the context argument in use.  The iterator argument concept is a different story.

In this discussion of the _.each function and in this article in general, we will be spending a fair amount of time examining different uses for an iterator function, as many of the collection functions have such an argument.  The basic concept of an iterator function is the same for all collection functions that use them – it is a bit of code that the Underscore collection function will call for each item in the given list argument.  The number and the nature of arguments that Underscore will pass to an iterator function varies with the particular collection function, as does the nature of the expected function return, if any.  Let’s go ahead and take a look at the code for the first of two _.each tests in the demo page.

The code above produces the following output:

In the test function shown above, note first the signature for the _.each function.  When calling _.each, we are passing a simple array of strings for the list argument.  The iterator function is defined on the fly, as an anonymous function.  The third argument to the _.each method – the context – is a reference to an object declared outside of the testEachInArray function. Lets look closely at the iterator function.  It takes three arguments – the Underscore.js documentation for the _.each method specifies this.  It has no return value.  Ok… but what is it doing?  The answer is, not very much.  It is merely printing to console the content of the three arguments for each iteration.  The Underscore.js documentation states  for the _.each method further states that when the list argument is an array, the first argument will be the value of the array element being processed for the current iteration, the second argument the index of that array element, and the third argument will contain the content passed as list (the entire array).

A major purpose of these _.each function examples is to show an iterator function in action – to give an idea of what sorts of things can happen when an iterator is being called.  In actual use, the iterator function passed to the _.each function would be doing some kind of work with each element in the array, as the _.each function iterated.  But what of the final argument, the context?  Note at line 19, the use of: this.contextValue.  The output of the function in the console proves that the this pointer for the iterator function has been set to the this pointer for the anExternalContext object, because contextValue prints out correctly.  Let’s take a look now at  the second _.each function example.

The code above produces the following output:

In the test function shown above, note that we are calling upon the same _.each function that we called in the previous test.  The difference here is that we are passing an object as the list argument, instead of an array.  This example is more or less the same as the previous one except that the usage pattern is slightly different because the list is an object this time.  Here we must make allowances in the iterator function for the fact that each iteration of _.each is working with a property of the given object passed as the list argument, instead of it being an element of a passed in array.  The Underscore documentation tells us that the three arguments passed to the iterator will (naturally) be a little different for an object passed as list than when an array is passed in the list argument.  Here, the first argument will be the property value being processed for the current iteration, the second argument will be the key for that value, i.e. the property name, and the third argument will again be the content passed as list, in this case an object.  The iterator function takes these differences into account (e.g. the use of JSON.stringify to get a string representation of the list object), but otherwise works the same as the iterator function worked in the first _.each example test.

The next function we will be looking at is the _.max function.  I chose this one because it is reasonably straightforward and representative of other similar functions (e.g. _.min, _.find), in addition to being very useful in its own right.  The iterator function is optional for the _.max function, but when used, its purpose is clear and it is easy to comprehend. Lets look at the first of two examples, one that does not employ an iterator function.

The code above produces the following output:

In the test function shown above, note first the signature for the _.max function.  It is the same signature that _.each uses, except that the iterator function is optional.  In this first example test of the _.max function, we are not passing an iterator.  There is no need for one, as the list we are passing is a simple array of strings and the _.max function needs no additional help to be able to iterate through and find the largest number in the array.  Lets look now at the second _.max function example usage, which does require an iterator function to be passed.

The code above produces the following output:

In the test function shown above, note that we are calling upon the same _.max function that we called in the previous test.  The difference here is that we are passing an array of objects as the list argument, instead of an array of numbers.  This example is similar to the previous one in that _.max is trying to iterate through the input list to determine the largest value in set.  However, the data is different here – both in content and in nature.  The _.max function cannot know on its own, the property name within the array of objects that it must access in order to determine the largest value. In some cases, each object might have any of several properties that would be correct.  Even in this simple test case, it is possible to pass the value of the person property instead of the value of the IQ property (the highest sorting person name would be the “max” value in that case) but we want to find the person with the max IQ in our example.  So, we create the anonymous function to help out.  The Underscore documentation for the _.max function states that the iteration function for _.max must take an argument representing the current object being iterated from the array of objects, and return the value of the property that _.max will use to cull the max value.

The last Underscore.js collection function that we will examine is the _.filter function.  It packs a lot of potential power, but requires the programmer to provide most of that power by providing an iterator function. You will find two example uses of the _.filter collection function below.  The first of these will be a simpler use case than the second.  Each of these two examples will be a little different from the previous examples in that an example is made up of two functions – one is a re-usable function and the other is the test driver for the function.  Let’s take a look now at the two functions that make up the first example use case for the _.filter function: simpleFilterLike and testSimpleFilterLike.

The code above produces the following output:

In the test function shown above, note first the signature for the _.filter function.  By now this pattern should look familiar.  In the scenario above, we are passing as the list argument a simple array of strings, containing phrases that will be searched.  We are also passing a non-optional iterator function that returns a result of true (per Underscore specification for _.filter) if a searched-for string is found with the currently iterated phrase.  We are not explicitly passing the searched for string.  A “likeCriteria” object is created within the body of the function that invokes the _.filter function.  Within this likeCriteria object, a “searchFor” property is set, containing the value that we will search for within the phrases.  We could have passed this likeCriteria object as the context and shared it’s contents with the iterator function that way, but there is no need to do so as the scoping rules of JavaScript already make the likeCriteria object contents available to the iterator function.

What is happening inside this iterator function?  For each iteration, the _.filter function is giving us a phrase to search in and we are doing so using the contents of the searchFor property in the likeCriteria object.  The iterator function is “teaming up” with the Underscore _.filter function by returning boolean true if the searched for string is in the phrase – effectively allowing the Underscore function to filter which of the array elements (i.e. which phrases) pass a truth test. Those phrases that pass the truth test will be returned in a filtered array that contains only the right contents from the original array – where “right” means that the searched for string was found in the phrase.  Now lets look at the second scenario, in which two more functions are used in a similar manner to the ones just examined to call upon the Underscore _.filter function to help do a search.

The code above produces the following output:

The two functions shown above are not very different from the first two _.filter leveraging functions we just looked at, there are just a couple of twists.  The first is that instead of a simple array of strings representing phrases, we are passing here an array of objects containing phrases.  The second twist is that we are employing two searchFor criteria, which means that this version of the “likeCriteria” object contains an array property instead of a property holding a single string.  Otherwise, this pair of functions works the same as the previous pair.

The main reason for providing this second example of how to leverage the Underscore.js _.filter function is that this example is much closer to a real world use-case for a utility filter function that returns information based upon matching “like” criteria.  In fact, a more robust and capable version of the “filterLike” function forms the heart of the search implementation for the main Uberiquity site.  You can test the Uberiquity search feature out if you like, by clicking on this Uberiquity Search link. If you were to go there, you could search for this very article by using the following search:  underscore AND search

How To Dynamically Swap External Style Sheets with JavaScript

Leave a reply

“There are very few things that surprise me.” – Judi Dench

Myself, I would not go as far as Dench – at least not in the world of Web development.  However, I will say that it isn’t everyday that I stumble upon something truly new (well to me) and delightful.  Nonetheless, not too long ago I discovered that it is possible to change external CSS style sheets for a running web page via JavaScript – i.e. dynamically, at will, after the page has loaded.  I confess I felt sheepish that I had not known this was possible.  Perhaps you didn’t know it either.  Read on if you wish and you’ll find out how.

Why is this ability to dynamically swap external style sheets useful?  There are at least three reasons:

  • In cases where a large number of individual CSS/style changes need to be applied all at once, this style sheet swap method might be faster to execute and easier to maintain.
  • The second reason is perhaps a specialized case of the general case stated above – this is a great way to implement dynamic theme changes that can be invoked by the user at any time.
  • The third reason is that this technique can assist with responsive web site design – i.e. changing the layout and sizes of element in a page to accommodate radically different screen sizes.  A more standard approach to responsive design relies on CSS media queries, which allows CSS to alter itself to accommodate different screen sizes.  Unfortunately, the sad reality of today is that many of the older mobile platforms (i.e. device, plus OS, plus mobile browser) do not properly support CSS queries.  Even on newer platforms, media query support can be quirky.  So one alternative approach is to substitute the technique being described in this article to allow for “responsive” CSS changes.  Instead of using media queries, JavaScript code can check the browser document size upon initial display and every refresh and then dynamically adjust font sizes, and specific DOM element dimensions by swapping in style sheets that have been created for best viewing at certain trigger sizes.

Actually accomplishing this feat is easier than one might expect – at the heart of it, there are only three standard, cross-browser DOM functions that need be employed for this purpose – removeChild, createElement and appendChild.  The basic idea is to remove one link element from the header of a page and add a new link element to replace it.  This methodology requires that two (or more) CSS files must already exist – no new CSS file is being created using this technique.  There are ways to dynamically create individual style sheet rules but this article does not cover that topic.  In any event – in my opinion – dynamically swapping entire style sheets is more useful.

To demonstrate actual swapping of external style sheets at run time I have built at a working demo page.  I will show every line of code and annotate how it works in the succeeding paragraphs, but it may be desirable to run the demo first.  Here is a link to the demo at my main site, uberiquity.com.  Check it out first or wait until after the explanation.  I’ll provide another link at the end of the article.

The demo consists of seven small files:

  • changecss.html
  • changecss.js
  • base.css
  • mono.css
  • red.css
  • green.css
  • blue.css

Only two of these files contain code and only one file, changecss.js, contains the actual code that enables the swap.  The base.css file contains all the non-swappable CSS used by the HTML for the demo, mostly formatting information that is non-color oriented – e.g. setting the sizes of things.  The other four CSS files represent themes.  It is these files that we can swap to change the theme of the demo page on the fly.  We’ll look at changecss.js first, then we’ll see how the single function in changecss.html calls the methods therein.

I like the names of functions and variables to be self-explanatory.  This means that to some people my function names may be annoyingly long.  I realize that this is a bit of a “religious thing” and not everyone likes the verbose approach – please forgive me if it rubs you the wrong way.  In any event, I list the content of the changecss.js file below.  It contains only two functions.

To make the swap, we need to first remove the existing external style sheet link.  For this purpose, we call upon the removeExternalStyleSheetLink function.  When first loaded, the demo page is linked to the theme CSS file named mono.css.  However, the demo user is allowed to swap in any of the other theme CSS files.  In order for this to happen, the demo first calls upon the removeExternalStyleSheetLink function to remove the link to mono.css, i.e.: <link id=”mono” rel=”stylesheet” type=”text/css” href=”mono.css” media=”screen” />.  Notice the rather unusual fact that the link has an id attribute – this is perfectly legal HTML (it’s just not done very often).  This id value of “mono” is the value that must be passed in the cssLinkId argument to the function.  The function uses it to get a reference to the DOM object of the external style sheet link that must be removed, and then does the removal by telling the parent of this DOM object to “remove me”.

Once the link to the original CSS theme file is gone, a new link element must be inserted into the head of the document to complete the swap.  For this purpose, we call upon the second function, createExternalStyleSheetLink.  The job of this function is to create a DOM object that will represent a link to the CSS file that is to be “swapped in”.  Like the removal function, the create function takes an id as an argument, because it is important to give this new link tag an id.  Giving it an id attribute makes it easy to reference it in case it too will be swapped out at some future time. The second argument, href, will of course contain the name of the CSS file that will be linked to (i.e. swapped in).  The function uses the standard DOM manipulation JavaScript functions createElement and appendChild to perform the work of creating the link object in memory and then inserting it into the proper place in the DOM. The proper place is in the head of the document, as denoted by the invocation: document.getElementsByTagName(“head”)[0].

Now let’s take a quick look at changecss.html.  I’ll list the entire file but the the main point of interest is the solo function named changeStyleSheet. Most of the rest of what is there merely exists to provide objects whose colors can be changed by a theme switch (i.e. proof of concept).  Note that I don’t necessarily advocate defining functions in HTML pages, but in this case, the function is quick and dirty and relies upon embedded page knowledge.  I’m doing a couple of other naughty things here, for instance, using a non-namespaced global variable.  Please bear in mind that I am striving for brevity and clarity of presentation – not necessarily trying to use best practices.

Before we look at the changeStyleSheet function, note again the link tag containing the id of “mono”.  That is the default theme that may be swapped out by pressing either the “Apply Red Theme”, “Apply Green Theme” or “Apply Blue Theme” buttons.  It is worth noting that the act of swapping the CSS files causes the page to radically change it’s appearance, but this does not cause a page refresh.

Because the functions in changecss.js have already been covered, there is nothing mysterious about the changeStyleSheet function.  We already understand how the removeExternalStyleSheetLink and removeExternalStyleSheetLink functions work, and the changecss.js function does little more than simply call them in succession.  The remainder of what the function does revolves around remembering the name of the CSS file for the current theme and displaying that information to the user.

The only remaining files to discuss are the CSS files – here is base.css.  This file never gets swapped out – which is of course the normal scenario for CSS files.  It performs a useful function in the demo by providing CSS that never changes, but its existence in the demo proves something as well.  The technique presented here is perfectly adept at “cherry picking” links – base.css remains undisturbed now matter how many times one swaps themes in the demo.

What follows after this paragraph are the listings of the four CSS files that could be considered to be “themes”.  There is nothing special at all about the makeup or structure of these files – they are just standard CSS files.  It is the JavaScript in changecss.js and the browser’s CSS engine itself that provide all the magic.

And there you have it – simple, sweet and powerful.  Here is that repeat link to the demo I promised I would provide.  If you haven’t yet done so, take a moment to play with it.

Node.js Part 2 – Asynchronous Calls in Node

“There’s no such thing as a free lunch” – El Paso Herald-Post, June 27th, 1938

The etymology of this quote, like many others, is not exactly clear-cut.  However it’s meaning and truth are indisputable.  Ok, you can dispute it, but even doubters would be forced to admit that it is very rare to get something good in return for giving nothing.  The same is true of obtaining the real benefits of Node.js.  You cannot reap the harvest without first doing some sowing of your own.  You need to commit to one thing if you are going to use Node in the spirit of it’s design – you have to be willing and able to make your blocking calls (mostly IO-related) in an asynchronous manner.  Furthermore, there is a wrong way to do this and multiple right ways to do it.  This article explains the difference and explains how asynchronous calls can be made correctly.

To write a non-trivial Node program one must understand something of how Node works, but not every detail.  A typical user application is more than just a single program, it is a composite of one or more discreet interlocking bundles of code.  A Node application is no different in this regard.  To deeply comprehend Node without abstracting our understanding we’d all have to go learn the internals of Node’s component pieces – Google’s V8 JavaScript engine, the libuv framework and a bunch of Node binding libraries written in C and JavaScript.  Fortunately (although study of these things would be interesting indeed) we don’t have to do this.

To understand Node well enough to really use it, we can rely on an abstraction of how it works.  We could think in terms of Event loops, Event Queues, Event Emitters and function call backs.  In fact, a lot of articles attempting to describe the way Node works simply throw up a few of these terms and continue with something to the effect of, “You are undoubtedly familiar with events and call backs since you encounter these ideas when writing JavaScript for a browser”.  It’s a bit of a cop out, but understandable – they are trying to abstract for you how Node works without going into gory details you don’t have to know.  If you are like me, explanations like this can leave you feeling vaguely uneasy.  They either too far or else don’t go far enough to provide a solid abstraction that one can use to understand the interface between a Node program and Node itself.

For the purposes of this article, we’ll use a simpler abstraction – one that focuses solely on the interaction between Node and your program code.  Node can be thought of as an Event Creator and Manager.  Every time your code makes an asynchronous call, Node creates an associated event that your code will take advantage of.   Naturally, a two-way communication must be maintained between your code and this Event Creator/Manager that is Node.  There are two ways in which you as a coder can communicate with Node:

  • Call back functions to allow Node to “talk back” to your code when an Asynchronous function completes.
  • Library calls that you invoke as input to Node, many of which can be blocking calls.

It is these blocking calls that are of interest in this article.  Typically these are IO sorts of calls but there are others.  Some examples include calls related to files, streams, sockets, pipes and timers.  Node has one single process thread of execution.  Its job is to quickly move through the ordinary parts of program code (and the code of other program instances that are using it) and queue up asynchronous events for blocking code.  This event queuing is part of a process that Node uses to off-load blocking work onto other external process threads.  As a coder, when you make a blocking call in your code, you have choices about the manner in which you invoke it:

  1. Usage of asynchronous vs. usage of synchronous function calls (asynchronous is thematic).
  2. Assuming asynchronous calling is chosen, how you control the order of execution for calls that are dependent upon each other.

The use of Asynchronous function calls is more than just the subject of this article – it is the heart and soul of Node itself.  The Asynchronous execution of long blocking functions – on external process threads that your own single program thread never has to worry about – is Node’s whole reason for existence.  This is what makes Node fast, efficient in memory usage and extremely scalable.  However again, there is no such thing as a free lunch – there is a cost for this.  That cost is the necessity of working with an event-driven callback model and taking extra care when calling the “I0″ types of functions mentioned above.

This article will focus primarily on explaining how to correctly use the asynchronous versions of function calls supplied by Node’s function libraries.  To do so, we will be looking at three different program examples.  Each example will be a variation on the same simple program.  It will first create a file and write some information to it, then it will re-open it and append some more information, then it will read that information and present it in the browser.  The three variations of this program will consist of a program using only synchronous calls, a program using only asynchronous calls in a naive manner, and the program using asynchronous calls in a proper manner. We’ll be looking at the actual runnable code for these a little later on in the article.  I’d like to first present a little analogy that helps explain and justify the need for taking the trouble to set up proper asynchronous processing.  If that sounds like a waste of time to you, then just skip on to the code examination portions of this article.

Here is an analogy that illustrates the differences between a synchronous approach and a proper asynchronous approach.  Let us equate a Node application to a small restaurant, one with only three employees – a Chef, a Waitress and a Manager. Fleshing out the analogy a little further, it is made up of:

  • A Chef – Node itself will play this part.
  • A Waitress – This is represented by the controlling thread of your program.
  • Food Orders – These are the Blocking/IO calls, presented to the Chef by the Waiter.  They are blocking because the Waitress must wait for the food to be prepared before she can serve it.
  • Food – This is data.
  • Customer(s) – This is the remaining code in your program that consumes the “food”.
  • The Manager – This is the coder.  He tells the Waitress and the Chef how they will do their jobs.

We will look at a slow synchronous restaurant, a naively run asynchronous restaurant, and a well run asynchronous restaurant.  For each scenario we will have just one diner ordering the same meal:  An appetizer of onion rings, followed by a main course consisting of a salad, a steak and a baked potato, with a dish of ice cream for desert.

Let’s look at a synchronous restaurant first.  In this restaurant, the manager has decreed that the Chef will only work on one menu item at a time.  He figures this will reduce complexity and possibility for error. To control this, the Waitress must submit Food Orders consisting of just one item and she must do so in the order needed to properly serve the diner.  So, the Chef makes just one thing at a time instead of being able to prepare multiple things at once.  There is a controlled ordering of the food items to prepare but each one must be completely finished before the Chef begins working on the next.  In the end, the customer would not be bowled over with the service but at least the meal would be correct, i.e. it would be delivered in good order and nothing would get cold or melt.  Consider however, how well this restaurant would do if it got busy.  One customer’s meal would be prepared at a time, and even that in discreet sub steps.  Service would be unacceptably slow.

Now let’s look at a naively run asynchronous restaurant.  The Manager, having been fired from his previous retaurant due to slow service, realizes that this Chef is going to have to be allowed to cook more than one thing at time.  He tells this Waitress that the chef must not be kept waiting, and to submit all Food Orders as soon as a customer has finished ordering.   However, the Manager still wants to avoid complex communication between herself and the chef.  Therefore, each Food Order is only allowed to have one item on it.  He figures that she can arrange the food items into meals when delivering them to each diner, since she knows what they ordered.  So the Chef is unfettered in this restaurant.  He can prepare as many items at once as seems reasonable to him, and furthermore he can choose what order to prepare them in. The results are not good.  He has no idea which items are destined for which customer, or even how many diners there are at the time.  He decides that ice cream is very fast to prepare, so he’ll dish it out first.  After that, he decides to start preparing everything else all at once – there isn’t much to cook.  It turns out that the potato is fastest – they are pre-cooked, so that gets handed off to the Waitress not long after the ice cream.  The salad follows shortly thereafter.  A bit later, the steak and the onion rings are finished about the same time, so he sets both at once on the pickup area used by the Waitress.  The end result here is that all of the food making up the customer’s order was delivered pretty quickly, but there were definite problems with the order of delivery.  The customer had a choice of eating desert first or letting it melt. He was then presented with a baked potato instead of the introductory salad, which he got next.  After a while, the waitress served up the rest of the main meal and the appetizer at the same time.  In the end, the customer would be upset and probably never return – that is if he even stayed for the entire meal.  This naïve asynchronous approach fails at even serving one customer, unless the customer orders something simple like just ice cream, or unless the order of food preparation completion just happens to match the expected order.  In a busy restaurant, the insanity would be even worse – and what’s more, service for some customers would probably not only be badly ordered, but slow.

Now we’ll look at a win – the well run asynchronous restaurant.  In this scenario our intrepid Manager has decided he doesn’t completey know what he is doing.  So he takes some classes and reads a book or two on managing the restaurant business.  He lands another new job, based upon the strength of his new training.  This time things are different.  He realizes that the Chef needs to work on more than one thing at a time and he also realizes that the Chef has to be given more guidelines as to how he prepares the Food Orders.  He knows he has to allow for a little more complexity in communication between Waitress and Chef.  He tells the Waitress to immediately submit Food Orders containing the entire meal for a diner and that the order of items within the Food Order must match the expected order of delivery for the meal.  So the chef is free to prepare more than one thing at a time and the order in which he may prepare them for a given Food Order is specified.  Furthermore, since food orders contain entire meals and the preparations he begins will only roughly have to coincide with the order of Food Orders as they are submitted, he is free to work at maximum efficiency while still more or less completing entire orders as they arrive. It won’t necessarily be first in first out – part or all of one Food Order might be completed before a prior Food Order.  However on the whole, Food Orders will more or less come out in the order in which they were submitted – and more importantly, the order of completion of items within an individual Food Order will always be correct.  In the end, orders come out in correct order and are delivered pretty quickly and even approximate a first in first out scenario.  Customer are satisfied, Manager and restaurant flourish.

It’s time to look at some code - we will look at three programs that attempt to do the same thing, though not at all in the same way. Each program is written with the idea that it will perform the following three basic steps in the order listed below:

  1. Create a file and add some data.
  2. Append some more data to the file.
  3. Read the full content of the file, then send that content back to the browser, along with some tracking of step order and elapsed time that were created as the program did it’s work.

Also, each program makes use of the same two helper functions.  I’ll just define them here once and not show them in each of the program examples, though you will see them being called.  There is nothing of any interest to discuss in these two functions, so I’ll not comment on them.

 

Our first example program is one that does it’s work by making synchronous File IO calls.  It runs correctly but would not scale well, nor would it play nice with other programs running on the same Node instance -  Every single File IO call is blocking – the main thread of Node must wait, babysitting this one instance of a program.

 

Our second example program is one that does it’s work by making asynchronous File IO calls, but it does so in a naïve manner.  It simply replaces the synchronous calls with their asynchronous versions.  This is not enough!  Since the events will execute asynchronously, there is absolutely no guarantee that they will return in the same order in which they were called.  Nothing has been done here to constrain that – the order of return is undetermined.  Step 2, “Append data”, might occur before Step 1, “Create original file data”.  Alternatively, Step 3, “Read and show existing file content” might occur before Steps 1 or Step 2 are complete, or on a busy system, perhaps before any part of their data were in the file at all.

 

Our third and final example program is one that does it’s work by making asynchronous File IO calls, and it does so correctly.  In the absence of special tools for this purpose (more on that later), the correct way to force one asynchronous call to follow the next is to chain them.  Let us say for example, as in the example below, that there are three successive asynchronous calls to be made and each one should only be invoked after the successful completion of the previous call.  What must be done is to put the invocation of the second call in the branch of code that is the successful completion of the first asynchronous call.  Likewise, the third asynchronous call should only be invoked in the branch of code that represents successful completion of the second asynchronous call.

 

Note the important difference between the properly coded asynchronous methodology and the use of simple synchronous calls.  In the Synchronous method, the entire execution thread of Node is forced to move through the blocking calls one at time, sitting idle will waiting a relatively long time for the results of each blocking call.  With the asynchronous methodology, this program still forces the sequential calling of these programs, waiting for the prior call to finish before beginning the next.  But what a difference.  In the asynchronous case, the single Node process thread is free to set up an event for each of these calls in turn, and it does not have to wait for them to return results.  It is free to continue it’s business of queuing events and making call backs for other programs – all that IO blocking is done on some other external process thread that your program knows nothing about, and that Node does not have to sit idly waiting for.

So there are indeed extra precautions to take when making depended asynchronous calls and they produce a kind of hierarchical hell (with all the indentations) in the source files that make them harder to maintain.  There are multiple Modules available from Github that attempt to provide solutions for this problem.  They provide an API that allows for a more organized approach to writing this sort of chained code i.e. avoiding the pyramidal, hierarchical hell.  I will mention one such project, perhaps the most popular, called Async (not Asynch.js which is something else).  The usage of such modules and of Async in particular, is beyond the scope of this article.

You may have noticed that my error handling in the three examples was either threadbare or non-existent. Example 1 actually had none – this is a combination of laziness on my part and a desire to keep the examples as simple and pure as possible.  In examples 2 and 3, there is some error handling due to the fact that is required by design, but it is pretty threadbare.  What, no try catch blocks?  Well, it turns out that try/catch/finally doesn’t really serve well in the asynchronous coding style that one must adopt for high volume Node coding.  As of Node 0.8, there is something new provided called Domains, that is meant to address this.  The usage of Domains is also beyond the scope of this article, but here is a link to a pretty good Domain slide presentation by Felix Geisendörfer at Speaker Deck.

I hope this article has been of some use to someone out there.  Remember that if you are writing synchronous, blocking function calls into your Node code, you are doing the wrong thing unless you own the Node instance and you know that it will never have to handle heavy loads.  Asynchronous, event-driven coding for server-side solutions is what Node was invented for in the first place.

Node.js Part 1 – Introduction

“Node news is good news.”

As you might gather from the rather cheesy paraphrasing above, I am very impressed with Node.js.  More than that, I am astonished by the open source community and code projects which have which have sprung up around Node.  The popularity of Node is exploding, and in this article I will offer you some basic knowledge about Node.js that may pique your own interest in this “hot” server side development framework.

So then, what is Node.js?  To properly answer that question will take this entire article and perhaps more – but we need to start somewhere, so I am going to list a few truths about node.js that will give you some context upon which to hang the rest of the article:

  • It is JavaScript for the server side – all things related to browser and DOM removed.
  • It is based upon Google’s V8 JavaScript engine, a platform abstraction layer called libuv, and a core library originally written in C but now mostly rewritten in JavaScript.
  • It was created in early 2009 by Ryan Dahl of Joyent (this company still maintains and develops Node.js).  Here is a link to Ryan’s original presentation of Node.js, courtesy of Youtube: http://www.youtube.com/watch?v=ztspvPYybIY
  • It is intended to be used primarily as a high performance back end coding platform that is utilized by front ends like browsers over HTTP or other front end applications using TCP sockets to transmit data to and from the server.
  • It follows an event driven, non-blocking I/O model.  What this means is that the event loop driving Node does not have to wait for IO (typically, IO is glacial in performance compared to other code execution).  This is an asynchronous rather than a synchronous approach, allowing Node to be highly scalable.  (I plan to delve more deeply into asynchronicity with Node.js in a future article.)
  • It’s a hugely popular open source framework upon which hundreds of other liberally licensed, freely available open source frameworks and add-ons.  (See my previous article about Github: Stepping into the New Age of JavaScript using Github).

Let’s try and defragment the above list a bit by focusing on the potential benefits of using Node.js. There are at least three compelling reasons for using Node.js:

  1. The single most compelling reason for using Node.js as the basis for the server half of a client-server solution is that it is highly scalable and highly performant under heavy request loads.  This holds true more so when the bulk of these individual requests tend to be small, rather than large,  Fortunately, that is the usage pattern of many, or perhaps the majority, of web-based applications.
  2. Another often cited reason for using Node.js is that JavaScript is a well known programming language, and usage of Node on the back-end allows all code, both client and server, to be written in JavaScript.  There are even efforts afoot to try and systemize the reuse of portions of code on both client and server (think data model definition and validation).
  3. If the aforementioned benefits of using Node are not enough for you, there is a biggie remaining - the plethora of useful and professionally maintained open source frameworks and libraries available for Node.js.  If there is a particular type of resource you need for your back-end code or some problems you need to solve by writing code, there is a very good chance that it has already been done for you.  I’ll have more to say on the subject of add-ons further on in the article.

Ok, that was a lot of wordage to wade through – let’s look at some code.  We will use some code examples as a springboard from which to begin understanding how Node.js works.  The smallest, simplest node.js program might look something like the following code snippet and exist in a text file named helloworld.js:

Of course this is a useless program, but it will serve to get us started.  Perhaps the first thing worth noting is that it uses standard JavaScript syntax (single quotes instead of double quotes would have worked just as well).  To execute this program on your system, Node.js would first have to be installed (we’ll see how to do this later).  To run it, you would use the command line or open a terminal or command window, whatever your OS provides.  After ensuring that that both Node and helloworld.js are in the executable path, enter the following: node helloworld.js.  Upon execution of the program, you would see the words Hello World in your console, i.e. the same UI you used to execute the program.

Let’s look at a program that does a little bit more – another program that could also be written in client-side JavaScript.  Again, this program must not reference any browser elements - these don’t exist in Node.js JavaScript.  The code snippet for this example is nearly as simple and pointless as the code in the first example, but it goes a little further in reinforcing that fact that server-side Node.js JavaScript looks the same as browser-based JavaScript.  You could put it in a file called helloworlddated.js and run it in a manner similar the previous example, i.e. by entering: node helloworlddated.js.  Here is the source for helloworlddated.js:

The resulting output in your console would be something like this: Hello World.  Today (day/month/year) is: 15/06/2013.  Running this small program gives a more convincing demonstration that Node.js is JavaScript for the server side… yet, it is not a thematic demonstration.  Sure, you might put some similar code into a Node.js program, or you might even create a server side program that runs now and again to perform some useful function.  However, the main idea behind the creation of Node.js was to provide solutions for high volume client-server types of apps.

What might a more thematic Node.js program look like?  Here is an extremely stripped down version of the beginnings of one such program:

This example program was taken from: http://nodejs.org/ , the home of Node.js at Joyent.  What is it doing?  It looks almost as if it is acting like some sort of miniature web server… which is exactly what it is, a little five line web server that serves up some dedicated content instead of HTML files.  This simple web server written in Node responds with “Hello World” for every request.  A very common pattern for Node.js programs is that they will not only provide web content or other content, but they also provide the mechanism to receive client requests and return server responses.  The good news is, that because of some built-in Node.js library code, creation of such a web server is a trivial matter.  Actually, even creating a full blown, actual web server is a fairly trivial matter, if certain add-on frameworks are also put into use.  (I will provide some more detail on this subject further below and will probably eventually write more articles about various modules that extend the basic functionality of Node).  I am getting ahead of myself though.  Let’s dissect the above code snippet, piece by piece.

Line 1:  var http = require(‘http’);

It looks like standard JavaScript syntax, but what does “require” do and what does that argument of ‘http’ mean?  The first thing to understand is that much of the functionality of Node is compartmentalized in to something called a module.  Node comes installed with several built in modules and you can download hundreds of others from Github.  In Line 1, the argument ‘http’ is the name of a module that pulls in functionality surrounding the use of http, in this case the ability to create a simple web server.  the function named “require” is a globally available Node.js function that allows you to pull the content of modules into your code.  As one might guess, it returns an object that can then be used to invoke the functionality exported by the module.

Line 2:  http.createServer(function (req, res) {

In Line 2, we use the http object returned by the require(‘http’) call in Line 1, to create an http server.  We can see the beginning of an anonymous function definition that takes two arguments – a request argument and a response argument. (Note: For those unfamiliar with anonymous JavaScript functions, you may simply think them as an unnamed functions that are dynamically declared at runtime.  This link to Helen Emerson’s blog at Helephant.com offers a very clear explanation.).  The anonymous function that is being passed here (definition completed on subsequent lines), will be invoked when Node receives an HTTP request on the port whose number is passed in Line 5.  This anonymous function will provide the response to that request.

Line 3:  res.writeHead(200, {‘Content-Type’: ‘text/plain’});

Line 3 is the first line of the body of the anonymous function began in Line 2.  When the anonymous function is executed, this line of that function writes an HTTP response header into the response.  You may have seen similar code in other types of back end solutions or if you have ever examined the workings of the HTTP protocol.

Line 4:  res.end(‘Hello World\n’);

Line 4 contains the end of the anonymous function body and it adds the string ‘Hello World\n’ to the body of the response.  It also caps off the response because the end method does that, in addition to accepting the Hello World string content.

Line 5:  }).listen(1337, ’127.0.0.1′);

Line 5 is the end of the createServer call and it chains a call to the listen method of said server after the createServer call is completed.  The first argument to the listen method is the port number that the server should use to listen for requests (a small leet-speak joke if you didn’t notice).  The second argument is the url upon which the server should be listening.

Line 6:  console.log(‘Server running at http://127.0.0.1:1337/’);

Line 6 provides output for the console about what has been done.  After the program has been started (as before this involves saving the code into a JavaScript file and running it from the console by typing; node <the javascript file name goes here>), copying that link from the console and running it from a web browser will cause the words “Hello World” to be displayed in the browser.

That wasn’t much work to code, even for a typical “Hello World” sort of program – but the reality is that program is ever so much more than a typical “Hello World” program.  It is the basis for a server-side program that might perform any of various kinds of behaviors or actions for a client side piece.  That particular example is an HTTP kind of animal, but an analogous TCPIP based program is just as easily created.  Here is an example of a simple TCP server which listens on port 1337 and echoes whatever you send it:

It is beyond the scope of an introductory article to try and bridge the gap from the previous thematic example to showing explicit examples of code that might back real world projects - but here is a link to an eye-opening page at Github: Projects, Applications, and Companies Using Node.  Scan the notes in the two large tables on the page that this link leads to and you will see a list of uses and functionality that empirically demonstrate what Node is capable of.  The list of usages is broader and deeper than one might expect - I think it’s pretty impressive.  Some readers might think this paragraph would be a natural place to end this first introductory article, but wait, there’s more!

There is a wealth of freely available external modules that can be used within a Node program.  I would be remorse in my introductory duties if I did not provide at least a glimpse of what kinds of functionality might be offered by those modules, how one might find out more about them and how they can be installed.  The original creator of Node.js and those that provided additional essential work on Node believed that Node should provide a basic, highly scalable, extensible engine for server-side work.  These external modules offer solutions (some of them competing with one another) that fill in the gaps of what is not built into Node itself.  There is a way that you can search for, find out about, and download nearly all of the external modules that have been created for use with Node.js – follow this link the Github Modules Search page.  However, In addition to downloading these modules directly Github, there is another, better way to get  them onto a system on which Node has already been installed - Node Package Management.  Node has a built-in command line program called npm that can be used to install modules (note that they must still be required into your code).  The syntax to install a module looks like this: npm install <some module>.  To learn more about how to use npm one can either search the web or else on the command line, type: nmp help install.

The functionality provided by this huge number of external modules could be divided into a number of different categories – but I decided upon just four.  My four basic categories are: Middleware, MVC Frameworks, Database Access, and my cop-out category, “Others”.  I will provide no examples for “Others” but a host of very useful library, utility and debugging-oriented modules would fall into this category.  In any event, these four divisions are highly arbitrary and probably woefully inadequate, but categorizing things in this manner offers a way to approach a summarization of what is really a rather extensive topic.

The term “middleware” is unfortunately, a sorely overloaded one.  It has had multiple meanings in the history of our industry and yet another is introduced in the context of Node.js.  In Node, the term is a loose descriptor for functionality that exists between the client portion of an app and the logical portion of the server-side app that makes up the application code.  I am going to give a couple of examples of what I consider to be middleware in Node and you can agree with me or disagree if you wish.  Fortunately, the “rose by any other name” principle applies to the two examples I am going to give you – they are both arguably best of breed in their class – regardless of what name you give that class.  The two examples I will put forth are Connect.js and Socket.io.  Connect is actually a framework of sorts itself, one that offers a surprising amount of functionality.  The bulk of this functionality centers upon fleshing out the things a web server can do that the http module does not do, or does not handle as well (i.e. powerful but elegant: logging, cookie handling, favicon handling, static file serving, error control, etc.).  Socket.io is one of several modules that are available for node that provide Web Sockets functionality.  Socket.io is one of the more popular if not the most popular and it is very good at gracefully degrading into alternative communication methodologies when older web servers and/or browsers are in play.

MVC stands for Model-View-Controller of course, and there are a number of Node.js external modules that offer MVC or MVC-like functionality.  (Note: The scope and definition of the MVC style of architecture is beyond the scope of this article, but here is a link at Wikipedia that offers a starting point of you are unfamiliar with it: Model-View-Controller.)  I am going to mention a few MVC-ish modules here but not go into great detail about any of them.  I will briefly discuss Geddy, Express.js and Sails.js.  Geddy is billed as the “original MVC” solution for Node and was quite popular at one point though is perhaps growing somewhat long in the tooth now.  It is more of a standard, Rails-like MVC solution than say, Express is.   Express.js is almost itself a framework for creating MVC frameworks – in fact some new MVC frameworks are based upon Express.  However, you can do MVC in Express.js, and out of all the MVC-like solutions I read about, it seems to be the most popular.  As of this writing, I have never yet used Geddy or Express (though I do plan to experiment with Express), so bear in mind that my opinions on Geddy and Express are based upon reading from multiple sources, not personal experience.  Sails.js is an up and coming solution that is generating a lot of interest.  Its claim to fame is the ability to easily generate a RESTful API, the delivery content of which is JSON.  This is a natural fit for creating hybrid MVC solutions in conjunction with a front end framework like the popular framework Backbone.js, a partial MVC solution that will natively digest JSON via internal JQuery AJAX calls.

The available modules for database access are in a way, astonishingly rich but in another way, surprisingly poor.  They are rich in that among them there is surprisingly broad support for what these days are being called “NoSQL” databases.  The examples I will list are MongoDB,  CouchDB and Redis.   MongoDB and CouchDB are both document-centric databases that accept, store and re-issue data in JSON format. (Note: If you are not familiar with JSON you should fix that ASAP.  Here is a link at Wikipedia that offers a starting point for beginning to understand JSON.)  Redis is a key-value in memory database that may be optionally persisted to file (the type of persistence in use by Redis is referred to as “durability”).  Between MongoDB and CouchDB, MongoDB seems to be more popular, though CouchDB has strong adherents.  I have played with MongoDB some and I like it.  It is worth noting that accessing MongoDB from Node is much easier if done via a simplifying API such as Mongoose.  Redis is purportedly the most popular key-value store that is out there at this time.  Ok, now we’ve come to the “surprisingly poor” part.  As far as I know, as of this writing, the only available module level support for relational database access for Node is for MySQL.  Doubtless this will be addressed soon, to at least provide support for Oracle and SQL Server.  That is not to say that your application code could not access such resources itself, for example, via web service.  Nonetheless, I consider this to be a fly in the ointment for Node.js.

We are nearing the end of the article and I am finally going to tell you how you can install Node.js - assuming I have interested you enough for you to give it a try.  The easiest way to get Node onto your system is to go to the main Node.js site sponsored by Joyent and either hit the big Install button on that page or else click the smaller Downloads button and choose an explicit installation on the subsequent page.  There are also other ways to get Node onto your system and install it.  You can Fork the project on Github and then Clone it down to the target system.  Alternatively, you could download the source code from Github and build it yourself.  The details of how to apply the latter two of the listed methodologies are beyond the scope of this article and are platform dependent as well.  Undoubtedly though, the best way to get started with Node.js is to install it from the Joyent sponsored site as described above.

In closing, I’d like to re-iterate Node’s strengths and also highlight the weaknesses I have mentioned. When high performance scalability is important, Node.js and judiciously chosen additional modules may offer a great replacement for more traditional solutions involving a combination of traditional web servers and architectures such as Ruby on Rails, Java Servlets or EJB-driven solutions, and ASP.NET style of pages.  Additionally It’s all JavaScript all of the time and offers a number of NoSQL solutions.  Every bit of this code is open source and free under licenses such as the MIT variant and other permissive GPL licenses.  A lot of this code is offered with dual license options as well.  On the other hand, if you are looking for built-in, strong relational database support or if you require a solution involving frequent heavy streaming between client and server, then Node is probably not the best choice, at least not yet.

Node is just warming up – this too may be a weakness – some of the modules you will be interested in will likely be pretty fresh and thus reliability and sustainability may be in question. However, I think most outsiders and insiders would say that it has already achieved critical mass, and this is just the beginning of a bright and beautiful future – Node is just warming up!