Vishal Parikh

TLDR; It’s an online dance party! Check it out. Source.

Danceynous (for Synchronous Dance Party) uses Meteor to make a real-time, interactive, YouTube playlist

Well, I can’t go a month without building a new side project, now, can I?  The release of Meteor last week made it doubly exciting.  I’ve spent about 30 hours working on Danceynous, and thanks to Meteor have come out with something pretty fun.

Danceynous is a social music playing app.  It is pretty similar to Turntable.fm, but aims to be a little more interactive.  Everyone can submit songs to a playlist and vote on them.  The playlist is reordered in real time as it plays, resulting in a fair amount of point battling.  The frontend and backend are powered by Meteor (with help from Backbone and jQuery).  The music is sourced from the YouTube player API.

There was a ton of buzz (in the developer community at least..) when they release the preview of Meteor last week.  I guess that shows how fed up people are with the state of javascript development.  I have been following Luna, Asana’s Javascript-based infrastructure for a little while now, hoping they would release it, but no dice.  I have been using Backbone for a few months now, and it is neat, but just not particularly life-changing.  Considering the whole thing is under 1300 lines of code, I wouldn’t expect it to be.

Meteor and Luna have a lot of similarities.  The idea is to unify the client-side app with the data being served on the server-side.  This way data transfer happens automatically instead of the client having to explicitly request data and the server having to explicitly serve it.  It turns out you need a lot of pieces to make this work well: Reactive front end with client side database caching, websocket emulation, client simulation on the back end.

In the end you get a much more clean and maintainable code base, with much less work needing to be done on the server side, and much less redundant coding.

I am going to briefly walk through the different pieces of Meteor that I used and talk about some of the challenges I faced and also what was remarkably easier compared to Rails+Backbone.

There are four pieces to Danceynous: Synchronized player, the real-time playlist, the chat room, and the YouTube search.  The first three are powered by Meteor; the last is purely client-side YouTube API.

The chat-room is incredibly straightforward; so much so, that I’m not even going to bother going into detail about it.  There is literally just a Chats collection with a bunch of message documents (‘documents’ are what MongoDB calls ‘records’) that get rendered as they are inserted into the collection.  Due to the natural reactivity of Meteor I don’t need to handle sockets myself to detect incoming messages.  My local database gets updated automatically, which triggers the render code, which inserts the new message.  MongoDB collections are stably sorted as well, so I don’t even have to do any “order-by” nonsense.

The real-time playlist is almost as easy as the chat.  There is a Songs collection.  Users can insert song documents into it.  Each song has a points attribute.  Players can increase and decrease the points on the song, which triggers resorting the list.  I use the $inc atomic method in Mongo to avoid any RMW race conditions.  Note that everything I have described so far is entirely coded on the client side.  A client creates a chat message; a client creates a song; a client adds points to a song.  Meteor, reacts to the client side changes and propagates them through the server to remote clients, updating their databases and triggering them to re-render.

The synchronized player is slightly more complicated.  The general idea is that each user has their local version of the song.  The videos are streamed directly from YouTube, so all the server can do is trigger the client to start loading the file and then hope for the best.  Users with slow connections may spend several seconds buffering, which puts them behind other users.  Users re-synchronize when a new song is loaded, which means the slow users may skip the last few seconds of the song.  There are two things that need to be done here.  We need to keep track of the time progressed through the song, so that someone who connects in the middle of the song can start at the appropriate time.  We also need a way to detect that a song has ended to trigger the synchronized loading of a new song (remember, the music playing happens entirely on the client side, so the server does not know how long the song is or when it ends).

To keep track of the time progression of the song, the client simply polls its local player periodically to check the time.  It then updates the song document with a time attribute (which is immediately propagated to the server). This time attribute is read by any new users who join the party and they start the song at that time.  The catch here is that since the time updating is done on the client side, we have potentially many users updating times that do not necessarily match.  One way to solve this is to pick one user as the “host”.  However, what do we do if the host disconnects?  We can pick a new host, but the problem is that it actually takes quite a long time to detect if a user disconnected (more on that later), which could cause problems.  If you have played Halo multiplayer, you have experienced this process (i.e. when screen goes black for 30 seconds after someone drops out, while they pick a new host).  I actually have all the clients attempt to update the time.  However, only the client who is farthest along actually gets through.  This is done by only updating the time if your  local time is greater than the server time.  The first client to reach time X will update the server.  All of the other, slower, clients will not perform any update.  Again, this is accomplished using atomic RMW Mongo commands to avoid race conditions.

We use a similar process for switching songs.  The first user to reach the end of the song updates the database to play a new song.  All the other, slower, clients receive this update and start playing the new song.  It is important to note that all of this is performed on the client side, with Meteor automatically syncing the clients with each other.  Think about how this would be done on a Backbone+Rails platform.  (Hint: it would involve a lot of POST /api/room/:id/current_time?time=t).

Of course, it wasn’t all gravy.  There are few things that I struggled with while using Meteor.  The first, that I alluded to earlier, is detecting user disconnect.  (This is important because I have a list of people currently in the room).  To be fair, this is a problem with all systems that use socket emulation.  The issue is that since the socket is emulated by long-polling the server, the server has a hard time detecting when the client actually disconnects, versus disconnecting as part of the periodic polling process.  The standard way to detect a disconnect is by having the client set a keepalive record in the database.  If the client disconnects, the keepalive record will expire and you can mark all owners of expired records as disconnected.  Unfortunately, since the long polling time is 25s, that is the minimum disconnect timeout.  I am using a 70s timeout right now to avoid false disconnects.

Another issue I had that is more Meteor specific is attaching things to DOM elements.  This is straightforward in conventional Javascript.  You just use a $(document).ready(dosomething) call.  However, since everything in Meteor is reactive, the document.ready paradigm does not apply.  When the “document” is ready, most of the HTML is not rendered yet.  In general, this is not an issue because you can just put everything in templates which are naturally reactive and will get called.  The template event system also handles event delegation in a fairly straightforward way.  There are some places, however, where this breaks down.

For example, the YouTube api needs to be passed an ID of a DOM element where it will put the flash player.  Passing the ID in the initialization does not work because the page isn’t rendered yet.  Even calling Meteor.flush() does not work if the the template that contains the ID depends on some record, as the records may not have arrived from the server yet.  Once the record arrives, the template is rendered, but I was not able to track down a way to detect the rendering and call a function to attach the player afterwards.  At first I used a timeout that was long enough that the page had rendered, but that was a terrible hack and caused me to give up a piece of my soul.  Ultimately, I put the ID in a section of the html where it did not depend on the presence of a record.  I then called Meteor.flush() to ensure that the tag was rendered.

A final issue I had with Meteor was that, in general, my code feels very unstructured.  Unlike Backbone, Meteor does not lend itself naturally to encapsulation.  This is due to much of the logic being in the Templating, and also the extensive use of global collections and session variables in order to enable the reactivity.  I am sure there is a way to write great, modular code with Meteor.  It just doesn’t come as naturally as it did with Backbone.

That sentence may be confusing since I said I am using Backbone on this project, but I am only using the Backbone router.  The Backbone Models and Views are superceded by Meteor reactive templates and Collections.

I have some other, more general, concerns about Meteor.  The first is client side performance.  The code ran fine on my Macbook Pro, but I am curious to see how well it runs on my 2.5 year old smartphone.  Also since Meteor is constantly pinging the server, how well does it perform on a flakey connection?  I imaging the user experience is better than a conventional framework because of the client side database that simulates a server response as it waits for the real response.

Also, as many people noticed last week, Meteor currently has no built-in way to effectively secure the database.  As in, I can open up firebug and type Users.remove({}) and delete all the users (please don’t do this.).  There are a few workarounds right now.  I can effectively disable the ‘remove’ function for certain collections.  I can use publishing (which I do) to restrict the records that users have access to (so you can only remove users who are in the room with you).  The conventional way to solve this problem would be to disable ‘insert’, ‘remove’, and ‘update’, and replace them with more restrictive functions that the server validates and then sends a response.  I am curious to see what the Meteor team comes up with and if they can manage to maintain the real-time feel on the client side and elegance on the server side.

Overall, I am very happy with how this project turned out.  I definitely see myself using Meteor for a serious project when it is more mature.