Services in the cloud are great, and I make some us of them when I find them appropriate or particularly useful. One of the cloud services I've used with some frequency is the del.icio.us bookmarking service, because I can access the bookmarks from any location. Having my data in "the cloud" means it isn't really mine, so I attempt to retain a copy of it to be on the safe side. I certainly don't know when a cloud service will decide to go "puff" and take my data along with it. In order to accomplish this, I'm implementing a trivial bookmarking application a la del.icio.us, with a CouchDB back-end. Now, before I continue a disclaimer: I'm not a developer, so there are plenty of things I won't get right, such as my lousy CSS, poor JavaScript, etc. Feel free to provide a better solution, but I hope you'll find this as interesting as I do.

Let's first have a look at the final result. You can follow along by installing a version of couchapp and CouchDB of course. Then go to the program's Github repository, clone the app, push it into CouchDB and launch the URL:

    git clone http://github.com/jpmens/scrumptious.git
    cd scrumptious
    couchapp push http://localhost:5984/scrumptious
    

By pointing your Web browser at the URL given you by couchapp, you'll be redirected to a show function, from which you drag the generated bookmarklet URL to your browser bar: (I have to "generate" the bookmarklet for you because it contains the URL to your CouchDB instance; the show function determines the full path and creates the bookmarklet containing that.)

Go ahead and drag the link into your bookmarks bar: mine is shown as the little recycle symbol. When you've done that, click on view all bookmarks, which will take you to the paginated list of the bookmarks stored in your CouchDB. (Yes, there are three there already. :-)

Clicking on the greater symbol takes you to a page describing the bookmark. Click on the bookmark title to take you to the URL saved with the bookmark. Within the description page, click on the "delete bookmark" to kill the document.

When visiting a site you wish to bookmark, press on the bookmarklet you saved in your bookmark bar when on the page. The text you select on the page becomes the bookmark's description, and you'll be propted to enter a list of (white space or comma-separated) tags for the page.

The bookmark is saved into CouchDB with a shortish _id (swiped from Jan's io), whereupon you get a confirmation page. (I'd have liked to have the update handler return a 302 redirection to the browser, but that isn't possible with CouchDB 0.11.)

The resulting bookmark looks like this:

One URL you may like to see is produced by a couchapp list: /_design/app/_list/bookmarksfile/all produces a bookmarks.html file you can import into some Web browsers. Now that we've seen the app at work, let me discuss its innards a bit.

Standalone applications

Couchapps are standalone applications. They consist of JavaScript and JSON documents stored in a CouchDB design document:

As a proof of concept, look no further than CouchDB's built-in administrative interface. Futon, as it's called, is just a collection of HTML, CSS, and Javascript files, and it is a fully functional database browser and JSON document editor, as well as the runner for CouchDB's functional test suite. The test suite is written in Javascript and runs from the browser. What else could more strongly indicate that Ajax and CouchDB go hand in hand?

I find that there is little documentation about couchapps, which is fair enough if you believe in "read the source" strategies, so I've created this trivial couchapp, and I'm going to point out the things I find most interesting, or situations where I had a hard time understanding what happens in couchapps.

del.icio.us

Del.icio.us is a service which lets me submit, describe, and tag URLs I find interesting. I can share these with the rest of the world or mark URLs as being private. They have a sexy Web interface with which I can create and manage my bookmarks, and there exist dozens of bookmarklets and clients for managing them. They also have an API.

The del.icio.us API allows me to submit an authenticated HTTP GET request with URL-encoded parameters containing the data for the bookmark I want to add to my collection. As an example, I can point any HTTP client (e.g. my Web browser) at a URL like

https://api.del.icio.us/v1/posts/add?
  url=...&description=...&tags=....

and create a new bookmark.

I thought it would be trivial to implement the del.icio.us API on top of a CouchDB database, which would have had the great advantage that I could use any of the dozens of del.icio.us clients to drop bookmarks into my CouchDB. Unfortunately, the guys over at del.icio.us decided to implement a not-so- REST-y interface to their service; to add a new bookmark, del.icio.us uses an HTTP GET request instead of a POST or a PUT request.

Without some sort of middleware which converts a GET request to a POST, there is no way to add a document to a CouchDB database -- it must be either a POST or a PUT request. I can call an update handler, but update handlers in CouchDB support POST and PUT only, and Jan Lehnardt quickly pointed out in no uncertain terms that I was barking up the wrong tree, trying to GET into an update handler. He was right in pointing out on #couchdb that I was trying to circumvent REST principles. (And I'm grateful to him for having put me right.)

So, in order to POST to a CouchDB, I had to create a browser bookmarklet which does a POST request. There is a huge amount of information on how to make a bookmarklet, but I finally swiped code from Smart Bookmarks and Bookmarklets, which provided just what I was looking for: one click and the JavaScript submits a POST request to my update handler.

Tools used

Editing JavaScript via the Futon interface is a no go; you'll go crazy attempting it. The only sane method is via the couchapp program which does a quite a bit of magic on the fly. What does couchapp actually do? Well, these are some of the things I've found so far.

  • Couchapp provides helper macros for getting code into your show and list functions. If you look at the file list/ls.js, you'll see a JavaScript comment that looks like this:

    // !json templates.index

That is parsed by the couchapp utility when you invoke a push and the content of the file templates/index.html is inserted into our ls.js JavaScript, much like an #include in C, for example. Go ahead and view the app design document with Futon. The file's content becomes a JSON object called templates with an name index and as value, the content of the file. Neat.

  • A similar macro facility called !code inserts JavaScript code from the specified path, which is relative to the top-level couchapp directory. (See example in updates/post.js.) When couchapp is run, it replaces the line with the content of the JavaScript file.

  • Files in the _attachments directory are dumped into your CouchDB database as attachments. You'll note I've stored the CSS for the HTML in such an attachment, for example.

  • I can have couchapp add documents to the CouchDB database; see three examples in the _docs directory. (This is why you get three bookmarks in your database when you push my simple app into your CouchDB.)

  • JavaScript files in the lib/ file-system hierarchy are added to the design document, from which you can load them by requiring them, as in var mustache = require("lib/mustache");. Omit the .js extension when doing so, or CouchDB complains about infinite recursion or something similar.

Couchapp contains a lot more magic: invoke couchapp generate whatever and explore the files created for you in the file system!

You'll note I make some use of mustache.js -- an indespensible templating system which thankfully allows me to keep code and HTML separate from eachother. Jan recently wrote a bit more about mustache.js, and the project's README contains the rest of what you need to get started.

Import from del.icio.us

You can import your existing del.icio.us bookmarks rather easily by grabbing an XML stream containing your own bookmarks and dumping that into a file mine.xml:

    curl -sf -u username:password -o mine.xml https://api.del.icio.us/v1/posts/all
    

Let's massage that XML with the following small program (delicious2couchdb.pl) by "translating" the XML entities to corresponding JSON objects ready for CouchDB:

    #!/usr/bin/perl
    
    use strict;
    use XML::Simple;
    use JSON;
    
    my $json = new JSON;
    my $ref = XMLin('mine.xml', ForceArray => 1);
    my @docs;
    
    foreach my $post (@{$ref->{post}}) {
        my $book = {
            type => 'bookmark',
            url  => $post->{href},
            date => $post->{time},
            tags => [ split(/\s+/, $post->{tag}) ],
            title    => $post->{description},
            description  => $post->{extended},
            hash => $post->{hash},      # Keep for reference
            imported => JSON::true,
            };
        push(@docs, $book);
    
    }
    
    my $all = {
        docs => [ @docs ],
    };
    print $json->encode($all);
    

The format produced by delicious2couchdb.pl is ready to be POSTed into CouchDB's _bulk_docs API:

    ./delicious2couchdb.pl | 
      curl -X POST http://localhost:5984/scrumptious/_bulk_docs -d@-
    

The funny-looking @- is for curl: data is read from the named file, and as the file is a single dash, curl reads the data from standard input.

Further reading

There is a lot more to discover here, and I'd like to point out some neat Couch Apps:

  • Jan Lehnardt's io (mentioned above already) is a CouchDB-based URL shortener.
  • J Chris A. created sofa, a CouchDB-based blog, which is what he uses.
  • Taskr is a task-tracking app by J Chris A.
  • More at CouchApp.org

The chapter on Standalone Applications in CouchDB: The Definitive Guide talks a bit about Couch Apps, and I don't want you to miss my CouchDB Reference Card

Database, CLI, Backup, CouchDB, NoSQL, and bookmarks :: 25 May 2010 :: e-mail

Comments

blog comments powered by Disqus