In addition to push-mail, calendar and PIM synchronization, BlackBerry devices connected to a BlackBerry Enterprise Server (BES) with Mobile Data System (MDS) offer an exciting technology which can be used to extend web applications to mobile BlackBerry devices via what is aptly named MDS Push. This paper discusses a real-world scenario developed on a Linux host. This document describes a framework written to facilitate the distribution of information to BlackBerry handheld devices. In essence, I'll be showing you how to easily push out a wealth of information which appears on the devices as depicted: inbox listthe page The technology described herein doesn't require a Unix/Linux environment; it can easily be implemented on Windows or Mac OS/X as long as the tool requirements are met. I've used Perl and MySQL because both are readily available for these platforms, but this could easily be ported to work with Perl and a different database backend, or any language that allows HTTP connections (PHP and Java come to mind). Furthermore, if you want to experiment with the technology without having a full-blown BES setup, you can use the BlackBerry Device Simulator together with the MDS Simulator to develop and experiment before rolling your application out to real user's devices. Using the simulators of course also greatly lowers testing cost, because a wireless data connection is neither needed nor required. Read more about the simulators and the JDE. In my tests, I utilized a BES running on Windows with a properly configured MDS. I used version 4.1, but this may work on a previous version; check the documentation for your version where appropriate.

Intro

Some of our BlackBerry users require up to date financial statistics which, as soon as they are available, should be pushed out over the air (OTA) to the users' devices. I've opted for what is called a Browser Message Push.

What is MDS Push?

The Mobile Data Service (MDS) is a software component of the BlackBerry Enterprise Server (BES) . I think of the MDS as a gateway between the wireless BlackBerry devices and the corporate network. The MDS can be used in the corporate environment to distribute (i.e. push) information and applications to devices utilizing the BlackBerry network infrastructure, and itMDS is used as a proxy between the BlackBerry browser (a.ka. MDS browser) on the devices and your corporate network and/or the Internet. Push applications can be created using any language that can initiate a network connection to perform an HTTP request. A push message is made up of a request with a set of headers and a body (the page to be sent to the device). In essence, there are three type of Browser push that can be sent to a BlackBerry device (see Pushing Web Content to BlackBerry Wireless Devices for an more verbose description of the different kinds of browser push supported by the MDS):

Browser Channel Push

A Browser Channel Push sends a web page and optionally two icons to the device. The web page is stored in the browser's cache, and an icon is displayed on the home screen of the device.

Browser Content Cache Push

With this type of push, a page is pushed to the cache of the BlackBerry browser, but the user is neither given any type of indication that a page has arrived, nor that a page has been updated.

Browser Message Push

A Browser Message Push arrives in the inbox of the device. When the user opens the message, the browser is launched to display the page, which is kept on the device until the message is deleted. This message is neither sent to, nor will it appear on the user's desktop inbox; it is sent directly to the device.

Using a Browser Message Push means that this message will appear in the user's message list (i.e. inbox) on the device, and it will be marked with a small icon to differenciate it from a normal email message. Similarly, the device home page will indicate that a new browser push message has arrived on the device. Device home page with MDS indicator The device inbox displays the browser push message by title; opening that message launches the browser to display the page: inbox listthe page

What to Push

In the scenario that follows, we'll be pushing ready-made HTML pages to the devices. These will be held in the message list on the device, allowing them to be viewed even if the device is not in a wireless coverage area. Any type of page supported by the BlackBerry browser can be pushed into the device cache. These include HTML and plain text. As described above, my users need a financial statistics page. The layout of the page was originally tested on a desktop browser (Mozilla Firefox to be precise), and it was tweaked for layout using one of the BlackBerry device simulators. What the page contains and what it looks like is almost irrelevant to the application at hand, if you keep a couple of things in mind:

  • The BlackBerry Browser which will later be used to view the pages which are pushed to the device, has fewer capabilities than your full-blown desktop web browser. Consider that carefully when developing the page which you'll be deploying to the devices
  • Choose the subset of HTML that is used in your page carefully. You may wish to consider using cHTML, or even WML.
  • The display of the BlackBerry devices is limited in size. You might have a whopping 20" monitor on your desk, but the device's screen is not much larger than a few stamps
  • Ensure that your BlackBerry browser is configured to support the features your pages will be using (think tables, JavaScript, etc.). On your BES you can configure your devices' browsers via an IT policy which is then applied to the devices. For the browser settings proper, look at Create Directory of BlackBerry Enterprise Server Users. This screen shot shows the IT Policy settings I deploy to our users: Browser Policy

The Little Big Picture

The diagram depicts the main components I'm using in this application. First the HTML page proper is produced. Since that part depends a lot on what information you wish to push out to your BlackBerry user base, and because that part is proprietary, I'm won't further discuss it here. Suffice to say, that at the end of the day we have an HTML page which will be viewable by the BlackBerry Browser, and that that HTML page will be stored in the database. This is not a technical requirement; it could just as easily (no, that is not true: more easily) have been stored somewhere in the file system, but I want it in the database. The Apache web server runs on Linux but might as well be hosted by your BES server on Windows. In our environment, it massages the statistical data which is then transformed to HTML on the one hand, and on the other hand, it hosts the getpage and notify programs which will be invoked by the MDS. Additionally it could host a web page on which you allow your users to subscribe to distinct pages. Little Big Picture In my scenario, subscriptions to this service are handled by an administrator. Due to the relatively high sensitivity of the data, I don't want users subscribing to the push service themselves, although that would be very easy to implement (see Creating a RSS/RDF Push Service for BlackBerry for an example implemented in the PHP language). The subscribe program is invoked by an authorized user to add a device PIN or a device-user's email address to the database. The getpage program is a CGI-type script that can retrieve the page, given its identifying tag (also called a page name or an appname or application name). The main workhorse here is the pushout program; it scans the database for new documents to send out and sends appropriate push requests to the Mobile Data System (MDS), which in turn pushes these messages out over the wireless network (OTA) to the devices. In newer MDS versions, the MDS can track and record delivery of the push messages to the devices, which is useful for determining whether the message actually reached the device. The notify CGI program will be invoked by the MDS with a device and message identifier which I'll use to correlate delivery to the original push message, therewith recording which push messages were actually delivered to users' devices. A log of deliveries and their correlated notifications will be held in the database.

The Backend Database

As mentioned above, the framework at hand uses a MySQL database backend and the supplied database schema was designed for that. It ought to be no problem at all to adapt the schema to use any other backend such as PostgreSQL or any other. There are four tables in the schema:

users

The users table holds a touple consisting of a userid (a unique user identifier such as jdoe or J9001) and a push destination pushdest. This column contains either the unique device PIN or the user's BES email address to which the push messages should be directed.

CREATE TABLE users (
  userid VARCHAR(128) NOT NULL,    -- user identifier
  pushdest VARCHAR(128) NOT NULL,  -- PIN or email on BES
  modif  TIMESTAMP(14),            -- modification timestamp

  PRIMARY KEY(userid)              -- unique
);

subscribers

Each user in the users table can be subscribed to one or more "pages", and this table records which "pages" the user has subscribed to. As mentioned above, in my case the administrator will be updating this table, but you could easily create a web page with which users can subscribe to pages themselves.

CREATE TABLE subscribers (
  userid VARCHAR(128) NOT NULL,         -- user identifier
  appname VARCHAR(128) NOT NULL,        -- application name

  modif  TIMESTAMP(14),                 -- modification timestamp
  PRIMARY KEY(userid,appname)           -- unique
);

pages

This table holds the data to be pushed to the devices; the "pages". Each page (also called appname, as in application name) is given a unique tag or name, and a title. The appname will be used as a channel identifier (see below) and the title will be used as the page name as seen by the user. In the screen shots at the top of this document, the title is "fupps.com: Turnover FY 1993". In addition to the application or page name, this table holds the page itself, encoded as Base-64 stream in the data attribute.

CREATE TABLE pages (
  appname VARCHAR(128) NOT NULL,   -- application name
  title   VARCHAR(128),            -- title of push message
  data   MEDIUMBLOB NOT NULL,      -- Base64-encoded HTML/text page
  modif  TIMESTAMP(14),
  PRIMARY KEY(appname)             -- unique
);

pushlog

Each push message sent to the MDS is recorded in the pushlog table, identifying the userid to whom the page was pushed. In addition, the appname is recorded as well as the (redundant) push destination pushdest. Each message pushed out is given a unique identifier called pushid; this is recorded as well. As soon as the MDS has delivered the message to the wireless BlackBerry device (or hasn't, in case of a failure), this table is updated to record that fact. The attribute status indicates success or failure of the message push.

CREATE TABLE pushlog (
  id INT NOT NULL AUTO_INCREMENT,       -- unique identifier
  userid VARCHAR(128) NOT NULL,         -- user identifier
  appname VARCHAR(128) NOT NULL,        -- application name 
  pushid VARCHAR(128) NOT NULL,         -- unique MDS push identifier
  pushdest VARCHAR(128) NOT NULL,       -- PIN or email pushed to
  status SMALLINT NOT NULL DEFAULT 0,   -- notification status
  cover TINYINT NOT NULL DEFAULT 0,     -- coverage (T/F)

  pushtime  DATETIME,                   -- when pushed
  notifytime  DATETIME,                 -- when notified
  PRIMARY KEY(id)                       -- key
);

CREATE INDEX pushlog_pushid ON pushlog (pushid);

Each of the tables above, may contain as many additional columns as desired; the code below uses only the columns defined above. In fact in my implementation, the users table contains all of the information gathered in Create Directory of BlackBerry Enterprise Server Users.

Configuration

All the Perl programs listed below use a small XML configuration file which ought to be self-explanatory. Still, here goes: the location of the backend database and the credentials are in the db element. The URL and the port number of the MDS is in the mdsserver element and the location of the getpage and notify CGI scripts are grouped in the web element. FIXMESYNTAX-syntax,bberry/push/config.xml,xml Don't forget to change the path to the config.xml in config.pl

#!/usr/bin/perl

package Conf;
use strict;
use XML::Simple;

our $config = XMLin('/etc/blackberry/push/config.xml');


die "Can't get configuration" unless ($config);

1;

Storing Pages in the Database...

As mentioned above, I have the pages in the backend database. The small task of reading the page and encoding it in Base64 is performed by insertpage.pl.

#!/usr/bin/perl
    # insertpage.pl (C)2006 by Jan-Piet Mens <jpmens at gmail.com>
    # Usage: insertpage appname
    # Reads stdin, converts to base64 and stores result in DB
    
    use strict;
    require 'config.pl';
    
    use DBI();
    use MIME::Base64;
    
    die "Usage: $0 appname\n" unless (my $appname = $ARGV[0]);
    
    my $DSN = "DBI:mysql:database=" .  $Conf::config->{db}->{dbname} .
            ";host=" . $Conf::config->{db}->{host};
    
    my $dbh = DBI->connect($DSN, $Conf::config->{db}->{user}, $Conf::config->{db}->{password}, 
            {'RaiseError' => 1, PrintError => 0});
    
    local($/) = undef;  # slurp
    my $encoded = encode_base64(<STDIN>);
    
    $encoded =~ s/\s+$//g;
    
    # Try inserting the page. If that doesn't work, update it. Since
    # appname is a primary key, the insert might fail.
    
    eval { $dbh->do("INSERT INTO pages (appname, data) VALUES ('$appname', '$encoded')") };
    
    $dbh->do("UPDATE pages SET data = '$encoded' WHERE appname = '$appname'");
    
    $dbh->disconnect;

... and Retrieving Them

A page stored in our pages table can be retrieved via CGI with the getpage program

#!/usr/bin/perl
    # getpage.pl (C)2006 by Jan-Piet Mens <jpmens at gmail.com>
    # This CGI program is called in a GET request with a single push identifier,
    # extracts the named document from a database and returns it to the caller.

    use strict;

    use DBI();
    use CGI;
    use MIME::Base64;

    my $q = new CGI;
    my $appname = $q->param('appname') || bailout(409, "appname missing");

    require 'config.pl';

    my $DSN = "DBI:mysql:database=" .  $Conf::config->{db}->{dbname} .
        ";host=" . $Conf::config->{db}->{host};

    my $dbh = DBI->connect($DSN, $Conf::config->{db}->{user}, $Conf::config->{db}->{password}, 
        {'RaiseError' => 1, PrintError => 0});

    my $sth = $dbh->prepare("SELECT COUNT(*) FROM pages WHERE appname=?");
    $sth->execute( $appname );

    my @row = $sth->fetchrow_array;

    if ($row[0] != 1) {
        bailout(404, "document $appname not found");
    }

    $sth = $dbh->prepare("SELECT data FROM pages WHERE appname=?");
    $sth->execute( $appname );

    @row = $sth->fetchrow_array;
    my $decoded = decode_base64($row[0]);

    print $q->header(
        -Content_length => length($decoded),
        'text/html','200 OK');
    print $decoded;

    $dbh->disconnect;
    exit;

    sub bailout {
        my ($status, $reason) = @_;
        print $q->header(-status => "$status $reason"), 
        $q->start_html('Problems'),
        $q->h2($reason);
        exit;

    }

Do note that we don't really use this program.

The push program

When a document is ready to be sent out, the pushout program is invoked with the tag identifying the document to send. pushout scans the database looking for subscribers to that document tag-type. For each subscriber found, pushout invokes an HTTP request to the MDS to push the document to the user's devices. For each of the invocations, pushout constructs a list of headers containing information which is submitted to the MDS. These headers instruct the MDS to handle the push in a specific manner:

Content-Location

This header is passed directly to the BlackBerry browser and not otherwise used. If the user on the device looks at the page address, the value of this header is displayed. Similarly, if the browser is forced to refresh the page, it will use the value of the Content-Location header as the URL from which to refresh. If, for some reason, you want to hide the URL, you can set this to file://some/place, thereby effectively hiding the information and also effectively disabling a user from refreshing the page's content.

Content-Type

This header specifies the MIME type of the document being sent to the browser.

X-RIM-Push-Title

The title of the browser message. For the messages we are pushing to the browser, the value of this header will be displayed in the message list. For push messages which install a new icon on the device home page, the icon will be named according to this value

X-RIM-Push-Type

The type of push message. Visit Pushing Web Content to BlackBerry Wireless Devices for a description of the four different kinds of browser push supported by the MDS

X-RIM-Push-Channel-ID

Each channel gets an identifier which can be used to delete the channel from a device. I will not be using this feature, but I specify a channel-id none the less

X-RIM-Push-NotifyURL

When the MDS pushes out a browser message, it will invoke this URL to post a notification regarding success or failure in pushing the message. This is optional, but I'm using the notification URL in order to determine whether a device received my page or not

X-RIM-Push-ID

When using X-RIM-Push-NotifyURL, the MDS will pass the value of the this header to the notification URL. That is the only way to correlate a push with a subsequent asynchronous notification. If the programmer doesn't specify this value, the MDS constructs its own.

X-Rim-Push-Use-Coverage

This appears to be new with MDS 4.1; it ought to indicate via the notification URL whether the device is in coverage or not. I may have misunderstood this, but I cannot get the MDS to return anything but true. The pushout program is otherwise quite straightforward:

#!/usr/bin/perl
    # pushout.pl (C)2006 by Jan-Piet Mens <jpmens at gmail.com>
    # Usage: pushout appname user

    use strict;

    use DBI();
    use MIME::Base64;
    use Digest::MD5 qw(md5 md5_hex);
    use LWP::UserAgent;
    use HTTP::Request;

    my $appname = shift @ARGV || die "Usage: $0 appname user|ALL\n";
    my $user   = shift @ARGV || die "Usage: $0 appname user|ALL\n";

    require 'config.pl';
    my $DSN = "DBI:mysql:" . 
        "database=" .  $Conf::config->{db}->{dbname} .
        ";host=" . $Conf::config->{db}->{host};

    my $dbh = DBI->connect($DSN,
        $Conf::config->{db}->{user},
        $Conf::config->{db}->{password}, 
        {'RaiseError' => 1, PrintError => 1});

    # Warning: the following statements allow SQL injection! Use
    # at your own risk!

    my $sql = <<ENDSQ;
       SELECT u.userid AS userid, u.pushdest AS pushdest, p.data AS data, p.title AS title
       FROM users u JOIN subscribers s JOIN pages p
       ON u.userid = s.userid AND p.appname = s.appname
         AND s.appname = ?

    ENDSQ

    $sql .= " AND u.userid = '$user'" if ($user ne 'ALL');

    my $sth = $dbh->prepare($sql);
    $sth->execute( $appname ) or die $dbh->errstr;

    my $c;
    while (defined ($c = $sth->fetchrow_hashref)) {
        my $userid    = $c->{userid};
        my $pushdest    = $c->{pushdest};
        my $title    = $c->{title};
        my $page    = decode_base64($c->{data});
        my $status;

        $title = $appname if (!$title);

        # Create unique push ID for MDS. If the MDS receives a push-id a second
        # time, it will bail out with status 400

        my $pushID = md5_hex(time() . $pushdest . $userid . length($page) . $title);


        $status = mds_push($userid, $appname, $pushdest, $page, $pushID, $title);
        log_push($userid, $appname, $pushdest, $status, $pushID);
    }

    $sth->finish();
    $dbh->disconnect;
    exit;

    sub log_push {
        my ($userid, $appname, $pushdest, $status, $pushID) = @_;

        $sql = <<ENDSQINS;
          INSERT INTO pushlog (id, userid, appname, pushdest, status, pushid, pushtime)
        VALUES (NULL, ?, ?, ?, ?, ?, NOW());
    ENDSQINS

        my $ins = $dbh->prepare($sql);
        $ins->execute($userid, $appname, $pushdest, $status, $pushID) or die $dbh->errstr;
        $ins->finish();
    }

    sub mds_push {
        my ($userid, $appname, $pushdest, $page, $pushID, $title) = @_;
        my $code = 0;

        my $url = $Conf::config->{web}->{getpage} . "?appname=$appname";
        my %headers = (
        'Content-Location'    => $url,
        'Content-Type'        => "text/html",
        'X-RIM-Push-Title'    => $title,
        'X-RIM-Push-Type'    => "Browser-Message",
        'X-RIM-Push-ID'        => $pushID,
        'X-RIM-Push-Channel-ID'    => 'fup-$appname',
        'X-RIM-Push-NotifyURL'    => $Conf::config->{web}->{notify},
        'X-RIM-Push-Use-Coverage' => 'true',
        );

        my $browser = LWP::UserAgent->new;
        my $get_response = $browser->get($url);
        my $content = $get_response->content;

        $content = $page;

        my $mds_url = 'http://' . 
        $Conf::config->{mdsserver}->{host} . ':' . 
        $Conf::config->{mdsserver}->{port} . "/push?DESTINATION=${pushdest}&PORT=7874&REQUESTURI=/";

        print STDERR "Pushing $mds_url\n";

        my $request = HTTP::Request->new;
        $request->method('GET');
        $request->uri($mds_url);

        foreach my $header (keys %headers) {
        my $value = $headers{$header};
        $request->header($header, $value);
        }

        $request->content($content);

        my $resp = $browser->request($request);

        if ($resp->is_success) {
        print "Push successful\n";
        } else {
        print "Error: Code = " . $resp->code . ": " . $resp->content . "\n";
        $code = $resp->code;
        }
        return ($code);
    }

Notification

The notification URL specified in the X-RIM-Push-NotifyURL header at the time of the push is invoked asynchronously by the MDS as soon as a message has been delivered, or in the case of a failure, if the message could not be delivered. In both cases, the notification URL is invoked with details of the intended delivery. In my experience, in the case a message cannot be delivered, the MDS will invoke the notification URL after quite precisely ten minutes, whereby the reason for the delivery failure is not always precise. The list of response codes are documented in DB-00502 in the BlackBerry knowledge base. Unfortunately, I've only seen status 200 (OK) or status 400 (General error) when working with the both the MDS simulator and on a live BES. Contrary to what is stated in the documentation, the notification URL is invoked by the MDS in an HTTP GET request (not a POST request). Invoked on my Apache server, I was able to see the following headers in addition to the usual headers passed by the web server to a CGI program:

HTTP_X_RIM_PUSH_STATUS

This contains a result code detailing success or failure of the push. The code 200 means success, any other indicates a failure. The list of response codes are documented in DB-00502 in the BlackBerry knowledge base.

HTTP_USER_AGENT

The user-agent of the invoker. On my 4.1 MDS I see "RIM MDS/4.0".

REQUEST_METHOD

The request method used to invoke the notification URL. This is a GET request with an empty QUERY_STRING. And by the way, the REMOTE_ADDR of the client is the IP address of the MDS server, and not that of the device.

HTTP_X_RIM_DEVICE_STATE

If the X-Rim-Push-Use-Coverage header was passed to the MDS during the push, this header details the coverage state of the device; true means in coverage and false means out of coverage. As mentioned above, I couldn't get this to indicate any other than true

HTTP_X_RIM_PUSH_ID

This header contains the value specified in the X-RIM-Push-ID header upon submitting the push. In my programs, it is a unique ID which I use to correlate the push with the subsequent (and asynchronous) notification.

HTTP_X_RIM_PUSH_DESTINATION

This header contains the email address or the PIN used as destination address during the push. The notifiy.cgi program uses the push identifier supplied in the X-RIM- PUSH-ID header to correlate the notification with the original push request.

#!/usr/bin/perl
    # notify.cgi (C)2006 by Jan-Piet Mens <jpmens at gmail.com>
    # This CGI program is invoked asynchronously by the MDS to record
    # delivery status notifications of pushed messages.

    use strict;
    use DBI;

    require 'config.pl';

    my $DSN = "DBI:mysql:database=" .  $Conf::config->{db}->{dbname} .
        ";host=" . $Conf::config->{db}->{host};

    print "Content-type: text/plain\n\n";

    my $dbh = DBI->connect($DSN, $Conf::config->{db}->{user}, $Conf::config->{db}->{password},
        {'RaiseError' => 1, PrintError => 1});


    my $status = $ENV{'HTTP_X_RIM_PUSH_STATUS'} || -1;
    my $coverage = $ENV{'HTTP_X_RIM_DEVICE_STATE'} eq 'true' ? 1 : -1;
    my $pushID = $ENV{'HTTP_X_RIM_PUSH_ID'};

    my $sql = <<ENDSQ;
       UPDATE pushlog SET status = ?, cover = ?, notifytime = NOW()
       WHERE pushid = ?
    ENDSQ

    my $sth = $dbh->prepare($sql);
    $sth->execute($status, $coverage, $pushID);

    $sth->finish();
    $dbh->disconnect;

    print "ok\n";
    exit 0;

Where to go from here

The small framework presented above has a lot of potential for improvement. As discussed, you might wish to add self-subscribing by users, which should be quite easy to implement. If you have other ideas for improvement (of which there are undoubtedly many) I'd be pleased to hear of them.

Resources

The following articles may be interesting reading

Comments

blog comments powered by Disqus