In addition to push-mail, calendar and PIM synchronization, BlackBerry devices
connected to a BlackBerry Enterprise Server (BES) with Mobile Data System
(MDS) offer an exciting technology which can be used to extend web
applications to mobile BlackBerry devices via what is aptly named MDS Push.
This paper discusses a real-world scenario developed on a Linux host. This
document describes a framework written to facilitate the distribution of
information to BlackBerry handheld devices. In essence, I’ll be showing you
how to easily push out a wealth of information which appears on the devices as
depicted: 
The technology described herein
doesn’t require a Unix/Linux environment; it can easily be implemented on
Windows or Mac OS/X as long as the tool requirements are met. I’ve used
Perl and MySQL because both are readily available for these
platforms, but this could easily be ported to work with Perl and a different
database backend, or any language that allows HTTP connections (PHP and
Java come to mind). Furthermore, if you want to experiment with the technology
without having a full-blown BES setup, you can use the BlackBerry Device
Simulator together with the MDS Simulator to develop and experiment before
rolling your application out to real user’s devices. Using the simulators of
course also greatly lowers testing cost, because a wireless data connection is
neither needed nor required. Read more about the simulators and the JDE.
In my tests, I utilized a BES running on Windows with a properly configured
MDS. I used version 4.1, but this may work on a previous version; check the
documentation for your version where appropriate.
Intro
Some of our BlackBerry users require up to date financial statistics which, as soon as they are available, should be pushed out over the air (OTA) to the users’ devices. I’ve opted for what is called a Browser Message Push.
What is MDS Push?
The Mobile Data Service (MDS) is a software component of the BlackBerry Enterprise Server (BES) . I think of the MDS as a gateway between the wireless BlackBerry devices and the corporate network. The MDS can be used in the corporate environment to distribute (i.e. push) information and applications to devices utilizing the BlackBerry network infrastructure, and itMDS is used as a proxy between the BlackBerry browser (a.ka. MDS browser) on the devices and your corporate network and/or the Internet. Push applications can be created using any language that can initiate a network connection to perform an HTTP request. A push message is made up of a request with a set of headers and a body (the page to be sent to the device). In essence, there are three type of Browser push that can be sent to a BlackBerry device (see Pushing Web Content to BlackBerry Wireless Devices for an more verbose description of the different kinds of browser push supported by the MDS):
Browser Channel Push
A Browser Channel Push sends a web page and optionally two icons to the device. The web page is stored in the browser’s cache, and an icon is displayed on the home screen of the device.
Browser Content Cache Push
With this type of push, a page is pushed to the cache of the BlackBerry browser, but the user is neither given any type of indication that a page has arrived, nor that a page has been updated.
Browser Message Push
A Browser Message Push arrives in the inbox of the device. When the user opens the message, the browser is launched to display the page, which is kept on the device until the message is deleted. This message is neither sent to, nor will it appear on the user’s desktop inbox; it is sent directly to the device.
Using a Browser Message Push means that this message will appear in the
user’s message list (i.e. inbox) on the device, and it will be marked with a
small icon to differenciate it from a normal email message. Similarly, the
device home page will indicate that a new browser push message has arrived on
the device. 
The device inbox
displays the browser push message by title; opening that message launches the
browser to display the page:
What to Push
In the scenario that follows, we’ll be pushing ready-made HTML pages to the devices. These will be held in the message list on the device, allowing them to be viewed even if the device is not in a wireless coverage area. Any type of page supported by the BlackBerry browser can be pushed into the device cache. These include HTML and plain text. As described above, my users need a financial statistics page. The layout of the page was originally tested on a desktop browser (Mozilla Firefox to be precise), and it was tweaked for layout using one of the BlackBerry device simulators. What the page contains and what it looks like is almost irrelevant to the application at hand, if you keep a couple of things in mind:
- The BlackBerry Browser which will later be used to view the pages which are pushed to the device, has fewer capabilities than your full-blown desktop web browser. Consider that carefully when developing the page which you’ll be deploying to the devices
- Choose the subset of HTML that is used in your page carefully. You may wish to consider using cHTML, or even WML.
- The display of the BlackBerry devices is limited in size. You might have a whopping 20” monitor on your desk, but the device’s screen is not much larger than a few stamps
- Ensure that your BlackBerry browser is configured to support the features your pages will be using (think tables, JavaScript, etc.). On your BES you can configure your devices’ browsers via an IT policy which is then applied to the devices. For the browser settings proper, look at Create Directory of BlackBerry Enterprise Server Users. This screen shot shows the IT Policy settings I deploy to our users:
The Little Big Picture
The diagram depicts the main components I’m using in this application. First
the HTML page proper is produced. Since that part depends a lot on what
information you wish to push out to your BlackBerry user base, and because
that part is proprietary, I’m won’t further discuss it here. Suffice to say,
that at the end of the day we have an HTML page which will be viewable by the
BlackBerry Browser, and that that HTML page will be stored in the database.
This is not a technical requirement; it could just as easily (no, that is not
true: more easily) have been stored somewhere in the file system, but I want
it in the database. The Apache web server runs on Linux but might as
well be hosted by your BES server on Windows. In our environment, it massages
the statistical data which is then transformed to HTML on the one hand, and on
the other hand, it hosts the getpage and notify programs which will be
invoked by the MDS. Additionally it could host a web page on which you allow
your users to subscribe to distinct pages. 
In my
scenario, subscriptions to this service are handled by an administrator. Due
to the relatively high sensitivity of the data, I don’t want users subscribing
to the push service themselves, although that would be very easy to implement
(see Creating a RSS/RDF Push Service for BlackBerry for an example
implemented in the PHP language). The subscribe program is invoked by
an authorized user to add a device PIN or a device-user’s email address to the
database. The getpage program is a CGI-type script that can retrieve the
page, given its identifying tag (also called a page name or an appname or
application name). The main workhorse here is the pushout program; it scans
the database for new documents to send out and sends appropriate push requests
to the Mobile Data System (MDS), which in turn pushes these messages out over
the wireless network (OTA) to the devices. In newer MDS versions, the MDS can
track and record delivery of the push messages to the devices, which is useful
for determining whether the message actually reached the device. The notify
CGI program will be invoked by the MDS with a device and message identifier
which I’ll use to correlate delivery to the original push message, therewith
recording which push messages were actually delivered to users’ devices. A log
of deliveries and their correlated notifications will be held in the database.
The Backend Database
As mentioned above, the framework at hand uses a MySQL database backend and the supplied database schema was designed for that. It ought to be no problem at all to adapt the schema to use any other backend such as PostgreSQL or any other. There are four tables in the schema:
users
The users table holds a touple consisting of a userid (a unique user identifier such as jdoe or J9001) and a push destination pushdest. This column contains either the unique device PIN or the user’s BES email address to which the push messages should be directed.
CREATE TABLE users (
userid VARCHAR(128) NOT NULL, -- user identifier
pushdest VARCHAR(128) NOT NULL, -- PIN or email on BES
modif TIMESTAMP(14), -- modification timestamp
PRIMARY KEY(userid) -- unique
);
subscribers
Each user in the users table can be subscribed to one or more “pages”, and this table records which “pages” the user has subscribed to. As mentioned above, in my case the administrator will be updating this table, but you could easily create a web page with which users can subscribe to pages themselves.
CREATE TABLE subscribers (
userid VARCHAR(128) NOT NULL, -- user identifier
appname VARCHAR(128) NOT NULL, -- application name
modif TIMESTAMP(14), -- modification timestamp
PRIMARY KEY(userid,appname) -- unique
);
pages
This table holds the data to be pushed to the devices; the “pages”. Each page (also called appname, as in application name) is given a unique tag or name, and a title. The appname will be used as a channel identifier (see below) and the title will be used as the page name as seen by the user. In the screen shots at the top of this document, the title is “fupps.com: Turnover FY 1993”. In addition to the application or page name, this table holds the page itself, encoded as Base-64 stream in the data attribute.
CREATE TABLE pages (
appname VARCHAR(128) NOT NULL, -- application name
title VARCHAR(128), -- title of push message
data MEDIUMBLOB NOT NULL, -- Base64-encoded HTML/text page
modif TIMESTAMP(14),
PRIMARY KEY(appname) -- unique
);
pushlog
Each push message sent to the MDS is recorded in the pushlog table, identifying the userid to whom the page was pushed. In addition, the appname is recorded as well as the (redundant) push destination pushdest. Each message pushed out is given a unique identifier called pushid; this is recorded as well. As soon as the MDS has delivered the message to the wireless BlackBerry device (or hasn’t, in case of a failure), this table is updated to record that fact. The attribute status indicates success or failure of the message push.
CREATE TABLE pushlog (
id INT NOT NULL AUTO_INCREMENT, -- unique identifier
userid VARCHAR(128) NOT NULL, -- user identifier
appname VARCHAR(128) NOT NULL, -- application name
pushid VARCHAR(128) NOT NULL, -- unique MDS push identifier
pushdest VARCHAR(128) NOT NULL, -- PIN or email pushed to
status SMALLINT NOT NULL DEFAULT 0, -- notification status
cover TINYINT NOT NULL DEFAULT 0, -- coverage (T/F)
pushtime DATETIME, -- when pushed
notifytime DATETIME, -- when notified
PRIMARY KEY(id) -- key
);
CREATE INDEX pushlog_pushid ON pushlog (pushid);
Each of the tables above, may contain as many additional columns as desired; the code below uses only the columns defined above. In fact in my implementation, the users table contains all of the information gathered in Create Directory of BlackBerry Enterprise Server Users.
Configuration
All the Perl programs listed below use a small XML configuration file which
ought to be self-explanatory. Still, here goes: the location of the backend
database and the credentials are in the db element. The URL and the port
number of the MDS is in the mdsserver element and the location of the
getpage and notify CGI scripts are grouped in the web element.
FIXMESYNTAX-syntax,bberry/push/config.xml,xml Don’t forget to change the path
to the config.xml in config.pl
#!/usr/bin/perl
package Conf;
use strict;
use XML::Simple;
our $config = XMLin('/etc/blackberry/push/config.xml');
die "Can't get configuration" unless ($config);
1;
Storing Pages in the Database…
As mentioned above, I have the pages in the backend database. The small task of reading the page and encoding it in Base64 is performed by insertpage.pl.
#!/usr/bin/perl
# insertpage.pl (C)2006 by Jan-Piet Mens <jpmens at gmail.com>
# Usage: insertpage appname
# Reads stdin, converts to base64 and stores result in DB
use strict;
require 'config.pl';
use DBI();
use MIME::Base64;
die "Usage: $0 appname\n" unless (my $appname = $ARGV[0]);
my $DSN = "DBI:mysql:database=" . $Conf::config->{db}->{dbname} .
";host=" . $Conf::config->{db}->{host};
my $dbh = DBI->connect($DSN, $Conf::config->{db}->{user}, $Conf::config->{db}->{password},
{'RaiseError' => 1, PrintError => 0});
local($/) = undef; # slurp
my $encoded = encode_base64(<STDIN>);
$encoded =~ s/\s+$//g;
# Try inserting the page. If that doesn't work, update it. Since
# appname is a primary key, the insert might fail.
eval { $dbh->do("INSERT INTO pages (appname, data) VALUES ('$appname', '$encoded')") };
$dbh->do("UPDATE pages SET data = '$encoded' WHERE appname = '$appname'");
$dbh->disconnect;
… and Retrieving Them
A page stored in our pages table can be retrieved via CGI with the getpage program
#!/usr/bin/perl
# getpage.pl (C)2006 by Jan-Piet Mens <jpmens at gmail.com>
# This CGI program is called in a GET request with a single push identifier,
# extracts the named document from a database and returns it to the caller.
use strict;
use DBI();
use CGI;
use MIME::Base64;
my $q = new CGI;
my $appname = $q->param('appname') || bailout(409, "appname missing");
require 'config.pl';
my $DSN = "DBI:mysql:database=" . $Conf::config->{db}->{dbname} .
";host=" . $Conf::config->{db}->{host};
my $dbh = DBI->connect($DSN, $Conf::config->{db}->{user}, $Conf::config->{db}->{password},
{'RaiseError' => 1, PrintError => 0});
my $sth = $dbh->prepare("SELECT COUNT(*) FROM pages WHERE appname=?");
$sth->execute( $appname );
my @row = $sth->fetchrow_array;
if ($row[0] != 1) {
bailout(404, "document $appname not found");
}
$sth = $dbh->prepare("SELECT data FROM pages WHERE appname=?");
$sth->execute( $appname );
@row = $sth->fetchrow_array;
my $decoded = decode_base64($row[0]);
print $q->header(
-Content_length => length($decoded),
'text/html','200 OK');
print $decoded;
$dbh->disconnect;
exit;
sub bailout {
my ($status, $reason) = @_;
print $q->header(-status => "$status $reason"),
$q->start_html('Problems'),
$q->h2($reason);
exit;
}
Do note that we don’t really use this program.
The push program
When a document is ready to be sent out, the pushout program is invoked with the tag identifying the document to send. pushout scans the database looking for subscribers to that document tag-type. For each subscriber found, pushout invokes an HTTP request to the MDS to push the document to the user’s devices. For each of the invocations, pushout constructs a list of headers containing information which is submitted to the MDS. These headers instruct the MDS to handle the push in a specific manner:
Content-Location
This header is passed directly to the BlackBerry browser and not otherwise used. If the user on the device looks at the page address, the value of this header is displayed. Similarly, if the browser is forced to refresh the page, it will use the value of the Content-Location header as the URL from which to refresh. If, for some reason, you want to hide the URL, you can set this to file://some/place, thereby effectively hiding the information and also effectively disabling a user from refreshing the page’s content.
Content-Type
This header specifies the MIME type of the document being sent to the browser.
X-RIM-Push-Title
The title of the browser message. For the messages we are pushing to the browser, the value of this header will be displayed in the message list. For push messages which install a new icon on the device home page, the icon will be named according to this value
X-RIM-Push-Type
The type of push message. Visit Pushing Web Content to BlackBerry Wireless Devices for a description of the four different kinds of browser push supported by the MDS
X-RIM-Push-Channel-ID
Each channel gets an identifier which can be used to delete the channel from a device. I will not be using this feature, but I specify a channel-id none the less
X-RIM-Push-NotifyURL
When the MDS pushes out a browser message, it will invoke this URL to post a notification regarding success or failure in pushing the message. This is optional, but I’m using the notification URL in order to determine whether a device received my page or not
X-RIM-Push-ID
When using X-RIM-Push-NotifyURL, the MDS will pass the value of the this header to the notification URL. That is the only way to correlate a push with a subsequent asynchronous notification. If the programmer doesn’t specify this value, the MDS constructs its own.
X-Rim-Push-Use-Coverage
This appears to be new with MDS 4.1; it ought to indicate via the notification URL whether the device is in coverage or not. I may have misunderstood this, but I cannot get the MDS to return anything but true. The pushout program is otherwise quite straightforward:
#!/usr/bin/perl
# pushout.pl (C)2006 by Jan-Piet Mens <jpmens at gmail.com>
# Usage: pushout appname user
use strict;
use DBI();
use MIME::Base64;
use Digest::MD5 qw(md5 md5_hex);
use LWP::UserAgent;
use HTTP::Request;
my $appname = shift @ARGV || die "Usage: $0 appname user|ALL\n";
my $user = shift @ARGV || die "Usage: $0 appname user|ALL\n";
require 'config.pl';
my $DSN = "DBI:mysql:" .
"database=" . $Conf::config->{db}->{dbname} .
";host=" . $Conf::config->{db}->{host};
my $dbh = DBI->connect($DSN,
$Conf::config->{db}->{user},
$Conf::config->{db}->{password},
{'RaiseError' => 1, PrintError => 1});
# Warning: the following statements allow SQL injection! Use
# at your own risk!
my $sql = <<ENDSQ;
SELECT u.userid AS userid, u.pushdest AS pushdest, p.data AS data, p.title AS title
FROM users u JOIN subscribers s JOIN pages p
ON u.userid = s.userid AND p.appname = s.appname
AND s.appname = ?
ENDSQ
$sql .= " AND u.userid = '$user'" if ($user ne 'ALL');
my $sth = $dbh->prepare($sql);
$sth->execute( $appname ) or die $dbh->errstr;
my $c;
while (defined ($c = $sth->fetchrow_hashref)) {
my $userid = $c->{userid};
my $pushdest = $c->{pushdest};
my $title = $c->{title};
my $page = decode_base64($c->{data});
my $status;
$title = $appname if (!$title);
# Create unique push ID for MDS. If the MDS receives a push-id a second
# time, it will bail out with status 400
my $pushID = md5_hex(time() . $pushdest . $userid . length($page) . $title);
$status = mds_push($userid, $appname, $pushdest, $page, $pushID, $title);
log_push($userid, $appname, $pushdest, $status, $pushID);
}
$sth->finish();
$dbh->disconnect;
exit;
sub log_push {
my ($userid, $appname, $pushdest, $status, $pushID) = @_;
$sql = <<ENDSQINS;
INSERT INTO pushlog (id, userid, appname, pushdest, status, pushid, pushtime)
VALUES (NULL, ?, ?, ?, ?, ?, NOW());
ENDSQINS
my $ins = $dbh->prepare($sql);
$ins->execute($userid, $appname, $pushdest, $status, $pushID) or die $dbh->errstr;
$ins->finish();
}
sub mds_push {
my ($userid, $appname, $pushdest, $page, $pushID, $title) = @_;
my $code = 0;
my $url = $Conf::config->{web}->{getpage} . "?appname=$appname";
my %headers = (
'Content-Location' => $url,
'Content-Type' => "text/html",
'X-RIM-Push-Title' => $title,
'X-RIM-Push-Type' => "Browser-Message",
'X-RIM-Push-ID' => $pushID,
'X-RIM-Push-Channel-ID' => 'fup-$appname',
'X-RIM-Push-NotifyURL' => $Conf::config->{web}->{notify},
'X-RIM-Push-Use-Coverage' => 'true',
);
my $browser = LWP::UserAgent->new;
my $get_response = $browser->get($url);
my $content = $get_response->content;
$content = $page;
my $mds_url = 'http://' .
$Conf::config->{mdsserver}->{host} . ':' .
$Conf::config->{mdsserver}->{port} . "/push?DESTINATION=${pushdest}&PORT=7874&REQUESTURI=/";
print STDERR "Pushing $mds_url\n";
my $request = HTTP::Request->new;
$request->method('GET');
$request->uri($mds_url);
foreach my $header (keys %headers) {
my $value = $headers{$header};
$request->header($header, $value);
}
$request->content($content);
my $resp = $browser->request($request);
if ($resp->is_success) {
print "Push successful\n";
} else {
print "Error: Code = " . $resp->code . ": " . $resp->content . "\n";
$code = $resp->code;
}
return ($code);
}
Notification
The notification URL specified in the X-RIM-Push-NotifyURL header at the
time of the push is invoked asynchronously by the MDS as soon as a message has
been delivered, or in the case of a failure, if the message could not be
delivered. In both cases, the notification URL is invoked with details of the
intended delivery. In my experience, in the case a message cannot be
delivered, the MDS will invoke the notification URL after quite precisely ten
minutes, whereby the reason for the delivery failure is not always precise.
The list of response codes are documented in DB-00502 in the BlackBerry
knowledge base. Unfortunately, I’ve only seen status 200 (OK) or status 400
(General error) when working with the both the MDS simulator and on a live
BES. Contrary to what is stated in the documentation, the notification
URL is invoked by the MDS in an HTTP GET request (not a POST request). Invoked
on my Apache server, I was able to see the following headers in addition
to the usual headers passed by the web server to a CGI program:
HTTP_X_RIM_PUSH_STATUS
This contains a result code detailing success or failure of the push. The code 200 means success, any other indicates a failure. The list of response codes are documented in DB-00502 in the BlackBerry knowledge base.
HTTP_USER_AGENT
The user-agent of the invoker. On my 4.1 MDS I see “RIM MDS/4.0”.
REQUEST_METHOD
The request method used to invoke the notification URL. This is a GET request with an empty QUERY_STRING. And by the way, the REMOTE_ADDR of the client is the IP address of the MDS server, and not that of the device.
HTTP_X_RIM_DEVICE_STATE
If the X-Rim-Push-Use-Coverage header was passed to the MDS during the push, this header details the coverage state of the device; true means in coverage and false means out of coverage. As mentioned above, I couldn’t get this to indicate any other than true
HTTP_X_RIM_PUSH_ID
This header contains the value specified in the X-RIM-Push-ID header upon submitting the push. In my programs, it is a unique ID which I use to correlate the push with the subsequent (and asynchronous) notification.
HTTP_X_RIM_PUSH_DESTINATION
This header contains the email address or the PIN used as destination address during the push.
The notifiy.cgi program uses the push identifier supplied in the X-RIM-
PUSH-ID header to correlate the notification with the original push request.
#!/usr/bin/perl
# notify.cgi (C)2006 by Jan-Piet Mens <jpmens at gmail.com>
# This CGI program is invoked asynchronously by the MDS to record
# delivery status notifications of pushed messages.
use strict;
use DBI;
require 'config.pl';
my $DSN = "DBI:mysql:database=" . $Conf::config->{db}->{dbname} .
";host=" . $Conf::config->{db}->{host};
print "Content-type: text/plain\n\n";
my $dbh = DBI->connect($DSN, $Conf::config->{db}->{user}, $Conf::config->{db}->{password},
{'RaiseError' => 1, PrintError => 1});
my $status = $ENV{'HTTP_X_RIM_PUSH_STATUS'} || -1;
my $coverage = $ENV{'HTTP_X_RIM_DEVICE_STATE'} eq 'true' ? 1 : -1;
my $pushID = $ENV{'HTTP_X_RIM_PUSH_ID'};
my $sql = <<ENDSQ;
UPDATE pushlog SET status = ?, cover = ?, notifytime = NOW()
WHERE pushid = ?
ENDSQ
my $sth = $dbh->prepare($sql);
$sth->execute($status, $coverage, $pushID);
$sth->finish();
$dbh->disconnect;
print "ok\n";
exit 0;
Where to go from here
The small framework presented above has a lot of potential for improvement. As discussed, you might wish to add self-subscribing by users, which should be quite easy to implement. If you have other ideas for improvement (of which there are undoubtedly many) I’d be pleased to hear of them.
Resources
The following articles may be interesting reading