Sunday, November 23, 2014

Offline First Storage Constrains and Different Use Cases

A few weeks ago I spoke at Nationjs about "Responsive is Over, Offline First." Raymond Suelzer @RSuelzer asked "Any recommendations on handling the 5mb limit in local storage?"

Constrains are a positive in design. So the question is when working on an offline first sites how to think about the storage limit of local storage. This can be as low as 5MB.

First, even 5MB can be considered a lot of memory. The Apollo computer that got man on the moon had only 16k. So think positively about local storage.

Second, local storage is not your only option. Depending what you are trying to store, application cache is a good alternative. It does not have a great reputation but it is worth considering especially for images and some other files.

Third, your application, data or content may exceed the storage limit, that does not mean it cannot work offline. There are many different types of offline use cases.

One case is a website of content. No interaction and no syncing of data back to the application. Pouchdb's current site is a case in point. I recently accessed the site and then got on a cross country flight with no internet access. I navigated from page to page with no trouble.

Another case is interaction with poor connectivity. Think of turning on and off Airplane Mode as you use the app. The user experience is good so long as the input is not lost and synced when a connection is reestablished. It would be a rare case when the input exceeds the storage limit. While the user may not be able to access the entire site when offline the data is safe.

Another approach is to consider your app as having two different modes: offline and online. Write the app as an offline app first with enhancements when online. For example, if you app shows a photo gallery, perhaps that feature is only available when online.

There are lots of different approaches to offline first besides being a 100%, stand alone app when offline.

Monday, November 03, 2014

Halloween Joke - 2014

When kids trick or treat at our house we ask them to tell a joke. We have the reputation as the 'joke house.' These are the best from this year.

  • What do you call a pig that knows karate? Pork chop!
  • Why cannot you trust an acupuncturist? They stab you in the back.
  • Knock, knock. Who's there? Trigger. Trigger who? Trigger or Treat.
  • Why did the skeleton cross the road alone? Has no body.
  • Say windshield wipers. You wet your diapers.
  • Knock, knock. Who's there? Boo. Boo who? Don't cry, it's only a joke.
  • How do you make a tissue dance? Put a litter boogie in it.
  • What is a vampire's favorite boat? Blood vessel.
  • Why didn't the skeleton cross the road? No guts.
  • Why did the coach go to the bank? He wanted his quarter-back.
  • What did the skeleton waiter say to the customers? Bone appetite.
  • What did the skeleton travel agent say to the customers? Bone voyage.
There you have the best. 2012 and 2006 are also available.

Tuesday, October 28, 2014

Getting Started with Hapijs and Testing

API driven development allows for easy testing of the backend and a variety of front-end options. Hapijs for nodejs is a great solution when taking this approach. This is a step-by-step tutorial on Getting Started with Hapijs and Testing.

This tutorial takes a different approach than most. Testing is important for serious developement. If you think coding should start with a test, then you'll like this tutorial because that is how it starts.

Note: updated for Hapi version 8.x

Installing Hapijs

To start with testing not only do we need to install hapijs but also a few plugins. Installation is still very easy. And like other nodejs projects, a package.json file gets created.

First, create a directory for this tutorial. I use the name "hapijstut". You can choose any label. Then move into that directory.

Type the following to create your package.json file. But don't enter the defaults when prompted (see below):

npm init

When prompted for the test command enter the following. For the other prompts, default is fine.

./node_modules/lab/bin/lab -c

Next we want to install hapi and the joi plugin that we'll be using shortly. So type this:

npm install hapi joi --save

Finally, install the "lab" and "code" plugins needed for testing. Since these are not required for production the --save argument is now "--save-dev".

npm install lab code --save-dev

This should result in a package.json file that looks like this:

  "name": "hapijstut",
  "version": "0.0.0",
  "description": "Getting Started with Hapijs",
  "main": "index.js",
  "scripts": {
    "test": "./node_modules/lab/bin/lab -c"
  "author": "Your Name ",
  "license": "BSD-2-Clause",
  "dependencies": {
    "joi": "~4.7.0",
    "hapi": "~7.1.1"
  "devDependencies": {
    "code": "~1.2.0",
    "lab": "~5.0.1"

Setting Up a Hapi Test

Code tests are actually pretty simple. Describe the expected result and see if the actual result matches. Or if it doesn't match.

Setting up code tests is work because you have to write code. But after you've written a few, it gets easy. That's why it is a good idea to start learning Hapijs by coding some tests. Let's setup tests.

Create a new directory in our hapijs project directory ("hapijstut"). It must be called "test" ("hapijstut/test"). In that directory you'll create a new file called "first.js" which uses the Lab plugin for testing hapijs code.

var Lab = require("lab");
var lab = exports.lab = Lab.script();
lab.experiment("Hello", function() {
    // tests

This test needs more coding, but we can run it to make certain that test runs. So on the command line type:

npm test

You should see a result of "0 tests complete".

0 tests complete
Test duration: 0 ms
No global variable leaks detected
Coverage: 0.00% (0/0)

We can run tests, now lets code a test.

The Shell For A Hapi Test

Before a test can be code a shell is needed to hold the test.

The test code is placed inside the function that is an argument of lab.experiment. Lab.experiment is a way to group tests. Even though we have only one test, it is a good practice to use lab.experiment because you will add more tests as you develop your application. We'll do so later in this tutorial.

The first argument of lab.experiment is a description of the test.

lab.test("Testing for 'Hello World'",
    //test code here

Take these four lines of code and replace the "//test" comment in the first.js file. Then run our test again using "npm test" as we did above.

The result will be:

Failed tests:
  1) Hello Testing for 'Hello World':
    Timed out (2000ms)
1 of 1 tests failed
Test duration: 2004 ms

This shows that tests will timeout after 2 seconds if the code is not written properly. The test code needs to execute the done() function which is passed to it. If you want to see the test pass, you can replace "//test code here" with "done()". Run the test ("npm test") and you'll see that the test will complete.

Success. We've run our first test and it failed! When writing tests, you'll see Failed many times. This is actually how you start testing. Write the test and make certain it fails. Then write the server code and make sure it passes the test. In short, a failure is the first step towards success.

Testing The Server

Below we will code a hapijs server that will repond with "Hello World". To test that server our test code must "see" that server. This requires one line of code. At the begining of our first.js file we add the following line just after "var Lab = require('lab')". This pulls in the server code which will be in the "server.js" file in the parent directory.

var server = require("../server.js");

Now if you run the test, it will fail because the file "server.js" cannot be found.

To make the test pass, let's write the code the server and put it in the file named "server.js" in our project directory.

var Hapi = require("hapi");
var server = new Hapi.Server();
server.connection({ port: 8989 });
var hello = function (request, reply) {
  reply('Hello World');
server.route({ method: 'GET', path: '/', handler: hello });
console.log('hello server http://localhost:8989');

The above code requires "hapi" just like any nodejs program that uses a module. Then a server is defined to run on port 8989. Next a handler is defined to reply "Hello World" when a request is made to the server. The route is set next which uses the "hello" handler. Finally the server is started and a message is sent to the console.

Run "node server.js" to execute the code and then point your browser at "localhost:8989" to see "Hello World".

Back to testing. The server code works fine but the serve code needs to go into the test code. The method to do this with nodejs is to use the server code as a module. So at the end of the server.js code the server is exported. Here is the revised server.js code:

var Hapi = require("hapi");
var server = new Hapi.Server(8989);
var hello = function (request, reply) {
  reply('Hello World');
server.route({ method: 'GET', path: '/', handler: hello });
console.log('hello server http://localhost:8989');
module.exports = server;

While the test passes, you'll see that the result shows "missing coverage". Let's now add the test code.

Coding The Test

Our test code will expect to see a request to our server return "Hello World". The Lab plugin needs an assertion library to do this part. This tutorial uses the "Code" plugin - which we added to the project at the start of this tutorial. This gives us the ability to expect a result and see if we got what we expected.

In the code below the function part of lab.test is now complete. The options are set for doing a GET request from the route of "/". The server module which was required in the third line is then injected specifying the options and with a typical responce callback. The code plugin method of expect is used to examine the result of the response and see if it is equal to "Hello World". Hopefully it is and our test can pass. After that line runs successfully then the test is done().

var Lab = require("lab");
var lab = exports.lab = Lab.script();
var server = require("../server.js");
var code = require("code");

lab.experiment("Hello", function() {
  lab.test("Testing for 'Hello World'",
      var options = {method: "GET", url:"/"};
      server.inject(options, function(response){
        var result=response.result;
        code.expect(result).to.equal("Hello World");

Run the test (npm test) and you should see the result of

result="Hello World"
1 tests complete
Test duration: 16 ms
No global variable leaks detected
Coverage: 100.00%

There are actually a few more things that this server should produce. The status code it returns should be a 200. We could test that the length of the string returned is 11. Add these expectations just below the other code.expect line.


Run the test and everything passes with 100% coverage. This is just what we wanted to see.

Continue Coding So Add Another Test

The "Hello World" server is now working as expected. Time to start coding our next API. For this case we want a request to "/status" to return the "{message: 'ok'} object. So let's write the test first. Add this code to a new file in the "/test" directory and name it "second.js".

var Lab = require("lab");
var lab = exports.lab = Lab.script();
var server = require("../server.js");
var code = require("code");

lab.experiment("Status", function() {
  lab.test("Testing for 'ok' status code.",
      var options = {method: "GET", url:"/status"};
      server.inject(options, function(response){
        var result=response.result;

Now when you execute the test (npm test) both first.js and second.js are run. No configuration to modify. Simply add the new test file to the "/test" directory. The test runner looks for any ".js" files in the "/test" directory and runs them.

Of course when you add this our tests will now fail because we've not yet written code for the "/status" route.

1 of 2 tests failed

To pass the test add another route to the server.js file.

var check = function (request, reply) {
  reply({message: "ok"});
server.route({ method: 'GET', path: '/status', handler: check });

Note that the hapijs server is now returning an object. Run the test and it will now pass.


As you can see from adding this last bit of code, it is much easier to add additional tests. Hopefully this tutorial has you starting to write tests for your hapijs project and you'll continue to have test coverage for your code.

Other Tutorial

Hapijs for Proxy Routes and mapUri

Monday, October 27, 2014

Vox homepage should be the VoxDeck app

Melissa Bell at Vox wrote about "The home page is dead! We're building one anyway." The homepage is an important topic for news organizaions and most get it wrong in my opinion.

The home page has been obsolete for a long time. The primary way readers come to articles is by Google and social media. Less and less people start at the home page. But obsolete does not mean dead (see McLuhan). For example, you can still buy buggy whips. The home page is not dead, it still has a role to play. It should now be an app.

Most news organizations and in particular newspapers think of the home page as the front page of a newspaper. It is like they are building it for the archives so they can say "This is what happened on this day." These organizations keep trying to fight the last war better. So they get the user experience all wrong.

News sites are getting better because previously every click lead to a static article template page. The news is more than articles. Rich interactive stories and data presentations are common. The most popular story last year (2013) for The New York Times was interactive. So this "JavaScript Journalism" needs to be a consideration in the design.

News organizations produce huge amounts of content. Other sites can have nice navigation and whitespace galore because they are trying to present their limited amount of new content in the best possible way. The design considerations for news are different. 

People want to wade through news sites. So they return often. Sometimes they miss a day. There are topics that don't interest them. Think of a relative who starts reading the paper with the sports section and not the front page. This would speak to personalization. But people rarely want to personalize for themselves. They don't want to be programmers, they want it programmed for them.

Finally, people use many different devices and the experience should carry over from screen to screen. The cloud is not so much about storing data on remote serves as it is syncing the experience from device to device. This is a feature of nearly all popular apps like Dropbox, Evernote, email, etc. News needs to be offline first.

Home Page User Experience

This is what the User Experience should look like for news home page apps.

What is new for me? Show the new stories since my last visit. This does not mean hide the old stories, but highlight what is new since I visited three hours ago or since yesterday or last week. Don't waste my time with headlines I've already read. This is fairly simple to do.

Technical details:.Store in a cookie the timestamp of the last visit. Use a data-timestamp property for each article. Then use a short JavaScript routine to add a highlight class to each story that is new. In a prototype that I did, every story had a gray background except highlighted stories had a white background. Then sync this time stamp into a member's profile for syncing the experience across devices.

Show me the breadth of the news. Readers of newspaper sites are not told how many articles were published in the last 24 hours. In print, you know as soon as you pick up the paper. Show me lots of articles. Curate the order of the articles, not the ones I can see.

Show me that I'm up to date. With the paper you know when you finished reading every section. for many it is an important part of the news experience. So display a progress bar, or the like, showing how much of the news I've scanned and read. Give the experience of an accomplishment. 

Yesterday's news. Contrary to the Rolling Stones' song, people do want yesterday's paper. If readers are on vacation for a week or two, they want to be able to get caught up. Techmeme does this almost perfectly. Click on the date (which is not obvious) on the Techmeme page and roll back time to any previous home page. Every news site home page should have this feature.

Timelines and storylines. Stories do not stand alone. Perhaps I missed the first story in the series. Or I missed the news of the bill that passed or the death of a important figure in the story. Don't make me feel dumb because I didn't read a story ten weeks ago.

Next article, please. I've never understood why news sites make it so difficult to read the next article. With the paper my eyes just scan to the next headline. After I scroll down to the end of an article, please show me the next article. Let me keep scrolling. No need to click. Use the same gesture used to read the article to read the next article.

Read me later. Sometimes I get busy. Let me save an article to read later. Even on a different screen. Storing the unique id of a story is not a difficult technical challenge. It gives the news site another visit which I'm sure the business people will love.

The mosaic of news.  This all speaks to a layout of columns of articles like tweets in TweetDeck. For most newspapers there would be a column for each of the sections like Sports. If a reader frequently read a columnist or blogger, the personalization could add this as a new column. A list of articles could be filter by section, blog, author or any other metadata. Or even a search.

Most of the articles in a newspaper could be displayed on one page. Think of it as a single page web app for news. I called my concept app PostDeck. For Vox it could be VoxDeck. Other newspapers would be TimesDeck or TodayDeck. VoxDeck could even bring in a column from SB Nation and other Vox media.

Apps as news. Readers expect more than articles. Weather is a prime example. Columns don't have to be an index of articles. A column could be the weather app. Or breaking news app. Or twitter stream. Or election results. Or feedback. Or most frequently read. Or it could be a data driven story that has a long life.

Ads as news. Ads are broken on the web compared to print. I can clip an ad or coupon from the paper. I can turn back a page and re-read an ad. I know that every Sunday has my electronics flyer. Contrary to the common view, people want ads, not the intrusion. Put ads in a column, let people scroll through them. Make them searchable. Read them later. Make them social with comments. Challenge ad agencies to make ad apps for columns. Lots of room for innovation for the poor experience we get now. Just like in the paper make ads first class citizen with articles and apps.

Satisfy active readers. I don't follow baseball. As an active reader, let me customize the columns so I get hockey news but turn off baseball. Others might want all the stats of a live game as an app. Let me drag and drop the order of the columns. Let active readers customize and then feature their customization in promoting this news app. 

Long live the news home page app

Home pages for news sites have a lot of life left in them. They just should be considered as news apps and we'll see a renaissance for news home pages. 

Friday, October 24, 2014

Couchdb Filter Replication Error: "changes_reader_died" Debugged

Couchdb replication allows for filtering. It produces a nondescript error of "changes_reader_died" with endless attempts at replication. The error is a result of a poorly named query key (parameter).

This can be seen when a document in the _replicator database contains the following. The query key is set to "key".

"query_params": {
    "key": "user"

You may have to delete your _replicator database to stop the replication loop even with "continuous": false.

The solution was to change the key from "key" to another label. In the example below it is changed to "mailbox".

"query_params": {
    "mailbox": "user"

With this change the dreaded "error" :"changes_reader_died" was debugged and I could relax.

Monday, July 14, 2014

My statement to the FCC on Net Neutrality

As a former FCC employee of the Cable TV Bureau, I'm all in on Net neutrality. As an Internet user, net neutrality is vitally important to me, my family and my community. The FCC should use its Title II authority to protect it.

ISPs should not be able to profit from so called fast lanes. The internet was designed so that the end user pays, not the content providers upstream. The ISPs want a drastic and draconian change to the business model of the internet. They are trying to use their monopoly power and this is why the FCC was created - to protect communication from monopoly power - be it Bell or CableTV. Please protect the internet and the business model that has server the US and the world so well.

Thank you.

Monday, July 07, 2014

Passport Error with Google: Error: invalid_client

Nodejs module passport allows loggin into your app with OAuth2 for Google. But I kept getting an error of "invalid_client".

Google eror message screen
The solution was rather simple: check for typo in clientID in the passport.use(new GoogleStrategy({

In my app there was an extra number at the start of the clientID string.