Test Driven API Development Thoughts

This was the basis for my lightning talk at #nodedc on December 14, 2011. It received positive feedback.

---

I wish I was better at Test Driven Development, I guess most developers do to. For me Rails started it, jQuery pushed it, but I still don't do it. But I'm trying. I'm talking build the test, then write the code. I may have a way to do the walk.

I have this idea for TDD for APIs. I want to take the idea of TDD outside the realm of a language or framework. In consuming an API the underlying language does not matter. Therefore, using TDD for developing APIs or developing with APIs should be outside the language or framework.

This project could also be a template for good practices in creating RESTful API. For example, a RESTful API is suppose to be about one resource. While that is good in theory, in practice it often works out differently. So however the design works, it needs flexibility.

There are probably three part to the project: an engine, a reporter and builder. The engine would read a json message, run the tests and report back. The builder would create the json message for the engine. The builder form would allow you to enter a URL/resource and test it. The result would be filled into the expect result field, which could be overwritten. This information could be used for documentation as well.

It could be especially useful for apps that depend on a variety of APIs from different sources. Run the tests regularly to make certain the APIs are still functioning as expected.

Or maybe this already exists? Or maybe this depends on Mocha? Or other existing packages?

Anyway, that is the concept. Thanks to the @nodedc for listening.

 PS One other feature, it should be able to run the same tests against a variety of environments: dev, qa, production, server01, server02, etc.

PSS. Since a RESTful API is like GETting a web page, it probably has broader uses than just APIs.

The GitHub Revolution

We shape our buildings; thereafter they shape us. - Churchill

Once again, Mikeal Rogers has a thoughtful and insightful post called "Apache Considered Harmful." I have no opinion or personal interaction with the Apache Foundation. Rather my focus is on his view of git versus subversion. He calls it "The GitHub Revolution." I could not agree more even though I'm not very active on GitHub.

On GitHub the language is not code, as it is often characterized, it is contribution. GitHub presents a person to person communication system for contributions. Documentation, issues, and of course code, travel between personal repositories. 

The communication medium is the contribution itself. Its value, its merit, its intentions, all laid naked for the world to see. There is no hierarchy or politic embedded in the system. The creator of a project has a clear first mover advantage but the possibility is always there for its position to be supplanted by a fork, creating a social imperative to manage contributions in a satisfactory manor to her community. 

GitHub is truly a system of anarchism, in the most classic sense of the term. It is a system of communication and contribution that is without a central organization or institution of governance. Sure, it is hosted, developed, and maintained by someone but they do not enforce any set of governance or process over the users of the system. 

It is my belief that we are, right now, in the middle of a very large evolution in the ecology of open source. The language of contribution has infected a new generation of open source contributors. Much of the potential first imagined by open source pioneers is being realized by high school kids on a daily basis who contribute effectively with less effort than has ever been required.

The post goes on to discuss the bias of Apache to Subversion as a repository.

The Git versus Subversion has a long history but that has mainly been a comparison of features. For me that is like comparing the features of television versus radio, or books versus ebooks. Mikeal's argument is a more important and an interesting view. I'm looking at the influence of the underlying technologies.

Git is not Subversion. They are different media, different technologies. Different media do not allow the same content. Just as a newspaper cannot tell a story the same way as television, so too Git projects will not be the same as Subversion projects. You cannot cram a GitHub project into subversion. People are also using GitHub for writing blogs, books and many different ways than simply a code repository. As Mikael points out there is substantial communications in the hub. The technologies are different. What is created is different.

The GitHub revolution is part of a larger cultural change. As Om Malik pointed out, Dribble and SoundCloud are part of this revolution. The results of Hackatons, Camps and Startup Weekends are impressive. These projects, services, projects and communities are our new cultural artefacts which are changing hiring practices and people's identity.

The definition of Open Source is also changing perhaps to the breaking point. We probably need a new term to describe the content being created on these platforms. Times are changing.

A technology choice is important and Mikeal has provided a strong argument for choosing GitHub. In addition, I expect that in the future we may see less companies choose the Apache way, just as a generation ago we saw companies no longer see the need to move from NASDAQ to the NYSE.

Speaking after Steve Jobs introduces the Mac

On January 22, 1984 the Mac was introduced with the very famous Apple SuperBowl ad. It was spectacular. Steve Jobs did his famous keynote two days later to introduce the Mac. 27 years ago the web did not instantly let customers watch Steve Jobs and his presentation skills were just becoming known. Not many people saw the Mac keynote. So he hit the road to SoftCon to show the software support for the new computer.

SoftCon, was held on month later in New Orleans. I regularly gave talks about software retailing having started the first software store in 1979. We had a small chain of stores at that time. I was scheduled to speak on the good and bad of software packaging.

Enter Steve Jobs. In a very large room he introduces the Mac to the software industry. This was a big moment. He showed the $1.5 million dollar Ridley Scott ad, professional slides and had a large staff.  He was well prepared and had masterful timing. Thousands of people in the room applauded when he finished.

I was on next. I had to speak after Steve Jobs. It really didn't matter that my presentation was 35mm slides after professional photos. I had no Ridley Scott ad. I had nothing. The topic of software packaging was insignificant after seeing the Mac for the first time. The legibility of software titles was not as important. I hurried through my talk and sat down. It was exhausting.

Soon after I bought a Mac. 

The New Framework Discussion (Part One)

mikeal posted "A new direction for web applications" And I left the following comment.

I agree about MVC is not the right fit. For me it breaks down even for Rails with JavaScript on the client. Model is database term that does not fit async data. RESTful is certainly part of the mix as are other HTTP inputs and IP streams. These don't fit the model analogy either. 

The "new" framework is probably much more of a mosaic than a flow (or flowchart). Not certain how well the neighbors (parts of the mosaic) know each other. Probably some zoning laws, but everyone is able to do their own thing (except after midnight) so long as certain ordinances are not broken. 

My other thought is that these new apps are much like a room full of people. Conversations between a few people are great. Lectures work. But getting a room to act together requires something like Roberts Rules of Order. 

Writing internet code is certainly has a long way to go.

I wanted to expand on that post and comment about the New Framework. I think there are several influences shaping its development.

1. Async. The first big push was Ajax. The latest is node.js. Data no longer has a flow or a sequence, it is more quantum. REST is the structure for async.
2. Speed. Look at the popularity of Steve Souders and Nicholas Zakas talks and books on speed. I hope the new framework fits the techniques they and others are discussing. For example, fewer requests doesn't always play well with frameworks that generate lots of directories and files.
3. Events. Even a small app has a population of events. The new framework must provide a logical structure to these events.
4. CSS, HTML and JS. Most of us work with all three. For example, what use to be in JS is now often in HTML5. The new framework will have to consider all three.
5. History. Presenters are going back and looking at the development of other technologies and looking for good patterns. Douglas Crockford was looking at the transition from manuscripts to books in his talk at CapitolJS. This search is common for any new technology. Marshall McLuhan showed that a new technology retrieves an older technology. It will be interesting to see what is retrieved by the new framework.
6. Testing. Of course.

You can probably argue that there are other forces at work here. For example, I could argue pro and cons of templating. 

I'm sure someone is working on the new framework right now, it is "just now well distributed" yet.

CapitolJS 2011 [#JSConf East and #capjs]

One attends a JavaScript conference to learn how to use the language better. The conference is not intended to teach the beginner. There are assumptions of a working knowledge of the language, probably extensive experience in coding and an interest in node.js. However, I think there were themes or lessons in the presentations at this conference that would help any JavaScript developer.

The future of the language is amazingly bright. Mike Taylor showed that even though the browser Navigator has very unreliable properties that the future new properties (e.g., online, geolocation, etc.) hold great promise. Pamela Fox had example after example of APIs that JavaScript can access for functionality, public data and user data. And Brendan Eich demonstrated the work that Intel has done to access parallel-processing power with an engine called RiverTrail (on github) [Infoworld article][http://www.infoworld.com/d/application-development/intel-river-trail-adds-parallel-dimension-javascript-173347].

Performance in JavaScript is now in the hands of developers now. Nickolas Zakas promoted the use of both defer and async for script loading. So testing must include performance testing with no action taking longer than 100ms. This is the max for a responsive system. Alex Seston also emphasised the need for non-blocking code while on the topic of third party JS. Alex Russel went deep into working web engines in browsers so coders know what is behind some simple DOM changes.

JS, the programming language of the web, is amazingly rich. Douglas Crockford looked at how we code from the viewpoint of Head and Gut (see book Science of Fear) and how the Gut often gets in the way. Using tools and the good parts of the language (language subsetting) we can use our Head more and produce less errors. "Programming requires more precision than writing in English." Language conventions take time. For example the line breaks and spacing introduced by Medieval manuscript copyists. This language history perspective was echoed in Mike Taylor's talk and emphasized by Brendan Eich in noting how young the language is. It is mindblowing what the JS community can code in 140 bytes on Jed Schmidt's site (http://140byt.es.com).

Growth is in mobile and it effects all. Rebecca Murphey showed the new toura mulberry framework for creating content rich mobile apps built atop phonegap. http://toura.github.com/mulberry/ Joe McCann works in the mobile area and provided a comprehensive view of both coding and using node.js with Express, Stylus, Smoosh, and Commander as tools for creating any app. it is clear for the Head to win over Gut that automating tools are needed.

Node.js is still growing. Mikeal Rodgers ran down the most popular packages for node including npm, optimist, request and of course socket.io. But there is still room for improvement, especially in debugging. And again, node was mentioned by most presenters. Ben Combee showed how to access the node running in webOS and stated that webOS will soon be updated to version 0.4.11. Paolo Fraqomeni talked about how the design of node is especially suited for the enterprise and the event emitter.

As I see them available, I'll link to the decks for each presentation.

Douglas Crockford
Ben Combee
Nicholaz Zakas
Mikael Rogers
Jed Schmidt
Mike Taylor UPDATED
Rebecca Murphey
Alex Sexton
Alex Russell
Joe McCann
Paolo Fragomeni
Pamela Fox
Brendan Eich UPDATED

The conference was terrific not only because of the speakers but thanks to the work of my friends Laura and Chris Williams. Also because of sponsors there was time to network at parties both before and after the event.

Thanks. A wonderful community.

UPDATE: Group Photo

Distinction between sites and apps is RSS

James Pearce wrote a substantial post about the distinction between sites and apps. He says the distinction matters only to developers and not for users. I think there is another measure of the distinction than his fat versus thin client. This is my response. 

The distinction is only relevant, as you say, for developers. The reason is simply because "interface is content." Or perhaps put another way "interface is the message."

Take some content and provide a simple HTML interface. Put the same content in a JS/CSS3/HTML5 interface and the message is different. People will "read" it differently and have a different user experience.

So the distinction becomes blurred in my view as soon as you cross the pure HTML line. Ask yourself at what point does adding JS/CSS3/HTML5 take the content from site to app? I suggest there is not a brightline using that (ie. fat client) as a measure.

However, there may be another measure. Again, it may not be a brightline, but it does help.

Sites can support an RSS feed while apps don't. 

---

Let me add to the comment that I left.

The post argues that the distinction fat versus thin, that the code resides on the browser versus the server. It also argues that users don't care.

First, take the New York Times or any newspaper or news source. Does it really matter if I read the story in an app or off their site? They support RSS so I can read the story on other sites, in RSS readers and apps that support RSS.

Then take an app like Pulse or other news apps. I can read an MSNBC story with the app or on the MSNBC site. But I don't find Pulse stories on MSNBC. Pulse cannot support an RSS feed because it does not create stories.

So I would argue that the user experience between the Pulse app and the New York Times app is substantially different because the users know the source of the content. Therefore, as a developer, you'd design the app to the user expectation.

Now that is a simple, legacy example.  Let's add some web features. Many sites, not just news sites, all users to comments. Comment systems support RSS feeds or other notification methods (e.g., email).  Certainly comments are not an app.

More modern day are sites like Quora. The content is not news stories but there is new, relevant content. They use the headline of "Feed" on the main page. Clearly Quora is a site.

And finally, let's look at the classical app: the todo app. Put a todo app on any smartphone for personal use and all of the generated content can stay on the phone. It would be a useful app without any interaction with the internet. Even if you sync the todos with other devices which would require servers somewhere, the bulk of the app in on the phone. But more importantly the data source is on the phone. You could sync with RSS (or a JSON equivalent) but is that feed really useful for the public? It is core to the purpose?

In summary, you've got a site when the content supports a RSS feed. The purpose of an app is confined to the device. The distinction my not matter to developers, but user know when a site can support an RSS feed, even if they don't know what RSS is.

Update to Roger McNamee's TED Talk, Watch This One

This version of the talk includes a hypothesis about many personal screens that leads to discussing the cloud. To borrow a phrase, "It is the screens, stupid."

http://fora.tv/2011/06/28/Elevation_Partners_Director_and_Co-Founder_Roger_McNamee#fullprogram

Why JavaScript and #jsconf matter

JavaScript is the key to HTML5 as described above.

My JSConf ToDo List

With great anticipation, I will attend JSConf and NodeConf. To get the most out of that time, there are a few items I want to brush up on prior to hitching up the wagon and heading West. I've got a list, some big and some small and in no particular order.

1. Full Sail Ale. jsbeer prep is first on the list.
2. Eloquent JavaScript. Finish reading especially the object oriented chapter.
3. jquery mobile alpha4. Convert some code including j360.
4. Sencha. Run through the tutorials and screencasts.
5. learn about SASS
6. jquery plugin. Write a very simple plugin.
7. outrigger app site. Create landing page about outrigger app.
8. outrigger app. Finish demo data. Ready phonegap feature.
9. get new Android phone and install couchdb. Make iphone into an ipad touch for Civ
10. learn git well enough to be comfortable with github
11. put j360 into git and write readme
12. use build.phonegap for j360 and outrigger
13. colossal caves into github
14. hours reading nodejs blogs and reinstalling on my laptop
15. put code into a js module, maybe my search parser
16. publish regex attribute for input tutorial for Coding JS Apps http://jsapps.posterous.com/
17. fix raydaly.com and raydaly.org
18. publish "New Enterprise Network" talking paper
19. use ace
20. publish on Coding JS Apps an article on using couchdb as development environmen
21. new photo for my blogs, twitter, google, quora and facebook
22. hitch up the wagon

I'll probably add to the list and update this post. As I reread the speakers applications, I'll probably get inspired to do more. This blog or my Coding JS Apps blog will have the results.

Also I need to review what I wrote after last year's JSConf:  a pesky list of ToDos.

15 Years of WebApps in JavaScript

15 years ago on January 23, 1996, HTMLjive - an HTML editor written in JavaScript was release. It is considered the first serious JavaScript program. In short order, it went viral, was on websites around the world and translated into several languages.

HTMLjive was a simple HTML editor but it incorporated a preview window, three modes of HTML coding and on-screen help. The inline HTML editors, like the one I'm using right now to write this blog, have added features, but HTMLjive had the basic features you take for granted today.

I am often asked, "Did you expected JavaScript application (web apps) to become popular?" Yes. Shortly after HTMLjive was released, I starting writing the book "Using JavaScript" with several co-authors. One of my great memories is giving an author talk called "JavaScript Makes Your Site An Application" at a local book store a few months later. So, absolutely.

The community that supports JavaScript has come a long way since. Brendan Eich, father of JavaScript, has received the recognition he deserves. Douglas Crockford gave us JSON and is the JavaScript evangelist. Ajax pushed the language back into popularity and Google showed what could be done with Google Maps and Gmail. Now the annual JSconf features the greatest advances, technologies and speakers. Most of the people that deserve a mention for their contributions to the community were or are speakers at JSconfs.

JavaScript is no longer just for the browser. My web apps, both mobile and desktop, have JavaScript on both the browser with libraries like jquery and on the server with couchdb and node.js. Thank you to the community. It continues to be a joy to code in JS because of you.

Dave's plan to rethink RSS has potential. I've always liked the simple concept of title, link, description (text) and categories. Making a RESTful, JSON API would be great. I've been dabbling towards that with a project I've been working on that started with Zlides.com and OPML combined with JQuery Mobile. So I've been giving this some thought.
Dave likes Twitter. The Twitter API is proven itself and lessons have been learned. Lots of people know it. So my comments below reflect my usage of the Twitter API, particularly the search API.

1. JSONP. The ReallySimple API should support JSONP. The primary benefit is cross-domain support so I can write browers apps hosted on any site and use the reallysimple.org api. This would make it much easier to write mobile apps using JavaScript, HTML5 and CSS. For an example, look at the Twitter Search wiki and how they support JSONP. Also look a jquery's support for JSONP.
2. Categories. RSS has always supported categories and I was rather suprised not to see them in the API. Much of web 2.0 worked leveraged categoris and I think it is an important addition.
3. RESTful. Several people have commented on using the RESTful verbs. I certainly support that for GET and POST but since DELETE and PUT are harder to implement there should be more discussion of that.
   
4. Extending Really Simple. RSS supports namespaces and there should be someway to extend the Really Simple api. Steve Moyer commented about having extra fields like _rev in the API. In my view, extra fields would be fine but probably ignored by most clients. It would probably be better to suppress them, but in some implementations there are cases were the extra fields would add a feature. More discussion needed.
5. getPost. Given an idPost, I would want to get the JSON for just that post. The id could be the permalink. And perhaps, maybe I supply a link and get back an array of posts which could be all of the comments made to a post.
6. Enclosure. Very encouraged to see it supported. Important in this world of videos, photo blogs and other media files.
7. OAuth. Twitter and others have moved away from basic authentication for a reason. I would suggest supporting OAuth like twitter.
   
8. No date. The format of title, description and link is so powerful that it applies to more than just reverse chronological blogs. The web content management system (CMS) that I wrote had RSS feeds as its foundation. [I'll have to add the ReallySimple API to it.]  And see example of Wikipedia below. However, I would think that this might be an expected field. Or perhaps it is better to ignore it for right now given the discussion that will start about date format.
   

ReallySimple maps to ReallySimple HTML pages as well. It supports title, link, description and hopefully category. This also maps directly to standard Web 1.0 content web pages where one HTML page contains content on a subject. For example, this Wikipedia page on Daly Languages has a title and obviously a link. The content is inside the body tag, less the navigation, footers and banners. Categories in the metadata and displayed on the page. So I should be able to take the primary content of this page and easily put it into a JSON message using the ReallySimple fields. Lots of features could be built off having an entire page in JSON.

I look forward to using the ReallySimple API with mobile apps and Couchdb apps.

Cross posted from: http://raydaly.posterous.com/really-simple-api-first-comments

Recommended Beers of 2010

This was a great year for beer in Northern Virginia. Full Sail is now distributed and Great Divide's beers are more widely available. Also, Bob Tupper got back into business after a few years of dryness. Finally, Church Key opened a few block from my office in DC. Hope you have a chance to try some of these beers this year.

Great Divide - Collette   This is a special. Grab as much as you can when you see it. I spent a night driving from store to store buying everything that was left. The focus on this beer is not the hops, but the yeast. It is a different taste. This is a farm house ale with yeast in the bottle. My friends laugh at me because my pours end up with overflowing heads. Great in the spring or summer. We had it the other night with Raclette and it was excellent.

Great Divide - Titan IPA We first discovered this perfect IPA from Great Divide before the Collete. It is great when a brewery makes a beer so good that you want to try everything that they produce. The balance in the IPA between the hops is totally refreshing and not overwhelming like others fighting the hop wars. I'm never disappointed when pulling one of these out of the fridge.

Appalacian - Hoppy Trails IPA  This IPA made me really appreciate the IPA style. It made be go back and try some others again like Full Sail. The semi-high alcohol punch of an IPA has to be balanced with a strong hoppy flavor supported by a robust malt. A regular in my fridge.

Tupper's Hop Pocket Ale  Bob Tupper introduced me to so many different beers at the Brick. I was thrilled when Old Dominion started brewing his Ale because I knew he liked Celebration Ale. This was my friend's favorite until Tupper pulled it from Old Dominion. After a long hiatus, Bob found a brewery and his Ale is back. Dominion did a better job, but still wonderful to have it back.

Schlafly - Christmas Ale Normally this spot would be reserved for Sierra Nevada's Celebration Ale. But this spot opened up this year because of the disastrous change in the recipe. But Schlfly's might have made it anyway. The hint of orange made it a wonderful taste in a full ale for the season. I recommended it to everyone in my annual Christmas Beer tasting this year.

Heavy Seas - Winter Storm I've enjoyed this for the past several Christmas seasons. It is now my favorite Christmas ale with a great Extra Special Bitter flavor. I'll stock up this ale. Nearly a perfect cold night brew to warm your bones. Arrg!

Breckenridge - Extra ESB I love ESBs. My favorite was Fredrick's Red Ale, but that is now defunct. Had another great one in New Orleans about 10 years ago. It is so hard to find a good one, nevertheless a perfect one. Breckenridge's is not perfect but better than good. It is amazingly consistent. ESBs can be so smooth. I keep this one around.

Full Sail - Pale Ale. About 20 years ago at the Brickskeller the bartender introduced me to Full Sail Ale and Red Seal Ale. They were mana from heaven. They would be my first request when walking down those stairs and seeing all the cans. But distribution stopped and I finally gave up the search. I had to visit the brewery when in Oregon but never expected to see it back on the East Coast. This was the great surprise of the year. And try their IPA as well. Both wonderful.

Full Sail - Session Found this at a bar and have stocked up with both the regular and the dark Session. It comes in a Red Stripe like bottle and is a tasty lager with only 4% alcohol.  Often I don't need the full IPA punch so this is a tasty alternative. Expect it to be a regular this summer.

Bell's - Oberon  Our summer beer for the past few summers. I cannot help but think about a warm summer day and the pleasant taste of this wheat beer. You know summer has arrived when Oberon is on the shelves.  Coming again soon.