Augmented Reality has a lot in common with 3D movies. Both were invented decades ago, both are enjoying a renaissance, and both, as we will see, can be used in ways that transform the experience--or simply as attention-attracting gimmicks. Invented in 1968 by computer graphics pioneer Ivan Sutherland, AR was once the sole domain of academic researchers, requiring serious programming skills and expensive equipment like head-mounted displays. Now, the learning curve and hardware costs have precipitously dropped--putting augmented reality in the mainstream of commercial application development.

Mark Billinghurst, director of the Human Interface Technology Laboratory New Zealand, has helped propel this revolution. He is best known for his collaboration with Hirokazuo Kato on ARToolKit, a library that solves one of the key problems of augmented reality by enabling an AR application to determine where users are looking in the real world.

“I always knew there would come a point where that barrier to entry would be lowered--and it has,⁠”⁠ Billinghurst told me, speaking from Christchurch. Evolving display hardware has made a dramatic difference. The smartphone, Billinghurst noted, has more computational power than the high-end graphics computer that once sat on a few researchers⁠’⁠ desks. Smartphones are being joined by the camera-equipped iPad2, gesture-driven gaming consoles, and webcam-equipped PCs and laptops, both of which remain the platform of choice for AR-augmented e-commerce.

Flash makes a difference

If hardware advances have driven down AR costs, so have development tools--most notably those that take advantage of Flash. “⁠Around ten years ago, you started seeing computer vision-based augmented reality appearing on the PC, but mostly as native, standalone applications developed with C++,⁠”⁠ Billinghurst said. “⁠Then around three years ago, the Flash platform developed to the point where you could start using it for computer vision--and people started porting applications over to it.⁠”⁠ For end users, Flash meant an AR experience without having to download additional software. On the development side, Flash opened the door to the hundreds of thousands of software engineers familiar with the platform who could potentially create AR applications.

These days, Billinghurst said, you can create AR applications in Flash ActionScript code, with calls to a Flash AR tracking library, such as the GPL-licensed FLARToolKit--an ActionScript 3 version of the ARToolKit--which enables an application to overlay content atop a tracking marker. That content is produced by a Flash compatible 3D graphics library, such as Papervision3D.

“On mobile, you might download one of the free AR browsers, such as the junaio browser from metaio or the Layar Reality Browser from Layar--both cross-platform applications supporting Android, iOS, and other mobile operating systems. From there, the developer creates server-based code, typically written in PHP or another scripting language, that specifies the points of interest and the appropriate overlays.”

A second option is to build a native application for the Android or iPhone platform. “⁠In this case you⁠’ll need to be a competent Java programmer for the Android platform or a C++ or Objective C programmer for the iPhone, and you⁠’ll need to download a tracking library on one of those platforms and then build your native application.⁠”⁠ Versions of ARToolKit are available for both mobile platforms. “⁠Once that⁠’s done you can upload to the appropriate marketplace. A native application requires a lot more work, but it provides a lot more flexibility in terms of what your application can do. Your interface doesn⁠’t need to look like a generic AR browser interface―it can look like whatever you want.”

Billinghurst says that the next breakthrough for AR tools may put the technology in the hands of non-programmers--enabling authors, artists, and other creative people to invent their own AR experiences. “⁠For example, we developed a piece of software called BuildAR that lets ordinary people, including children, assemble simple AR scenes using a simple drag and drop mouse interface on their PC.”

The AR landscape

Today⁠’s universe of AR applications is often divided by platform, with the PC and its dedicated cousin, the kiosk on one hand, and mobile on the other. Because they typically run iOS or Android, camera-equipped tablets are seen as part of the mobile camp--even though you can⁠’t quite stuff one in your pocket. But whatever the platform, not all AR applications are equally promising, and what first grabbed people⁠’s attention about the “⁠new⁠”⁠ AR may not represent its future. Consider the magazine covers and product boxes that, held up to a camera-mounted display, come alive. These “⁠digital holograms⁠”⁠ are often the first experience many people have with AR, and that experience can be eye-popping--which is why so many companies have been attracted to AR as a marketing ploy.

Will over-exposure turn that ploy into a gimmick? Here, 3D movies are instructive. While 3D is here to stay, the best examples, like Avatar and Werner Herzog⁠’s Cave of Forgotten Dreams, use the third dimension not just as eye candy, but as an integral part of the movie. The best AR experiences will do the same. “⁠The lower hanging fruit has been the AR marketing experience, where there is often very little interactivity with the content,⁠”⁠ Billinghurst said. “⁠Two years ago, it would have been sufficient for a marketing company to show a 3D model on top of a magazine with the company⁠’s products. Now that⁠’s become old fashioned. As people become more familiar with the development platforms and the technology, I think you⁠’ll start seeing richer AR experiences.”

Those experiences fall into four different categories: e-commerce, games, information, and professional. The last of these is worth mentioning up front--because it hints at where the technology is headed. These are state-of-the art applications designed for specialists--surgeons and engineers, among them. “⁠You are starting to see AR providing professional training to, for example, show the names of the component parts of an engine, or to help with a surgical procedure,⁠”⁠ said Billinghurst. “⁠These are still largely custom software running on PCs, because they must provide a higher quality AR experience than you can get on a mobile device or through a browser. But I think these sorts of applications will spread to lower end devices over the next five to 10 years.”

AR development with an SDK

To get a better idea of how AR development works, I spoke with Ben Blachnitzky, director of research and development for metaio GmbH, which, along with Layar, makes one of the most popular augmented reality browsers, as well as related SDKs and other development tools. An early step in the development process is for the software to be able to identify the object to be tracked. Objects with strongly demarcated colors and lines are good candidates, while more subtle, gradated images, are not. “⁠One of the first things the SDK tells you, on a percentage basis, is how trackable your object is,⁠”⁠ he said. The developer also provides the actual dimensions of the object, thereby enabling the software to accurately track it. This data, including the image and measurements, is stored in an XML file.

Once the software “⁠knows⁠”⁠ the object it is tracking, it generates a matrix of coordinate values that show the center of the object with respect to a virtual camera, as well as its rotation. That data can then be used to manipulate the rendered, virtual image, thereby syncing or “⁠gluing it⁠”⁠ to the real-time image. What you see on the display is a blending of the two.

Blachnitzky said that while you don⁠’t have to be an advanced developer to accomplish this, some programming is required. “⁠You need C# or Visual Basic, plus some HTML and ActionScript to put it on a website. But you don⁠’t have to know about tracking, visualization, or rendering. The developer⁠’s main job is in providing the user interface--figuring out where to put the controls and what those controls do.”

One of metaio⁠’s prime examples of a practical AR application is a virtual dressing room developed last year for the American department store JC Penny. The clothes are rendered into 2D, the customer positions herself so that it fits, scaling the image larger or smaller to match. Rather than tracking a pattern, the software tracks hand movement, making for a convenient interface.

The next obvious step would be to use a 3D image instead, and Blachnitzky says we are not far off. “⁠A 3D image might appear to attach to the body, turn when you turn, and might even wrinkle appropriately as you raise an arm. We⁠’re not there yet, but we⁠’re surprisingly close.⁠”⁠ What the technology needs is a way to identify not just hand motion, but a specific body shape. That could be done by photographing a customer from every angle, or having him or her wear a gridded shirt that would establish the coordinates.

AR Games: Ogmento’s Paranormal Activity

Based in New York and Los Angeles, the game company Ogmento has hung its hat on AR. The company is best known for Paranormal Activity, an iPhone game some reviewers have said is better than the movie franchise it was based on.

“Unlike most games where you sit on the couch and play in front of the screen, this game takes place in the real world,⁠”⁠ said Ogmento CEO and co-founder Ori Inbar. “⁠The back-story is that demons and ghosts are taking over the world and your job is to create sanctuaries that will protect you favorite neighborhood spots.⁠”⁠ Players cast spells by drawing a shape on a paper that is then tracked by the software when the phone is in camera mode. The game also uses Google Maps, with an overlay showing areas of demonic activity in red, and sanctuaries in blue. “⁠A third part of the game are missions that are incorporated in real-life locations. The game automatically creates those for cemeteries or churches, grocery stores and hospitals around you, and that leads you to the places as far as the story line. It may say there⁠’s a weird looking guy in the grocery store―go investigate. When you get, there you get another mission.”

Ogmento has experimented with more site-specific versions of this idea. At the Game Developers Conference in San Francisco, the company placed missions at the Moscone Center venue, Alcatraz Island, and a nearby hotel. “⁠In New York, we put missions in some famous haunted places, like Washington Square,⁠”⁠ Inbar said. “⁠I learned from developing the game that 20,000 people were buried there around 200 years ago.”

Part of the inspiration for these types of games came from Inbar⁠’s realization of how much time his son was spending indoors in front of a screen: watching TV, playing video games, browsing the Internet. “⁠He was missing life on the streets, playing with friends. I thought: what if we take some of the stuff that attracts kids in video games and bring it into the real world.⁠”⁠ Using an entre from one of Inbar⁠’s partners, Ogmento pitched the idea to Paramount Pictures, which had acquired the rights to the film. The plot, about a couple who captures a demonic presence in their bedroom on a video camera, seemed a shoe-in for a video game with AR elements. The studio agreed. “⁠They loved it,⁠”⁠ Inbar said.

Initial development took six months, and the team was ultimately surprised how much they managed to cram in: geolocation, mapping, and of course augmented reality and 3D graphics, which were created with the Unity 3 engine. “⁠For the AR part of the game, we came up with the idea of extracting the content from the real world. You sketch a shape on a paper and the graphics represent what you just drew, becoming part of the game⁠’s imagery. It⁠’s a form of player-generated content, because every time you draw the shape, the game will be a bit different.”

Ogmento⁠’s follow-up game will also incorporate AR elements, but will be more casual and to a broader audience--but the idea of playing in the real world, not on the couch, will remain the same. What about Japan? “⁠We have no location-specific plans at this point, but the game can be generated anywhere ― we have players in Guam. We are tracking the activity around the world and there is a lot going on in Japan. It⁠’s one of the top three or four most active countries right now. We would love to do more in Japan, as well as Korea.”

Sidebar - Virtual tags and text transformation

While much of the attention for AR centers on 3D graphics, another, quieter form is in superimposing labels on real world items. Amateur astronomers were early beneficiaries through applications like Star Walk (Vito Technology), which provides virtual labels for stars, planets, constellations and meteor showers. Other mobile AR apps can do similar tricks here on earth by correlating location information with the image captured on a video camera.

A cousin of these apps can be glimpsed in Google Goggles, which can use a mobile device⁠’s camera to search for information on an object, including business cards, paintings, and landmarks. The app also translates from one language to another. Google notes that not everything is yet searchable: food, cars, plants and animals, among them.

Sidebar - Putting AR to work on websites: Zugara’s Matt Szymczyk

Los Angeles based website firm Zugara has taken the AR plunge. Founded in 2001 as an interactive Web developer, the firm got its start doing interactive marketing for the Sony Playstation, and continued with work for Lexus, Toyota, Casio and other American arms of Japanese companies. But after experimenting with AR in 2009, the firm repositioned itself as an AR software developer. The move put Zugara⁠’s CEO, Matt Szymczyk, in an interesting position--he a proponent of the technology, but with a distinct view of how best to put AR to work on websites.

Here are excerpts from our conversation:

How should website designers look at AR: as a promising technology, a gimmick, or perhaps both?

Any online technology can be a gimmick. It⁠’s the strategy behind it and how you are deploying it that matters. I⁠’ve been meeting with companies for two years, and am seeing a shift. Now they are asking smarter questions, like how AR can help with product development. People are starting to see the longer trajectory of AR, not just what⁠’s happening now. But it⁠’s definitely not just marketing. I think marketing has worn it out already. That⁠’s why we are focusing on e-commerce. Augmented reality can really help in areas where you want to go online and see how something works.

What are some examples?

There⁠’s a Converse mobile app to see what a shoe would look like on your feet. IKEA has an app that simulates how furniture would look in your home. Total Immersion worked with Olympus to let you interact with a 3D representation of a camera. The application serves both as an e-commerce vehicle and as a training manual. Samsung lets you see how an LCD screen would look on your wall.

The point of these AR applications is to help e-commerce retailers increase the sales conversion rate and decrease returns, which are the two biggest pain points. In a physical store, the sales conversion rate is something like 20 to 40 percent, while online, it is about 2-3 percent. We see AR as a way to help bridge that gap. When you are in a store, you can hold an item up to yourself to see how it looks. AR makes it possible to do something like that online, because you can step away and see the apparel item overlaid on you. These kinds of things represent a whole new “⁠funnel⁠”⁠ in the sales cycle for purchasing online.

You are not a big proponent of mobile AR. Why not?

Nobody is going to dispute that mobile is the future of AR, but the hype around mobile AR is ten times worse than the hype around AR in general. There are too many expectations on what mobile AR is going to do. There are issues surrounding processing power, battery life, and the user interface. There are also multiple development platforms to account for ― iOS, Android, Windows and Blackberry. The iPad, technologically, still doesn⁠’t meet the minimum specifications needed to run the average AR application on a desktop. Besides, are people really going to walk around using their phone as a viewfinder? Probably not.

Is the iPad2 in the same boat as mobile?

Google just conducted a study on how people actually use their tablets. The number one thing was gaming. From the gaming side, there might be a lot of practical applications with AR using the front forward facing cameras. But as you go down the list, you see shopping is less than 50 percent: people aren⁠’t going to shop on an iPad. With any hardware device, you need to first understand how consumers will use it.

What should website developers know about incorporating AR?

Follow the KISS principle―keep it simple stupid. If you are doing AR for a quick engagement, you want to have the fewest barriers possible. Augmented reality represents a revolution for how humans will interact with computers, but most people aren⁠’t ready for it yet. This has brought the focus back to usability and interface design for AR, both of which have been overlooked at the expense of technological advances. We learned a lot of those lessons over last three years. Older people are not all familiar with their web camera. Younger people, around the median age of 28, quickly grasp gestural control, where older people tend to take longer to adapt to it.

Does it suggest that gestures may be the future user interface for augmented reality?

Yes, it⁠’s all going to be gestures. You won⁠’t be sitting in front of the computer rotating a piece of paper and image.