Sony Revamps AR Technologies With Robotics [Movie]

May 21, 2011
Tetsuo Nozawa, Nikkei Electronics
A superimposed hologram
A superimposed hologram
[ If it clicks, the expanded picture will open ]
"Momo," a character for Sony's PostPet service, is superimposed on a recognized object. This is something that even traditional AR technologies can do.
"Momo," a character for Sony's PostPet service, is superimposed on a recognized object. This is something that even traditional AR technologies can do.
[ If it clicks, the expanded picture will open ]
Momo moved away from the recognized object and started walking in the room. This cannot be done with traditional technologies.
Momo moved away from the recognized object and started walking in the room. This cannot be done with traditional technologies.
[ If it clicks, the expanded picture will open ]
When a menu card placed on a table in a restaurant is viewed through a smartphone ("Xperia arc SO-01C" of Sony Ericsson), the message "Touch Here" appears on the card.
When a menu card placed on a table in a restaurant is viewed through a smartphone ("Xperia arc SO-01C" of Sony Ericsson), the message "Touch Here" appears on the card.
[ If it clicks, the expanded picture will open ]
When the message is touched, a menu whose pages can be turned with a finger appears. For regular customers, it can show recommendations. The superimposed information does not disappear even when a smartphone is moved. So, the information can be brought to home.
When the message is touched, a menu whose pages can be turned with a finger appears. For regular customers, it can show recommendations. The superimposed information does not disappear even when a smartphone is moved. So, the information can be brought to home.
[ If it clicks, the expanded picture will open ]
The image of flowing water is superimposed on a table. The flow changes when it hits a book.
The image of flowing water is superimposed on a table. The flow changes when it hits a book.
[ If it clicks, the expanded picture will open ]
A dress at a boutique is recognized. When the screen is touched, a variety of coordinates are displayed on the smartphone.
A dress at a boutique is recognized. When the screen is touched, a variety of coordinates are displayed on the smartphone.
[ If it clicks, the expanded picture will open ]
When the door of a store (such as a restaurant) is recognized, it is possible to display Twitter-like comments about the store.
When the door of a store (such as a restaurant) is recognized, it is possible to display Twitter-like comments about the store.
[ If it clicks, the expanded picture will open ]

Sony Corp developed a technology that combines augmented reality (AR) technologies and robot technologies that the company has been developing for the "Aibo" and "Qrio" robots such as to recognize space.

This announcement was made May 19, 2011. And Sony will try to apply the new technology, which it calls "Integrated AR Technology (SmartAR)," to advertisements, games and businesses.

According to the company, the SmartAR has the following three features: (1) a capability of recognizing objects without using markers, (2) "Kousoku Pittari" (high-speed matching) based on a high-speed tracking technology and (3) "3D Space Recognition."

Probability used for matching

The first and second features were developed by improving the accuracy of a technology that enables to use objects to be shot by a camera, etc as information for retrieving AR information and adjusting its position. It eliminates the need for two-dimensional markers, which have been commonly used for traditional AR technologies.

Specifically, Sony realized a high-speed recognition technology that is less subject to changes in the way objects are lighted and their postures by employing the company's own image recognition algorithm that requires a smaller amount of calculation and a matching technology using probability.

Furthermore, Sony enabled to track objects at high speeds by using a technology that matches local characteristics obtained from the image of an object used as a trigger (an object that triggers AR information to appear).

Recognizing, recording space

As for the third feature, 3D Space Recognition, once it recognizes an object used as a trigger, it also recognizes objects and space around the trigger object. Then, the technology deals the information of the surrounding objects and space as part of the trigger object.

It is necessary to prepare the image data or reference data of a trigger object beforehand. But the information of the space around the trigger object is input into a database after the trigger object is recognized, Sony said.

This technology has several advantages. First, even when the trigger object goes out of the view of a camera, it is possible to continue to superimpose AR information. With traditional technologies, AR information disappears once a trigger object goes out of the view of a camera.

Also, the technology eliminated the need to register data of an entire object that will be recognized in a database, enabling to use an entire room as a space on which AR information can be superimposed.

Furthermore, it became easier to interactively work with AR information. The recognized 3D space keeps being updated, and AR information can be superimposed based on the updated space information.

In a demonstration that Sony had this time, the image of flowing water superimposed on a table changed depending on whether a book is placed on the table. And the image of a ball that was at first bouncing on a table started bouncing on a book.

High hopes for multi-core smartphones

However, Sony said that it is not planning to use the new technology for actual services at this point because of the following three problems.

First, because it is an AR technology that does not use markers, it is necessary to pre-register the image information of a trigger object in a database. This will require some preparation.

Second, there is a question whether it is possible to superimpose AR information within a short enough time via wireless connection. In the demonstration, the image information of objects was stored in the memories embedded in a smartphone and a PC.

However, to commercialize the technology, the image information has to be exchanged between an image database on the Internet and a smartphone, etc via a wireless network. This will slow response speed because wireless networks cause delays. Moreover, if a large-scale database is used, it will take even more time in consideration of the time required for retrieving image information.

The third problem is related to the processing power of smartphones.

"The 3D Space Recognition will not fully function with existing smartphones because they do not have enough processing capability," Sony said. "But if it becomes possible to use a smartphone equipped with a dual-core or quad-core processor, this problem will be solved."

Also, the company will continue to make efforts to develop a lighter algorithm.


When the angle of the book changes, the moving directions of AR balls change too. And the balls bounce on the book even when it is away from the table.