Sony Reveals Latest Features of Its AR Software

Oct 31, 2012
Tomohisa Takei, Nikkei Electronics
SmartAR can recognize the structure of a real space and display a virtual object (a monkey in this case) as if it exists in the space.
SmartAR can recognize the structure of a real space and display a virtual object (a monkey in this case) as if it exists in the space.
[ If it clicks, the expanded picture will open ]
"Hatsune Miku -Project Diva- f," Sega Corp's game software for the PS Vita, has a mode in which Hatsune Miku, a virtual singer, dances and sings in the image of a space taken by a camera.
"Hatsune Miku -Project Diva- f," Sega Corp's game software for the PS Vita, has a mode in which Hatsune Miku, a virtual singer, dances and sings in the image of a space taken by a camera.
[ If it clicks, the expanded picture will open ]
"Hako -Open Me-," game software that Sony Computer Entertainment plans to release at the end of 2012, is a puzzle game in which a box placed in a real space is opened by a player. It also offers a play mode in which two players share the space and cooperate to open a box.
"Hako -Open Me-," game software that Sony Computer Entertainment plans to release at the end of 2012, is a puzzle game in which a box placed in a real space is opened by a player. It also offers a play mode in which two players share the space and cooperate to open a box.
[ If it clicks, the expanded picture will open ]
Nogizaka 46 AR is a smartphone application for promoting CDs of "Nogizaka 46," a Japanese idol group. When a photo of the poster is taken with a smartphone, a member of the group appears on the screen. The video is superimposed by using the "LiveAction AR" technology.
Nogizaka 46 AR is a smartphone application for promoting CDs of "Nogizaka 46," a Japanese idol group. When a photo of the poster is taken with a smartphone, a member of the group appears on the screen. The video is superimposed by using the "LiveAction AR" technology.
[ If it clicks, the expanded picture will open ]
Churaumi AR Camera is an AR application that Okinawa Churaumi Aquarium will start to distribute for smartphones. When a photo of the postcard posted in the center of the board is taken, a 3D model of a whale shark appears in the space. The size of a real space is estimated based on the size of the postcard, and the scale ratio of the whale shark (total length: 8.5m) is displayed so that the user can feel the size of the shark.
Churaumi AR Camera is an AR application that Okinawa Churaumi Aquarium will start to distribute for smartphones. When a photo of the postcard posted in the center of the board is taken, a 3D model of a whale shark appears in the space. The size of a real space is estimated based on the size of the postcard, and the scale ratio of the whale shark (total length: 8.5m) is displayed so that the user can feel the size of the shark.
[ If it clicks, the expanded picture will open ]

Sony Corp announced the current state of the development of SmartAR, which is software for realizing augmented reality (AR), at a press conference in Tokyo Oct 29, 2012.

Sony has been improving the software to display AR images on mobile devices without lags since the previous announcement that the company made May 19, 2011 (See related article).

SmartAR is software that can extract the feature points of an image taken by a camera, match them with registered images and recognize the structure of a space while estimating the movement of a camera. It does not require "markers," which work as triggers that are used to select information or video to be superimposed. Also, it can display an AR image as if a virtual object is placed on a real object.

As for the recognition of a space or an image and the process of displaying information to be superimposed, a frame rate of only about 15 fps had been achieved with a personal computer at the time of the announcement in 2011. However, as a result of the improvement, Sony realized a frame rate higher than 60fps with a mobile device.

It is now possible to display AR images that accurately follow input images, and users feel that the AR images are more natural, said Masaki Fukuchi, researcher at the comapny's Intelligent Systems Research Laboratory. Also, the improved software uses information obtained with an acceleration sensor and gyro sensor to accurately estimate the orientation and location of a mobile device.

After the improvement of the software, it was employed for the main function of Sony Computer Entertainment Inc's "PlayStation Vita" portable game console, and several games using SmartAR were released. Moreover, the software is now used to promote the sales of the Sony Group's movie and music software. And Okinawa Churaumi Aquarium plans to start distribution of Churaumi AR Camera, an AR application for smartphones, Nov 1, 2012.

As functions related to SmartAR, Sony added (1) the "LiveAction AR," which displays a 3D photographed image such as human image so that it seems to exist regardless of the angle from which it is viewed, and (2) a large-scale image recognition technology that transmits image information extracted from an image taken by a camera to a server and matches it with registered images in the server.

The first function has already been employed for "Nogizaka 46 AR," an application for promoting CDs of "Nogizaka 46," a Japanese idol group. And the second function will be used for a campaign that Culture Convenience Club Co Ltd will start to promote the rental and sales of DVDs and Blu-ray discs in November 2012.