Virtual Reality System Offers a Wide Perspective

Originating Technology/NASA Contribution

Robot Systems Technology Branch engineers at Johnson Space Center created the remotely controlled Robonaut for use as an additional “set of hands” in extravehicular activities (EVAs) and to allow exploration of environments that would be too dangerous or difficult for humans. One of the problems Robonaut developers encountered was that the robot’s interface offered an extremely limited field of vision. Johnson robotics engineer, Darby Magruder, explained that the 40-degree field-of-view (FOV) in initial robotic prototypes provided very narrow tunnel vision, which posed difficulties for Robonaut operators trying to see the robot’s surroundings. Because of the narrow FOV, NASA decided to reach out to the private sector for assistance. In addition to a wider FOV, NASA also desired higher resolution in a head-mounted display (HMD) with the added ability to capture and display video.

Partnership

Founded by former Johns Hopkins University optics researcher, Dr. Lawrence Brown, Sensics Inc., received a NASA Small Business Innovation Research (SBIR) program award in 2003, and began Phase I of their FOV interface for NASA’s Robonaut, which also included development of a wide FOV camera. Baltimore-based Sensics created an HMD with a wide FOV, using a patented optical design to combine several tiled images into a high-resolution, three-dimensional panorama. A follow-on Phase II SBIR project culminated with Sensics delivering the piSight HMD to Johnson Space Center in February 2008, after delivering the prototype the year before.

The Sensics piSight HMD, essentially a high-tech viewfinder, offers an additional panoramic, high-resolution camera array that allows remote control and telepresence, immersing the HMD operator in the robot’s workspace in real time. The operator sees through Robonaut’s “eyes” and controls its movements, working safely from a distance. “Astronauts wearing a piSight will feel just like they are experiencing the world from Robonaut’s perspective, due to the full field-of-view, which is an exciting advance for the NASA robotics program,” says Brown.

Product Outcome

With the piSight, virtual surroundings appear in the viewfinder and respond to head movements. By coordinating head and joystick movements, the user flies or coasts up, down, forward, or sideways. Some versions of the piSight work with motion trackers, which eliminate the need for a joystick and, instead, follow the user’s movement realistically.

When used with the piSight, InterSense Inc.’s SoniWing tracks user movements, employing technology similar to some video gaming systems. With the addition of the SoniWing, the user’s entire body moves within the virtual environment, allowing greater freedom than systems restricted to head motion, and the environment responds realistically to the user’s location and movements. Sensics also offers customized systems that track a user’s eye movements, which help vision researchers learn which images or objects in a scene hold a user’s interest, and also offers a video see-through solution for applications in which users need to see the virtual world and reality simultaneously.

The piSight HMD is light (weighing less than 2 pounds), stable, comfortable, and adjustable to an individual’s head and eye shape, but Sensics still wants to make the technology more lightweight and efficient. Currently, piSight can be used with a variety of virtual reality software and platforms, including OpenSceneGraph on Linux and WorldViz products on Windows.

The piSight also offers high resolution “better than an HDTV display,” according to Brown, with the NASA model displaying 12 screens per eye; lower-end models have as few as 3 screens per eye. Other commercially available HMDs usually offer fewer screens, fewer cameras, or poorer resolution, which leads to either a much smaller field of vision (i.e., no peripheral vision) or a poor-quality image that appears pixilated or stretched. With more screens per eye, however, each display can offer a slightly different orientation, forming a concave arrangement that wraps around the eye, and projecting a different and more realistic perspective. In a higher-end model, the user can see in high resolution for 150 degrees, which is almost as wide as the natural range for humans. “The operator basically sees a field-of-view that has virtually no limitations,” Brown notes.

The piSight camera array captures panoramic, high-resolution live video, which a user views while wearing the piSight HMD. Video is compressed, sent over a network, and then displayed inside the HMD. This interface may be able to support astronauts and engineers on Earth with various tasks like repairs and maintenance in high-risk environments. An astronaut in orbit, for example, could use the piSight remotely to control a robot on the surface of the Moon and examine the surrounding environment in real time. The system could also enhance safety on Earth by enabling remote operation of machinery in biohazard, defense, or medical environments, without requiring physical access to the site. In an emergency involving hazardous materials, a human operator could use the piSight system to remedy the situation from a safe location.

Sensics is finding success in the commercial sector, with sales of its products topping $1 million in 2007. Sensics is currently marketing the piSight to larger companies, since system prices start at about $27,500. Depending on customization and features, such as additional camera arrays for a wider field-of-vision, the units can cost up to $100,000.

Commercial applications for the piSight are only limited by customers’ imaginations. “Customers have been selecting the piSight systems for a variety of simulation, training, design, and research applications,” says Brown. An automobile designer uses the HMD to “sit” in the driver’s seat of an automobile and explore the design in a realistic, immersive experience before production begins. Other companies are beginning to use the piSight for training purposes that involve learning specific tasks in situations, such as teaching a quarterback a specific play or showing a soldier how to operate a certain piece of equipment. Clearly, NASA will not be alone in reaping the benefits from virtual reality systems like the piSight.

piSight™ is a trademark of Sensics Inc.

SoniWing™ is a trademark of InterSense Inc.

Linux® is a registered trademark of Linus Torvalds.

Windows® is a registered trademark of Microsoft Corporation.