FlightGoggles is a development environment that allows the design, implementation, testing and validation of autonomous super-vehicles. FlightGoggles provides exteroceptive sensor simulation based on the Unity3D engine as well as vehicle dynamics and inertial sensor simulation capabilities. It features photorealistic exteroceptive sensor simulation, generated using photogrammetric scan of real environments. It allows traditional simulation, hardware-in-the-loop simulation, virtual-reality simulation and augmented-reality simulation capabilities. FlightGoggles is available to the public open source.
Photogrammetry powers the two unique virtual environments that comes with FlightGoggles. Photogrammetry is a technique that creates virtual environments through their photographs. To create both of the two FlightGoggles virtual environments, we took countless photographs of real-world assets. We created 3D graphics assets from these photographs.
The next animation (animated on your scroll) shows the graphics asset generated along with the pictures that were taken to generate it. (The pictures are shown as small rectangles.) As you scroll page the asset will rotate for you to get a better view.
True photorealism is enabled by the photogrammetry technique. In this way, every pixel that is rendered is indeed looking at some transformation of photos from a real-world object. If applied well, the realism can be truly exceptional. The next animation (animated on your scroll) shows one section of one of the FlightGoggles Virtual Environments.
FlightGoggles features two photorealistic virtual environments: 1. The Abandoned Factory Environment, 2. The Stata Center Environment. While they serve different purposes, both are designed with photorealism and autonomous super-vehicles applications in mind.
The Abandoned Factory Virtual Environment is designed to resemble a run-down large interior space, envisioned similar to typical drone racing arenas - perhaps a bit more run down than usual. It features several visually-challenging elements, such as water puddles, rubble, broken objects, and lighting effects.
It is created by constructing virtual replicas of eighty different objects, such as columns, walls, rubble, barrels, and more. The environment is then put together by creating around a thousand instances of these eighty different objects, hence replicating some of the objects many times. You can find more information here.
The Stata Center Virtual Environment is a true virtual replica of two floors of MIT's Stata Center - a Frank-Gehry-designed building. Stata Center is one of the landmarks of Boston area. It is the home of the Laboratory for Information and Decision Systems (LIDS) - our home laboratory. This environment is, we believe, one of the visually most compelling virtual environments available today.
The Stata Center environment is carefully crafted from tens of thousands of photographs. The environment is hand-cleaned in order to perform rendering in real time. You can find more information with visual benchmarks here.
AlphaPilot challenge used FlightGoggles with its Abandoned Factory Virtual Environment for down-selecting the best nine teams that will compete in the main challenge. AlphaPilot was a competition co-organized by Lockheed Martin, NVIDIA, and the Drone Racing League (DRL). FlightGoggles simulation challenge is the main test that the teams are asked to complete in order to make it to the race.
The video shows the top twenty teams that submitted solutions for the simulation challenge. We overlaid their trajectories, so that you can watch the competition. All drones in this video are autonomously navigating this complex simulation environment without any human intervention.
The FlightGoggles development environment also enables development in virtual reality using motion-capture facilities. An aerial robotic vehicle with on-board embedded computers is flown in a motion-capture facility. All exteroceptors, including cameras, are synthetically generated, and transmitted to the vehicle while the vehicle is in flight. In this case, all complex dynamics acting on the vehicle, such as the aerodynamics of the vehicle, mechanics of the motors, and electrochemistry of the battery, are real, while all exteroceptive sensors are simulated.
FlightGoggles Arcade is a drone-racing video game, developed for research purposes. Specifically, our aim is to capture how human pilots make their decisions in drone racing games. It features FlightGoggles Stata Center Environment, which is a digital replica of MIT's Stata Center. The game has a number of courses with varying challenge levels.FlightGoggles Arcade will be released for Mac and Windows shortly. Please stay tuned! The video below is a teaser, showing the main menu and game play.