I first learned about synthetic data generation from an article that described how researchers had used 3D renderings of eyeballs under a multitude of lighting conditions, in different orientations, of various colors, etc. to determine the direction of gaze for real world video analysis. As it turns out, it’s very difficult to determine, simply from security footage or for eye tracking applications precisely where a given person is looking — and there’s no good way to collect that data. You can estimate it by hand-labelling a human’s best guess at where the eyes are gazing frame-by-frame, but the level of inaccuracy is too high to produce useful data. As they say, “garbage in, garbage out.”
Well, synthetic data is some of the most pristine data you can get your hands on because you get a pixel-perfect labeled data set right from the outset (because you generated it!). No outsourcing frame-by-frame images to literally thousands of error prone and probably incredibly bored humans. And luckily, because synthetic data is generated by a computer, once you have a system to procedurally generate that data, it’s a massively scalable data source.
A massively scalable data source is going to be critical for training and testing autonomous vehicle (AV) software. While it is indeed possible to drive millions of miles to gather data about the public roads (as Waymo, Uber, and others have spent untold millions of dollars to do), relying purely on real-world data is not a scalable approach.
Why? Roads, traffic conditions, weather, pedestrians, obstructions, and a whole host of other dynamic elements are in constant flux and present a gargantuan number of variations that make it incredibly difficult for AV software to safely navigate. And every time AV software companies want to push out a point update (e.g. Version 1.1 of your autonomous vehicle software), they will need to scalably test their new algorithms to ensure that their updates work safely across the trillions of possible scenarios a car can encounter on the open road. Just as much as your iOS or Android point updates sometimes inadvertently crash your entire phone based on an unforeseen bug, autonomous vehicles can and, unfortunately, will inevitably introduce bugs into their navigation stack while updating systems.
Realist rendering and the underlying pixel-perfect labeled synthetic dataset.
Parallel Domain is building software that can procedurally generate worlds that, due to their synthetic nature, are both scalable and come with pixel-perfect labels (see the labeled stripe on the image to the left). In doing so, Parallel Domain enables AV companies not only to drive billions of simulated miles before they push their software to the real world, but also generate more training environments where the AV software struggles. Say, for example, an update decreases safety when making a left turn in heavy traffic or if the sensors on the car get confused in fog, rain, or snow, then Parallel Domain’s software can generate millions of small variations of those exact scenarios to help the company improve performance without risking public safety.
Real-world map data can be used to generate virtual worlds that mirror the public roads.
The RRE team and I are incredibly excited to be supporting Kevin McNamara and his world-class procedural graphics team as they build the digital infrastructure not just important, but vital to the safety of our autonomous transportation future.