Penn engineers build AI software that helps robots use eye-like cameras
Event cameras react in millionths of a second and see in dim and bright light at once, but their data has been hard to use. Penn's F3 tool turns it into something ordinary robot software can read.
Engineers at the University of Pennsylvania say they have built AI software that lets robots make much better use of event cameras, according to a university story published Oct. 29. The work appears in the journal Science Robotics.
What makes an event camera different
A regular camera takes a full picture at set intervals, such as 30 times a second, whether or not anything has changed. An event camera has no shutter. Each of its pixels works on its own and speaks up only when the light it sees gets brighter or darker, much like the light-sensing cells in an eye. If nothing changes, it stays silent.
That design brings several advantages, according to a widely cited 2019 research survey of the technology. Event cameras note changes within millionths of a second, so fast motion does not turn into a blur. They can see detail in very dark and very bright parts of the same scene, such as a road at night with oncoming headlights, where a regular camera washes out or goes black. And they use little power.
The catch is the data. Instead of pictures, an event camera produces tens of millions of separate signals a second that look like a storm of dots. Until now, engineers have had to write special software for each task, which has kept the cameras mostly in research labs, the Penn team said. Sony AI's ping-pong robot Ace, for example, uses event cameras to track a ball's speed and spin, but its vision system works only for table tennis.
What F3 does
The new tool, Fast Feature Field, or F3, acts like a translator. It turns an event camera's stream of dots into a format that the huge amount of software already written for regular cameras can be adapted to use. Kostas Daniilidis, a Penn professor of computer and information science and a co-author, compared it to the JPEG and MPEG formats that made pictures and videos easy to share across devices. F3, he said, "gives event cameras a common language."
F3 is a neural network, a type of AI that learns from examples. The team trained it to predict which signals would come next. To do that well, it had to learn the layout of a scene and how things in it were moving. About 95% of the possible positions in the data are empty at any moment, so F3 only does math where signals occur, which keeps it fast.
How it could make things better
The team tested F3 on recognizing objects, judging distances and tracking motion, using data from cars, four-legged robots and drones on city streets at night and in crowded indoor spaces. Programs using F3 ran about two to five times faster than earlier methods, and F3 handled high-definition camera data 120 times a second, the university said.
That speed and the camera's ability to see in poor light could help self-driving cars, delivery robots and drones react sooner and avoid more mistakes. "Fast Feature Field could make the robots that increasingly fill our skies and streets much safer by helping them see the world more accurately," said Pratik Chaudhari, an associate professor and the paper's senior author. The team is already using F3 to build drones that fly through cluttered, poorly lit spaces.
Eventually, the researchers hope to build F3 into event cameras themselves. "Our goal is to make using an event camera as straightforward as using a conventional one," Chaudhari said. The first author is doctoral student Richeek Das, and the work was funded in part by the National Science Foundation.
Source: Penn Engineering, "The Missing Link Between Event Cameras and Robot Vision," by Ian Scheffler, Oct. 29, 2026; Gallego and others, "Event-based Vision: A Survey".