Pokémon Go was the globe’s inaugural augmented-reality blockbuster. Launched in 2016 by the Google spinoff Niantic, the AR twist on the colossal Pokémon franchise swiftly became a worldwide sensation. From Chicago to Oslo to Enoshima, gamers took to the streets in eager anticipation of capturing a Jigglypuff or a Squirtle or (if exceedingly fortunate) a rare Galarian Zapdos hovering just out of reach, projected onto the everyday landscape.
In essence, we’re discussing an immense quantity of individuals directing their smartphones at a vast array of buildings. “Five hundred million individuals installed that application within 60 days,” remarks Brian McClendon, CTO at Niantic Spatial, an AI enterprise Niantic unveiled in May last year. Per the video-game company Scopely, which acquired Pokémon Go from Niantic concurrently, the game still attracted over 100 million players in 2024, eight years post-launch.
Currently, Niantic Spatial is leveraging that vast and unmatched collection of crowdsourced data—images of urban landmarks tagged with precise location markers gathered from the devices of countless Pokémon Go players globally—to create a type of world model, a trendy new tech that anchors the intelligence of LLMs in actual settings.
The firm’s newest offering is a model that claims to accurately identify your location on a map within a few centimeters, based on just a few snapshots of buildings or other landmarks in sight. The company aims to utilize it to assist robots in navigating more accurately in areas where GPS is unreliable.
In a significant trial of its technology, Niantic Spatial has recently partnered with Coco Robotics, a startup deploying last-mile delivery robots across various cities in the US and Europe. “Everyone believed that AR was the future, that AR glasses were on the horizon,” states McClendon. “And then robots became the audience.”
From Pikachu to pizza delivery
Coco Robotics operates approximately 1,000 flight-case-sized robots—designed to transport up to eight extra-large pizzas or four grocery bags—across Los Angeles, Chicago, Jersey City, Miami, and Helsinki. According to CEO Zach Rash, the robots have successfully completed more than half a million deliveries to date, traversing a few million miles in all types of weather conditions.
However, to compete with human couriers, Coco’s robots, which maneuver along sidewalks at roughly five miles per hour, need to be as dependable as possible. “The optimal way we can fulfill our role is by arriving precisely when we indicated we would,” asserts Rash. And that entails not getting lost.
The challenge that Coco encounters is its inability to rely on GPS, which can be unreliable in urban environments as radio signals bounce off structures and interfere with one another. “We conduct deliveries in many dense regions with high-rises, underpasses, and freeways, and those are precisely the places where GPS tends to falter,” explains Rash.
“The urban canyon is the absolute worst scenario for GPS,” states McClendon. “When you observe that blue dot on your smartphone, you will frequently notice it drifting 50 meters, which places you on a different block heading in a different direction on the opposite side of the street.” That’s where Niantic Spatial steps in.
For several years, Niantic Spatial has been utilizing data amassed from users of Pokémon Go and Ingress (Niantic’s earlier phone-based AR game, launched in 2013) to develop a visual positioning system, a technology that determines your location based on your visual field. “It turns out that having Pikachu realistically dart around and enabling Coco’s robot to navigate safely and accurately through the environment is fundamentally the same problem,” remarks John Hanke, CEO of Niantic Spatial.
“Visual positioning is not an entirely new technology,” states Konrad Wenzel at ESRI, a company specializing in digital mapping and geospatial analysis software. “However, it’s apparent that the more cameras we have deployed, the more effective it becomes.”
Niantic Spatial has trained its model using 30 billion images taken in urban settings. Specifically, the images are focused around hotspots—key locations identified in Niantic’s games that players were encouraged to visit, such as Pokémon battle arenas. “We possess a million-plus locations globally where we can pinpoint you exactly,” states McClendon. “We know your precise standing to within several centimeters and, most critically, where you are looking.”
The result is that for each of those million locations, Niantic Spatial has thousands of images captured in roughly the same vicinity but from different perspectives, at various times of the day, and under diverse weather conditions. Each image includes comprehensive metadata that determines where in space the phone was at the moment it took the image, including the phone’s orientation, whether it was stationary or moving, its speed and direction, and more.
The firm has leveraged this dataset to train a model capable of predicting its precise location by considering what it observes—even for sites beyond those million hotspots, where reliable sources of image and location data are limited.
In addition to GPS, Coco’s robots, equipped with four cameras, will now utilize this model to ascertain their current location and direction. The cameras are positioned at hip height and face all directions simultaneously, providing a slightly different viewpoint compared to that of a Pokémon Go player, but adapting the data was uncomplicated, claims Rash.
Competing companies also employ visual positioning systems. For instance, Starship Technologies, a robot delivery company established in Estonia in 2014, asserts that its robots utilize their sensors to construct a 3D representation of their surroundings, identifying building outlines and streetlight locations.
However, Rash is confident that Niantic Spatial’s technology will provide Coco a competitive advantage. He believes it will enable his robots to position themselves correctly in designated pickup zones outside restaurants, ensuring they do not obstruct anyone’s path and stop directly outside the customer’s door instead of a few steps away, which may have occurred previously.
A Cambrian explosion in robotics
When Niantic Spatial embarked on developing its visual positioning system, the initial intention was to apply it to augmented reality, states Hanke. “If you are utilizing AR glasses and wish for the world to align with your viewpoint, you require a method to achieve that,” he notes. “However, we are now witnessing a Cambrian explosion in robotics.”
Some of these robots might need to coexist with humans—such as at construction sites and on sidewalks. “If robots are ever going to integrate into such environments without disrupting human activity, they must possess a comparable level of spatial awareness,” states Hanke. “We can assist robots in determining their precise location after being nudged and jostled.”
The partnership with Coco Robotics marks the beginning. What Niantic Spatial is establishing, according to Hanke, are the initial elements of what he refers to as a living map: an incredibly detailed virtual representation of the world that evolves as the actual world changes. As robots from Coco and other companies navigate their surroundings, they will provide fresh sources of mapping data, enhancing the detail of digital replicas of the globe.
However, as Hanke and McClendon perceive it, maps are not only growing in detail; they are increasingly utilized by machines. This transforms the purpose of maps. Traditionally, maps have been employed to aid individuals in locating themselves in the world. As they transition from 2D to 3D to 4D (consider real-time simulations, like digital twins), the fundamental principle remains consistent: Points on the map correlate with points in space or time.
Nonetheless, maps designed for machines may require becoming more akin to guidebooks, laden with information that humans consider standard. Companies like Niantic Spatial and ESRI aim to incorporate descriptions that inform machines what they are observing, with every object identified alongside a list of its attributes. “This era is focused on constructing useful descriptions of the world for machines to grasp,” states Hanke. “The data we possess serves as an excellent foundation for developing an understanding of how the world’s connective elements function.”
There is considerable excitement surrounding world models currently—and Niantic Spatial is aware of it. LLMs may appear as know-it-alls, but lack common sense regarding interpreting and interacting with daily environments. World models aim to rectify that. Some companies, such as Google DeepMind and World Labs, are creating models that instantaneously generate virtual fantasy realms, which can subsequently serve as training grounds for AI agents.
Niantic Spatial asserts it is tackling the issue from an alternate perspective. Extend map-making sufficiently, and you will inevitably capture everything, claims McClendon: “I’m highly focused on attempting to recreate the real world. We’re not there yet, but that’s our objective.”
