The U.S. Military Is Building Its Own Metaverse

The U.S. defense technology company has started the exploration of the metaverse, applying the metaverse to military simulation exercises.

On May 10, two fighter pilots conducted a high-altitude protoverse experiment. Thousands of feet above the California desert, aboard a pair of Berkut 540 jets, they don custom AR headsets connected to a system that overlays a ghostly image of a refueling plane flying alongside them in the sky. glowing image. One pilot then conducted the refueling operation using the virtual tanker while the other looked on. Welcome to the fledgling military virtual world.

It’s not just Silicon Valley that is plagued by virtual universe mania these days. Just as technology companies and other brand names scramble to develop strategies for virtual worlds, many defense startups, contractors and funders are increasingly talking about virtual worlds, even if its definition and purpose are not always clear.

The key technologies needed for virtual worlds—augmented and virtual reality, head-mounted displays, 3D simulations, and artificial intelligence-built virtual environments—are already found in the defense world. The result is nowhere near as refined, cute and spacious as Mark Zuckerberg's vision of a virtual world, but that's part of the point. There's a good chance the underlying technology will take off, even if it has problems in the civilian sphere.

For example, a combination of augmented reality, artificial intelligence, and video game graphics enables fighter pilots to practice dogfighting against virtual opponents (other nations) while pulling several G's. Red 6, the company that developed the technology, says it provides a more realistic test of a pilot's abilities than a traditional flight simulator. "We can fight any threat we want," said Daniel Robinson, founder and CEO of Red 6. "And this threat can be controlled remotely by an individual, or it can be controlled by artificial intelligence."

Red6's AR technology has to work in more extreme conditions, with lower latency and higher reliability than consumer-grade AR or VR headsets. Robinson added that the company is now developing a platform that will allow many different scenarios to be rendered in augmented or virtual reality. "What we're building is a real military virtual world," he said. "It's like a multiplayer video game in the sky."

Ideas related to the Metaverse are already part of some of the latest military systems. For example, the high-tech helmet on the new F-35 fighter jet includes an augmented reality display that displays telemetry data and targeting information on video footage around the aircraft. In 2018, the U.S. Army announced it would pay Microsoft up to $22 billion to develop a version of its HoloLens augmented reality system for warfighters, the Integrated Visual Augmentation System (IVAS).

In recent years, virtual and augmented reality have become routine aspects of military training. In 2014, the USC Office of Naval Research and the Institute for Creative Technologies developed Project BlueShark, a system that allows sailors to steer ships and collaborate in a virtual environment. Another program called Project Avenger is now used to help train U.S. Navy pilots. The U.S. Air Force is using VR to teach pilots how to manage aircraft and perform missions. VR is also being used to help treat chronic pain and post-traumatic stress in veterans. Boeing has created an AR environment that lets mechanics practice working on the plane before boarding the real plane.

More recently, the U.S. military has begun exploring more complex virtual worlds. There is also growing interest in connecting and combining virtual worlds in ways similar to metaverse thinking. In December 2021, the U.S. Air Force held a high-level meeting of more than 250 people in a virtual environment at various locations from the United States to Japan. "The promise is to integrate these technologies," said Caitlin Dohrman, general manager of defense at Improbable, a company that develops virtual world technology that created a wargame with more than 10,000 individually controlled objects for the UK military wargame. A huge virtual battlefield of characters, and also in cooperation with the US Department of Defense (DOD). "It's an incredibly complex simulation, especially given the level of fidelity the military requires," Dohrman said. "You could have live players in the simulation, or [characters] could have artificial intelligence enabled, which is usually what the military does."

Palmer Luckey, founder of VR company Oculus, which Facebook acquired in 2014, said Zuckerberg's decision to go all-in on VR and virtual worlds created huge expectations in the business world. “Everyone is on their quarterly company call, like a week or two later, and investors ask them, 'What's your metaverse game?',” he said.

In 2017, Luckey co-founded defense company Anduril. Despite all the recent virtual universe hype, there's still a lot of potential for defense, in part because military training is so important and expensive, he said. But he said the technology doesn't have to be surreal to be useful, and he wants Anduril to focus on using it only when necessary. "Everything we do with VR is better than any other option," he said. That includes using VR to train people to operate Anduril's drones, or using data from ground-based sensors to display information about an area, he said.

Like Zuckerberg's planned metaverse, newer military systems rely heavily on artificial intelligence to function. In October 2020, AR technology developed by Red6 was used to pit real fighter pilots against aircraft controlled by AI algorithms developed as part of the Defense Advanced Research Projects Agency's (DARPA) AI dogfighting project. The AI Top Gun, created by another startup called EpiSci, learned how to out-strategize its opponents through a process of trial and error. The AI pilot eventually developed superhuman skills and was able to defeat its human counterpart every time.

Another DARPA project, called Perceptual-enabled Task Guidance, aims to create an AI assistant that can observe a soldier's behavior and provide advice via voice, sound or graphics. Such a system would need to understand the real world, in contrast to the augmented reality system developed by Boeing that only works in specific contexts. DARPA program director Bruce Draper said the real value of the technology the military is exploring lies in merging the real and the virtual. "The metaverse is mostly virtual, and virtual worlds are great for training, but we live in the physical world," he said. "The military domain is inherently physical, not an abstract metaverse."

But efforts to merge the virtual and real worlds have run into problems. In March 2022, a leaked Microsoft memo reportedly revealed that staff working on IVAS, the U.S. Army's version of the HoloLens AR headset, expected it to receive a strong response from users. An audit released by the Department of Defense in April 2022 concluded that the Army may be wasting money as a result. Jason Kuruvilla, senior communications manager at Microsoft, shared several statements from top military figures declaring the potential of IVAS. He also referred to a 2021 DoD report that discusses the importance of rapidly developing IVAS to address issues along the way.