Rebooting the Pentagon

In 2006 Raj Shah was a captain in the US Air Force, piloting an F-16 fighter jet during the insurgency phase of the Iraq War. Not long into his tour of duty, he noticed a problem with the display screen in his cockpit: the signal from GPS satellites let him see a map of the terrain below, but there was no moving dot or icon to indicate his location in relation to coordinates on the ground. At times, on missions near the Iran–Iraq border, he couldn’t tell which country he was flying over. This was a dangerous situation: at five hundred miles per hour, one stray minute on the wrong side of the border could place him within range of Iranian air-defense weapons.

Back in his barracks, Shah had an early pocket PC called an iPAQ for playing video games. He loaded it with digital maps and strapped it to his knee while he flew. The software in that $300 gadget let him see where he was—basic information that the gadgetry on his $30 million plane could not provide.

Shah suddenly realized how far Silicon Valley had leaped ahead of the nation’s largest defense contractors in certain vital aspects of high-tech prowess. He also saw that this posed a danger to national security: the US military had long maintained an edge over its adversaries through technological superiority. Commercial software, like the tracker in his iPAQ, was available worldwide; at some point, possibly soon, the US would lose its edge—and could lose the next war.

Ten years later, after some time at business school and a cybersecurity firm, Shah was recruited to run a tiny Pentagon-funded start-up called Defense Innovation Unit Experimental (DIUx), whose mission was to cram some of Silicon Valley’s adventurous, streamlined methods into the military’s sclerotic weapons-building apparatus. In their new book, Unit X, Shah and his coleader in the project, Christopher Kirchhoff, tell the story of how DIUx began, met hostile resistance from nearly all corners of the defense establishment, but in the end triumphed—sort of, a little bit, though by the authors’ estimate not nearly as much as it should have.

Citizens who are not well versed in the Defense Department’s ways might assume that as the world’s threats have multiplied and our budget deficits have soared, officials would welcome something like DIUx: an agency in which the nation’s most creative innovators devise new ways to deter and fight wars more effectively and inexpensively. Alas, that is not the case. The Defense Department is a bureaucracy; like all bureaucracies, its main interest is self-protection, which largely means fending off outsiders who want to change how it operates. And so Defense Department officials, the contractors they funded, and the legislators in states where weapons were built and provided jobs viewed DIUx—whose explicit aim was to transform the weapons-procurement business—as a threat.

Shah and Kirchhoff write that the biggest challenge in overcoming this problem is “the inherent conservatism of the military.” There is something to this: as they note, the British navy at first rejected steamships, cavalry units dismissed the practicality of tanks, and jet-fighter pilots resisted the advent of ballistic and cruise missiles and, more recently, drones. But the hostility toward DIUx, as described by Shah and Kirchhoff, was something different, directed not at military revolutions (which can be expected to stir opposition from protectors of the status quo) but simply at new ways of performing old tasks.

Besides, for roughly a half-century after World War II, the US military thrived on, and at times instigated, innovation. Radar, the atomic bomb, nuclear-powered submarines, and the Internet began in—and were, in some cases, invented by and for—the military. The microchip, though devised by scientists at two corporations (Texas Instruments and Fairchild Semiconductor), could not have entered the commercial marketplace without NASA’s space program and the Air Force’s Minuteman intercontinental ballistic missile, which generated enough demand for chips to spur economies of scale that made them cheap enough to install in consumer products. All through the cold war, many defense companies, while never hotbeds of competitiveness, flourished as manufacturers of civilian goods as well and were at least open to adopting technologies and techniques from that realm.

This changed after the end of the cold war. As defense budgets plunged (there really was a “peace dividend”), many companies got out of the war business, and a small number of defense contractors—the ones that did very little but war business—came to dominate the field. Over the next few decades, those firms merged into a mere handful. This was deliberate Pentagon policy. In 1993, Secretary of Defense Les Aspin and his deputy, William Perry, gathered the leading aerospace executives at what came to be called “the last supper” and told them that the defense budget would continue to tumble and that it would be best for them to consolidate.¹ Eventually, four of the largest companies—Lockheed, Martin Marietta, Northrop, and Grumman (already the products of mergers and acquisitions involving fifty-one companies)—merged into two (Lockheed Martin and Northrop Grumman).

This was meant to increase efficiencies, but as often happens with monopolization (forced or natural), it had the opposite effect. As global tensions increased and defense budgets swelled beyond even those of the cold war years, the bloated, sluggish weapons-procurement machinery was too bloated and sluggish to respond with agility. Major weapons systems now take years, sometimes decades, to rumble from contract bidding to research and development to procurement and deployment, by which time their technical specs are often outmoded. The Defense Department’s annual budget has soared to $849.8 billion, yet many defense analysts (and not just hawkish ones) conclude that the US military is ill-prepared to fight, and might well lose, a large war that erupts with little warning.

Shah and Kirchhoff don’t cover this history but delve deep into its consequences. They begin, after a quick account of Shah’s F-16 improvisation, with the moment in February 2015 when Ashton Carter became Barack Obama’s fourth and final secretary of defense. Carter was a Yale- and Oxford-trained physicist and defense policy analyst. As far back as 2001 he had written, “Tomorrow’s defense innovations will largely be derivatives of technology developed and marketed by commercial companies for commercial motives.” Yet in his first two jobs under Obama—undersecretary of defense for acquisition, technology, and logistics, then deputy secretary of defense—he had noted, to his alarm, that his early prophecy had failed to come true because of institutional resistance. Soon after ascending to the department’s top job, he flew to Silicon Valley and implored executives to get involved with defense contracting. Most of them were reluctant; a few had tried but were put off by the military’s indifference. Besides, after Edward Snowden’s leaks revealing their complicity with the National Security Agency’s post–September 11 mass-surveillance programs, they feared aggravating the opposition to defense work from their employees. Carter promised them a new start—and created DIUx as a test.

Shah soon took over, assisted by engineers from Google and other companies, and he was authorized to run the agency like a venture capital fund. “Our offices,” he and Kirchhoff write, “looked like a startup, not an air force facility,” with programmers dressed “in jeans and hoodies” working at “open tables instead of in cubicles.” Recalling his own experience, Shah asked military personnel what help they needed on the battlefield, then scoured Silicon Valley for those who could come up with solutions. (This was the opposite of the Pentagon’s normal method, which was for bureaucrats to spend a year or so devising a program with a long list of requirements, then send it out for bidding to contractors, who would take months more—without ever consulting the people who would end up having to use the resulting weapon.)

At first DIUx had tight limits on how much money it could invest in any one project. Still, some of them, esoteric at first glance, had potentially vast consequences. Early on Shah discovered something shockingly antiquated about one of the Air Force’s most basic functions—lining up the flight paths of its fighter jets and cargo-transport planes with the tanker aircraft that refuel them in midflight so they aren’t forced to land. This is a very complicated task, and at the time Air Force personnel were coordinating 1,500 refuelings a day over Afghanistan, Iraq, and Syria. Yet to plan these operations, they were moving magnetic pucks around on a whiteboard, just as their forebears had done during World War II. Northrop Grumman had won a contract to overhaul this system; by the time Shah saw the whiteboard, the company had spent $745 million—twice the original estimate—over ten years with nothing to show for it, and the Air Force was now asking Congress for more.

Shah found a small Silicon Valley firm to work on the problem. In four months, at a cost of $1.5 million, its coders turned in a working product: software that could enable planners in Air Force headquarters to match a fighter plane to a refueling tanker in less than a minute (as opposed to several days). Shah and his DIUx colleagues saw themselves as “part of the Rebel Alliance,” in a reference to the Star Wars franchise. When they delivered the tanker-refueling app, “we felt like Luke Skywalker dropping a proton torpedo into an exhaust port of the Death Star.”

They faced intense resistance from the Air Force officer managing the Northrop Grumman program and from staffers on the House subcommittee overseeing the defense budget. Eventually they found a higher-up to break through the blockage. One innovation after another brought on similarly fierce battles. In the unit’s first two years, Shah took fifty-five red-eye flights from California to D.C. because there were “fires to be put out.” On one such flight he took with Kirchhoff, they discovered that their government credit cards had been canceled.

Still, the unit had more victories than defeats, across expanding plains of operation. In Project Maven, Amazon, Microsoft, and Google—hardly big names in defense contracting—jointly developed computer vision algorithms that made it easier to track the movements of ISIS militiamen. Shield AI, developed by an MIT engineer and his brother who had been a SEAL officer in Afghanistan, was a small drone that could fly into a building and transmit video signals back to special-operations troops so they could see whether enemy insurgents were inside before they knocked down the door.

By the end of 2018 DIU was a full-blown unit, without the “x” for “experimental,” and was “starting to breach the walls of the Pentagon fortress.” In part this was because the secretary of defense, retired marine general James Mattis, had spent the previous two years at Stanford, where he learned of DIU and was impressed with its work. Still, Mattis’s enthusiasm could go only so far; he became mired in myriad crises, and President Donald Trump didn’t care about any of this. Shah and Kirchhoff, whom I briefly met around this time,² describe a meeting at the White House that Shah attended along with “a few dozen of the country’s biggest tech executives,” supposedly to brief the president and his cabinet on emerging innovations. “Instead,” they write, “the first twenty-five minutes were wasted introducing people and praising Trump, who soaked up the adulation.” As for the substance, Trump “wasn’t interested at all.” He stood up halfway through the briefing and said, “You’re doing a tremendous job. We’re doing great technology. And next week we’re going to announce our great health care plan.” At which point he walked out, along with the entire cabinet, except for the commerce secretary, who carried on for another twenty minutes.

After a while, Mattis resigned from the Pentagon over far larger disagreements with Trump’s policies.³ The Pentagon bureaucracy remains a dominant fixture, regardless of which party is in charge. The Defense Department, even under Mattis, was working with just three of the top AI firms. DIU’s total portfolio of programs amounted to just 0.01 percent of the DoD budget.

Ironically, at the end of 2024, in the weeks leading up to his second term as president, Trump almost (perhaps unwittingly) put the DIU agenda into motion when he short-listed a major Silicon Valley figure—Trae Stephens, the cofounder and executive chairman of Anduril and a former associate at Palantir, two of the most successful venture capital firms to win defense contracts, mainly in AI, robotics, and autonomous drones—to be deputy secretary of defense, the official who runs the Pentagon’s day-to-day operations. In the end, though, Trump selected Stephen Feinberg, a more conventional billionaire hedge fund director whose firm, Cerberus Capital Management, has invested enough in defense to raise possible conflicts of interest but not in ways that suggest an innovator’s instincts.

It was a missed opportunity for transformation, as in the years since Shah’s futile meeting with Trump many more Silicon Valley innovators have come around to defense. In particular, as Shah and Kirchhoff recount, Russia’s 2022 invasion of Ukraine “set in motion a new gold rush” in defense investing. That year tech companies pumped $33 billion into defense technology, twice as much as in 2019. The war—the largest in Europe since World War II—became, in some ways,

the war DIU had envisioned, fought with drones, satellites, and artificial intelligence, hackers on both sides launching cyberattacks, and Ukrainian citizens using smartphone apps to alert their military to enemy positions.

Capella Space, a small firm that received DIU funds for building miniature satellites to monitor North Korea 24/7 even through thick cloud cover, was now flying a constellation of satellites to observe the Russian invasion as it unfolded. When Vladimir Putin denied that he was intending to invade, it was a Capella image of his buildup that the Pentagon released to CNN. Microsoft’s rapid response team plunged into the war business to assist US Cyber Command with a host of cybersecurity and surveillance programs. Various Bay Area venture capitalists, many of whom had only just started speaking with anyone from the Pentagon, were getting nonstop requests for night vision goggles, Kevlar vests, and more. The rockets fired by HIMARS, a mobile launcher that the US supplied to Ukraine, were guided to their targets by drones that had been ordered from Amazon. In all, thirty new products created by DIU start-ups were, and still are, being used in the Ukraine war.

Even Google, which had dropped out of Project Maven after three thousand employees signed a petition urging that the company dissociate from anything involving the military, reentered the defense business via DIU. The money was too good, and the cause—helping Ukrainians break free from Russian imperialism—was appealing.

Silicon Valley’s roots, back in the 1960s, were in defense. Now the connection was coming full circle, with little protest from within.

The battle for Ukraine isn’t entirely a hypermodern war. After some breathless prose suggesting that it is, Shah and Kirchhoff quote Bill LaPlante, the Pentagon’s acquisition chief during the Biden administration, pooh-poohing the notion. “The tech bros aren’t helping us too much in Ukraine,” he said in an industry interview. The war was instead coming down to old-school clashes of tanks, artillery, and ammunition.

Shah and Kirchhoff allow the point, to some extent, noting that the conflict has been a “hybrid” war—“commercial technologies” deployed “in tandem with traditional, exquisite weapons systems, both to enhance their effectiveness and enable their defeat.” They also make the case that this fusion we’re seeing in Ukraine is a preview of wars to come, notably if China mounts an invasion of Taiwan. The Chinese army has invested heavily in the sorts of weapons that the Pentagon, in part because of DIU’s persistence, has only just started to buy in bulk: minidrones, cyber and countercyber tools, and augmentations of AI.

The bad news is that, as with most breakthroughs in defense technology, other countries’ militaries have adapted, either through imitation or asymmetric tactics. The Ukrainians fended off Russia’s invasion and mounted an effective counteroffensive, aided by commercial technology that helped them track Russian movements and that improved the accuracy of their weapons. But the Russians soon found ways to jam the surveillance sensors, fend off the drones, and launch some of their own new-tech weapons. And so the fight has turned into a slugfest, where advances and retreats are measured a few kilometers at a time. The new devices help both sides ward off defeat, but they provide no magic bullet for victory.