This spring, during the war with Iran, a US intelligence report made its way around the country warning of a Chinese ship in the Middle East carrying components of a nuclear weapons program. The US military reacted swiftly, putting together plans to stop the vessel, according to four people familiar with the incident. Two of those people said armed members of the US military were getting ready to board the ship. One of those two and a third person familiar with the matter said military planes had taken to the air.
Just prior to the scheduled operation, officials examined a report compiled by a special operations command analyst and discovered that AI had assisted in producing it — and that a chatbot the analyst relied on had incorrectly identified the cargo aboard the vessel. CNN was unable to determine what the misidentified cargo actually was.
The report, according to one of the sources, was “entirely false.” But it also “almost started a war,” the source said. Any US operation against a Chinese vessel could have risked spiraling into an armed conflict between the two nations.
How the AI report was made
A person contacted a chatbot to ask about intelligence reports on the ship’s manifest that came from US Special Operations Command Pacific, which is located in Hawaii. It remained unclear whether the chatbot was a commercially available one or a product made by the US government.
“The internal tools are mostly just copies of the commercial stuff wearing lipstick,” a former senior US official familiar with the AI systems used by military and intelligence analysts.
The machine combined public information from various sources with classified government data to form its judgment on what the vessel was transporting.
Once the findings were assembled, the analyst employed AI once more to present them in the form of a standard intelligence report, the type that carries the trust of military officials, before sending it on.
Neither US Special Operations Command Pacific nor the Pentagon answered a request for comment.
The Pentagon’s AI push
In January, Defense Secretary Pete Hegseth released his agency’s “Artificial Intelligence Acceleration Strategy” in a bid to speed up the military’s use of AI. Hegseth said in a speech announcing the strategy: “We will unleash experimentation, eliminate bureaucratic barriers, focus our investments and demonstrate the execution approach needed to ensure we lead in military AI.”
The plan calls for wide adoption throughout the armed forces, instructing the department to distribute AI through multiple programs so that it becomes widely accessible, according to “democratizing AI experimentation and transformation across the Department by putting America’s world-leading AI models directly in the hands of our three million civilian and military personnel, at all classification levels,”.
According to several US officials who know the situation well, the approach to verification across federal agencies is not centralized. Instead, distinct offices employ their own methods and work under separate instructions and safety requirements. That means there is no single standard guiding how the US checks the data produced by these systems. As a result, the various AI applications now spread across the military and intelligence community differ greatly in how reliable and useful they are.
The wider debate over AI
For several weeks now, officials in Washington have engaged in vigorous debate over AI, prompted by serious warnings from engineers and chief executives at Silicon Valley firms about the risk that AI might escape human control, with results that could end civilization entirely.
The danger of human error
The encounter with the Chinese vessel highlights a distinct danger, one that arrives suddenly rather than gradually: people acting badly because they believe machine-generated information that is wrong or misleading. According to reports, the armed forces are quickly adopting AI for guidance on who or what to target, a field where fatal errors loom large.
“AI in targeting is definitely something that is ramping up and there is no real guidance for how having a human in the loop will prevent civilian casualties or fratricide,” another source familiar with the military’s current policies said.
According to one source, the type of “hallucination” produced by the analyst’s tool has not appeared alone within the intelligence community; it has surfaced repeatedly since these tools first spread through government.
Some former intelligence officers, even those who generally back the use of AI within the military, say the technology has pushed analysts to churn out and spread intelligence at a quicker pace, which increases the risk of errors. Several sources noted that younger analysts, raised with these tools, are especially prone to trusting them without question.
“AI allows you to get to a bad idea faster,” one of the sources said.
The order of events
- A US intelligence report circulated warning of a Chinese ship in the Middle East carrying nuclear components.
- The US military put together plans to stop the vessel.
- Armed members of the US military prepared to board the ship.
- Military planes took to the air.
- Officials examined a report compiled by a special operations command analyst.
- They discovered that AI had assisted in producing it.
- A chatbot the analyst relied on had incorrectly identified the cargo aboard the vessel.
- Just prior to the scheduled operation, the operation was called off after the report was found to be entirely false.
What happens next
No public statement from the Pentagon has been made regarding the matter. US Special Operations Command Pacific declined to answer when asked for a response.
The story raises questions the Pentagon has not answered:
- Is the chatbot a commercial product or a US government tool?
- How did it fuse open-source intelligence with secret signals intelligence?
- What checks exist on the output of these systems?
Source material: “US Military had close call after using AI for hallucinated intelligence report,” CNN.
Get the Notebook.
The day's best stories and every fresh verdict, in plain English, in your inbox by seven. One email a day, no more.