Out of the Cold: Mainstreaming Intelligence Sourcing

How intelligence archives can challenge settled views

When I was a graduate student at Columbia University in the 1970s, there was an epistemological “joke” that we students sometimes referenced when choosing research topics. It went like this:

A man is walking down the street at night and comes across an obviously inebriated man on all fours searching the ground under a street light.

“What is the problem,” says the first man, “can I help you?”  

The inebriated man replies, “I am searching for my keys that I lost over there,” pointing to the inky darkness.

“Well, if you lost them over there,” said the first man, “why are you looking for them over here?”  

“The light is better over here,” came the reply.

Not particularly funny, the joke does demonstrate a serious point: we tend to search in those areas rich in data rather than where data are covered in the inky darkness of secrecy. This is a rational response to imperfect and limited access to information, particularly for those just starting out on their scholarly careers.

But such an approach also risks missing the “keys”—the relevant explanations—that we so ardently seek.

This is one of the dilemmas of intelligence research and its role in historical and theoretical research more broadly. Empirical “darkness” can discourage scholars from pursuing intelligence-based topics, or even from including them in their more general projects. Presumably prevented by the “redaction barrier” from gaining any data of value, there are simply fewer incentives for scholars to bother with intelligence-based research.

Much research is conducted because “the light is better” on such subjects. This approach is understandable, but seriously incomplete.

Fortunately, this is changing. Established repositories such as the Cold War International History Project, the National Security Archive, and the Texas Technical University Virtual Vietnam Archive, among others, continue to collect and make available formerly classified information, with the CWIHP specializing in translated materials.

The State Department, the CIA, and other US government agencies have regular declassification programs, often prodded into action by intrepid scholarly groups. The presidential libraries play an important part. The CIA, and even the highly secretive National Security Agency (NSA), has recently released highly informative internal histories that provide a far richer historical record than just a few years ago. The CIA, in January 2017, also placed all eleven million pages of its CIA Records Search Tool (CREST)  online.

Former communist countries, and even contemporary communist regimes as in China and Vietnam, have also begun to loosen their grips on previously protected information.

It is easy to become discouraged at the glacial pace of declassification, but as one who has been at this subject for decades, there is also great satisfaction mixed with that frustration at what is available now compared to those grad student years.

Seemingly small disclosures can have important empirical consequences, as I have found in my recent work. In 1967, there was a heated and accusatory debate between dissident analysts in the CIA and military intelligence and the leaders of both groups over the Order of Battle (OB) in Vietnam, that is, the forces available to the enemy in the war. (What constituted “forces” was hotly disputed, but does not concern us here.) 

One of the charges made against General William Westmoreland and his staff was that they suppressed intercepted signals intelligence (SIGINT) to mask levels of infiltration, denying access even to President Lyndon Johnson and his security staff. Newly released materials, however, make clear that this could not have been the case.

A 1968 interview with Mr. Arthur McCafferty, a CIA and NSA employee serving with the National Security Council (NSC), informs us that President Johnson had, in fact, installed a direct communication link between the collectors of SIGINT in Vietnam and the NSC, bypassing the NSA, CIA, or any other mediating analytical organization. This had been initiated in 1964 following the Gulf of Tonkin incidents, and expanded in 1966 when Walt Rostow became NSC adviser. As General Westmoreland said at the time, he could not have suppressed the SIGINT because there were too many “customers” that had the same information. But because of the highly classified nature of the source of the information, he could not fully explain why publicly.

This one example could cause us to think in new ways about:

• President Johnson’s view of the intelligence surrounding the Gulf of Tonkin incidents and his use of intelligence more generally;
• the role the NSC played in shaping policies over time;
• the integrity of the intelligence system;
• Westmoreland’s personal role in the OB disputes of 1967;
• the relative weight given to SIGINT compared to other forms of information;
• the degree of intelligence warning given to the White House about the forthcoming Tet Offensive of 1968.

Other uses of signals intelligence in the policy process are also discussed in the memo. Such intelligence disclosures can change the way we think about the policy process as we learn more about its complexities.

Historians and analysts of intelligence have toiled in these fields for years, but with some of the most valuable information hidden in the darkness of classification and redaction. It is time to mainstream intelligence history and analysis and integrate it further into diplomatic history and international relations analysis. Still too few, but many more of those keys are under the lamppost now.