by Doctor Science
Eric Lichtblau and Scott Shane of the NY Times reported yesterday:
A wide-ranging surveillance operation by the Food and Drug Administration against a group of its own scientists used an enemies list of sorts as it secretly captured thousands of e-mails that the disgruntled scientists sent privately to members of Congress, lawyers, labor officials, journalists and even President Obama, previously undisclosed records show.The existence of such an operation doesn't shock cynical me, but I am truly astonished by how it was uncovered:
What began as a narrow investigation into the possible leaking of confidential agency information by five scientists quickly grew in mid-2010 into a much broader campaign to counter outside critics of the agency’s medical review process, according to the cache of more than 80,000 pages of computer documents generated by the surveillance effort.
The documents captured in the surveillance effort — including confidential letters to at least a half-dozen Congressional offices and oversight committees, drafts of legal filings and grievances, and personal e-mails — were posted on a public Web site, apparently by mistake, by a private document-handling contractor that works for the F.D.A.Update the Evil Overlord List: "My secret programs must not be discoverable by Google. If I *must* have a webpage for a covert operation, it will have the following code in the header:
While the surveillance was intended to protect trade secrets for companies like G.E., it may have done just the opposite. The data posted publicly by the F.D.A. contractor — and taken down late Friday after inquiries by The Times — includes hundreds of confidential documents on the design of imaging devices and other detailed, proprietary information.
The posting of the documents was discovered inadvertently by one of the researchers whose e-mails were monitored. The researcher did Google searches for scientists involved in the case to check for negative publicity that might hinder chances of finding work. Within a few minutes, the researcher stumbled upon the database.[bold mine]
<meta name="robots" content="noindex, nofollow, noarchive" />My husband and I have been trying to figure out how you could post that much surveillance material "by mistake", and how Google could have found it. Our theories so far:
<meta name="googlebot" content="noindex, nofollow, noarchive, nosnippet" />
- The contractor and the FDA agreed that the contractor would collect the material and put it online so people at the FDA could look at it via the general Internet and decide what to do with it. Clearly, whatever else happened, permissions weren't set properly for the site. I bet the FDA users had to log in to see the home page, but apparently no-one checked whether the other 80,000(!) pages were password-protected, too. So we start with a big Security FAIL.
To that Security Fail, we add the fact that search-engine robots weren't blocked. AHAHAH, I think I see why -- I bet they were using Google as the site's search engine. They didn't have the *content* of the documents in their database, only indexing information (subject(s), author, etc.), so they couldn't turn off the googlebot, they needed it for the site to be useful.
Google itself says If you need to keep confidential content on your server, save it in a password-protected directory. You can still use Google's site search app, but it'll be slow and a trifle less convenient.
- But given that the material was supposed do be "secret" and unlinked, how did the public googlebot get to the site to index it? We've come up with 4 possibilities so far:
- Someone trying to get to some other page on the contractor's site made a typo, got to the secure material by mistake, and started poking around. They later posted a link somewhere to something they found.
- Authorized users (FDA or the contractor) sent links back and forth in Gmail, which Google uses to "seed" the googlebot webcrawler.
- One or more authorized users went to the site via Google Chrome, and that's how the URL got into the Google system.
- An authorized user was *really, really* dumb, and posted a link somewhere public to a document buried in those 80,000 pages. Probably to make a point in a blog comments section.
The moral of this story: password-protected directories are slow and a pain, but they are your *friend*! Google, on the other hand, is *not* your friend, no matter how hard it pretends to be.
As for the actual news substance of the story, what Charlie Pierce said:
I have a suggestion for the Constitutional Law Professor In Chief.
Knock off this scarifying pissantery. Today.
Some slopes are not necessarily slippery, but some of them are luge runs, and this is one of them. If you allow one part of the executive branch — the intelligence community, let's say — to act beyond the Constitution, and you do so with such regularity that it seems to become the political status quo, well, then you license every department of the executive branch to behave the same way. And thus does the FDA take upon itself some of the essential functions and justifications of the CIA, as ludicrous as that sounds in theory.