by fiddler
(not an April Fool's post, despite the date)
Richard A. Clarke, former counter-terrorism czar for both the Clinton and Bush administrations, had some strong words about the US Chamber of Commerce's aborted plans for discrediting its critics, which included spying on families, using malware to steal information, faking documents to embarrass its liberal opponents, and creating and using 'sock puppet' personas to infiltrate their targets.*
Clarke said of the US Chamber's plans to hack, impersonate, spy upon and steal from its perceived opponents:
"I think it’s a violation of 10USC. I think it’s a felony, and I think they should go to jail. You call them a large trade association, I call them a large political action group that took foreign money in the last election. But be that as it may, if you in the United States, if any American citizen anywhere in the world, because this is an extraterritorial law, so don’t think you can go to Bermuda and do it, if any American citizen anywhere in the world engages in unauthorized penetration, or identity theft, accessing a number through identity theft purposes, that’s a felony and if the Chamber of Commerce wants to try that, that’s fine with me because the FBI will be on their doorstep in a matter of hours."
(Although spokesmen for the US Chamber deny that it was planning anything of the kind, leaked memos say otherwise, including this December memo from Berico Technologies to the law firm Hunton & Williams that provides a status report on the work that Team Themis (Berico, Palantir and HBGary Federal) was doing for the US Chamber, and this January memo that discusses pricing quotes and preparation for their project presentation to the US Chamber. So it's pretty clear that whatever the US Chamber says, it's attempting to slam the barn door after the horses aren't just out but are playing tag in the field.)
However, what Clarke asserts as illegal for any US citizen apparently is not so for the US government. It might be inethical, intrusive and deceptive, but not illegal.
One of HBGary's other projects, as revealed in the memos, was a bid for a contract with the US Government to supply software that would let it manage an army of sock puppets, which could be used to impersonate people online. Each person could manipulate up to ten sock puppets. The RFP also sought a secure virtual private network to supply randomly selected IP addresses; static IP address management, so that each sock puppet could be managed by more than one person and present a believable identity, as well as servers to support the system. According to the RFP, the software was to be used at McDill Air Force Base in Florida, Kabul, Afghanistan and Baghdad. But that doesn't mean the sock puppets would be restricted to those areas -- they could be in computers all over the world.
From ComputerWorld:
It's recently been revealed that the U.S. government contracted HBGary Federal for the development of software which could create multiple fake social media profiles to manipulate and sway public opinion on controversial issues by promoting propaganda. It could also be used as surveillance to find public opinions with points of view the powers-that-be didn't like. It could then potentially have their "fake" people run smear campaigns against those "real" people. As disturbing as this is, it's not really new for U.S. intelligence or private intelligence firms to do the dirty work behind closed doors....
And, from Rawstory.com:
Ultimately, and perhaps in part because of the timing of the email release, HBGary did not receive the contract; instead it went to Anonymizer, a company acquired in 2008 by intelligence contractor Abraxas Corporation, because they had existing persona management software and abilities.In the continuing saga of data security firm HBGary, a new caveat has come to light: not only did they plot to help destroy secrets outlet WikiLeaks and discredit progressive bloggers, they also crafted detailed proposals for software that manages online "personas," allowing a single human to assume the identities of as many fake people as they'd like....
In another document unearthed by "Anonymous," one of HBGary's employees also mentioned gaming geolocation services to make it appear as though selected fake persons were at actual events.
"There are a variety of social media tricks we can use to add a level of realness to all fictitious personas," it said.
The Guardian UK has been following this:
The US government's plan to use technology to create and manage fake identities for social interaction with terrorists is as appalling as it is amusing. It's appalling that in this era of greater transparency and accountability brought on by the internet, the US of all countries would try to systematise sock puppetry. It's appallingly stupid, for there's little doubt that the fakes will be unmasked. The net result of that will be the diminution, not the enhancement, of American credibility.
But the effort is amusing as well, for there is absolutely no need to spend millions of dollars to create fake identities online. Any child or troll can do it for free. Millions do. If the government insists on paying, it can use salesforce.com to monitor and join in chats. There is no shortage of social management tools marketers are using to find and mollify or drown out complainers. There's no shortage of social-media gurus, either....
Hell, if the government wants to spread information around the world without being detected, why doesn't it just use WikiLeaks? Oh, that's right. Secretary of State Hillary Clinton called WikiLeaks disclosures "not just an attack on America [but an] attack on the international community". The leaks, she said, "tear at the fabric" of government.
Yes, indeed, they tore at the fabric of the Tunisian government and helped launch the revolts in the Middle East and a wave of freedom – and, we hope, democracy – across borders. The movement of liberation we are witnessing came not from war and weapons or spying and subterfuge but from a force more powerful: transparency; openness; honesty.
And, in another article in the Guardian:
The project has been likened by web experts to China's attempts to control and restrict free speech on the internet. Critics are likely to complain that it will allow the US military to create a false consensus in online conversations, crowd out unwelcome opinions and smother commentaries or reports that do not correspond with its own objectives.
The discovery that the US military is developing false online personalities – known to users of social media as "sock puppets" – could also encourage other governments, private companies and non-government organisations to do the same.
The Centcom contract stipulates that each fake online persona must have a convincing background, history and supporting details, and that up to 50 US-based controllers should be able to operate false identities from their workstations "without fear of being discovered by sophisticated adversaries".
Centcom spokesman Commander Bill Speaks said: "The technology supports classified blogging activities on foreign-language websites to enable Centcom to counter violent extremist and enemy propaganda outside the US."
He said none of the interventions would be in English, as it would be unlawful to "address US audiences" with such technology, and any English-language use of social media by Centcom was always clearly attributed. The languages in which the interventions are conducted include Arabic, Farsi, Urdu and Pashto.
Centcom said it was not targeting any US-based web sites, in English or any other language, and specifically said it was not targeting Facebook or Twitter....
The sock puppet management software is part of Operation Earnest Voice, which US Central Command hopes will “counter extremist ideology and propaganda, and to ensure that credible voices in the region are heard,” according to a statement by Gen. Petraeus.
I'm not reassured by the statements that the sock puppet software will be used only in foreign languages and in dealing with web sites or social networks originating outside the US. The Internet is international; Americans look at sites originating outside the US every day. Also, the uses to which the sock puppets might be put include more than just surveillance on a social networking site, intrusive as that may be. They could act as a military-controlled virtual flash mob imitating a specific political party within a foreign country in order to destabilize its politics, or as a gang of trolls to shout down legitimate individual voices of protest whose message may call for freedom and democracy in countries where the US is too closely aligned with the repressive government.
Don't get me wrong -- there are legitimate reasons for police (as a branch of government) to use Facebook and similar sites, such as for community policing, to track cases of harassment and bullying, and to catch sex offenders soliciting children online. And police are already dealing with online impersonation cases, such as a man impersonating a firefighter, and, in Canada, a sock puppet impersonating a popular hockey player and making offensive comments.
But online undercover work by ordinary police is not the same as a government pursuing public policy ends by manipulating online personas. There's also no guarantee that these sock puppets will be entirely mythical persons as opposed to impersonations of real people. Thinking ahead, it's possible to envision a virtual debate of slanders, which the real people couldn't control and wouldn't participate in, to destroy both debaters in favor of a third party; or the overthrow of a people's rebellion by sock puppets impersonating their oppressors.
USCentcom may do exactly what it says it will with the software-- but there's no way for us to know that. There are no visible checks and balances here. And, since the ability to put government-based sock puppets into social networks now exists, it's always possible that someone will do it here, sooner or later. We have another election coming up in less than two years, and dirty political tricks are commonplace in American elections, much as I wish they weren't.
From The Tech Herald:
It’s been widely reported that terrorists are aware that their online movements are monitored. They actively post false information within their ranks to ferret out impostors. With this software, it would appear that the U.S. wants to take part in this game too. Yet, it isn’t clear what this propaganda push will accomplish in the long term, other than government approved comment spam.
On the other side of this coin, the persona management software of the scale discussed recently is as dangerous as it is powerful. Aaron Barr is still taking a public beating over his research and ideas, but it is important to remember he is well versed in social media and intelligence gathering.
At the same time, he cannot be the only one with the idea to collect and sell information, to anyone with cash in-hand, simply by using social networks. Moreover, his idea to develop personas to further information collection cannot be seen as a unique idea either.
But sales is only part of it, this technology can be used to promote. It can be used to advance an idea or grassroots movement. The government could use this strategy sure, but there is more value to the private sector when it comes to this type of use. Viral promotion can be used to leverage a brand, or it can be used to pass agendas....This alone should make everyone cautious. The concept isn’t new or original, but it isn’t something that was publicly discussed until recently.
The only way to prevent this technology from being abused is to enact tough laws that protect the privacy of anyone online. Adding to that is a need for social awareness, as in be sure you know what personal information exists about you online and know who it is you are associating with. Both options are far easier to write down than they are to accomplish sadly.
Both Anonymous and US Rep. Hank Johnson (D-GA) are concerned as well:
Interestingly, [Johnson] seems aligned with Anonymous in wanting to get to the bottom of some information that came out of the HBGary emails that showed that U.S. Central Command - the Pentagon group running the wars in Iraq and Afghanistan - put out a bid for software that can create and manage multiple social network personalities. Central command says it's to counter violent extremist ideology and enemy propaganda and will be used in Iraq and Afghanistan.
But both Anonymous and Johnson are worried what uses the software might be put to. Anonymous has launched what it calls Operation Metal Gear to investigate what the software does and why it's being developed. "We believe that Metal Gear [the Anonymous code name for the software] involves an army of fake cyber personalities immersed in social networking websites for the purposes of manipulating the mass population via influence, crawling information from major online communities (such as Facebook), and identifying anonymous personalities via correlating stored information from multiple sources to establish connections between separate online accounts, using this information to arrest dissidents and activists who work anonymously."
Since Anonymous claims responsibility for helping disrupt government Web sites in Egypt and other countries to lend support to uprisings, it falls into the category of those who might be arrested.
Johnson makes a connection between the plans to discredit individuals and organizations and the Metal Gear software for which Central Command put out the bid. "When those contractors using that kind of technology, developed pursuant to government contract and utilizing American tax payer dollars, then turn the tools into domestic surveillance and marketing to business organizations, with the goal of discrediting and disrupting and actually destroying organizations that disagree with their clients, doing that domestically is like turning spying tools on the very people who paid for them," Johnson says to forbes.com.
"You should not use tools developed to get at foreign terrorist agents on American citizens who are choosing to exercise their First Amendment rights."
*I wrote about the role of HBGary Federal in crafting the US Chamber's plans here and here.
I'm guessing that the US government might be able to pull off destructive effects with its sock puppets in English, but operating plausibly in other languages? I think the Guardian's right-- the outcome of the program will be that the US government just spent a lot of money to make itself look stupid and evil.
Posted by: Nancy Lebovitz | April 01, 2011 at 07:08 PM
Our government already looks stupid and evil to many, and spending a lot of money is the American Way.
What's not to like?
Posted by: bobbyp | April 01, 2011 at 09:03 PM
Bobbyp, having more of our hard-earned tax money wasted in order to achieve stupidity and further evil isn't what we need right now.
Posted by: fiddler | April 01, 2011 at 10:32 PM
Great post fiddler. But the paradox that you describe, of this being illegal for the Chamber of Commerce may be rooted in the distinction that is often made between foreign and domestic. I believe there is no legal impediment to the US government spying on foreign nationals (Gary has written extensively on this, and would be able to give chapter and verse), and I imagine that some might argue that the ability to interfere with the formation in the run up to a war constitutes an important ability for psy-ops. That the line bleeds over is also unsurprising, but functionally, it is understandable. This links to the Rolling Stone article about another case and here is a different take by Thomas Ricks and I tend to agree with the point that the information landscape is quite different, so we might have to rethink this.
Again, this is not to say that sock puppeting is an intelligent strategy, and it is probable that there is no solution to the problem, but I think that the problems of cyber nationalism are very real. I think the answer is in cultural exchanges and personal ties, but in a world where governments don't have the luxury of spending on these types of programs, I'm not sure how they would be accomplished.
Posted by: liberal japonicus | April 02, 2011 at 12:17 AM
I meant to link to this article as well, which illustrates some of the tension.
Posted by: liberal japonicus | April 02, 2011 at 12:19 AM
LJ, I don't think of it as a paradox as much as irony -- hence the title, which is what parents say to children.
Posted by: fiddler | April 02, 2011 at 08:46 AM
Fiddler, I'm idly wondering, no big deal, just curious, if you read this comment or the following four extremely relevant comments I left on that post of yours.
Mostly just wondering if you keep track of the comments on your posts, and read them before comments close. Also if you're aware comments nowadays close after two weeks.
(No one may have told you; and, pshew, it took me something like seven years to nagging to finally getting that eternal spam magnet fixed; one itch finally scratched after all those years.)
Posted by: Gary Farber | April 02, 2011 at 10:32 AM
And, yes, I'd like to discuss this in the context of intel efforts at tracking al Qaeda and all the other jihadi groups, but probably not this morning. I have written about it before, though, yes, among many other related issues.
But also haven't woken up yet, and have to go out later, do other stuff first, so: when I can.
Certainly a fine post, Fiddler! Thanks muchly!
Posted by: Gary Farber | April 02, 2011 at 10:36 AM
Gary, I have been away all day at a conference and am only now back at the computer. I have not been reading back to check two weeks worth of comments because I was unaware that they were open for so long. I have been thinking that people were unlikely to continue to comment after conversation had moved on to other posts, as has been the custom at other blogs/online sites. And lately, due to family obligations, I have been hard put some days to get online here at all. So I'm sorry if I'm not replying as quickly as you are commenting.
It would be extremely helpful if notice of comments were to show up in my email, so that I would be aware that they existed. If there is a way to accomplish this, I have not yet found it.
Posted by: fiddler | April 02, 2011 at 05:18 PM
LJ, thanks for the links (which I will read when I get a little time, probably in the next couple of days.)
The question that kept arising while I was writing this post was more ethical and philosophical than anything else: in a country whose government is supposed to be of the people, by the people and for the people, how much and what should the government be able to do that it has forbidden to the people?
Posted by: fiddler | April 02, 2011 at 05:23 PM
"in a country whose government is supposed to be of the people, by the people and for the people, how much and what should the government be able to do that it has forbidden to the people?"
Although Richard Clarke cites, 10 USC, it's not clear what action violates what provision of the law. 10 USC is a huge title of the United States Code, one that deals with the Armed Forces. What the heck is he talking about?
Posted by: sapient | April 02, 2011 at 05:49 PM
Thanks fiddler, and there was nothing pressing in the links, just some interesting points of contact with what you mention. In fact, the WaPo Spy Talk blog has a few more links which may be more closely related to what you have written, especially the first one. here, here and here.
As I said, Gary has tons of stuff that he has posted about the dividing lines and various tests in surveillance. To me a lot of this stuff is a hangover from the idea of a national polity, which then allows a nation to grant rights over and above any considerations it would grant to non-citizens. It has always seemed a bit strange to me, that person X gets certain civil rights because of passport and person Y gets none of those, and the tendency has been to try and erase or ignore those civil rights in key cases, such as Jose Padilla, so what interests me is what sort of things people feel a government can of should be able to do to non-citizens that it can't do to citizens, which seems like a slightly different notion.
At any rate, I think it is good to have people with different levels of time and ability to interact because it underlines the notion that we all have different situations with different views and insights.
Posted by: liberal japonicus | April 02, 2011 at 05:59 PM
I'm sorry, but unless the sock puppets are imitating real people, how is it illegal to create and manage them? Just want some clarification because, although I see the problem, I'm having trouble distinguishing this from what I see all the time on blogs.
Posted by: sapient | April 02, 2011 at 06:09 PM
sapient,
bearing in mind that Clarke is not a lawyer, (and neither am I) I am thinking, given the context of the interview, that he is referring to the failure to disclose foreign support while lobbying the US government. I don't know what part of title 10 this is, but I think that AIPAC has been accused of regularly running afoul of it.
I'm not sure, but I don't think there is any law on impersonating someone except in cases of impersonating a public official, so getting someone to sockpuppet on a forum doesn't carry any legal ramifications.
Posted by: liberal japonicus | April 02, 2011 at 06:13 PM
Reading that again, I'm probably not correct, because he is talking about identity theft. I'm not sure if that is a part of TItle 10 (nor am I sure that foreign support is part of Title 10, but it seems more likely than identity theft)
Posted by: liberal japonicus | April 02, 2011 at 06:18 PM
lj, okay, I am aware of that law, but still don't understand fiddler's post where she says ""in a country whose government is supposed to be of the people, by the people and for the people, how much and what should the government be able to do that it has forbidden to the people?"
Isn't she talking about sock puppetry software here? For example, she says this: " There's also no guarantee that these sock puppets will be entirely mythical persons as opposed to impersonations of real people. Thinking ahead, it's possible to envision a virtual debate of slanders, which the real people couldn't control and wouldn't participate in, to destroy both debaters in favor of a third party; or the overthrow of a people's rebellion by sock puppets impersonating their oppressors."
I disagree that the government would be allowed to impersonate real people online, but that it's against the law for private people to do that.
I'm just troubled that the post raises alarms based on suppositions that really aren't factual.
Posted by: sapient | April 02, 2011 at 06:25 PM
Although Richard Clarke cites, 10 USC, it's not clear what action violates what provision of the law. 10 USC is a huge title of the United States Code, one that deals with the Armed Forces. What the heck is he talking about?
I don't think Clarke was speaking about sock puppets at all. If you follow fiddler's ThinkProgress link, you can say that Clarke's answer is in response to a question about cyberwarfare attacks. His reference to pen testing and identity theft is consistent with cyberwarfare. Cyberwarfare (ugh, I hate that word) attacks are pretty clearly illegal. I don't know the statute number, but unauthorized access of a computer system across state lines is a federal crime.
Posted by: Turbulence | April 02, 2011 at 06:53 PM
LJ, after a bit of Google News checking, I've found several places in which it's illegal for an individual to impersonate another person online. In California, it's illegal to pretend to be someone else on Facebook. A Colorado man is in hot water for impersonating President Obama in order to get himself a copy of Obama's selective service registration card. And an employer who impersonated one of his employees on Twitter may have violated federal law. In the last case, the employee discovered her Twitter account was being used while she was in the hospital recovering from an accident; her employer was using it to promote his business.
Posted by: fiddler | April 02, 2011 at 07:53 PM
But fiddler, is that what the government is doing? Impersonating real people? That's not what I gathered from your post. It seemed to me that they're creating fake people.
Posted by: sapient | April 02, 2011 at 07:56 PM
Sapient, there's no way to know whether they're creating fake people for flash mobs, or creating accounts under the names of real people who may or may not be aware of this. Either is technically possible with the software. All we have been told is that it's not happening in English, and is mostly happening abroad (save for that one military base) though the sock puppets could be in use anywhere.
I see the difference between creation of fake people for the purpose of maneuvering policy agendas and creation of impersonations of real people for the purpose of maneuvering policy agendas to be one of degree only -- it might be technically more difficult if they were manipulating personas of real people, because the situations are more likely to come into public view or be discovered in an untimely manner. It may well be little different from part of what the US Chamber of Commerce wanted to do to its assumed enemies -- create false information under the names of real people in order to cause confusion, distress and distrust. Or the sock puppets may be increasing apparent interest in one or another policy alternative in order to change minds and push the position of an official or a government in a direction that the US desires, whether that's what the locals want or not.
Do I have absolute evidence that this is happening? No. Nobody does, except those who are working with the technology and they're not talking. Is it possible? Yes. Could the software, or something very much like it, find its way into domestic use at some point? Yes, whether it's the government using it or big business or a political party or someone else with sufficient money and resources. When two companies think of how to do something, at roughly the same time in the development of the art, it's likely others will do so as well -- the more so if the new invention is both a technical advance and a moneymaking idea.
Creating false personas online isn't illegal; it's what's done with those personas that matters. If a fraud is conducted by a sock puppet, it's still a fraud; ditto any other serious illegality.
Posted by: fiddler | April 02, 2011 at 08:58 PM
fiddler, it's certainly a possibility that we should be aware of. With all of the bad things that are happening in the world, I'm not inclined to worry about things for which there's no evidence that they're actually happening. Certainly, there are laws on the books already to prevent certain abuses, and I'm in favor of enforcing laws - whether government or corporations are doing the illegal acts. What do you propose doing about it other than that?
Honestly, in the scheme of horrible things to think about, this one is pretty low on my list. There are plenty of things happening for which we have tons of evidence already (from environmental disasters, stomping out the middle class, a mendacious media, gun violence, wars, etc.) Obviously, if there's a way to prevent cyber abuses by governments and wealthy corporations without violating people's first amendment rights, I'm totally in favor. I guess I'm wondering what other horrible things we should start worrying about for which we have no indication that they're actually happening. Not to say that concerned and knowledgeable people shouldn't propose legislation or figure out remedies in advance of some misdeed. Do you have a proposal?
Posted by: sapient | April 02, 2011 at 09:52 PM
It is not a new tactic to discredit people by 'impersonating' them in the media of the day. I think false flag operations are a clearly related phenomenon btw. Currently I have no specific real world example at hand*, only literary ones all running along the line of
1. Person A publishes something that Person B (often the government) either dislikes or wants to stay secret.
2. Person B publishes other material (lots of it!) under the name of Person A that is patently false and/or likely to have the public turn against Person A (like endorsement of certain points of view or behaviour the public considers unacceptable).
3.(optional) Person B, either openly or in disguise as Person C, attacks the material from step 2 emphasising that due to the nature of the step 2 material no credecne should eb given to the step 1 material.
4. (optional) Person B again under the name of Person A 'defends' himself against step 3 in the most clumsy way possible.
5. repeat step 2-4 until Person A has lost all credibility in the eye of the public.
*the treatment of Al Gore in the media during the presidential campaign comes close though. Claiming Gore said something outrageous (Gore lied!) then claiming that Gore's statements that he never said it were a second lie => Gore is a notorious habitual liar who lies about his own lies.
Posted by: Hartmut | April 03, 2011 at 05:51 AM
Hartmut, you make a good point. I mentioned the mendacious media as one of our current nightmares - something that we're confronting now. Without it, your scenario is much less likely. If we can figure out something to do about their malicious influence, the sock puppetry issue will be a minor problem. Commenters have much less influence. (Although one reason I do comment is to make it known that there are people out there with my views.)
Posted by: sapient | April 03, 2011 at 08:20 AM
You're right, it's not a new tactic. We had a recent example right here, that I unpublished. But Matthew Yglesias and various other people (Ace of Spades HQ; it's an equal-opportunity sockpuppet) have not unpublished.
Sure, you can't rid yourself of every malicious or spammy comment. And if you googled a certain string, those comments I unpublished would still show up as being on OW, because Google keeps things around for a while.
That's the game, I think: the people who do this sort of thing are, at least in part, intent on getting certain things so that they show up on or other search engines.
Posted by: Slartibartfast | April 03, 2011 at 09:09 AM
sapient,
I'm generally sympathetic to where you are coming, but 'there is no there there' in this case misses the argument, which is that when the government seeks to influence opinion by creating false personas, certain norms are broken, regardless of the presence of laws. While there are laws, as fiddler points out, the California law against Facebook impersonation is of recent vintage, and most other cases have been that the 'harm' created is when the false person somehow dilutes the 'brand' of the person's name. There is a really tough line here, because parody is allowed if it is clearly labeled, so it depends on the person to bring a claim rather than being something that the state treats as a crime (I'm not sure if the law recognizes that distinction, but while the police might go somewhere to prevent an assault, I can't imagine they try to prevent online impersonations) Here is a frex, and there were any number of facebook impersonation sites that are only starting to get attention.
In a recent case that sent a chill throughout social-networking sites, Fouad Mortada, a Moroccan computer engineer, was sentenced to three years in jail for setting up a Facebook profile in the name of Prince Moulay Rachid, second in line to the Moroccan throne.
Twenty-four junior-high students in Edmonton were expelled or suspended last year for posting profiles in the names of two teachers on the social-networking site Nexopia, but they did not face criminal charges.
The Canadian statute defining personation specifies that the charge only applies if the impersonator intends either to personally profit or to harm his victim in the process.
"The Criminal Code makes it really clear that impersonation alone isn't bad," said Mr. Johnston, who is not involved with the Brandon case. "It has to be impersonation with criminal intent."
Based on the scant details police are releasing about the Brandon case, Mr. Johnston said it was hard to find much evidence of criminal intent. "At the end of the day, this kid may not be guilty of any criminal offence even though it may be an outrage that he'd do such a thing."
So fiddler is correct, it is what is done with the online personas and not the creation as such. Still, it takes an incredibly high bar of truly bad conduct to violate the law, I think. and merely churning out opinions in order to overwhelm a site, or creating an atmosphere where a target may be unable or unwilling to respond.
Generally, to deal with these cases, the world at large has taken an approach similar to that of plagiarism. In specific circumstances (where the person is in an environment where they can be subjected to sanctions), they can be punished with being kicked out of the guild (academia and journalism are the two I am thinking of), there isn't much the law could do. Stephen Glass was not prosecuted, nor was Jayson Blair.
Similarly, with the creation of a persona, there can be some opprobrium that serves as punishment but that is the extent of it. And people have created personas and then filled them in, that bring to mind Kurt Vonnegut's dictum in Mother Night, "We are what we pretend to be, so we must be careful about what we pretend to be." Patrick O'Brian is one example, and Red Thunder Cloud is another.
I'd also point out that, as George Allen (Sr.) used to note, The Future is Now.
Weiss is one of more than 100 scientists, policy-makers and journalists, many linked to stem-cell research, whose identities have been purloined to create a convincing — but bogus — network of apparent friends on Facebook. The victims have no control over the profiles that carry their names, and the perpetrators — and their motive — remain unknown.
Posted by: liberal japonicus | April 03, 2011 at 09:38 AM
The first 'frex' link didn't show up, it is here and is about the closing of the Fake Steve Jobs blog.
Posted by: liberal japonicus | April 03, 2011 at 09:52 AM
I used to check back for months, but stuff would show up years later; you'd always see it on the sidebar, of course, but that was in the days when I simply always read the blog, pretty much; it was a great relief to have ObWi finally shift to so short a period of comments being open as just two weeks as to make it so easy to check back. In the case of one's own posts, it just means an extra minute every couple of weeks, unless one has made a dozen posts a day each day.
But we all have only so much time, habits, and, of course, familiarity with practices at a given blog.
And, yes, a Guide For The Perplexed New ObWi Poster (and commenter) would be a good idea at this point. :-)
Sorry, I didn't realize you wouldn't be looking, or would have dropped you a note. I erred by assuming that of course you'd keep checking your own post every few days until the two weeks were up, since it only takes, at the very least, 30 seconds on the last day. I should have realized this wouldn't be obvious. My bad.
The whale phobia long term thread was definitely the most bizarre. Read through some of the later comments, at least.
Blogs used to leave them open forever, and plenty still do. I thought it was a bad idea, but it's long custom. I thought it was obvious that it became spam magnet trap, but it took about 7 years to get that view seen and the change made here.Posted by: Gary Farber | April 03, 2011 at 11:03 AM
But you'd have to go find it, and then ask someone with the SuperUser password to install it for you. I think such changes to allow for greater options for commenters would be a fine idea, and, again, I've been agitating for such widget-updating and a revision of the template in general -- though not that particular option, but you're perfectly correct that many blogs have it, and offering it as an option is an excellent idea -- since 2005.
You could try writing Eric Martin and saying you see the need. Same as regards any other improvements you might like to suggest. Copy all of us, or not, as you like; I'd certainly be highly interested in following any such discussion as might result.
Alternatively, if you can find said widget, let me know, and can discuss with Slart installing it with Eric's permission.
While you're at it, I should go AGAIN find the widget to allow for clicking automatically linking, bolding, italicizing, etc., to point Slart and Eric to. If you find it before I do that again, please do email.
It would drive me bananas to have hundreds more emails showing up in my email box, mail filters or no, on top of the 1000+ I already get (same reason I don't have Facebook notifications turned on to show up in my mailbox, or ANYTHING I can turn off), but everyone is different, and if you want hundreds of emails per day for a few days, I suspect there's a Typepad widget that would do it.Posted by: Gary Farber | April 03, 2011 at 11:12 AM
It's a valuable thing to point out; sooner or later someone will be first, and it will be interesting to watch that, and see how it plays out.
I'll bet there will be a lot of news coverage.
On the Newton's Third Law #4, the continuing story, with update post, since you posted February 23, 2011, and my subsequent comments were February 28, 2011 at 03:15 AM, and later, on the matter of the Persona Management Software.
Solicitation Number: RTB220610
Agency: Department of the Air Force, on March 03, 2011 at 12:27 AM, and then March 19, 2011 at 12:02 PM, it didn't occur to me to think that you might think I was posting too quickly.
But as I said, no complaint! I meant what I wrote:
I just had assummed you kept up with comments on your posts, but then it seemed as if perhaps you did not, so I asked, and now I know, and that makes perfect sense, so thanks for clarifying! I'm all good! Hope you are, too! It's a misdeameanor, and somewhat narrowly tailored (also only as of January, 2011): I don't think this law has actually been tested yet; I haven't seen any evidence that anyone has yet been charged, though it's certainly possible I'm missing such an account. But I'd be surprised not to have run across it.Posted by: Gary Farber | April 03, 2011 at 11:30 AM
This was a case of going into a federal database and lying, not a matter of posting to the internet. Of course it's illegal to misrepresent yourself to the Social Security Number Verification Service, but the SSNVS is not the internet.
To subsume both things under the heading of "it's illegal for an individual to impersonate another person online" is to smush together very disparate acts under the heading of doing things "online."
Similarly it's illegal to use someone else's credit card online, but that doesn't mean that if, say, someone here switched posted as "debbie" and so did someone else, that they could be arrested, or that if, well, anyone here posted under any name, that this would inherently be illegal. It isn't.
And this isn't a case using a sock puppet, nor of a criminal offense; it's a civil lawsuit, and it's on grounds that, allegedly:
Moreover, it hasn't even gone to trial yet.So there's no law against sock puppeting, just a civil suit based on an allegation of an alleged violation of a theory, which hasn't gone to trial.
That's a lot of thin reeds. It doesn't support a claim that it's illegal for U.S. citizens to sock puppet.
Agree! This implies it's relevant to someone sock-puppeting. It isn't. It's an entirely different beast.Posted by: Gary Farber | April 03, 2011 at 11:43 AM
Also, I should point out, that since this Colorado example was cited, that the only thing that has happened is that:
Er, citing that a blogger has an opinion is not exactly the same as citing an actual law, or trial or charge, or act of any sort by any legal authority.Hey, I can also say that eating Milky Way bars "may violate federal law" and it'll be exactly as true that a blogger said this. I wouldn't then point to that as evidence that eating Milky Way bars is apparently a federal crime.
I'd tend to look for a cite to U.S. code, and some actual trials and convictions, other than in "internet buzz." :-)
Posted by: Gary Farber | April 03, 2011 at 11:47 AM
Gary, re comments: what I would like would be the ability to have comments only from my own posts show up in my inbox, or in a designated RSS feed. I am not a programmer, so I have no idea if this is possible.
Posted by: fiddler | April 03, 2011 at 12:40 PM
I think the concern over federal government sock puppets has a whiff of the moral panic of the week ala a Time magazine cover story to it.
As far as I can tell, this whole notion of the feds impersonating real people was just invented by fiddler: no one else seems to have written about it. Is that right fiddler?
I can't help but think that sockpuppeting software is a huge waste of money. I mean, think about how much effort it takes to convince someone who disagrees with you about an issue online....it is a massive time sink. You have to marshal facts and evidence, you have to develop a reputation as trustworthy, you have to respond to comments and questions. And if you try to do it on the cheap by slinging talking points, ornery folks will eat you alive. And in many comment threads (like most major newspaper comments sections), the signal to noise ratio is appalling while the volume is huge so the odds that a reasonable number of people will see your comment are tiny. This has to be the least cost effective way to influence people that I've ever seen. What's more, the government has been really pushing for operations that minimize head count and this is so not that. I'd worry a lot more about government programs that don't require lots of staff, like mass telecom spying.
In all honesty, the RFP for sockpuppeting software reminds me of the CIA's crazy experiments with dosing random unsuspecting people with LSD for years to see if it would lead to mind control. In both cases, you have a big government entity that is scared about social changes it can't control, has bought into hype about new technology that it doesn't really understand, and thinks that if it just throws money at the new technology, solutions to the problems that terrify it will magically appear. It is nutty magical thinking and there's no reason to take it seriously. Just like we shouldn't seriously fear that the CIA will develop mind control serum based on LSD, we probably should not worry overmuch about a vast army of government sockpuppets.
Even if we assume that the feds are seriously interested in sockpuppeting, impersonating real people is a dumb thing to do because it has high costs and no benefits. What benefit do you get from impersonating someone in a comments section? I mean, if the feds started impersonating Gary Farber here, it would be pretty obvious and the war between real and faux Farber would completely undermine whatever message faux Farber was pushing. If they tried to impersonate someone we did not know, how would that benefit them? I mean, if Sally Alvaraz if Phoenix AZ is thought of as a very trustworthy person by her neighbors, how does that credibility affect me reading her comments online. So there's no benefit but there are costs: every time you spoof a real person, you risk a lawsuit, lawsuits bring with them discovery proceedings which lead to tracing IP logs which leads you right back to machines that are owned/leased by the feds at which point the whole operation is blown and made public. It only takes one irritable judge to be outraged over defamation to make this whole house of cards collapse.
Posted by: Turbulence | April 03, 2011 at 02:52 PM
Turbulence, I doubt that the US military would waste its time in creating personas that would not have some policy, political or military influence, whether those personas are fictional or impersonations.
Also, my extrapolation of the uses of the technology is not beyond reason -- if I am the first person to think of it, what of that? The technology is being used to further policy aims, hence for political purposes. I think it's possible that, given money and the availability of the same or similar technology, it could be used within the US by political organizations or technocrats, for instance to influence elections. Considering the various sorts of dirty tricks that have been pulled during elections for the past few decades, the use of software allowing a crowd of sock puppets to create false interest in one or another political party or candidate isn't unlikely at all. If a technology is available, and it can influence politics, it is likely to be used for that purpose, whether that's a good idea or not.
Posted by: fiddler | April 03, 2011 at 03:27 PM
I doubt that the US military would waste its time in creating personas that would not have some policy, political or military influence, whether those personas are fictional or impersonations.
Are you seriously suggesting that the US military would not waste money? Surely I'm misreading you here. I'm really curious: do you think that all large organizations never waste money on nutty projects that are doomed or is there something special about the US military in particular?
Also, my extrapolation of the uses of the technology is not beyond reason -- if I am the first person to think of it, what of that?
I've often heard that from various cranks and conspiracy theorists. That doesn't mean you are wrong, but it does raise questions....
As for whether it is beyond reason, I think the notion of the federal government using online sockpuppets on a large scale is actually beyond reason for all the reasons I described in my previous comment. You haven't really addressed any of them.
The technology is being used to further policy aims, hence for political purposes. I think it's possible that, given money and the availability of the same or similar technology, it could be used within the US by political organizations or technocrats, for instance to influence elections.
How exactly would that work? More to the point, why on earth would anyone do such a thing when there are much more cost effective ways of influencing elections?
I mean seriously, do you really think most people read internet comments? Do you really think manipulating internet comments is a good way to reach voters?
Considering the various sorts of dirty tricks that have been pulled during elections for the past few decades, the use of software allowing a crowd of sock puppets to create false interest in one or another political party or candidate isn't unlikely at all.
I'm curious: how many sock puppet hours on Obsidian Wings do you think it would take to convince a single voter to change their vote? Can you give me a number? One? Ten? One thousand?
If a technology is available, and it can influence politics, it is likely to be used for that purpose, whether that's a good idea or not.
The technology has been available for years. The RFP describes what is basically a few person-weeks worth of work. There's no evidence that anyone has ever bothered, which makes sense since the cost to benefit ratio is absurd.
Posted by: Turbulence | April 03, 2011 at 03:44 PM
Turbulence, you obviously disagree with what I've written; I have no problem with your disagreement. In fact, I feel a bit flattered that you take so much time and effort to tell me so. Thank you for your thoughtfulness.
Posted by: fiddler | April 03, 2011 at 04:35 PM
I think what you're suggesting here is a conspiracy theory, and a fairly silly one at that. Since you're unwilling or unable to engage serious questions about the merits of your arguments, the odds that they are correct are...extremely low. Now, if you find all that flattering, well...ok.
Posted by: Turbulence | April 03, 2011 at 04:57 PM
Fiddler, you might want to google about China's possible use of sock puppets to both stoke and calm nationalism online. Here is one example. Here is the Chatham House discussion of Chinese Online nationalism, it doesn't talk about sock puppetting, but it does say this
But rather than be a force for democracy and liberalisation, evidence suggests that the internet is actually being utilised by the state to reinforce its position. On one level, the internet is a key tool in the authorities’ attempts to convince the Chinese people that it is constructing a fairer, more predictable, and open form of government that has the people’s interests at its heart. On another, online nationalism largely chimes with the regime’s attempts to construct a new state identity as China as ‘different’ – a China that the West doesn’t understand and a China that faces a hostile international community. To be sure, exuberant expressions of nationalism can at times create some headaches for policy makers, but these debates do not threaten the existing order. With self-censorship restricting the focus of debates, and the state retaining the ability to control who can access what, it is perhaps no surprise that foreign ministry spokesman, Qin Gang, told Western reporters that ‘Many people have a false impression that the Chinese government fears the Internet. In fact, it is just the opposite’ – and presumably he did not mean by this that the internet fears China.
Also, while not sock puppetting, the reports of the 50 cent Army or 50 cent Party are probably driving some of this, because generally, the US military pattern has been to use technological advantages to try and overcome manpower disadvantages. However, the US would have to overcome not only linguistic challenges, but also a far different social media environment. However, I don't see these are impossible. We've recently had spam here that takes an (understandable!) portion of a previous comment and puts it up with a spam url. iirc, it didn't take the break at the paragraph but some way into the paragraph. We've deleted all the ones from the last outbreak, so I may be misremebering, but it seemed that a relatively complete part of a previous comment that could stand alone was taken. Perhaps someone is doing this by hand, but when it hit our comments, it hit roughly 10 or 15 threads simultaneously, with each thread taking a different comment to sock puppet. And manipulating Google search results has just been cracked down on, so creating a situation where the government view of the news gets top billing seems to be a relatively trivial task. Getting 20,000 online personas to hit a like button is not all that hard and a 'me too!' post is trivial.
Posted by: liberal japonicus | April 03, 2011 at 04:58 PM
I've already confessed at various times that "Gary Farber" is actually several South Korean goldfarmers in internet cafes, a couple of young women in Singapore, a dude in Brazil, an old South African woman, two NSA monitors, and an escaped mental patient in a North Dakota public library.
I suspect several of us are multiple personalities.
Perhaps that's already been going on! It could explain a lot!Posted by: Gary Farber | April 03, 2011 at 04:58 PM
Posted by: Gary Farber | April 03, 2011 at 05:09 PM
LJ, thanks for the links. What the 50 Cent Army is doing is fascinating; it would be possible to do that kind of political maneuvering with the military sockpuppets -- if their controllers could work that well in a foreign language and culture. The spam attack you describe sounds technically complex -- multiple threads, multiple comments at once -- though it may not have been that hard for the spammers to program.
Posted by: fiddler | April 03, 2011 at 05:14 PM
And manipulating Google search results has just been cracked down on
This is wrong. Google has had a large team of people working on anti-seo/fraud measures for many years now.
so creating a situation where the government view of the news gets top billing seems to be a relatively trivial task. Getting 20,000 online personas to hit a like button is not all that hard and a 'me too!' post is trivial.
This is also quite incorrect.
Posted by: Turbulence | April 03, 2011 at 05:15 PM
On the flip side, to back fiddler's concerns of potential domestic misuse, I'd bring up these programs, and... see some of the links.
There are so many programs, past and recent. Intelligence Support Activity.
So much infiltration of domestic protest movements.
My problem discussing this stuff is knowing a bit too much. I can't very practically say read all the links here and get back to me, but I've written a lot more, and written only a little about what I know. Etc.
Some further links.
Abuse of intel operations, activities, and infiltration, are always valid concerns, given the history ever since the Palmer Raid era, and dear young J. Edgar.
Posted by: Gary Farber | April 03, 2011 at 05:21 PM
Gary, have you written any followups or have related links to this? I could only find it because I remembered Scientology+Google, but other searches with the keyword google pull up 'you must open a Google/Blogger account', so I'm a bit overwhelmed.
Posted by: liberal japonicus | April 03, 2011 at 05:31 PM
Rita Katz, that's who I was thinking of. I mentioned her here, in passing in one of those Evil Haditha Round-ups that Proved I Wanted American Soldiers Mutilated And Dead, according to DaveC, saying "Which reminds me that I need to do a longer post on Rita Katz soon, but that'll be later."
Apparently a lot later. This is why I've tried to learn not to promise posts I haven't written.
But, hey, it's not as if that post wasn't chunky and nutritious enough, as it was, or as if I've lacked periods of posting a lot.
Meanwhile, here is a useful article on the private jihadi trackers.
I'm still sure I wrote about Rita Katz sometime back in 2002-3, but since tags hadn't been invented yet, or at least implemented on Blogger, and I kinda hadn't really planned ahead for thousands and thousands of posts, I didn't realize it would get so hard to find old stuff of mine eventually.
No matter. Never mind. Rita Katz. And, in any case, somewhat tangential to Fidder's post, in any case, but, then, there are so many ways I could shoot off in different directions from it it, whether back-citing myself, citing elsewhere, or just endless sorts of discussion of genuine U.S. government surveillance of and infiltration of domestic organizations, including, of course, vast amounts in the past ten years, some of which, again, I've blogged about, and much of which has been written about extensively. There's no shortage of reading matter by plenty of folks on it. It's a question of how narrowly we want to focus discussion.
Posted by: Gary Farber | April 03, 2011 at 06:32 PM
Perhaps you missed the news. None of us can follow everything, or even half of it!
That could be so, but it might be more useful when so asserting if you elaborated with some reasons, and as you like to ask for, and as many of us do, some cites.If I might also suggest, you might want to take a break from responding to LJ, unless you feel compelled to. It's just a idea.
Or perhaps you might try, just to be perverse, spending a week doing nothing but finding ways to agree with LJ nicely, just as a mental/emotional stretching exercise.
I'm sure you wouldn't want to give anyone the impression you were going out of your way to find excuses to pick fights with him, so perhaps such an exercise or break might be a way to prevent anyone from imagining such a thing.
It's just a thought from one commenter to another, and you're absolutely free to disregard it, of course.
And yet the quoted sentence and your last sentence here, Turbulence, are not contradictory, and they're both true. Yes, Google has always had anti-SEO measures and work hard at it, because, after all, search is the essence of what they do, and yet there were innumerable stories about theirPosted by: Gary Farber | April 03, 2011 at 06:47 PM
LJ:
Yes, that's the sort of problem I have now with searching my own writing and Amygdala.One can only use Google in a practical way to search for unusual terms, or at least uncommon ones, and if terms have made their way into my template, which includes the sidebar, then while one can use the "minus sign" function to exclude those, then that would also prevent searching every page of the blog, which isn't terribly useful.
I'm sure a real programmer could come up with a way to search the body of the text and not other parts of the template, but I'm Not That Person.
So my memory is my most reliable guide at this point, in some ways, on my own writing, and given not just almost 9000 Amygdala posts, and god knows how many tens of thousands of comments here, if not... I suspect I've written well over 100,000 ObWi comments since 2003 -- though I should stress that I'm a bit dyslexic about numbers, commonly make simple arithmatical errors, and in general when ever I use a number that I'm not quoting, or refer to a time span, given how crap my sense of time is, I'm very unreliable, and should be doubted -- it can be tricky for me to figure out what I have and haven't written, and where. Which is also why I end up repeating myself a lot; often it's easier just to rewrite the same damn thing from scratch then to find an old iteration, and yet other times I simply forget that I quoted that same point or told that story, er, sometimes in the same damn thread. Or the day before A tad embarrassing, that.
And my memory fails in the other direction: I'll sometimes think I've written something, and only meant to, and I'm just remembering what I read, and intended to write, or wrote in my head, or maybe even wrote and then hit post and it accidentally vanished, or somesuch.
(That may be the case with my thinking I wrote about Rita Katz in 2002; I know I read a lot about her back then; maybe I'm just thinking of one of those innumerable pieces I Wrote In My Head, or In The Shower. :-))
But more directly: I don't recall, off-hand writing anything more about Scientology/ists and google manipulation, simply because although I know quite a lot about the way Scientologists/y scam and the organization has functioned, etc., it's precisely because they have such a huge array of opponents Already Handling That Fight that I've not felt any need to start duplicating their efforts on my own. Life is short, and I'm content to let others, for the most part, blog about Scientology, unless, of course, there's a particularly wonderful story, or I'm moved by whim.
Just as I'm content to let others correct Wikipedia, irritating as it is to frequently see errors and huge omissions. But life is short, and ever shorter.
Xenu preserve us.
Posted by: Gary Farber | April 03, 2011 at 07:00 PM
Thanks Gary, I should use those operators more often, but Google has spoiled me by letting me get things with selected keywords rather than trying to exclude results and old habits are hard to break. Thanks again.
Posted by: liberal japonicus | April 03, 2011 at 07:12 PM
Just want to say that the name "Operation Earnest Voice" is brilliant.
Whoever names these military and intelligence programs is one droll SOB.
Posted by: russell | April 03, 2011 at 07:55 PM
And yet the quoted sentence and your last sentence here, Turbulence, are not contradictory,
I think one clear reading of the quoted bit is that the manipulation of google search results used to be ignored but has just now been cracked down upon, where the 'just now cracking down' refers to the recent efforts to block content farms. By this reading, the comment is in fact incorrect since Google has been working continuously to crack down on search result manipulation over the years.
Now, I suppose other readings of LJ's comment are possible and some of them are not incorrect, but such readings did not occur to me, which makes me suspect that they won't occur to others either. As you've pointed out several times in the past, you tend to read very very quickly and IMHO, often sacrifice substantial accuracy, so I don't necessarily find your first readings to be representative of us slower more careful readers. But YMMV.
Perhaps you missed the news. None of us can follow everything, or even half of it!
Actually, I'm very much aware of it.
That could be so, but it might be more useful when so asserting if you elaborated with some reasons, and as you like to ask for, and as many of us do, some cites.
OK, the problems with so creating a situation where the government view of the news gets top billing seems to be a relatively trivial task. Getting 20,000 online personas to hit a like button is not all that hard and a 'me too!' post is trivial start with:
(1) the government view of the news already predominates; the problems with access journalism and complete journalistic subservience to government actors let alone the fact that many media companies (starting with GE/NBC) have owners that depend on significant government contracts are well documented. Providing a solution to a problem that doesn't exist is not terribly interesting.
(2) you can't alter Google's news ranking by having 20,000 online personas hit a like button. You might be able to alter facebook's rankings, but again, facebook has a large anti-fraud team in place precisely to prevent corporations and governments from abusing their service in this manner.
(3) If you or LJ or fiddler want to have a discussion about precisely why manipulating Google rankings is not "a trivial task", I'm happy to participate, but it would really help if those who don't work in this industry read up on the basics. You can start with an introductory text like this and then we can move on to more advanced topics like distributed fraud detection. But I'm really not eager to go into great detail when presented with a 'it is totally trivial' comment by someone who doesn't work in the field.
(4) Even if you could do what many of the worlds biggest companies have been dying to do, namely, manipulate search rankings effectively, this activity has low benefits and high cost. It is extremely hard to do this without revealing yourself to Google; they have better analytic tools and more computational resources to ferret out this sort of fraud than you do. And when they find out, they're not likely to keep quiet. But even if you're successful, what exactly is the benefit? People who search for a news topic will get government friendly stories placed above neutral stories? Does anyone really think that will substantially affect the voting public? Really?
(5) It is not entirely clear, but it seems that a portion of LJ's comment suggested that the military could trade manpower for technology presumably with the use of better natural language processing in their spambots. This idea of automated bots participating in random conversations on the internet understanding comments and generating new ones well enough to persuade people is...amusing.
Posted by: Turbulence | April 03, 2011 at 08:06 PM
Gary, my knowledge of some aspects of intelligence work dates from before the Internet, so is probably not relevant here. I have friends who work in various aspects of national security (but can't talk about it) and computer/data/internet security (but can't talk about it). Thanks for the links; I hope to learn more from them.
Posted by: fiddler | April 03, 2011 at 08:07 PM
The Art of Naming Operations, GREGORY C. SIEMINSKI
From Parameters, Autumn 1995, pp. 81-98.
I also have a good .pdf around here somewhere that's very detailed, but can't find quickly, and is surely more than you want to know, anyway.
As may the above, but it will tell you plenty, and is entirely readable, not technical.
But if you want something even lighter, yet factually accurate, insofar as I know from most, without cross-checking the ones I don't, the The 25 Least Inspiring Military Operation Names from cracked.com.
Winston Churchill wrote a famous (if you're into this sort of thing) memo on the topic:
As usual, I could keep going, with OPERATION REDUNDANT EXHAUSTION DEAD HORSE.Posted by: Gary Farber | April 03, 2011 at 10:48 PM
Thanks also for the rest of your comments, Turbulence.
Thank you for that insight. I'm glad to know that more careful readers than myself, such as yourself, are out there to keep me on my toes. There are few things I approve of more than careful readers.Posted by: Gary Farber | April 03, 2011 at 10:54 PM
Posted by: Gary Farber | April 03, 2011 at 11:10 PM
Operation Flimsy Bunnyhug has possibilities....
Posted by: russell | April 04, 2011 at 10:03 AM