Thursday, November 29, 2007

S.F. Chronicle admits to deceptive comment-deletion policy, offers bizarre excuse, then lies again

[UPDATE 1: S.F. Chronicle Webmaster speaks!]
(Sat. 12-01-07, 7:30pm)

[All updates can be found below at the end of this entry]

In response to the scandal caused by our earlier exposé at Investigate the Media, the Webmaster for the San Francisco Chronicle has given an interview to local news site SFist, admitting that the Chronicle's Web site, SFGate , did indeed have a policy of deleting certain users' comments in such a way that the commenters themselves did not know they had been deleted. The Webmaster, Eve Batey (whose official title is "Deputy Managing Editor for Online"), had this to say:
The software we use for article comments isn't an SFGate creation -- it's provided by an outside company with which SFGate has contracted. When we gave them our requirements for article comments, we made it very clear that we really, really needed a way to indicate that a comment had been deleted -- something as simple as having the text of the comment replaced by "This comment has been deleted due to violations of SFGate's Terms and Conditions" would have done the trick nicely.

However, this wasn't something the company was able to provide to us immediately. We at the Chronicle and the folks at SFGate weighed this problem, and decided not to let this keep us from moving forward on article comments. This nagged at me and at my colleagues, that deleted comments would just "be disappeared," but we felt such a sense of urgency to add article comments to the site that we pushed this worry away, and hoped that this transparent deletion function would be added soon.

Unfortunately, the commenting company hasn't been able to provide us with this tool yet, and suggested that we use their "block user" function as a stopgap measure. (This "block user" function is what you see your colleagues in the blogosphere calling us out about.) The "block user" function blocks all comments made by a user from view by anyone but themselves (upon login).

So, what we've been doing is deleting TOC-violating comments from folks who only occasionally violate our policy, but in cases where mass disappearances of comments would make the article comment conversation completely incomprehensible, we opted to use the "block user" function. This function has been used very, very sparingly (and only a few of us have access to this function), and only for those few folks who have repeatedly violated the TOC.

Clearly, however, even though this only has impact on a few users, it was the wrong thing to do -- and that, in our eagerness to have discussion and conversation on the site, we failed to take into consideration those users who would feel hurt and deceived by having their comments blocked from view.

I'm glad that this issue has been raised because I think that this will help make our commenting software providers understand the importance of having a function that makes it clear that comments have been deleted. We've stopped using the "block user" function as of today, even at the risk of having comments "disappear" and at having some article comment section conversations suffer as a result.
Just as we suspected, the commenting software for SFGate was supplied by a third-party vendor (perhaps Prospero or Topix, both of which are known to offer this feature). Notice the bizarre excuse she provides -- that no one could figure out a way to delete a comment and replace it with a notice that it had been deleted, except for using the unintentionally nefarious "block user" function. How could that possibly be? Indicating deleted comments with a standard notice (such as "This comment has been deleted") is commonplace on nearly every blog and commenting platform. So excuse me if I don't entirely believe this bizarre excuse.

Also note that Batey reveals a detail that we suspected was true, but which we had no proof of until now: that all crytpo-deleted comments were hidden automatically by the software, and only applied to "graylisted" users, those who were victims of what Batey called the "block user feature." Any comments that were individually deleted by human moderators simply disappeared entirely, with no evidence that they had ever been made. Which means that if any user sees a notice in an SFGate comments thread that says "This comment has been removed by SFGate," it means that someone out there is still graylisted and getting their comments deleted without their knowledge.

Though this may at first appear to be a double victory -- getting the Chronicle to admit to its deception, and then getting them to change the policy -- the celebrations may be premature. Because even two days after Batey announced that they were no longer graylisting anyone, notes that say "This comment has been removed by SFGate" are still cropping up in Chronicle threads.

To see whether or not Batey was telling the truth, I made an intentionally innocuous comment on this thread about an earthquake in the Caribbean. Sure enough: when I viewed the thread as "jimjams" the comment remained visible, yet when I viewed it simultaneously (on a different browser) as an anonymous unlogged-in user, the comment was gone, replaced by "This comment has been removed by SFGate." Here is a screenshot proving this: the top browser window is Safari, with me logged in as "jimjams," which shows the comment visible; and the bottom browser is Firefox, with me not logged in at all, which shows the comment as deleted -- at the same time (click image to see it full-size):

Notice that the time of the article (Nov 29 at 1:42pm, circled in blue) is a full two days after Batey claimed the Chronicle had stopped graylisting commenters. And that when I'm logged in as jimjams (circled in green) the comment is visible (circled in green); but when not logged in (circled in pink) the comment is shown as being deleted (circled in pink). Which proves that Batey was lying when she said that they had stopped using the "block user" feature as of November 27.

Now, since I only know how my own account functions, I can't say for sure if any other formerly graylisted commenters are still being blocked. Perhaps "jimjams" uniquely is being punished for raising this issue. So I invite any readers who think they were graylisted in the past to check again now, to see if you're still being crypto-deleted; post your findings in the comments section of this thread.

[UPDATE 1, Sat. 12-01-07, 7:30pm]: S.F. Chronicle Webmaster speaks!

Eve Batey, the Webmaster for the San Francisco Chronicle's site SFGate, has dropped in and made some illuminating comments on this thread. She pleads -- rather convincingly -- that the problem lies with the software company which supplies SFGate's commenting platform, and not with the SFGate editorial team, which had no intention of doing anything underhanded. She also states she unintentionally misspoke in the SFist interview when she claimed she was unblocking all formerly blocked users, unaware at the time that there was no way (or so she says) of getting or creating a list of blocked users, so there was no way to identify them so they could be unblocked. As an attempt to "do the right thing," she has offered to unblock any currently blocked user who contacts her via the email address she posts in her comment.

Read all she has to say in comments #1 and #14 of this thread and judge for yourself. She vows to change the SFGate policy regarding crypto-deletions, eventually eliminating them altogether, and I'm willing to give her the benefit of the doubt until evidence proves otherwise.

Friday, November 23, 2007

The San Francisco Chronicle deceives its readers through comment-deletion trickery

[UPDATE 1: Software Exec Brags About Crypto-Deletion Feature.]
(Sat. 11-24-07, 12:30pm)

[UPDATE 2: Reader Documents "Graylist" of Banned SFGate Users Who Don't Know They're Banned.]
(Sat. 11-24-07, 2:20pm)

[UPDATE 3: The Scandal Spreads: Other Sites Caught Red-Handed Doing the Same Trick.]
(Sat. 11-24-07, 3:25pm)

[UPDATE 4: S.F. Chronicle admits to deceptive comment-deletion policy, offers bizarre excuse, then lies again.]
(Thurs. 11-29-07, 2:58pm)

[All updates can be found below at the end of this entry]

The San Francisco Chronicle has recently activated a devious system by which it deceives commenters on its website, Here's how it works:

If you make a comment on an article posted at SFGate, and if the site moderators then subsequently delete your comment for whatever reason, it will only appear as deleted to the other readers. HOWEVER, your comment will NOT appear to be deleted if viewed from your own computer! The Chronicle's goal is to trick deleted commenters into not knowing their comments were in fact deleted. I'll give evidence below showing how they do this.

Why would SFGate do such a thing? Because ever since public input was first allowed at SFGate, many commenters who had their comments deleted would come back onto the comment thread and point out that they had been silenced for ideological reasons -- i.e. they weren't sufficiently "progressive" -- or because they had pointed out ethical lapses at SFGate and the Chronicle. Or any number of other reasons that the Chronicle did not want known. So, to pacify these problematic commenters, the SFGate moderators came up with a very clever and underhanded coding trick to prevent deleted commenters from ever finding out that they had been silenced.

Now, I'm certain that there are plenty of comments on SFGate that indeed merit deletion, and plenty of commenters who say patently offensive things. No question about that, and no one is questioning the Chronicle's right to delete such comments. But there are many other comments that get removed for no apparent reason, except for their political stance, or because they strike too close to home -- pointing out flaws in the article's reporting or writing itself, or ethical or moral misdeeds on the part of the Chronicle editors or management. Deleting comments such as those would be bad enough, but the Chronicle really crossed the line with their new technique of essentially lying to any commenter who has been deleted by not allowing them to even know they were deleted -- so they don't subsequently complain.

The flaw in this new system -- and how I discovered the trickery -- is the "Recommended" rankings under each comment. Readers are permitted to "Recommend" comments they like, and the most popular comments can accumulate dozens of "Recommends." Also, comments near the beginning of any thread often get the most "Recommends." Several times over the last few weeks I noticed something odd: on the first page of comments, people would voice an opinion on an article, then I would make a comment that essentially voiced the same opinion as commenters before and after me; but if I checked back several hours later, or the next day, I would notice that these other commenters would have accrued 20 or 30 or more "Recommends," whereas I would either have 0 Recommends or (if I had recommended my own comment, which many people do) just 1 Recommend.

Why was this happening repeatedly? One explanation is that my comments were terrible, and thus did not earn any "Recommends," but in most cases they were not much different from other comments, and were at least as well-written. But today I discovered by accident the real reason. The night before I had made a comment from my personal computer about this SFGate article. When I checked back this morning, I noticed once again that my comment was the only comment which had only 1 "Recommend." I didn't give it much thought, but later in the day I revisited the same thread from a friend's computer. To my surprise, I discovered that my comment had been deleted, when viewed on this other computer. Then, later, I returned to my own computer, revisited the same thread -- and my comment had mysteriously reappeared, at least from my point of view.

Suspicious, I then took the following steps: I deleted the "cookies" that SFGate installs on users' computers to identify who they are, then logged out of my account, and then revisited the same thread on my own computer. As I suspected, the comment was no longer visible, replaced by the moderator's notation "This comment has been removed by SFGate." Then when I logged back in to my account, and viewed the thread as "me" again -- the comment was once again visible.

I also confirmed this by viewing the comment thread using a different browser (Firefox) which did not have any SFGate cookies installed yet, and on which I had not logged in to my account. Sure enough, the comment appeared as deleted; while exactly simultaneously, using my original logged-in browser (Safari) the comment was not deleted.

In other words, whenever I viewed the comments thread as "myself" (i.e. logged in under my account name, which in this case was "jimjams"), my comment remained visible; but whenever I viewed the comment thread either anonymously (i.e. not logged in) or from a browser with no SFGate cookies or (most importantly) from some other computer, then my comment was gone -- deleted by the moderators.

So the end result is that the only person who can see a deleted comment is the person who originally made that comment. To everyone else in the world -- the comment is gone, deleted, non-existent. And the only conceivable purpose for this is to trick commenters into not knowing their comments had been deleted.

Thus, I issue this call to anyone who has ever suspected that their comment was deleted at SFGate, or who ever was stuck at 0 "Recommends" near the beginning of a comment thread even though you made an excellent or incisive comment: take the steps I describe above, and you almost certainly will have the same experience that I did: your comment will be deleted everywhere except your home logged-in computer.

To provide some evidence of my claims, I have taken the following screenshot which shows exactly what I'm describing; the direct link to this particular comments thread is here. Click on this image to view it full-size:

This screenshot shows two different browser windows open simultaneously: The left side shows how the comments thread appears to me when I'm logged out, having cleared my cookies; and the right side shows how the comments thread appears to me when I'm logged in with cookies turned back on. Notice that my comment (the one by "jimjams") is missing on the left side yet present on the right side; and that all the other comments have a greater number of "Recommends" on the right side, meaning that that window was opened necessarily at a later time -- and yet that's the side with the visible comment, meaning that it must have become visible AFTER having already been deleted.

Please note that this is not a debate over whether or not this one particular comment of mine merited deletion -- I disagreed with the article's author, and called him an "idiot" for completely misrepresenting the issue, which I suppose the moderators felt was too extreme. No, the issue is that they tried to hide the fact of the deletion from me through chicanery -- and I suspect that they pull the same trick on other deleted commenters too, in order to pacify them.

Do you suspect that has ever happened to you on a comments thread at SFGate? If so, try the various steps described above (deleting cookies, logging out, and re-viewing the thread; or viewing it with a different browser, or on a different computer). Readers are invited to post their experiences here as comments on this blog. And I encourage you to post this URL ( in SFGate comments wherever you think it's appropriate.


Reader "Afkovach" sends in these two screenshots: the first one shows how his comments are visible when he is logged in to SFGate; and the second one shows that they appear as deleted when he views SFGate after logging out. This confirms my experience.


[UPDATE 1, Sat., 11-24-07, 12:30pm]: Software Exec Brags About Crypto-Deletion Feature

The PBS blog "MediaShift" recently had an interview with Rich Skrenta, the former CEO of a commenting-forum-software company called Topix, in which he bragged about this capability in his company's product:
There’s a lot of tricks in it. For instance, if you are banned from the forums, you can actually still post, and see your own posts, but other people don’t see them. That’s a neat social trick, because if you know you’ve been banned, most people will work around that. They’ll clear their cookies and work to figure out how to get around the block; but if they don’t know they’ve been banned, and they seem to be able to post, it won’t do any harm to the environment.
Another software company called Prospero supposedly also makes commenting software with this crypto-deletion feature. However, I don't know for sure if SFGate uses either the Topix or Prospero software. (Hat tip: MonkeySon)

[UPDATE 2, Sat., 11-24-07, 2:20pm]: Reader Documents "Graylist" of Banned SFGate Users Who Don't Know They're Banned

ITM commenter "Bricology" has just documented that, at least in some cases, the comment-deletions on SFGate are automated; that all comments from certain users who have been secretly banned from the site are immediately deleted automatically; but that such deletions are not visible to the banned commenter himself. Thus, he never knows that he has been banned.

Bricology took these three consecutive screenshots: click on each one to see full-size versions.

The first picture shows an innocuous comment on a thread that is visible when the commenter is logged in to his SFGate account. The second screenshot shows that the comment has already been deleted just a minute later -- when viewed as a non-SFGate member who is not logged in. (Notice the timestamps at the bottom right of the images.) And then the third picture shows the exact same comment reappeared on SFGate once more shortly afterwards -- after the commenter re-logged in to his account.

Since the comment was non-abusive and entirely innocuous (on purpose, as a test), and since it was deleted immediately, and yet still visible to the owner of the account, this proves that certain users are on a "graylist" of banned users who are not informed they are banned. Kudos to Bricology for discovering this.

[UPDATE 3, Sat., 11-24-07, 3:25pm]: The Scandal Spreads: Other Sites Caught Red-Handed Doing the Same Trick

The DJ Konservo site has just posted screenshots taken from the mainstream blog ThinkProgress which prove that the exact same comment deletion deception is being implemented there as well.

We're also getting reports from readers that newspapers in Canada, Washington state and Houston may also be using the same underhanded software gimmick -- preventing deleted or banned commenters from ever discovering that they have been deleted or banned.

[UPDATE 4, Thurs. 11-29-07, 2:58pm]: S.F. Chronicle admits to deceptive comment-deletion policy, offers bizarre excuse, then lies again.]

In a new posting on Investigate the Media, we discuss an interview given by the Webmaster for SFGate, in which she admits that the Chronicle did indeed crypto-delete "graylisted" commenters, as we claimed, then goes on to offer a bizarre excuse for this behavior that makes little sense, and finally falsely states that they have ended the practice, which in fact they haven't. Go to the new thread to read all the details.