« 2005 Dubious Feat Awards | Main | Gold Bullion Exchange Traded Funds »
December 26, 2004
Spam, Blogging and Lazy Lawmakers
For several months, I have been writing that bloggers like me have been under attack, and I have demanded, to zero response, that lawmakers protect us. I have even questioned the integrity and the work ethic of these lawmakers, but nobody seems to care. Isn't it ironic that the day I decided to shut down Trader Wizard, because I was prevented from accessing my own files, that Movable Type, the blog publishing platform I use, made the following statement:
Comment spam load issue 12.16.2004 Hi everyone, my name is Jay Allen and I am the Product Manager for Movable Type. I'm writing today to address—what else?—comment spam. This is an issue that, as many of you know, I have spent several thousand waking hours working on since its first appearance back in the fall of 2003, both as the author of MT-Blacklist and as the maintainer of the Comment Spam Clearinghouse. This is an issue which Six Apart takes very seriously, as evidenced not only in the improvements in Movable Type v3.x but also, in some part, by my hiring to this position.
Over the last month, we have been devoting a great deal of resources to solving the comment spam problem once and for all and making it a non-issue, not just for us in the Movable Type/TypePad world, but also for all weblogs regardless of publishing tool. Our preference is towards solutions that scale to the entire weblog medium, not those which merely move the burden from one site to another, from one tool to another, or from spammers to users.
Identifying the Problem
Recently, however, there have been a number of reports about the escalating effect of comment spam on Movable Type installations, especially evident in shared hosting environments. At first, we assumed that these problems were caused mainly on legacy systems (i.e. MT 2.x) running without the benefit of the modern anti-spam measures (e.g. TypeKey, comment moderation, MT-Blacklist v2.x, etc.) built to protect Movable Type installations. After further analysis and load testing, we've actually found that this is not the case.
In fact, we have found that there is a fairly major bug (in terms of effect, but not code size) which causes page rebuilding even in the case of a comment submission which would be moderated and hence should have no effect on the live page. This means that even if you are using comment moderation in Movable Type and even force moderation in MT-Blacklist, your server load is impacted just as if a comment had been posted to the live site. This bug has been fixed in development.
In addition, we have found another less severe instance of unnecessary database connections which would normally be associated with dynamic pages, even if dynamic templates are not in use. This would adversely affect any customer not using static pages by adding the overhead of dynamic files on top of the normal load caused by rebuilding of static files. This has also been fixed in development.
These two bugs are, in high probability, the causes of the extreme server loads that our customers have been experiencing under the load of a severe spam attack.
We are currently testing these fixes both in-house and with a number of web hosts who were among the first affected by the problem. We will have these fixes released to you as soon as the testing is complete. There is no higher priority to us than making sure that our customers and their websites are protected from the effects of these malicious attacks. We expect to give you a firm date for availability of this patch within 48 hours.
What To Do Now
In the meantime, one way you can help protect your system and mitigate the effects of both problems is by enabling dynamic templates. Under normal conditions, there are many factors to consider in choosing dynamic templates vs. static templates. In general, the higher your site's traffic is, the more beneficial static templates are to you. However, since spam attacks are rapid requests that would cause rebuilding in the case of static pages, the sweet spot is moved far towards dynamic templates, even for high traffic sites.
If you would like to change your templates to dynamic, you should check out the Dynamic Publishing section in the Movable Type documentation and also Elise Bauer and Arvind Satyanarayan's tutorial on the subject.
When setting up your dynamic pages, choose the "Build Only Archive Templates Dynamically" option. This choice means your archive templates will not be rebuilt upon comment or TrackBack submission. If you still experience high loads, you can choose the third "custom" option and set all of your templates to dynamic.
This is also a good time to mention our TypeKey authentication service which has proven very effective in stopping weblog spam. If you are interested in setting up TypeKey, check out our public TypeKey tutorial posted today on ProNet.
What's Next
While we realize that these recommendations may not be your normal preference, they should keep your servers responsive despite any severe attacks until we can release the patch. We are sorry for any inconvenience this may have caused and I assure you that we are working tirelessly to remedy the situation.
On a more personal note, I would have preferred the circumstances surrounding my first post here to be somewhat (or completely) different, but there will be time once this issue is solved for me to address the past, present and future of this software as it deserves.
I want to thank you all for your perseverence through these serious problems and for helping us see clearly where the problems lie.
Posted by Jay Allen | Permalink
I'd like to make a simple observation because it very much fits within my work for social equity.
If these spam attacks against bloggers were made against say the personal computers of the White House or the leaders of the U.S. Congress or the homes of the CEO's of the Dow 30 companies, how long would it be before the perpetrators would be arrested and the key to the lock thrown away?
As the rich and powerful get more so, they further distance themselves from the rest of us. They say one thing, while they do another. They say and do whatever is in their self-interests because they know they control the lawmakers.
The rich and powerful no longer want to be a part of the society they presently control; they now live in gated communities and put up fences around their properties.
Because they have to.
If people at the top of power and authority hierarchies can't see where I am coming from on this, you had better smell the roses now because pretty soon the great unwashed will be taking them from you.
Digital technology is a great enabler " for both sides of the fence.
So, to the servants of the rich and powerful, I'll ask one more time: why don't you stop telling us what laws you are presently debating and, based on the laws already on the books, start organizing a campaign to arrest the perpetrators of spam crimes against the blogging community, which is costing us in the aggregate billions of dollars annually in lost productivity and failed transactions.
Posted by Posted by Bill Cara on December 26, 2004 10:29:46 AM | Category: Blogging World