Dear PM blog readers
As you are all no doubt painfully aware, there have been some major technical problems affecting this blog lately, and I appreciate that this has been very frustrating not only for you, but also for Eddie and the PM programme team. As the executive at the ³ÉÈË¿ìÊÖ responsible for blogs performance, first and foremost, I'd like to offer my apologies and thank you for your continuing patience.
To give you some context, the PM blog is one of a limited number of ³ÉÈË¿ìÊÖ blogs which are part of a pan-³ÉÈË¿ìÊÖ trial, all of which have been affected by technical problems at various times over the last few months. More information about the trial can be found here and here.
During a trial of this kind, problems are, to some degree, inevitable, and they help shape our plans to develop and support further blogs at the ³ÉÈË¿ìÊÖ. If we are to continue to offer blogs, it's crucial that we come across these gremlins sooner, rather than later. Now, about those gremlins...
The biggest problem we've had to overcome has been defending ourselves from . In a similar way to companies that target email addresses and email software, most blog software (not just the ³ÉÈË¿ìÊÖ) suffer periodically from overwhelming volumes of spam and junk comments. Yesterday for example we received over 30k of these messages containing, and I'm sure you're familiar with them, links to money-making schemes and porn. Because of the sheer volume and range of our service (we run over 40 blogs with 3m users every month), this has gone a long way to clogging up the system and causing many of the errors that you have had to deal with and reported to us.
This is also the reason that we switch periodically (across many of our blogs) to what we call pre-moderation. We employ a team of moderators 24/7/365 across all our forums/blogs and in this mode (which we usually use for children's services) then we ask them to check all comments before they are published to ensure they don't break house rules or that we don't publish any of the sometimes illegal junk/spam comments. Usually for long periods this ensures that comments are published almost in "real time" but recent peaks in volume, due to the some of the issues above, have caused severe delays lasting over an hour for users. We have also had small periods of downtime when our blog software has been offline for a few hours.
It's worth noting that these various problems are cumulative and interrelated, which means it has taken us longer than we'd normally expect to get on top of them. Furthermore, since blogs within the ³ÉÈË¿ìÊÖ exist only on a trial basis at the moment, we are unable to allocate the same kind of technical resource to them as we would to a full service; to do so would, in the long run, detract from other important projects. Once again, it is not only PM which has been affected, but all ³ÉÈË¿ìÊÖ blogs, however briefly.
As Eddie and Marc have already hinted, there are technical improvements in the pipeline over the next few months which should make commenting on all blogs quicker and more reliable. We'll try to keep you posted about those improvements and any future significant disruptions to the service but for now, I'll put as much time aside as I can over the next week to answer further queries that you might have via the comments below.
Finally, I'd just like to thank you again for your patience, and to add that the resourcefulness you've exhibited during these problems is really impressive. Sincere thanks to Johnnie for setting up an alternative Beach over at - the enthusiasm and creativity of our audience is really encouraging.
RSS feed
Email us
Listen to PM
Jem by name and Jem by nature. Thank you so much for your comments, they are really appreciated. It makes it so much easier for me to be patient now I now what's going on, and why. May I take this opportunity to admit that I am guilty of multiple postings today, due to the fact that at the time of posting I got the 502 error message every time. I promise that in future I will try only once. If the message gets through, all well and good. If not, I'll grin and bear it, in the knowledge that the Blog is evidently in good hands. Thank you for your time.
Jem, Thank you for coming here and explaining the situation. Now that we can see that you obviously take care of the blogs seriously, I'm sure that we will be a little more patient.
As for further questions, can you give us some "rules of thumb" on what to do when we encounter:
a) A 502 error when commenting
b) A 503 error when commenting
c) The progress bar at the bottom disappearing, but the page not changing to the "your comment has been received page".
If we know whether to try and post again, then we may be able to reduce the multiple identical posts (which must make the work of the moderators harder).
Thanks again
FFred
Yep agree with Gillian (1) - thanks for the update, its much appreciated
Thank you Jem Stone, much appreciated and are you as sparkly as our Helen?
Jem - I was was going to say what Gillian said - only not as eloquently. Thanks for the info.
Thank you Jem for giving us the full picture.
Thank you Eddie and the PM Team for putting up with our gripes.
In other places, as well as email address and contact details etc, they display a small graphic, with a few characters on. A commentator has to read that graphic, and type it in a little box. If the characters are wrong, then that entry is binned.
(Usually the graphics are randomly selected/generated?. They also have backgrounds so that they are hard for OCR software to read correctly, relying on humans' pattern recognition skills).
Surely this would be a good first defence against what must be automated "spam" attacks - to generate that many postings in a day?
I second all of the above. I also repeat my earlier suggestion that y'all have a look at the Grauniad's Comment/blog software.
ed
08/02/2007 at 13:26:18 GMT
Thank you Jem. Just to carp a little, could not the (very welcome) explanation been made rather earlier, in fact as soon as it was realised that these problems existed? We did seem to lose some people unfortunately.
In many ways the blog is a sort of conversation not only between ourselves but also with all of you, so it makes sense to be as up front as possible. It also makes us feel a bit more included which is always an attractive feeling.
Deepthought (John W) - 7
The problem with the use of the "Capcha" technology you describe is that isn't accessible to those using screen readers or other alternative web browsing methodologies and therefore gives you problems a) reaching the widest possible audience and b) may well be in breach of the disability discrimination act.
The 'Recent Comments' list currently reads:
Jason sends this: Fearless Fred had to guess which frogger made it in. Snow much?
:-)
Everything I wished to say, on reading your explanation Jem Stone, has been said above, but I cannot resist chiming in with a huge bouquet of thanks! Only fair, since I've been hurling brickbats around in the hope of attracting attention.
We now appreciate that you must have been trying to get to the "bot" of the problem first, instead of rushing to give us a half-baked excuse.
As has been said elsewhere, above all it is important that you may not have alienated first-time posters who have joined us and been "rejected". I just hope they look back in and see your thread. Will Lord Mair perhaps make mention of this on PM tonight?
Thanks again, long live ³ÉÈË¿ìÊÖ Blogs, surely an experiment worth continuing? Let the Healing Begin :o)
Yep. Deepthought. Absolutely Some blog software (eg: Blogger) now uses Captchas (Completely Automated Public Turing test to tell Computers and Humans Apart - to try and prevent both spam and attempts to post malicious comments and its pretty successful. Blogger (and typepad, wordpress) also offer various levels of registration...Registration (or logging in) which is almost the layer above that (as used currently on Have Your Say comments and ³ÉÈË¿ìÊÖ message board posts) is being considered for the replacement comments system we are working on and which is being tested this month. However when we launched the trial we wanted to reduce, what we call the barrier to entry of users being able to comment to as low as possible which is why we avoided these techniques. Both have their merits and had their supporters when we started widely launching blogs at the ³ÉÈË¿ìÊÖ almost a year ago.
Hi Gillian, RJD, FFred, Fiona - first thanks for reading my probably overlong apologies/explanation.
Ffred. yes. the dreaded http error codes
">
I'm afraid there's not an awful lot you can do.
As much as possible our moderators should pick up on duplicate messages although I appreciate how annoying it must be to either not know if a comment is published or if they are published twice..
The reasons for the error itself are partly connected to the spam, and the number of db calls that commenting using this software is causing.
Anecdotally I've been hearing that bbc.co.uk is also having a particularly busy day in terms of traffic due to (we think) users looking for weather / snow info.
All i can say is that we are monitoring this v.closely and trying (or rather my colleagues) a number of fixes. (Yep. It took me many goes to get this into the system so I'm aware that its seizing up again).
...and so say all of us!
Jem, you are a diamond.
Fifi ;o)
thanks Jem,
might be an idea to call ³ÉÈË¿ìÊÖ blogs in "Beta" or something to set expectations.
I know Eddie did say it was part of a trial, but that was once, ages ago. So I certainly forgot, and anyone new wouldn't have known.
I know I was 'hrrmphing' about this a bit, and mainly cos of an expectation of service levels. Now I know how (low!) to set my expectations!!!
:¬)
and I think "On Trial" next to Eddie's photo might be worthwhile regardless....
("Beta" sounds like "beetles" and frogs eat them....)
Re Whist (14)
Please avoid the racial slurs. We French eat many things, but not beetles! :o)
Rosalind. - Yes. Sorry. We should have responded earlier.
Jem
We all love the blog and Eddie - hence the complaining.
It's been difficult for Eddie, keeping us informed
-thanks to you both.
Mollyxx
whisht - we did think about saying that the blogs were in beta. but you're right. most users have no idea what it means.
Jem,
Thanks for coming back with that (and also for your efforts, I forgot to add earlier).
I must confess I thought I had registered to use this blog already, althought I have registed for some ³ÉÈË¿ìÊÖ bulletin boards or whatever (I've forgotten which now!). While registration is not that hard, it would remove the spontaneous blogging that occurs at times, e.g. the one-off bloggers who make valid points, such as the speed camera debate yesterday. On the other hand, once they've blogged once, maybe they will come back again.
Ha! got the malaceous comment page on first attempt, so here goes....and a second time...
Thank you for the explanation Jem.
A, x.
Thanks for the explanation! What a great name - gem stone, is it really your name?
Mary
Thanks for the comprehensive and thoughtful update - are you sure you work for the Beeb? (Only kidding)
Jem,
Many thanks for the detailed explanation of the blog problems, I can now stop hitting my head against the wall so hard, in the cell.
Brian
Jem, just to add to the thanks for the detailed explanation. I guess it's easy to forget what a huge organisation the Beeb is and how worldwide its reach (although we do have some pretty far-flung froggers). Sadly I suppose it was inevitable that someone would think it a good use of their time to attack the blogs.
We'll keep hanging on and look forward to Blog version 2 (or should that be v.1 if this is beta?).
Ed Iglehart - looking at the code, the Guardian uses exactly the same software as the ³ÉÈË¿ìÊÖ are - Movable Type. They're just configured slightly differently.
Jem might like to know that just renaming Movable Type's comments scripts to something other than the default settings, will reduce the amount of rubbish the ³ÉÈË¿ìÊÖ Blogs get - alas automated scripts look for the default file names and in this case, they find it :)
hmmmm...... captcha huh? didn't know it was called that.
And the "Accessible" alternative requires a plugin to play audio! (not very accessible)
mulls, thinking over with a beer....
how about this:
Instead of an image, why not have a simple sum and the user needs to put the answer in a field eg "5 minus 1 is... "
That surely is something a bot can't understand but a 10 year old can (our lowest user group?) I know people are scared of sums but anything below 10 is surely ok?
However, I've just made this up - but if it gets past your Information Architect people (and isn't something bots would eat up) then I want a job!!
:¬)
Hi Jem, thanks for responding to our chorus of doom. I've never heard of beta, so obviously the techie stuff goes right over my head, but can I assume the blog motto from now on is "things can only get beta"?
We may have to give you your own special title, if you are going to be so helpful. You may have noticed that Marc, (nominally in charge of the blog, but really, he's forever disappearing without even asking...) is our New Blog Prince. NBP for short. We had a Blog Queen, but she left, & frankly, things haven't been quite the same since. So standby for some sort of nomination, if you keep up the current standard of caring.
"things can only get beta" - now there'sa strapline, Annasee!
Whisht - how can they refuse you?
Jem -- Blog Diamond Geezer.
Andrew - Yep. We do use the same software platform as The Guardian's family of blogs. Thanks for the suggestion of renaming the comment scripts. We've grappled with this several times to little effect, I'm afraid.
Annasee, Aperitif, Valery P - ;)
Good morning Jem Stone - it's addictive isn't it? Are you going to stick to your own thread (so to speak) or join us on the Beach today, if the barbed wire hasn't gone back up?
Blog Diamond Geezer - I like it Appy. Too much to hope Jem is a cockney, I suppose?
You see, Jem, told you it wouldn't be long before you had a nomination. We don't muck about ,here in Blogland. (except when we're supposed to be working, that is - have you noticed how the Nation's productivity has plummeted since August???)
Jem -
Hahaha! Love it
JEM:
I'm writing a comment for you on this blog because the blog I usually comment on seems to have been so affected by this problem that it now published absolutely NO comments whatsover. You left a comment on this blog earlier, apologising for problems. I'm speaking of the Will and Testament blog. I've taken a look at other bbc blogs and many seem to be admitting comments still. Can you explain why Will and Testament is now a comment fee zone? Some of the previous threads have run to more than a hundred comments. Now, there are none. Strange?
Jem and Helen
Totally agree Helen - I've been trying to post on W&T without success - I've even emailed William Crawley a couple of days ago - without response from him as yet!!
Perhaps the Blog Trial has ended first in N Ireland?
We need some explanations - please.
alan
This has turned into a real Jem Stone love in - maybe Jem could have his own radio show covering the internet and new technology?