Content Management Systems used by public sector websites found lacking

Verva (Verket för förvaltningsutveckling / Swedish Administrative Development Agency) recently published a survey of a number of CMSs (Content Management Systems) used by Swedish public sector websites.

The survey evaluates to which extent the content management tools adhere to the guidelines in Vägledningen 24-timmarswebben (in Swedish), the Swedish national guidelines for how public sector websites should be constructed. The following CMSs were evaluated:

  • EPiServer 4.51
  • SiteVision 2.2
  • Sitecore 5.2
  • Polopoly 9.3.5
  • Plone 2.1.2

Few standards and accessibility aware web professionals will be surprised by the results of this survey. All of the evaluated CMSs were found lacking with respect to web standards and accessibility, some of them severely. The results of the survey and a full report are available in Brister i publiceringsverktyg leder till att information förloras (in Swedish).

This survey makes it all too clear that many CMS developers are still stuck in the old ages and have not updated their web development skills for years. At least not those related to web standards, accessibility, or usability.

The results of the survey also make the marketing used by certain CMS vendors seem quite offensive. Some vendors claim full adherence to accessibility guidelines and web standards, others say their product enforces the separation of content from presentation. Neither is true.

These CMS vendors are either ignorant enough to actually believe their own marketing, or they simply do not care about accessibility and use it only as a buzzword to sell their product. I suspect it is more often the latter than the former.

While modern web professionals have the skills necessary to see through these marketing practices, the people in charge of many public sector websites do not, and cannot be required to. This leads to taxpayers’ money being spent on substandard, inaccessible, and outdated products. Sure, it is possible to build accessible and standards compliant websites on top of most of the CMSs evaluated in this survey, but it requires a lot of patching and fixing and filtering that shouldn’t be necessary.

CMS vendors in general have a long way to go. A few suggestions:

  • Do not settle for the HTML, CSS, and JavaScript produced by whatever IDE your backend developers are using.
  • Hire client side web developers with a solid understanding of web standards and accessibility.
  • Spend some time and money on upgrading the skills of your current developers.
  • Make sure your default templates are fully accessible and web standards compliant. No matter what you say, many web developers will use the default templates as a starting point.

Knowledge of web standards, accessibility, and usability is a must for CMS developers in the 21st century.

Posted on April 3, 2006 in Accessibility, Content Management, Web Standards


  1. I couldn’t agree with your suggestions more. A group of us are trying to help churches build a CMS that accomplishes a push to web standards called Ekklesia 360. Unfortunately, very few people have come to a place to understand the need to use web standards still…

  2. What about Drupal? ( There are a number of xhtml strict themes available. In fact, the template engine is based on php, so you can customize the output however you want. This is what Zeldman himself had to say:

    Open source content management platform that cares about accessibility and standards.

    I would seriously consider taking a hard look at Drupal, as through my research I found it to be the best CMS out there, especially the new 4.7 version.

  3. A simple, amen.

  4. I’d love to see an English translation of the survey results.

  5. It’s unusual for me to defend CMS, but I do think the problem here is not CMS nor the CMS vendors, but rather the implementation partners.

    You could take any perfect standards-compliant CMS (if a such existed), and nothing would prevent the unskilled web professional from creating a non-accessible, non-standards website.

    Sure, it would help if all the samples from the CMS vendors were compliant and used HTML + CSS best practices. All I’m saying is that it seems too easy to push all the blame on the vendors and the systems.

    Cheers, Janus

  6. Having not really used a CMS to any real degree prior to about two months back (apart from my Wordpress blog if they counts) I spent some time investigating what was available and would recommend Its not one of the larger or flashier CMS’s (such as Joomla etc) but it outputs really nice, clean XHTML and is quite easy to use - alowing direct access to the code in a sensible markup structure that most reasonably-html/css fluent people should be able to master without too much trouble. There are less extensions available for some of the others but then it really depends on the scope of your job.

  7. April 4, 2006 by Delia

    You want to see a BAD content management system specifically geared towards and sold to local governments…? And no, this is four days too late to be an April Fool’s joke… U.S. public sector web site

    To Janus in comment #5 above…how can you not blame a vendor who continues to sell this CMS disaster? Try validating it…

    If you hunt around, you’ll find the other towns also using this CMS…all similarly horrendous. Sites built in 2001 sport the same exact code as sites built at the end of 2005.

  8. I have found Xaraya to be an excellent choice for making web standard, accessible sites. It only recently went version 1, so many of your readers may not have heard about it (I’ve no affiliation btw, but am tempted to get involved).

    A key feature I like is the XML style blocklayout templating language, which once grasped is extremely powerful, allowing aparently limitless retrieval and arrangement of the stored data (around which you can place your well-formed, semantic XHTML) - so you’ve only yourself to blame if the site is lacking in any respect ;)

    Personally, I feel the default templates suffer from divitis and too many classes, but like I’ve said that’s easily cured.

    I’d be very interested to hear what CMS / framework you use or recommend Roger?

  9. CMSs have been known to be late in catching up with standards. They are busy building vertical functionalities or fixing bugs to make the package more attractive. Standards compliance is at a lower priority. However, some like eZ publish, typo3, Apache Lenya, opencms have adopted the standards pretty well.

  10. Drupal should have been reviewed as well.

  11. April 4, 2006 by Tommy Olsson

    The survey was very odd. How can they say that Polopoly is the least accessible CMS when everything it outputs is created by the templates you create yourself? You have full control.

    On the other hand, it’s hardly surprising that the CMS that Verva use themselves landed in the top spot. I wonder how much Elektropost paid for that piece of advertising…

  12. I believe that Tommys comment is good. That is the major problem with one of the CMS tested. The templates is built in a WYSIWYG GUI. This makes really hard to build good templates that are accessible and top notch. The GUI produce valid code but not very lean code.

  13. I don’t know how well Plone did (I can’t read swedish) but with Plone you can make your site as crappy, non-standard compliant or as kick-ass w3c compliant as you want. With Plone it doesn’t make sense to make statements about how well or how bad it fares when it comes to web standards. I work with people who implement my designs in plone. I deliver them standards compliant XHTML which they transform into Plone templates. The end result is exactly as good as my work. If the stuff I delivered them is 100% standards compliant and accessible, so will the site be.

    Therefore I agree with Janus. It’s the implementation partners who are the ones to blame if they produce garbage output, not the CMS. No serious developer/designer would ever use ‘standard templates’ anyway so it’s all really up to the person building the site with whatever CMS.

  14. Thanks for a great pointer, Roger.

    This quantification of features is a good starting point for more “scientific” ways of evaluating a CMS. I learnt a few things from the rigid way the comparison worked.

    The point about the importance of defaults in publishing systems is valid. The (as yet unattainable) CMS dream is that the implementor is never in the picture.

    It’s interesting (and smart) that the CMS overview only took the Semantic Web details (as expressed in the Swedish guidelines for making gov’t web content) into account, and did not confuse the overview by listing additional properties, such as SOAP integration, software/hardware environment requirements, price, or Web2.0-ness.

  15. April 4, 2006 by Roger Johansson (Author comment)

    Rick: I had a look at Drupal not too long ago, but I wasn’t that impressed. Maybe things have changed since then.

    Janus, Marco: Sure, the implementation partner is often to blame for not making the templates valid, semantic, and accessible. But what about the cases where a CMS uses a WYSIWYG editor that produces bad markup? Should the implementation partners have to build their own WYSIWYG editor? And when the functionality that comes with a CMS produces invalid code, should the implementation partner really have to recreate everything? Wouldn’t it be better if the CMS vendors fixed these problems instead?

    Tommy: The reason for Polopoly’s low score is that a default installation does not offer a way to create semantic markup. You need to install a separate editor for that.

    Verva’s use of EPiServer has nothing to do with it grabbing the top spot. Getting a score of 58% only shows that it is the least broken CMS according to the test criteria.

    Steve: I have working experience with Plone and EPiServer, and if Plone was to include a decent WYSIWYG editor it would be a really good CMS accessibility wise. It’s incredibly difficult to develop for though, unless you are a Python and Zope expert.

    EPiServer is decent once you have built new templates and implemented filtering to compensate for the stuff its WYSIWYG editor and the ASP.NET controls output. It only works properly in IE/Win though, which sucks.

  16. April 5, 2006 by Tommy Olsson

    Roger, with Polopoly you create the output templates yourself. It’s Java classes and JSP pages. You have full control. There are no default templates.

    EPiServer uses .NET, which under certain circumstances generates code outside your control. This code can be non-compliant and inaccessible. (At least that was the case when I last looked at EPiServer.)

  17. A few years ago, I saw a movie theater website prepared using frontpage that sent Opera badly (but automatically, as it turned out) prepared gif’s instead of text. Unsuspecting and without checking the code, I sent the owners an email stating that their site was really bad in terms of useability, without even having to test it in IE. I got a response that showed they really had no idea and ‘everybody seemed to like it, so if I could clarify what was wrong’. Then it dawned on me, when one opened it in Iexplorer, it was text and images as normal. Probably they forgot to turn off or accidentally turned on a “feature” to send “minority” browsers the correct layout, that’s my theory at least, I never used frontpage. When I checked the code, the page was full of IE checks all over the place. Good thing is they changed it later to a less worse version (and probably independent of my request, because I must have looked like a fool to them anyway)

  18. April 5, 2006 by Roger Johansson (Author comment)

    Tommy: Apparently Polopoly does include a few default templates, or at least some kind of example website. If you look at the report it states that the markup used by a default installation of Polopoly is excellent.

    EPiServer’s use of .NET can be problematic since you have to rebuild most of the controls it uses unless you want your site to be based on Microsoft’s “HTML”. Once you create your own controls you have complete control of the output though. Well, after you have created filters for the output from the WYSIWYG editor, which you shouldn’t have to.

  19. Well, if the CMS uses a WYSIWYG editor that creates bad markup, sure that is a problem, which should be fixed by the CMS vendor.

    But, CMS vendors will only change if the user community demand it (which I hope they will).

    And, even if the WYSIWYG editor did create good markup, this would only solve part of the problem.

    Generally, the issue seems to be that most enterprises are unaware of the problem with bad markup.

    I don’t believe that simply blaming the CMS vendors will solve this.

    Cheers, Janus

  20. I am not surprised, not at all. Good link, thanks a lot!

  21. I was looking for a nice CMS for myself and recently found MODx CMS It’s the first CMS I know using AJAX, it’s robust, SEO friendly and easy. I’ve been trying to go online with Drupal, Movable Type, Wordpress, Textpattern, and some more CMSs/blogs, but none impressed me like MODx. I had no time yet to change my template, but the default is very good. Build by webstandards, it has also many accessibility features, and with it’s rich text editor you can make almost anything. But sure, the owner has full control, and if he fails on building/mentaining the website, everithing is lost.

    Cheers, Boby.

  22. I’m suprised Microsoft CMS 2002 didn’t make it on to that list.

    Or can people just assume that it produces horrible code?

  23. I’ve been working on my own CMS, which will force people to write valid markup (can’t do anything about the semantic side of things mind you): WolfContent.

    Its not even ready for release, but I’ve written an article about how some of it works…

  24. April 7, 2006 by Roger Johansson (Author comment)

    Janus: Oh, I agree. Only blaming the CMS vendors won’t help. Both end users and developers need to demand that they fix their products. After all, most are broken to some degree.

    Michael: Yup, every site based on Microsoft CMS that I have seen is a mess.

  25. Even if cms uses bad wysiwyg editors, they can automatically clean up the code serverside using Tidy from w3c. That’s what we’re doing in umbraco cms (an open source .net based cms) until we’re done implementing a new wysiwyg editor.

    Even though our site is terrible markup (we’re still working on a new site -> which should be a better showcase), umbraco makes it very easy to create valid markup. It doesn’t require Visual Studio, a Dreamweaver plugin or like-wise, but has a template engine where you just copy your html and css. It doesn’t generate a single line of code by it self - not even a generator meta element in the header :)

    All dynamic lists are created using xslt, which is a perfect match with css when it comes to navigations and like.

    We’re doing quite a lot in listning to the webstandards community - we’ve even tried to have Roger come to our umbraco conference in May (and we still cross our fingers that you’ll have the time :))

    Cheers, Niels…

  26. April 8, 2006 by Roger Johansson (Author comment)

    Niels: Cleaning up the WYSIWYG generated code is a very good idea. It’s a bit odd that so few CMSs do that.

    From what I’ve seen so far, Umbraco is promising :-).

    Still looking into whether my schedule will allow me to come to the conference or not. I’ll let you know ASAP.

  27. This might be a nice mod for Apache that cleans the code before sending it to the client.

  28. One word: Ellington.

  29. Incidentally, I used EpiServer to update some stuff on this week. Gawd, it’s horrible! The HTML it renders is akin to Frontpage anno 1998.

  30. April 13, 2006 by Roger Johansson (Author comment)

    Jens: mod_tidy looks really, really useful. Too bad so many commercial CMS vendors have bought into the Microsoft hype.

    Jeff: Interesting. i checked a couple of the showcase sites for Ellington, and the markup they use isn’t too bad :-). Do you have personal experience from working with Ellington?

    Johan: Ah, nice coincidence. Just last week I spoke at a couple of seminars and used (you have to add the “www” or the site won’t work) as an example of designers not making sure that the site is usable with CSS off. Visit the site, turn off CSS and try using the top nav to see what I mean. :-D

  31. I agree with the ones mentioning Drupal above. It should have been listed. It is easily one of the most powerful CMSes available today, and one of the reasons is Taxonomy. As a person who has many academic credits linguistics and interaction design/HCI, I must say it’s a concept aimed at the future, belonging to the future. There’s also the gazillions of modules ready available, adding plug in functionality with a snap. I use it to power my own site (all XHTML + CSS).

  32. April 30, 2006 by Roger Johansson (Author comment)

    Jakob (and others mentioning various CMSs that you think should have been included in the test): I agree that there are many more CMSs that should be tested. Due to time and budget constraints this project had to be limited, but anybody can test any CMS themselves since everything needed to do so is available for download from Verva’s site.

  33. There’s just such a jungle out there! I looked at Xaraya and MODX and neither had a page explaining the basic ideas underlying the design of the CMS, it seems you have to download it yourself and try it out. Xaraya also suffers from major usability issues, or at least their website does and it seems poorly documented as well. So just deciding which one to try and invest time in isn’t very easy.

  34. This is shameless self-promotion but we’ve developed a CMS that outputs standards compliant, semantically correct xhtml that is entirely CSS driven…And I don’t mean that in a way that most apps claim to support css only to give a bunch of tag soup to wade through. Deploying sites is super easy, once I’ve defined the structure of the layout, I don’t even touch the xhtml…It’s just straight CSS. We’ve also included dynamic creation of contextual CSS hooks so you can target anything you need to. And not in a bloated way either. If you’re interested, check out Sorry for the quasi-spam, but it’s really a great tool.

  35. I has been developing for the Swedish goverment in many years and most been responsible and developed a few important projects in the Swedish Armed Forces.

    First I must say that this survey is lacking big time since many places are not using any of those CMS solutions, for example my solution was developed by myself and served over 30000 people in total.

    The main problem is not this “24 hour department” thing but rather the non-professional decisions taken by many people at high positions to use incompetent consults for evaluations, surveys, development for a lot of money with often very little result if you keep in mind how many has been involved and how much money it costs. Also often purchasing solutions which is pure crap and without any knowledge how these can help them now and in the future, a part of what Statskontoret has decided is to save documents in XML format to make them future proof and yes there is going on a lot of projects to transform old/new publications and documents to XML but once again with help of incompetent consults or “I dont say here in public, but you can guess why some buy crap solutions and still know it”. Most of those publications that is saved in XML formats is not at all future safe, they are often dependent of specific base parsers and/or even applications to view those publications from their base format in XML.

    Many governmental departments and municality websites which is made with CMS solutions have big problems with their security after my researches for my own and the government, there seems not to be any good “evaluator” for such things like security and badly coded and/or configurated packages are slippling through and are getting used for displaying information in public.

    And just meantion CMS like Xaraja and MODx aint productive, I think even MODx was stolen from another CMS which I do not remember the name of right now. But they both are not so very flexible as they want their audience to believe.

  36. August 11, 2006 by Roger Johansson (Author comment)

    R: Of course the survey is lacking. It is only a scratch on the surface, but that’s all you get with limited funding.

    I fully agree that the large number of incompetent consultants that are used by Swedish public sector organisations is the main problem.

    In some cases it doesn’t matter how skilled and experienced the consultants are though. Crappy CMSs that do not allow normal web development methods seriously hurt the accessibility, usability, and general quality of many public sector websites. Those CMSs should either be fixed (at the vendor’s expense) or banned from the public sector.

  37. Thanks for that very quick response.

    I see we are thinking on the same things and agrees to all or atleast most.

    The thing is that, it is possible to ban softwares that may comprimise security and reveal protected information, not talking about high security information but “semi” security information which many municipalities and public sector departments doesnt know how to handle due lack of competence.

    Maybe one day Swedish public services will work much stricter and follow a guideline book with much tighter computer system security beside CMS.

  38. May 31, 2007 by Martin

    Even though the different CMSs that were tested had/has its share of bugs and badly thought through solutions, I would say the problem really is elsewhere. I think a lot of the problems with the sites in the public sector both in Sweden, Norway and other countries are caused by lack of knowledge from those who set up the CMSs and use them.

    Another issue that should be focused on is how close the CMSs developement are followed up. The reason Drupal and other CMSs are viewed as better solutions by some people here and other places is because of the activitiy around the developement. And the community contributes a lot due to the fact that many are open source.

    Would it be wrong or right to mention ExpressionEngine as well here?

    My two cents ;)

Comments are disabled for this post (read why), but if you have spotted an error or have additional info that you think should be in this post, feel free to contact me.