April 28th, 2007 - Sitemap Best Practices
Here’s the original draft of this article that appeared in BizTech Magazine.
You will find two different sitemaps representing the Stephen Hawking website on the internet. Far from parallel universes, one is a utilitarian collection of links that represents the hierarchical structure of his website. The other is pure eye-candy – a stylistic collection of images and graphical pathways illustrating all the dimensions of the physicist’s life, career, and writing.
At some point the site owner (assumedly not Mr. Hawking himself, though I bet he’d be awesome with cascading style sheets) reviewed both and chose one over the other. Score one for simplicity and none for aesthetics, because the stylistic version rests on an obscure domain, essentially unused, while the other is accessed each day by many Hawking devotees. In making this decision, the site administrators likely surmised that the image-rich sitemap, while visually impressive, is hampered by search engine indexing and site maintenance flaws. When it comes to sitemaps, function usually trumps form.
Sitemaps have traditionally served a simple purpose in an unexciting way. Sitemaps were an effective navigation tool in the pre-Google years to help users survey the site’s material at a glance and quickly access their desired information. Today sitemaps are also used by search engine optimization experts, who use a sitemap to enable an automated search engine spider to properly index all of a site’s pages. These days you can create an XML-format sitemap that is available only to search engines. It’s tempting for some site owners, confident that site visitors will find their way through navigational menus and search engines, to tuck the traditional, skeleton-like html sitemap into the closet.
Not so for Helen Whelan, President of Success Television and owner of www.successtelevision.com. “With a deep, content-rich site, the sitemap is a wonderful means of helping users find the article or videos that are most relevant to them,” she says. “We regularly track our sitemap metrics to see who’s coming there, where from, and where they visit next so we can improve upon our content offerings.”
Helen uses Google Analytics, a web statistics package, to monitor the ins and outs of her sitemap. She particularly likes the site overlay feature that graphically shows her which links visitors are clicking on within the sitemap. Her esteem for sitemap activity is not uncommon. The sitemap remains a popular navigation tool for surfers.
A sizable contingent of web surfers follow the navigation menu, search engine, sitemap pattern. This means they’ll look to the nav menu options first. If they can’t find what they’re looking for they’ll search and if the results disappoint them, they’ll try the sitemap. As such, the sitemap is their last hope before moving on to the next site.
There are pros and cons to each form of sitemap mentioned so far:
HTML Sitemap: An HTML sitemap is still the preferred method to address both the navigational needs of site visitors as well as the structural needs of search engine robots. An html sitemap can be compiled automatically using an indexing tool that analyzes the site and then generates an HTML sitemap. More often, site owners will create a custom sitemap that simplifies or clarifies the site’s information architecture. Still, if the sitemap properly encompasses the site’s high level pages, which in turn link to all underlying pages, an HTML sitemap will very likely help a search engine spider find all of your website’s pages.
XML Sitemap: Endorsed heavily by Google, an XML sitemap is a collection of links and page details that is custom-built for the indexing needs of search engine spiders. It essentially tells the search engine where all the site’s pages exist. If you use an XML sitemap, you can access specific information from Google showing all the pages that were indexed, at which time, PageRank information and top keyword information.
There are some drawbacks to this method. The XML file is not intended for human viewing and does not serve as a navigational tool for your visitors. Furthermore, Yahoo, MSN and other search engines have yet to endorse the XML format [update: both Yahoo and MSN now support XML format sitemaps]; these engines are better served with an HTML sitemap.
Flash or Image-map Sitemaps: As described in the Hawking example, these can be fun to look at and an excellent way to visually call attention to key site areas. Their drawback is how they index with search engines (poorly) and the difficulty of maintaining them.
A Combination: You could combine an HTML sitemap or an Image Map with an XML sitemap on your site. This provides you with full control over what the visitor sees and how Google interprets your pages. The only drawback is the additional maintenance overhead of updating two sitemaps instead of one for site changes.
Choosing the Right Sitemap For Your Site
The nature of your site should determine your sitemap needs. For a simple brochure-ware site of a dozen pages, you won’t need a sitemap to help lost visitors – your navigation can easily point the way. And if all your pages are already indexed by Google (you can see this by searching Google with site:www.yourdomain.com), you likely won’t need an XML sitemap either. For a news site or a corporate site with deep information (hundreds of pages), consider using an HTML sitemap to orient users, as well as an XML sitemap to ensure all of your URLs are getting indexed regularly.
For Gary Berger, owner of www.rxHelps.com, his 25-page site didn’t warrant an HTML sitemap, but he values the information he gleans from his XML sitemap. “Selling natural, non-prescription health products online is very competitive,” he says. “I use an XML sitemap because it gives me valuable feedback on Google indexing activity.”
Simple Steps for Building Sitemaps
Automated tools make sitemap creation a breeze. One of my favorites is GsiteCrawler, a free desktop application that will spider your site, index all the URLs, create an XML format sitemap and, if you like, automatically upload it to your site via FTP.
For an HTML sitemap option, AuditMyPC.com has a Java-based HTML sitemap creator. You simply input your URL and wait for the HTML sitemap to be generated, then upload it to your site. However, if your intent is to create a sitemap to help visitors navigate your site, you may be better off hand-coding a sitemap that appropriately frames the customer experience you’re looking for. Whelan chose this option for the Success Television site, a process that she says took a few hours from start to finish.
Get More Mileage from your Sitemap
The sitemap is ripe for some new and innovative uses. One alternative usage is to put the sitemap on your website’s 404 error page (the page a user sees if they’ve followed a dead link). By presenting them with the sitemap, they can quickly relocate themselves in the site architecture.