SEO is a great way to reach customers because search engines attract people already searching for exactly what you sell. That means they tend to have a much higher conversion rate than visitors from other sources.
Despite good intentions, many development shops unknowingly nuke the SEO value of the sites they work on.
Even if you’re not actively working on improving your SEO metrics, you’re probably not looking to drain what SEO juice you already have. That’s why it’s so alarming that despite good intentions, many development shops unknowingly nuke the SEO value of the sites they work on.
How? The main concern is duplicate content. If Google discovers your development site and crawls it, it quickly finds that every page there is a duplicate of one on your live site. Both domains are then penalized for duplicate content, driving your real site down the results page into oblivion. Additionally, if your development site is open to search engines that means it’s also open to customers and viewers. Customers placing orders on development sites never have their orders fulfilled, and viewers who bookmark a development site won’t see any new content that goes up on your live site. Both situations are to be avoided.
That’s not to say we should do away with development sites. Development sites are a key part of modern web development—when you’re building something new online, features can be disabled or flat out broken during the process, so having a safe place where you can make changes is incredibly important. You just need to be aware of the SEO risks and the strategies that exist to combat them before creating a development site.
Save That SEO Juice
Robots.txt is a text file that lives in the root directory of your website. Its job is to tell search engines what pages they can visit and which they should stay off. If you want to block search engines from viewing and indexing your development site, all you need to do is add a robots.txt file that looks like so:
User-agent: *Disallow: /
That’s great, but comes with a caveat. If you accidentally deploy the new robots.txt to your live site, search engines will stop crawling the live site too, leading you to fall off the results page entirely until you correct the issue. That’s even worse than the duplicate content problem it was meant to solve.
We mentioned briefly that if your website is accessible to search engines, it’s also accessible to visitors. Although making sure visitors know they’re on a development site is a boon to any site, it’s most important on E-commerce sites. Customers placing orders that never get fulfilled is a terrible customer experience and it results in lost profits for you. To address that issue, you can put up a large demo notice on each page. The E-commerce platform Magento makes this quite easy—simply flip a switch in the database and a large orange banner shows up at the top of the page explaining that the site is merely a demo and that any orders placed there will not be fulfilled. Still, an ideal solution would prevent users from ever getting to the development site at all.
Full Password Protection
The best way to stop your development site from hurting your SEO ranking and misleading potential customers is to use your server’s password protection features. This requires any visitor—you, a search engine crawler, or a random visitor—to enter a username and password before any content is rendered by the server. Since you won’t be distributing your password to search engine crawlers or visitors, there’s no way for them to see content on your development site. By password protecting your development server, you don’t run the same risk of accidentally deploying changes to your live site as you do with the robots.txt solution, so you can be a lot more confident pushing changes live too.
Still A Problem?
If your web development shop isn’t doing this already, you might be losing SEO value. It’s pretty easy to patch the holes made by a non-SEO optimized development site, but the longer they sit unpatched the worse your rankings get, so make sure to fix it ASAP!
If you have questions about the topic, if you’d like to talk about fixing the damage done by a poorly implemented development site, or if you’re interested in working with us on something else, let us know! Reach out on Twitter, via email, on the phone at (720) 593-4565, or by filling out the form below. We’ll be in touch promptly!