As we all know, there’s a huge invisible, intangible creature roaming the web reading everything out there! But the Google spider really isn’t that bad. All it’s doing is reading your website, jotting down the key points and then storing this information somewhere else. In fact, the Google spider is somewhat like a librarian.

For this purpose, part of SEO is to make sure your website is as easy to read for Google’s spider as it is for a human to read your site. For example your site could look amazing to the user, but the code used to write the beautiful landscape of words and pictures dedicated to your services could be as muddled and confusing as one of Dan Brown’s books.

Another one of the main points for optimising your site for the crawler is making sure that your site can be read as frequently as possible. Because you want to update that site a lot with fresh, valuable and quality content.

Two main factors of this, as confirmed by Gary Illyes (Googles Webmaster Trends analyst), are the following:

HTTP Status Codes

There are a range of different HTTP Codes. The most common is 404 (Page Not Found). But there’s also 301s, 302s, 200s and a lot more. Google can handle these codes quite well, but when it starts to get http codes in the 500 range, this could spell trouble.

The 500 range indicates that there’s something wrong with the server. As such, Google won’t want to risk any further harm to your website so will stop crawling you for a while, giving your website time to recover/get fixed.

If you do have 500 error codes, make sure you get them fixed right away and then “fetch” your website in webmaster tools to make sure that Google can now read your website again.

Connect Time

As above, when Google detects a slow connection to your web server it will assume the worst. That your website is experiencing issues and any further connections to the site will cause further issues to your server and then break it.

Due to this assumption, it will again limit the number of crawls that it will run on your website.

So, all in all, if you want Google to read your site correctly and frequently clean up your 500 error codes and make sure that your server is in tip top condition.

Blog Post by: Greg McVey