Sadly, web crawlers don’t look like this, but they are still exceptionally powerful little lines of code. Image courtesy of Pixabay

To be indexed or not to be indexed

The noindex meta tag tells crawlers whether you would like particular websites or webpages to be indexed or not. Websites with databases or private information will definitely need to use this code. Forums or media-heavy websites may also prefer their content to be excluded from indexes. The tag is also useful if you would like to exclude specific pages on your site from being indexed.

Protip: The ‘index’ tag is not required since bots will crawl websites on default.

I will follow you, wherever you may go

No, crawlers don’t listen to Ricky Nelson, but they will listen to instructions on which outbound links they may follow. Outbound links bleed link authority, which is a commodity not many webmasters can afford to lose. So a simple nofollow attribute will prevent crawlers from following all the outbound links emanating from your website.

Don’t keep all brochures

Supermarkets and retailers hate it when their customers bring old brochures or flyers with outdated pricing. Similarly, if you’re running an e-commerce site, caches of old pricing lying around for everyone to see is the last thing you want. So gently, but firmly, tell search engines, ‘noarchive’! This will prevent them from making cache copies of your website. This will also prevent archived websites from stories complete copies of your website on their servers.

The post Search Engines – Robots appeared first on Climate Southeast.

Sitemaps play an important role in SEO. Image courtesy of Pixabay

Use a sitemap

A sitemap is a hierarchical textual layout of the content on your website. It provides web crawlers with instructions on where to go, and where it can’t go. Since crawlers no longer need to waste time following all internal links from the homepage, they will work faster and more efficiently – and return the favour by giving websites bonus ranking points. They will also consume less resources from the website’s server – which is important for really large websites. Using a sitemap also ensures that bots won’t crawl duplicate content, which Google doesn’t particularly like.

Minimising crawl errors

Robots sometimes encounter errors while crawling through websites. The simplest way to find out more about these errors is by going to a search engine’s webmasters tool. Some of the most common errors include:

• 4xx error codes: This means a crawler followed a non-existent URL. This URL may belong to a deleted page, or the URL may have been changed. Webmasters should write codes or install a plugin for a permanent 301 redirect to resolve the issue. In terms of SEO, a 301 redirect will preserve the link authority of the old page and transfer it to the new page.

• 5xx error codes: This is when a crawler is unable to access a website due to server-related issues. This could signal underlying issues with your hosting provider, some please spend some time to investigate the issue.

Have an expert review your site architecture

Site architecture is a massive topic and issues and opportunities here can be quite complex. This is especially true when looking at a larger site (some sites have tens of thousands, hundreds of thousands or even millions of URLs indexed after all!). It could be a good idea to have a search engine optimisation company take a look at your site in greater detail to find any potential issues.

The post Search Engines – Architecture appeared first on Climate Southeast.

Google’s market dominance is due to the accuracy of its algorithm. Image courtesy of Pixabay

Anatomy of Google

To enable Google to deliver fast, accurate and relevant results, Google must first learn about the majority of the almost 1.7 billion websites available on the internet, and thereafter, organise and classify them.

To do this, Google periodically sends spiders and web crawlers, sometimes referred to as Googlebots, bots or robots, to go to a website, analyse its content and structure, download a cached copy, and follow the outbound links from the site. This process will enable Google to continuously enlarge its search database and maintain its accuracy.

Following this, Google uses its proprietary algorithm to read through the data in its search database. Using its reported 200 ranking signals, Google will then classify and rank these websites. Popular signals include keywords and backlinks.

Serving search results

Google takes just a fraction of a second to serve its search results. They could serve it even faster (not that you’ll notice) if they didn’t stop first to fetch relevant Google Ads from their advertisers. However, search results served are not uniform. They actually vary depending on location, date, and device. In addition, a user’s browsing history and previous searches can also affect search results. You can view the difference by running the same search query using the incognito mode on your browser or by using a VPN.

However, what cannot be denied is, Google’s search results are extremely accurate and relevant. Even Bing, despite the billions poured into research by Microsoft, cannot come near to the accuracy of Google’s search results.

Who can help me when it comes to Google and their search engine?

Whilst Google is infamously quite when it comes to exactly how to rank a website, they do give hints and clues every now and then. Above all else they want webmasters to create the highest quality sites as possible and leave the ranking of them in their hands. Having said that, it is always good to be proactive. The best way to start improving your position in Google is to seek the expertise and SEO services of a qualified SEO company.

The post Search Engines – Google appeared first on Climate Southeast.

Technically speaking though, search engines are software algorithms which crawl through the majority of the visible portion of the World Wide Web, before cataloguing and sorting the millions of websites they found for easy retrieval. About a third of all internet traffic originates from search engines. Currently, the global search engine market is dominated by Google with a 92% share of the market, followed by Yahoo (2.67%) and Bing (2.39%).

The search engine landscape before the arrival of Google. Image courtesy of Philip Bradley

What is crawling?

Crawling is a layman’s term for the activities of software bots (robots) and spiders of search engines. The robots and spiders are deployed by search engines to navigate through the internet and collect data about websites. The data include URLs, code structure, size of webpages, types of content, keywords, backlinks, and much more. This process is called indexing, and the amount of data collected enormous.

What happens next?

These data are then stored inside a search engine index, which are then fed to the algorithms used by search engines. The algorithms would subsequently classify and rank the websites. When users make queries on search engines, the data will be retrieved and presented on search engine result pages in a human-readable sorted list structure. These queries would also be used by search algorithms to further refine their search results.

To maintain accuracy, search engines will periodically send the robots and crawlers out again to record new websites and changes to current ones.

You spend more time trying to rank! Once you better understand how the search engine result pages (SERPs) work, it is time to start ranking. It could be a good idea to seek out a trusted seo company now.

The post Search Engines – Index appeared first on Climate Southeast.