-
Magento Tips: How to improve search engine optimization with Robots.txt
| Posted in Tips&Tricks by Aleksander Tretjak
Robots.txt serves as one of the factors in search engine optimization, although in Magento this file is not included by default. So, let’s fix this and create the file ourselves:
12345678910111213141516171819202122232425262728293031323334353637383940# indicates that rules are defined for all botsUser-agent: *# should not wait too <div style="display: none"><a href='http://cheapviagra-2013.com/'>buy viagra no prescription</a></div> long for a page to loadCrawl-delay: 10# Now come the rules: restrict robots from indexing the following pages:Disallow: /admin/Disallow: /app/Disallow: /downloader/Disallow: /errors/Disallow: /includes/Disallow: /lib/Disallow: /pkginfo/Disallow: /shell/Disallow: /var/Disallow: /api.phpDisallow: /cron.phpDisallow: /cron.shDisallow: /get.phpDisallow: /install.phpDisallow: /LICENSE.htmlDisallow: /LICENSE.txtDisallow: /LICENSE_AFL.txtDisallow: /RELEASE_NOTES.txt# These rules are supported only by major botsDisallow: /*?dir*Disallow: /*?dir=descDisallow: /*?dir=ascDisallow: /*?limit=allDisallow: /*?mode*Disallow: /*?SID=#Disallow: /checkout/Disallow: /onestepcheckout/Disallow: /customer/Disallow: /catalogsearch/Disallow: /catalog/product_compare/Disallow: /catalog/category/view/Disallow: /catalog/product/view/These are the basic rules. Robots.txt file should be customized and changed per each project individually.
Useful links:
http://en.wikipedia.org/wiki/Robots.txt
http://www.mcanerin.com/EN/search-engine/robots-txt.asp
-
Posts by This Author
Aleksander Tretjak
(Magento Certified Developer Plus)
Aleksander Tretjak is an e-Commerce development master with 5+ years of experience. His specialization is Magento- and Prestashop-based webstores. Aleksander is an expert in PHP, MySQL, JS, AJAX, L...
Comments 0