Question: Occasionally peripheral concepts get dropped in without enough explanation of what they are to those of us to whom they are new, e.g. I am a bit hazy on the robot.txt file.
Answer: My apologies! Robots.txt is a little hard to explain. It’s main purpose is to tell the search engine spiders like Google’s Googlebot or Yahoo’s slurp what they can and cannot index. Usually you just tell them to ‘allow’ indexing of everything, and then use ‘disallow’ to designate certain directories you do not want in the public, Google index. Finally, you indicate the presence of your sitemap.xml file in the robots.txt file. This file tells Google / Yahoo / Bing how to navigate your entire site.
Google’s Webmaster tools have a nifty utility to create your own ROBOTS.TXT file. Just log in to Google Webmaster Tools and follow the instructions there.
Got a question? Click this link to email Jason or call 888-993-1122. Dr. Jason McDonald is founder and Senior SEO / Social Media Director of the JM Internet Group. He teaches the SEO training classes for the group, and therefore provides most of the SEO tips for this blog. His goal with this blog is to provide an easy 'one-stop shop' for the busy marketer looking for tips, tricks, and secrets on how to get to the top of Google and Bing for free using proven SEO tactics. When not dreaming up SEO tips, Dr. McDonald lives in the San Francisco Bay Area with his wife, two dogs, a cat, four iguanas and twelve children (just kidding). Really you read down to the bio on this guy? Enjoy the blog.
on SEO | AdWords | Social Media