Question: To keep the general site human-friendly, where would you put the robots.txt file? In your sitemap?
Question: In the robots.txt file, would you want to indicate that non-product related pages (staff bios, resources for existing customers) should not be indexed? Do those types of pages dilute your density? Or is it best to incorporate keywords into those pages as much as possible?
Answer: The robots.txt file is decidedly not for the humans, hence its name (robots.txt). The universally accepted location to put it is in your top directory, as in http://www.jm-seo.net/robots.txt or http://www.jasonmcdonald.org/robots.txt. You do NOT link to it anywhere on your website, as it is only for the robots.
That said, what are its functions?
First and foremost, it tells the robots (a.k.a., Googlebot, Slurp (The Yahoo Spider), etc.) what not to look at. Following the disallow command you tell these indexing spiders what to IGNORE on your website. The syntax is as follows:
User-agent: This means this applies to all robots
Allow: / this means index everything, just to be clear, except what is disallowed belo
Disallow: /1 this means do not index the ‘1’ directory
Disallow: /password this means do not index the ‘password’ directory
Finally, the robots.txt file also indicates the location of your sitemap.xml file. And remember that it is (in a sense) the opposite. What you put IN the sitemap.xml file is what you DO want them to index.
And, for extra credit, I recommend that you exclude your PPC / AdWords ad landing pages, because you want 100% paid traffic flowing into these pages so as to not confuse your metrics!
As for the second question in this group, I would leave in your staff pages, etc., as they make your website appear more natural. I would only exclude very unrelated pages from the indexing process.
Got a question? Click this link to email Jason or call 888-993-1122. Dr. Jason McDonald is founder and Senior SEO / Social Media Director of the JM Internet Group. He teaches the SEO training classes for the group, and therefore provides most of the SEO tips for this blog. His goal with this blog is to provide an easy 'one-stop shop' for the busy marketer looking for tips, tricks, and secrets on how to get to the top of Google and Bing for free using proven SEO tactics. When not dreaming up SEO tips, Dr. McDonald lives in the San Francisco Bay Area with his wife, two dogs, a cat, four iguanas and twelve children (just kidding). Really you read down to the bio on this guy? Enjoy the blog.
on SEO | AdWords | Social Media