In this session, we discuss the importance and function of Robots Meta Tags for your website. You can find it on your Thesis Site Options > Robots Meta Tags, then you can choose if you want it to be noindex, nofollow, noarchive and nodir. This are the meta tags that search engines look for to index your site. Then we discuss why and how to use them.
Member: I just wanted to get some clarity on what each of those meant when you check them in WordPress.
Rick: Ok, well, you know I’ve got some… under Thesis SEO I’ve got some stuffs in here about that subject. But when you said WordPress, what you really mean is in Thesis. In Thesis Design Options?
Member: Correct. Yes sir.
Rick: Or I mean in Site Options? The whole point of Robots meta tags is to control what search engines look for and index. So if there’s something you don’t want indexed then you would want to mark it no-index. Now the reason why you have this under site option as well is because WordPress generates lots of different views of the same content. And Google doesn’t like duplicate content. And Google sees the different views of the same content as duplicate content. So what Thesis does is it gives you the opportunity to prevent Google from indexing various paths with the same content. And so in this case, you know you’re not going to index the sub pages, the year pages, the month pages, the day pages, the author pages, or the tag pages. Because every single post has an entry under tag pages, author pages, day pages and month and year and sub pages, and in fact if it gives us more than one tag then it’s got more than one reference under tag pages.
Member: So it’s best to not reference that to, like… because what you see is that Google ends up showing each of those documents.
Rick: Its first instinct is to say they are separate documents. And it doesn’t like separate documents that say the same thing. So I do not no-index category pages, because if you no-index your category page then what you’ve done is… well, I think categories are important way in which people find your stuff. And so this essentially mean that all of you content, all of your blog related content gets index twice. It gets index in its normal index along its normal URL. And then it gets indexed in your category archive. And it could be that you might choose not to have any kind of duplicate paths in the same content at which point no-index to all of those things. I choose to have a duplicate path for categories.
Member: You choose to have a duplicate path for categories?
Member: Ok, did you say do not do the archive no-index archives?
Rick: I’m saying that my recommendation would be either check all of these or check all but categories pages.
Member: Ok, I understand now.
Rick: And that way Google will either not identify any duplicate paths, or any multiple paths to the same content or it will eliminate. And that’s the option that I choose, to have a limited amount of duplication.
Member: Ok, very helpful.
Rick: So that’s index. no-follow is a way of preventing Google from traveling from a link of one of your pages to somebody else’s site. And it can apply to the comments and it can apply to the links you yourself put in your blog post. So if you write a blog post and you include a link in it that is linking to somebody else’s site this no-follow thing is automatically prevents or tells Google not to follow that link anywhere.
Now the problem with that is that… I mean the reason why you would do that is because you wanted to… you don’t want Google to ever leave your site. And you don’t want give a vote to somebody else for the quality of their site. And there is a concept of link juice where if…that you dissipate the link, the SEO quality of your site by linking of to somebody else’s. Actually I think that’s trash, you know Google doesn’t say that. I think that’s a concept that somebody else developed to sell stuff. But even if it worked thorough, it’s not very good citizenship because if you find somebody’s site is worthy of linking to then you ought to do them the services of letting google know that, that what you think about your site because that helps them. So if you’re going to link to them, you’re linking to them presumably to have a view. And if you’re linking them to help yourself because you want to say something in your content, then you ought to extend that favor to them and let them get the benefit out of having that link by google registering that link.
So I don’t have any no-follow tags on my site. You can see all these things are no-follow to tag pages, but in fact there’s no-index to those pages. So the no-follow is pretty much irrelevant. Because it’s not being indexed it’s not being followed. I don’t add no-follow to category pages. If I link to somebody to one of my post then I’m going to let them get credit for that link. And if somebody comments on one of my posts and I think the comment is good enough to stay on the post, then I’m going to let them get the benefit from that too.
Member: When you choose not to follow inside your post, then you’ve gotta go into the HTML code and put some no-follow for that specific link?
Rick: Well, you have a couple of choices. One you could choose to add no-follow to a page. Right, because you still have this robot meta tag choices inside of each individual page as well. So you could decide to have no-follow on that specific page or post. Or if you wanted to give it no-follow in a one link only, then yes you would add the no-follow property to the link. And you know…
Member: Very very helpful.
Rick: That what’s lots of… you could actually have stuffs set up to do that automatically. You can have your… you can install a plugin on your site that will automatically add the no-follow to every comment that added to your site. People can comment but you don’t want to give them the benefit of a link back to their site for their comment. And the same thing as true for example in the forum post. It is possible, though I think very bad manners to have a forum where people posts and to not allow them to… and not allow any link credit to their site of what they’ve posted. And in a non-moderated forum of course that prevents links spam. But in a forum like I have, where people pay to participate on the forum, you know, I just think it’s very bad forum to no-follow their links. So we don’t do it.
So that’s no-follow, no-index, did you understand the no-archive?
Member: No sir, I don’t.
Rick: Yeah, no-archive is… Google for the benefit of its readers will archive pages and store them. So that if something happens and then you get a 404 error when you try to go to a page. Google will offer to serve you the archive instead. And again you can choose to no-archive if you wish. But I never choose that, I figure somebody’s searching for what I’ve got and for one reason or another that they get a 404 error when they hit on my page but Google’s got it in its index then I’m happy to let Google serve them a copy of their archive instead so they can have access to the content.
Member: Ok, make sense that’s it. Thanks for everything.
Rick: Well, in many respects I mean except for the no-index, no-follow and no-archive are definitely good manners and behavior checks. So, anyway.
Member: Thank you very much.
Rick: You’re welcome. Have a good day.
Member: You too.