Build Your Own Business Website header image

Webmaster Tools Reports My Homepage is Blocked by robots.txt

Difficulty Level -

Filed Under Topics - ,

Listed Under Lesson Subjects -

Whoops, you've found some premium content!

Watch the opening clip of this video to preview it,
the full video is available to paid members.

In this session we discuss why Webmaster Tools reports the homepage being blocked by robots.txt. Then we troubleshoot why the disallow statement is showing up in the robot.txt file.

Video Transcript

Member:  My first question is that my webpage is not showing up the homepage where it looks like it is but Webmaster Tools, you told me, says it doesn’t look like it’s being indexed. I’m wondering how I… a) I got to fix it of course and b) how did I get it screwed up in the first place? All I ever did is inside WordPress when I was building the website under the appearance privacy, I just said do not allow search engines to call it. Then when I went live with it 3 weeks ago, I just changed that radio button. And it appears when you type in into Google, it shows that page where like I showed you on the forum. You’re saying there’s a slash disallow that says it isn’t. So my question is is this screwed up really and how do I fixed it and how did it get screwed up?

Rick:  As long as Webmaster Tools is telling you you have a serious error, right? That’s what it says, that little orange exclamation point that says you have a serious health issue then you have a serious health issue. The disallow with a slash means you are disallowing whatever the root file of the website is. So it could be index php, it could be index html and it could be home html. It doesn’t matter what it is. It’s telling it to disallow, that is do not index that page.

The way you solve that is to remove that disallow statement from your robots.txt file. Now in your case, you could simply delete robots.txt because you don’t have any other thing in there. Robots.txt is a file that is there to instruct search engines. There’s lots of things you can do with robots.txt but you aren’t doing any of those. The only thing you have in your robots.txt file is that disallow back slash or disallow slash.

Member:  I just have no idea how it got there so show me how to get rid of it.

Rick:  Okay well, I’m going to make you the presenter. We’re going to get rid of it using Filezilla. So I’m going to make you the presenter…

Member:  Okay, Filezilla instead of Webmaster Tools?

Rick:  Well, you could use Webmaster Tools to edit it. But under the circumstances, I mean, let’s just take a look at it first in Webmaster Tools. So you see it says, “Severe health issues are found on your site.” So if we click the click site health, it says, “Robots.txt is blocking important pages.” So go ahead and open up the site there.

Member:  This?

Rick:  Probably just double click on… yeah, okay that got you there.

Member:  Yeah and when I got to here, it said, “Any changes you make here don’t matter and it won’t be saved.” So I’m like, “What?” I didn’t know what to do from here.

Rick:  Actually, I believe that this disallow is coming from a plugin because it’s telling you the site map is which means you probably have a plugin that is creating that.

Member:  Yeah so all the videos I went through you know, for your original 6-week training thing… so I have the Google XML site map thing. I have it set up just like you did on the video.

Rick:  Okay so let’s look at that because I think it probably has a thing in there that says, “Do not index homepage or something like that.”

Member:  So where is it again? It’s under…

Rick:  Yeah, there you go. Got it. Okay so scroll down there and let’s just see what… scroll a little further down. Now you just installed this yesterday though, right?

Member:  No.

Rick:  Oh okay.

Member:  I installed this back when I first built the website, when I was going through your videos.

Rick:  Okay so let’s keep going down then. Okay so…

Member:  Keep going or…

Rick:  Yeah, keep going. Okay, WordPress standard content, include the homepage, include the post, include static pages, include categories.

Member:  So what I did, if I remember, I mean, this was weeks ago. I went and well, I watched the video originally. I copied basically what you did and then you told me and where the post things is maybe depending on how often it upgrades, go look at this video. So I watched the entire video again. I went back and I think I changed like one thing in here which is like this category or something.

Rick:  Okay so let’s go back up. It looks like this is written properly so open up Filezilla. I don’t have an explanation for where that disallow came from because it’s clearly not coming from here and this is using robots.txt to to tell Google and everybody else that there is a site map. So we don’t want to just delete it so what we’ll do instead is we’ll edit. So in the right hand side there, scroll down to robots.txt. You know, you could… in this case, now I… oh okay. So go ahead and open it up and open up your local copy of it with a text editor. Not Word but you know, Notepad or whatever.

Member:  Okay. I got to go find it when I do that.

Rick:  It’s up at the very top. It was at the top of that thing.

Member:  What now? You want me to open Notepad, right?

Rick:  Yes, I do. Right. See it’s… yeah, there you go.

Member:  I don’t remember the directory where this just says to web because I copied what you did. It’s like 80 miles deep once it’s… I’d have to look at the directory because I copied your video…

Rick:  Well you can see it right there in Filezilla.

Member:  Yeah, I know. We’ll have to go back and see what it was. So it’s…

Rick:  2011 websites Anatomie Wear WordPress. Okay, let’s open that up. Okay that does not have a disallow statement in it.

Member:  Okay.

Rick:  Well, that’s irritating as heck.

Member:  You think? Like I said, I’ve been chasing around and going, “What am I doing?”

Rick:  I would delete this then entirely. I would delete the robots.txt file.

Member:  Then just close…

Rick:  Just close that, close Notepad and over on the right hand side, I would just say delete. Okay and then go back over to Webmaster Tools and go down below and say… let’s see…

Member:  Test.

Rick:  Yeah go down… yeah, test. Okay now it shows blocked URLs is none. No, it doesn’t. It says Google bot is blocked from http access. robots.txt so go ahead and delete that slash.

Member:  Let me ask something that maybe not have anything to do with it. Does it have anything to do with originally… well, I don’t know what the difference is between http and https because it’s a…

Rick:  Oh yeah. Yes, there is a difference.

Member:  Somewhere in the process, I had to get a certificate you know, 2 months ago when I was… I had to do the site certificate thing for the shopping cart stuff to make it secure.

Rick:  Right.

Member:  Because the other part that’s screwy, this is what’s never made sense to me. If I go to here… if I hit… if it’s being blocked, how does this show up?

Rick:  Because your site map is not telling it to block it.

Member:  Okay.

Rick:  Your site map is actually listing it. And so it’s following through on the site map rather than on this instruction. The thing is that this instruction doesn’t exist anymore.

Member:  So how do we go back a page to the next higher level here? I’ve never understood where we were before where we can hit the like check health thing.

Rick:  On the left hand side there? Oh yeah, you can do it there or you can just select health.

Member:  Okay, health.

Rick:  Just click on it.

Member:  I did.

Rick:  Oh.

Member:  It just gives me the drop down.

Rick:  Okay so then you just back up to the… yeah, the dashboard and…

Member:  You see, the navigating this Webmaster Tools drives me nuts because I don’t even know how to type it in to Google’s Webmaster Tools. I’m not sure how to get here from my account or anything. Or is there… do I do something? Is this where I’m going, like to here?

Rick:  No, I think… you’re at the dashboard right now so it says, “No new messages or recent critical issues.” Go ahead and click on view all. You have no messages so my guess is you’re probably okay and all you really need to do is just let it cycle through now that robots.txt…

Member:  How do I get back to screen to see if it has a little yellow warning thing? That’s what screws me up. It’s not on the dashboard. Where is it?

Rick:  Oh, I see what you’re asking for. Go click on My Account in the upper right.

Member:  It’s like only there when I first log in or something.

Rick:  Okay, go to Products and then Webmaster Tools is down at the bottom.

Member:  That’s how you get to it, okay.

Rick:  That’s how you get to it.

Member:  Alright. I would have never known that. See, it still says that.

Rick:  I guess I’d give it a few days and see if that changes. Well, I mean, let’s just make sure it’s not there. Let’s look at Filezilla one more time just to make sure that there is no robots anything there.

Member:  Am I in the right directory here or whatever?

Rick: It looks like you’re in public html of whatever…

Member:  Anatomie Wear so it should be in here?

Rick:  Yeah.

Member:  So to wrap up the first one, I just need to go back to Google Webmaster Tools in a day or two and see if the little yellow guy goes away? The warning?

Rick:  Yeah.

Member:  Okay.

0 Comments… add one

Save $200 on Membership Now!

Start learning today for as little as
$0.82 PER DAY!
Subscription Options
0 comments… add one