Possible for a website to only be accessed by a link or typing the url?

[Bentley4]

Hi guys,

 

Is it possible for a website to only be accessed by a link or typing the url?

(in other words I don't want search engines to pick it up)

 

How should I do this?

 

B

[requinix]

Any sufficiently motivated search engine bot is indistinguishable from a normal user.

 

For the Lawful Good ones you can use a robots.txt in the root of your site with

User-agent: *
Disallow: /

For the others you'll have to do user-agent checking and use IP address bans.

 

But why don't you want search engines crawling the site?

[Bentley4]

I want to implement different versions of my site, one for potential employers, one for friends,...

I would give them a link if I wanted them to check it out.

When you said 'lawful good ones'; In other words it is possible for google, yahoo and bing?

[requinix]

Yes, those three will.

 

So you're saying you don't want potential employers to see what you're showing your friends?

[Bentley4]

Actually I'm not even going to use it for friends, just for people I don't know yet that I want  to show my profile in a non-work related context. (different structure of website, work: more emphasis on skills, non-work: more emphasis on intrests and hobbies.)

I also have a public one that everyone can look up, which gives less detail.

[Philip]

If you're not linking it publicly, and neither are they, there is no way for the crawlers to find your pages without doing a lot of guessing.

 

For example,

http://www.example.com/123jhl342j3lk/preview/ is going to be pretty hard to guess

[Bentley4]

If that is the case I don't understand the following: I have a www.myname.info page. I have never linked to it anywhere and I'm pretty sure noone ever has because its relatively new and is blank.

Yet when I type 'myname' it comes up as the 6th search result on google. I also have a flickr account with my name recently created, it contains no photos, haven't linked to it and I don't understand why anyone would link to it if it's still blank(so I assume noone did.) I can also find this account by typing in my name.

 

[SparK_BR]

He means you must create a page like "hobbies.php" but in the main page you simply don't link to it

so the bot will get to your index but if the index has no link to the other page, the bot won't see it

[Bentley4]

So basically you're saying that search engines(or rather webcrawlers associated to them) will always be able to find and list my main/homepage in search engines but not subdivisions if I don't link to them?

Thnx

[requinix]

Yes, but obscurity is not security. If the names are predictable like "admin" or "www2" or "private" then simply not linking to them doesn't guarantee (but greatly increases the odds) that they'll stay hidden.

 

Also: somebody else might, for whatever reason, link to whatever "hidden" page on your site they want. Can't defeat that.

[Philip]

Also: somebody else might, for whatever reason, link to whatever "hidden" page on your site they want. Can't defeat that.

 

Sure, and that is very possible. However you could add something like an htaccess or simple php password to protect it.

[requinix]

Sure, and that is very possible. However you could add something like an htaccess or simple php password to protect it.

Which is basically what Bentley should be doing from the start. Hiding stuff doesn't really work.