Jump to content


Photo

Web robots


  • Please log in to reply
4 replies to this topic

#1 otuatail

otuatail
  • Members
  • PipPipPip
  • Advanced Member
  • 960 posts

Posted 19 September 2006 - 03:02 PM

I wanted to get my website indexed by robots, but having seen some articles on the website this looks unlikely. According to some information some engines do not look at meta tags only the content. They avoid sites that have re-directions and cannot handle sites with a database backend as they cant access the data. This looks like they can see the WebPages at source on the server, not as pure html. They prefer to see the information at the top of the page. I have php at the top of mine. I have a session variable so I can create a hit counter, a simple redirection to under-construction so I can do development. A called function, to see if it is a robot and avoid a hit count. On top off all this I collect data to see which page has been visited. It appears that all this PHP is getting in the way. If I want my site indexed I have to strip out all code.

I have been asked to do a business site but don’t think I can get it on a search engine. Is there any way around this, or do I do strait html.


Paul.


#2 Daniel0

Daniel0
  • Staff Alumni
  • Advanced Member
  • 11,956 posts

Posted 19 September 2006 - 03:07 PM

They can't see the actual source of the script. It's not true that they don't like webpages with databases. They can only see what you see if you view the HTML source after it has been interpreted by PHP.

#3 SharkBait

SharkBait
  • Members
  • PipPipPip
  • Advanced Member
  • 845 posts
  • LocationMetro Vancouver, BC

Posted 19 September 2006 - 03:21 PM

Web crawlers are pretty much like automated browsers. They see what we see.  Though they pick up on META tags and keywords etc within the content of the website.

They cannot see the source code, they can't see PHP/ASP, just like a user cannot see your source code.

There is talk about how the crawlers don't like to index pages with variables in the URL ie: http://blah.com/scri...&view=thisplace but I've seen lots of pages that are indexed that way with google and yahoo, so they must do it, they might just perfer the http://blah.com/scri...p/asd/thisplace better.

Submit your site to the big search engines.  Make sure there is alot of content in the pages you provide. Also make sure things images etc have title/alt tags associated to them too. Also ensure the content on your pages actual makes sense to the rest of the site.  Google likes to stop indexing your site if you use words that are not relevent to the rest of the site. IE having and white background and below your footer adding white text with popular search tems like teen, sex, etc when your site is about flowers ;)



#4 otuatail

otuatail
  • Members
  • PipPipPip
  • Advanced Member
  • 960 posts

Posted 19 September 2006 - 03:34 PM

But a statment made said if they see a re-direction they will refuse because they think they are being conned. If they can't see php they can't see the re-direction.

If i wanted to force there hand i could (Knowing they are robots) deliver a difrent page to them and they wouldent know. (A page with cmore compresed information for them to see.)


#5 Destruction

Destruction
  • Members
  • PipPipPip
  • Advanced Member
  • 108 posts

Posted 19 September 2006 - 03:50 PM

"If they can't see php they can't see the re-direction"

Yes, they can.  A redirection, perhaps with header("Location: index.php") for example, sends a 302 HTTP Response Code to the browser with details for the redirect if I remember correctly.  Therefore the browser can see it, meaning the robots can also.

Dest




0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users