Make crawler look into dynamically created data

[sader]

Hi.

I started to interest about search crawlers and make my sites friendly for them.

 

Can someone point me what technology to use to make search crawler to craw over data such as guestbook entries, blog comments, forum posts I mean data that changes and grows all the time.

 

For example I see in some forums stuff like "Loged users Google[bOT], Yahoo[bOT]"

Ok they regist some special users(bots) in database, but how they make crawler to login and craw over database entries.

 

Should I generate site map file that contains bunch of links or (I find out this stuff fiew hours from now) use "robots.txt" ?

[jackpf]

Creating a sitemap is generally a good idea.

You can get websites that do this periodically for you. Can't remember any of the top of my head. Google will reveal all.

[gffg4574fghsDSGDGKJYM]

 

For example I see in some forums stuff like "Loged users Google[bOT], Yahoo[bOT]"

Ok they regist some special users(bots) in database, but how they make crawler to login and craw over database entries.

 

You can't make googlebot or yahoo to login to your website.

What you can do is server a different page if the client is a robot, serve the whole page, if the client is a browser (human visitor) ask for login to see the page. It's called cloaking.

Cloaking is serving different page for search engine that normal visitor. It's usually used to trick search engine and get higher ranking. It's normally used by black hat seo and have a very bad reputation. But if you are carefull you can use it right.

You will have to disable the google cache or else everyone will be able to see the page in cache and won't have to login.

OR

You can also offer half the page to everyone both robot and human and need a login to see the rest.

OR

Show all your page to everyone, need login only to add a comment/participate.

 

Creating a sitemap is generally a good idea.

You can get websites that do this periodically for you. Can't remember any of the top of my head. Google will reveal all.

 

For the sitemap i disagree with jackpf and many seo on that point. I don't think it's a good idea at all. Let me explain, if you use a sitemap and a page get into the search engine, it may not rank and it will be tricky to find out why. The goal of SEO isn't to get indexed is to rank. If a page or a website doesn't get indexed and rank it's a external or internal backlink problem there and you won't solve it by adding a sitemap. The sitemap may get the page indexed and it will lure you into thinking the SEO is right when it's not. A good external and internal backlinks structure won't have any problem getting indexed and ranking and won't ever need a sitemap. It's valid for both XML and HTML sitemap.

 

A page indexed by a sitemap that don't rank is useless (and will probably be removed from the index later since no one link it), work to get ranked by creating a good backlink structure instead.

 

Google and yahoo don't talk a lot. It give you precious information when it don't index your page, they tell you that something is wrong. If you always add a sitemap with all page in there, you lost that information.