User Agent Question

[blacksnday]

Normally when viewing User Agents for Browsers, it would look similar to:
[b]Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)[/b]

My question is, if it shows by first saying [b]User-Agent:[/b]
such as:
[b]User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)[/b]
would you believe it was real or fake, or do certain things really put
[b]User-Agent:[/b] within the UA id?

My reason for this question is to either block or allow it.
Within the past few days been recieving hits with that type of UA
and the few different IPs showing that UA have been basiclly
loading up to 10 pages per second
which makes me think it is not a valid UA.

[ober]

Not sure.  Keep in mind that some browsers (opera) can easily spoof the UA.  It's a shame, but some sites will not work because the webmaster actually goes out of his way to block UAs other than IE.  I run around half the time identifying opera as IE because of this.

[blacksnday]

Yea, FF also offers a plugin to change the UA.
My IP/UA blocker I use with extreme caution, hence the question.
The way I block UA's are based on Keywords such as:
WebWhacker, WWWCopy, BackStreet Browser etc...

And alot of programs use same UA and cannot be changed such as stuff like:
Java and
WebCapture (Adobe Acrobat web grabbing for preserving as a PDF) etc..

I dont actually ban by Full UA... just keywords that shouldnt be in the UA which can
help show its Bad.. and which still allows other stuff like
RssFwd to work with no probs :P

which brings to try and figure out if User-Agent is good or bad
because why would a UA string tell you it's a User-Agent when it is already known to be?
hrmm..

[ober]

Good question.  One I certainly don't have an answer for.

[shoz]

[quote author=blacksnday link=topic=107233.msg429909#msg429909 date=1157639496]
Normally when viewing User Agents for Browsers, it would look similar to:
[b]Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)[/b]

My question is, if it shows by first saying [b]User-Agent:[/b]
such as:
[b]User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)[/b]
would you believe it was real or fake, or do certain things really put
[b]User-Agent:[/b] within the UA id?

My reason for this question is to either block or allow it.
Within the past few days been recieving hits with that type of UA
and the few different IPs showing that UA have been basiclly
loading up to 10 pages per second
which makes me think it is not a valid UA.
[/quote]

It's probably an error in the bot that's being used to access the page. The "User-Agent" string should be part of the Headers "field name" and not a part of the value. The application is probably sending the header in the way shown below.
[code]
User-Agent: User-Agent: ...
[/code]

[blacksnday]

[quote author=shoz link=topic=107233.msg429950#msg429950 date=1157641942]
It's probably an error in the bot that's being used to access the page. The "User-Agent" string should be part of the Headers "field name" and not a part of the value. The application is probably sending the header in the way shown below.
[code]
User-Agent: User-Agent: ...
[/code]
[/quote]

So basically I would be safe to assume it is a Bot when it appears this way?
As the logs show it sure acts like a bot when loading 10+pages in less then 1second.

[shoz]

You can't really know that it's a bot based on the User-Agent string, but it can be a good guess. Some individuals download premade bots that have hardcoded/default User Agent strings and never bother (or don't know) to change them.

I don't know of any bugs in any apps MSIE or otherwise that put the "User Agent string in the header value but it's not impossible. So I wouldn't block on the user agent string you posted, but you'll have to decide for yourself.

[quote=blacksnday]
the few different IPs showing that UA have been basiclly
loading up to 10 pages per second
[/quote]

The behaviour that you describe above would be a better reason to decide that it's a bot.

If you're going to block anything you'll have to decide why you're doing it in the first place. Is there any real reason to [b]try[/b] to block the bots from accessing the page?

Blocking based on a rule saying for example that if an ip accesses more than x number of pages within x time block for x minutes is ok, but you'll probably want to make exceptions for google yahoo etc (By IP/User-Agent or using some other method). Depending on why you're doing it, it may not be worth the trouble.

There may be other ways to do something about it but nothing comes to mind at the moment.

[blacksnday]

[quote author=shoz link=topic=107233.msg429967#msg429967 date=1157644079]Blocking based on a rule saying for example that if an ip accesses more than x number of pages within x time block for x minutes is ok, but you'll probably want to make exceptions for google yahoo etc (By IP/User-Agent or using some other method). Depending on why you're doing it, it may not be worth the trouble.[/quote]

Currently I dont block/ban based on
ip accesses more than x number of pages within x time block for x minutes
(even though I got some alpha-type code coming for that soon)

At this time I probably will block the UA with User-Agent
and since I track all bans/who was blocked by what ban/what UA they had, etc...
I will be able to better determine at a later date if 'Good' people are being wrongly banned.

Thanks for the help!