DIR Return Create A Forum - Home
---------------------------------------------------------
Create A Forum - Support Forums
HTML https://support.createaforum.com
---------------------------------------------------------
*****************************************************
DIR Return to: General Support
*****************************************************
#Post#: 3176--------------------------------------------------
Re: Web Crawlers
DIR By: Alex P.
Date: January 6, 2012, 9:05 pm
---------------------------------------------------------
They'll be listed under guests. If you see a random guest every
now and then, it's quite possible they are bots. You just have
to check their IPs.
#Post#: 3177--------------------------------------------------
Re: Web Crawlers
DIR By: CB
Date: January 6, 2012, 9:30 pm
---------------------------------------------------------
Is there a difference between real IP's and those of Bot IP's?
What do we look for that tells us one way or another?
#Post#: 3181--------------------------------------------------
Re: Web Crawlers
DIR By: xboi209
Date: January 7, 2012, 1:30 am
---------------------------------------------------------
I can't confirm if this list is valid but this is a list of
known ip addresses for Google bots:
HTML http://www.iplists.com/google.txt
#Post#: 3192--------------------------------------------------
Re: Web Crawlers
DIR By: Alex P.
Date: January 8, 2012, 2:44 pm
---------------------------------------------------------
Click on the IP next to the name Guest, and it should bring up a
"Track IP" page.
Go down to "Look up IP on a regional whois-server" and click on
the bolded link. From here you will be taken to a lookup page
and it will tell you information about that IP address,
including who it is registered to. If it's google or yahoo, etc.
then it will tell you this.
#Post#: 3193--------------------------------------------------
Re: Web Crawlers
DIR By: CB
Date: January 8, 2012, 2:46 pm
---------------------------------------------------------
Thanks.
#Post#: 3262--------------------------------------------------
Re: Web Crawlers
DIR By: CB
Date: January 12, 2012, 6:57 pm
---------------------------------------------------------
I know how to get to and open the pages with regards to the IP
address, but what are the tell tale signs to look for once the
page is opened to know if it's a legit Human looking at the site
and one that is a Bot? I can't make heads or tails from all the
jargon on the page.
#Post#: 3265--------------------------------------------------
Re: Web Crawlers
DIR By: xboi209
Date: January 13, 2012, 7:07 pm
---------------------------------------------------------
We can't really differentiate between human and bot guests, only
by known ip addresses that we know if it's a bot
#Post#: 3266--------------------------------------------------
Re: Web Crawlers
DIR By: CB
Date: January 13, 2012, 7:20 pm
---------------------------------------------------------
--- Quote ---
> ...only by known ip addresses that we know if it's a bot
--- End Quote ---
Really not sure what you mean here. The way it's worded is that
you are able to look at an IP address and know it's a Bot. So,
if that's the case, then what is it in the address that tells
you it's a Bot?
#Post#: 3268--------------------------------------------------
Re: Web Crawlers
DIR By: xboi209
Date: January 13, 2012, 8:23 pm
---------------------------------------------------------
Well some people try to track what ip addresses are Google Bots
for an example. They make a list and we read that list to know
which ip addresses a Google Bot uses. However it ip addresses
change from time to time unfortunately. Here's something I found
on Google's website:
HTML http://support.google.com/webmasters/bin/answer.py?hl=en&answer=80553
*****************************************************
Page 2 of 2
DIR Previous Page