Page Index Toggle Pages: 1 Send TopicPrint
Normal Topic Search Engine Identifiers (Read 6096 times)
Red Barchetta
New Member
*
Offline



Posts: 46
Location: Miami, FL. USA
Joined: Oct 4th, 2014
Gender: Male
Re: Search Engine Identifiers
Reply #7 - Dec 6th, 2014 at 2:01am
Print Post  
This is related, and possibly has more applications? GeoLite Legacy Downloadable Databases are updated once a month. How interesting it would be to have a bot or crawler's location automatically identified? One step further, how about a new user's location (country) automatically being entered into the application? 

Maybe its just something I am interested in as my forum is geared more toward local users, but I'll just toss this out there to see if anyone else has any thoughts or interest, like in YAMMS.
  

Florida Classics and Muscle Car Automotive Forum Administrator
Back to top
WWW  
IP Logged
 
Red Barchetta
New Member
*
Offline



Posts: 46
Location: Miami, FL. USA
Joined: Oct 4th, 2014
Gender: Male
Re: Search Engine Identifiers
Reply #6 - Nov 22nd, 2014 at 10:15pm
Print Post  
You can get the SE, Bot or Crawler's name from here:

http://udger.com/resources/ua-list/crawlers-ip

This is the source that I have used to expand my list and identify the items in the previous list.
  

Florida Classics and Muscle Car Automotive Forum Administrator
Back to top
WWW  
IP Logged
 
Red Barchetta
New Member
*
Offline



Posts: 46
Location: Miami, FL. USA
Joined: Oct 4th, 2014
Gender: Male
Re: Search Engine Identifiers
Reply #5 - Nov 22nd, 2014 at 10:12pm
Print Post  
I have also expanded my Search Engine list:

4seohunt|4SeoHuntBot
aesop|AESOP_SpiderMan
abacho|AbachoBOT
acoon|Acoon Robot
boson027|AhrefsBot/5.0
ahrefs|AhrefsBot/5.0
ia_archiver|Alexa
alexa|Alexa archiver
vestris|AlkalineBOT
altavista|AltaVista
scooter|AltaVista
sv.av|AltaVista
tarantula|AltaVista
alta-vista|Altavista
av|Altavista
apercite.fr|Apercite
aport|Aport
girafa|Aranha
archiver-web|Archive.org
ask|Ask Jeeves
askjeeves|Ask Jeeves
directhit|Ask Jeeves
teoma|Ask Jeeves
atomz|Atomz
axmo|AxmoRobot
baidu|Baidu
baiduspider|Baidu
net263|Baidu
buscaplus|Buscaplus Robi
ip3000|C-PBWF-ip3000-crawler
canseek|CanSeek
christcrawler|ChristCRAWLER
clush|Clushbot
crawler|Crawler
pinpoint|CrawlerBoy
powerinter|DIIbot
daadle|DaAdLe ROBOT
deepindex|DeepIndex
ditto|DittoSpyder
dotbot|Dotbot Research
dotnetdotcom|Dotbot Research
earthcom|EARTHCOM
travel-finder|ESISmartSpider
ezresults|EZResult
eurip|EuripBot
muscat|EuroFerret
arachnoidea|EuroSeek
euroseek|EuroSeek Arachnoidea
exabot|Exava
architext|Excite
atext|Excite
excite|Excite ArchitextSpider
alltheweb|FAST-WebCrawler
fastsearch|Fast Crawler
yelo.no|Findexa Crawler
searchhippo|Fluffy the spider
fybersearch|FyberSearch
galaxy|GalaxyBot
gendoor|GenCrawler
genieo|Genieo/1.0
geona|GeonaBot
gigabot|Gigablast
backrub|Google
google|Google
googlebot|Googlebot
mirago|HenryTheMiragoRobot
inktomi|HotBot
inktomisearch|Hotbot
hubat|Hubater
istarthere|I Start here
igde|Igde
iltrovatore|IlTrovatore-Setaccio
incywincy|IncyWincy
infoseek|InfoSeek
infoseeksidewinder|InfoSeek
ultraseek|InfoSeek
verno.ueda.info.waseda.ac.jp|Iron33
domanova|Jack
joocer|JoocerBot
fireball|KIT-Fireball
knowledge|Knowledge
linkfluence|Kraken
lexis-nexis|LNSpiderguy
ActiveBookmark|Link Checker, Monitor
ALink|Link Checker, Monitor
AMeta|Link Checker, Monitor
ASPSearch|Link Checker, Monitor
BlogBot|Link Checker, Monitor
BMChecker|Link Checker, Monitor
Bookmark|Link Checker, Monitor
Check&Get|Link Checker, Monitor
CheckWeb|Link Checker, Monitor
CNET_Snoop|Link Checker, Monitor
DRKSpider|Link Checker, Monitor
DISCo Watchman|Link Checker, Monitor
DoctorHTML|Link Checker, Monitor
EmailSiphon|Link Checker, Monitor
EmailWolf|Link Checker, Monitor
FavOrg|Link Checker, Monitor
FreshLinks|Link Checker, Monitor
HTMLParser|Link Checker, Monitor
InternetLinkAgent|Link Checker, Monitor
InternetPeriscope|Link Checker, Monitor
javElink|Link Checker, Monitor
jdwhatsnew|Link Checker, Monitor
Lambda|Link Checker, Monitor
LinkAlarm|Link Checker, Monitor
Linkbot|Link Checker, Monitor
Linkman|Link Checker, Monitor
LinkProver|Link Checker, Monitor
LinkScan|Link Checker, Monitor
LinkSweeper|Link Checker, Monitor
LinkVerify|Link Checker, Monitor
LinkWalker|Link Checker, Monitor
MoveAnnouncer|Link Checker, Monitor
mylinkcheck|Link Checker, Monitor
NetLookout|Link Checker, Monitor
NetMechanic|Link Checker, Monitor
elsop|Link Checker, Monitor
netmechanic|Link Checker, Monitor
NetMind-Minder|Link Checker, Monitor
marvin.netmind|Link Checker, Monitor
gary.netmind|Link Checker, Monitor
meg.netmind|Link Checker, Monitor
inyanga.netmind|Link Checker, Monitor
leo.netmind|Link Checker, Monitor
gemini.netmind|Link Checker, Monitor
NetMonitor|Link Checker, Monitor
Netprospector|Link Checker, Monitor
Rational|Link Checker, Monitor
Robozilla|Link Checker, Monitor
SiteBar|Link Checker, Monitor
SpurlBot|Link Checker, Monitor
SurfMaster|Link Checker, Monitor
SyncIT|Link Checker, Monitor
Watchfire|Link Checker, Monitor
WatzNew|Link Checker, Monitor
WebSite-Watcher|Link Checker, Monitor
WebTrends|Link Checker, Monitor
Weblink|Link Checker, Monitor
Xenu's Link Sleuth|Link Checker, Monitor
Z-Add Link Checker|Link Checker, Monitor
ActiveBookmark|Link Checker, Monitor
ALink|Link Checker, Monitor
AMeta|Link Checker, Monitor
ASPSearch|Link Checker, Monitor
BlogBot|Link Checker, Monitor
BMChecker|Link Checker, Monitor
Bookmark|Link Checker, Monitor
Check&Get|Link Checker, Monitor
CheckWeb|Link Checker, Monitor
CNET_Snoop|Link Checker, Monitor
DRKSpider|Link Checker, Monitor
DISCo Watchman|Link Checker, Monitor
DoctorHTML|Link Checker, Monitor
EmailSiphon|Link Checker, Monitor
EmailWolf|Link Checker, Monitor
FavOrg|Link Checker, Monitor
FreshLinks|Link Checker, Monitor
HTMLParser|Link Checker, Monitor
InternetLinkAgent|Link Checker, Monitor
InternetPeriscope|Link Checker, Monitor
javElink|Link Checker, Monitor
jdwhatsnew|Link Checker, Monitor
Lambda|Link Checker, Monitor
LinkAlarm|Link Checker, Monitor
Linkbot|Link Checker, Monitor
Linkman|Link Checker, Monitor
LinkProver|Link Checker, Monitor
LinkScan|Link Checker, Monitor
LinkSweeper|Link Checker, Monitor
LinkVerify|Link Checker, Monitor
LinkWalker|Link Checker, Monitor
MoveAnnouncer|Link Checker, Monitor
mylinkcheck|Link Checker, Monitor
NetLookout|Link Checker, Monitor
NetMechanic|Link Checker, Monitor
elsop|Link Checker, Monitor
netmechanic|Link Checker, Monitor
NetMind-Minder|Link Checker, Monitor
marvin.netmind|Link Checker, Monitor
gary.netmind|Link Checker, Monitor
meg.netmind|Link Checker, Monitor
inyanga.netmind|Link Checker, Monitor
leo.netmind|Link Checker, Monitor
gemini.netmind|Link Checker, Monitor
NetMonitor|Link Checker, Monitor
Netprospector|Link Checker, Monitor
Rational|Link Checker, Monitor
Robozilla|Link Checker, Monitor
SiteBar|Link Checker, Monitor
SpurlBot|Link Checker, Monitor
SurfMaster|Link Checker, Monitor
SyncIT|Link Checker, Monitor
Watchfire|Link Checker, Monitor
WatzNew|Link Checker, Monitor
WebSite-Watcher|Link Checker, Monitor
WebTrends|Link Checker, Monitor
Weblink|Link Checker, Monitor
Xenu's|Link Checker, Monitor
Z-Add|Link Checker, Monitor
LinkLint-checkonly|Link Checker, Monitor/
LinkLint-checkonly|Link Checker, Monitor/
linklint-checkonly|LinkLint.org
linklint|LinkLint.org
linknz|Linknzbot
magma|LookBot
fuzine.mt.cs.cmu.edu|Lycos
lycos|Lycos_Spider_(T-Rex)
majestic12|MJ12bot/v1.3.3
mp3bot|MP3Bot
msnbot-media|MSN Search
msnbot|MSN Search
search.msn|MSN Search
looksmart|MantraAgent
search.live|Microsoft Live Search
mojeek|MojeekBot
intags|Mole
webtop|MuscatFerret
nationaldirectory|NationalDirectory-SuperSpider
navadoo|Navadoo Crawler
websmostlinked|Nazilla
loopimprovements|NetResearchServer
northernlight|Northern Light Gulliver
objectssearch|ObjectsSearch
omgilibot|Omgili
szukaj|OnetSzukaj
openfind|Openfind piranha,Shark
portaljuice|PJspider
picsearch|PicSearchBot
picosearch|PicoSearch
plonebot|Plone Spambot
qweery|QweeryBot
daum|RaBot
supersnooper|Robot@SuperSnooper
scoutjet|ScoutJet
scrubtheweb|Scrubby
search4free|Search 4 Free
search-10|Search-10
searchbyusa|SearchByUsa
charlotte|SearchMe Visual Search
searchme|SearchMe Visual Search
searchspider|Searchspider
seznam|SeznamBot
sightquest|SightQuestBot
similarpages|Similar Pages
Slurp|Slurp
sogou|Sogou
entireweb|Speedy Spider
sphere|Sphere Scout
traficdublu|Spider TraficDublu
maxbot|Spider/maxbot
spidermonkey|Spider_Monkey
rambler|StackRambler
surfnomore|Surfnomore Spider
mapper.teradex|Teradex_Mapper
hoppa|Toutatis
tutorgig|Tutorial Crawler
cuill|Twiceler
twiceler|Twiceler
uksearcher|UK Searcher Spider
vivante|Vivante Link Checker
orange-ftpgroup|Voila Spambot
voila|Voila Spambot
voilabot|Voila Spambot
80legs|Voltron
wasalive|WASALive-Bot
wire.co.uk|WIRE WebRefiner
worldsearchcenter|WSCbot
webalta|WebAlta Crawler
webcrawler|WebCrawler
webwombat|WebWombat
whizbanglabs|WhizBang! Lab
wisewire|WiseWire
yourbettersearch|YBSbot search engine indexer
yahoo-mmcrawler|Yahoo!
yahoo|Yahoo!
yandex|Yandex
yanga|Yanga WorldSearch
nhn|Yeti
yeti|Yeti
yetiBot|Yeti
naver|Yeti(NaverRobot)
youdao|Youdao
youdaobot|Youdao
wisenut|ZyBorg
abcdatos|abcdatos_botlink
ah-ha|ah-ha crawler
almaden|almaden crawler
antisearch|antibot
walhello|appie
singingfish|asteris singingfish
crawl.baidu|baiduspider
france.misesajour|france.misesajour
geckobot|geckobot
getrax|getRAX
petersnews|ip3000
kuloko|kuloko-bot
look|lookbot
webseek|marvin/infoseek
mozdex|mozDex(ComCast)
noxtrum|noxtrumbot
navi.ocn.ne.jp|nttdirectory_robot
speedfind|speedfind ramBot xtreme
whatuseek|whatUseek
winona|whatUseek
yacy.net|yacybot
  

Florida Classics and Muscle Car Automotive Forum Administrator
Back to top
WWW  
IP Logged
 
Red Barchetta
New Member
*
Offline



Posts: 46
Location: Miami, FL. USA
Joined: Oct 4th, 2014
Gender: Male
Re: Search Engine Identifiers
Reply #4 - Nov 22nd, 2014 at 10:10pm
Print Post  
I have started adding IP blocks to my firewall when ever I find the "The User ID you specified does not exist or you entered a wrong password." error right after the "Slider Captcha: You have failed the safety function Slider Captcha!" error. Its amazing how many "Guests" have disappeared from my forum. Sofar I have blocked these, mostly from Russia and China.

213.252.170.254,222.186.21.1-222.186.21.254
  

Florida Classics and Muscle Car Automotive Forum Administrator
Back to top
WWW  
IP Logged
 
Red Barchetta
New Member
*
Offline



Posts: 46
Location: Miami, FL. USA
Joined: Oct 4th, 2014
Gender: Male
Re: Search Engine Identifiers
Reply #3 - Nov 2nd, 2014 at 5:07pm
Print Post  
My research has shown me, that most of my visitors are actually search engines.   Sad

"AhrefsBot/5.0 requests a new password."

huh?
  

Florida Classics and Muscle Car Automotive Forum Administrator
Back to top
WWW  
IP Logged
 
Dandello
Forum Administrator
*****
Offline


I love YaBB 2.7!

Posts: 1759
Location: The Land of YaBB
Joined: Feb 12th, 2014
Gender: Female
Re: Search Engine Identifiers
Reply #2 - Oct 26th, 2014 at 4:31am
Print Post  
That looks like a promising addition to the honeypot. Thanks  Smiley
  

Perfection is not possible. Excellence, however, is excellent.
Back to top
WWW  
IP Logged
 
Red Barchetta
New Member
*
Offline



Posts: 46
Location: Miami, FL. USA
Joined: Oct 4th, 2014
Gender: Male
Re: Search Engine Identifiers
Reply #1 - Oct 26th, 2014 at 4:23am
Print Post  
  

Florida Classics and Muscle Car Automotive Forum Administrator
Back to top
WWW  
IP Logged
 
Red Barchetta
New Member
*
Offline



Posts: 46
Location: Miami, FL. USA
Joined: Oct 4th, 2014
Gender: Male
Search Engine Identifiers
Oct 26th, 2014 at 4:23am
Print Post  
Here is a thought. Would it be possible to have bots automatically added to the "Search Engine Identifiers" list using the bot trap in YABB? This way unknown or unlisted bots can be identified and added to the Search Engine list instead of the Guest List. I got this idea from a previous message about the bot trap here on YABB, and on Elxsy. (Link will be posted later when I have access).
  

Florida Classics and Muscle Car Automotive Forum Administrator
Back to top
WWW  
IP Logged
 
Page Index Toggle Pages: 1
Send TopicPrint