Issue: Reported # Of Clicks Are Much Higher Than Expected Options

[Arlen Coupland]

Hello,

*edit* - this subject has some important updates, please see below

Summary of Issue ::
It has been found that there are a few robots/spiders which do not identify themselves in a common manner and will ignore the ROBOTS.txt file. This can result in clicks from these 'rogue' robots/spiders to be tracked, which can result in higher click stats than is expected.


Who is affected ::
If you find that your CTR (Click Through Ratio) is of an overly-high number, you are most likely affected. Only sites which are targeted by certain spiders will experience such a result. The majority of spiders, which can be easily identified, should already be properly blocked from click results.
This issue will not normally cause a small difference in click statistics - the difference would be relatively large and easily noticed compared to normal click statistics.

Blocking clicks from additional spiders ::

First off, you should make sure that a robots.txt file is placed in the root directory of OpenX. OpenX has already included robot files inside certain folders, however for these to work you must have your domain pointed directly at the OpenX directory or the OpenX admin/delivery directories.

If you have installed OpenX into a subdirectory of your domain ( www.example.com/openx ) then you will need to place a copy of robots.txt into the root directory ( www.example.com/robots.txt )

The content of robots.txt in the root directory of OpenX is:
User-agent: *
Disallow: /
You want to move this file into the root directory of your domain, and change the 'Disallow' part to match the foldername you use for OpenX
User-agent: *
Disallow: /openx/

However, not all spiders/bots follow the rules of robots.txt - but this will certainly help.

For the rogue bots :::

The main test to do is to query your raw data to look for any IPs which have many clicks.

To do this, execute the following query on your database:

CODE

SELECT ip_address, user_agent, COUNT( * ) AS count
FROM data_raw_ad_click
GROUP BY ip_address
HAVING count >5

You will want to make sure that "FROM data_raw_ad_click" uses the actual name for your table - you might need to add a prefix like 'oa_data_raw_ad_click' or 'phpads_data_raw_ad_click'

You can also change the 'count > 5' line. It means it is looking for any IP with more than 5 clicks.

The result of this query will show you the IPs and user_agents (browser details) of any IP with more than 5 clicks. This can help identify any IP which might be causing click-inflation.

You can then enter any IP you wish into the "IP blocking" feature, which is located at:
Settings -> Main Settings -> Statistics and Maintenance Settings , called "Global Prevent Statistics Logging Settings"
Please note: If fields on this settings page are greyed out for you, please upgrade to Openads 2.4.3

Enter each IP onto a new line. You can enter the exact IP or you can replace a number with a wildcard '*' character, like: 127.0.0.*

It is best to block only the IPs which seem to be causing an issue for your adserver.

[Arlen Coupland]

Below I will attach a list of IPs of known spiders. The list is not meant as a list of IPs you should automatically place in the blocked IP list. Doing so will result in a very long list and could cause slower delivery.

You should discover your own list as described above. The list which follows is only intended as a reference point. It is not expected that all of the IPs on the list will cause click-inflation: Most will not.

Also, the IPs and exact user_agents will not remail static as the companies may change their settings.

[Arlen Coupland]

*edit* -- before using this list, see the new posts below.

Google 72.14.199.*
AdsBot-Google (+http://www.google.com/adsbot.html)

Walhello 80.60.35.*
appie 1.1 (www.walhello.com)

Walhello 81.205.39.*
appie 1.1 (www.walhello.com)

Walhello 84.104.216.*
appie 1.1 (www.walhello.com)

Walhello 84.104.217.*
appie 1.1 ( www.walhello.com)

Walhello 84.104.39.*
appie 1.1 (www.walhello.com)

Walhello 84.104.40.*
appie 1.1 (www.walhello.com)

Walhello 84.104.43.*
appie 1.1 (www.walhello.com)

Brandimensions 204.92.59.*
BDFetch

Brandimensions 216.183.91.*
BDFetch

Brandimensions 64.26.160.*
BDFetch

Brandimensions 70.25.237.*
BDFetch

Brandimensions 72.14.163.*
BDFetch

Brandimensions 74.97.49.*
BDFetch

Intelliseek 64.158.138.*
Blogpulse (info@blogpulse.com)

Commission Junction 216.34.209.*
CJNetworkQuality; http://www.cj.com/networkquality

Commission Junction 64.70.54.*
CJNetworkQuality; http://www.cj.com/networkquality

Google 84.58.171.*
CSCrawler/http://www.kde.cs.uni-kassel.de/lehre/ss2005/googlespam/crawler.html RPT-HTTPClient/0.3-3

Yahoo! 202.93.76.*
DoCoMo/2.0/SO502i (compatible; Mozilla 4.0; MSIE 6.0; yahooseeker-jp-mobile AT Yahoo!JAPAN)

Envolk 98.173.26.*
envolk/1.7 (+http://www.envolk.com/envolkspiderinfo.html)

Envolk 68.108.249.*
envolk/1.7 (+http://www.envolk.com/envolkspiderinfo.php)

Envolk 68.6.111.*
envolk/1.7 (+http://www.envolk.com/envolkspiderinfo.php)

Envolk 70.169.191.*
envolk/1.7 (+http://www.envolk.com/envolkspiderinfo.php)

Envolk 68.6.101.*
envolk[ITS]spider/1.6 (+http://www.envolk.com/envolkspider.html)

Envolk 71.102.140.*
envolk[ITS]spider/1.6 (+http://www.envolk.com/envolkspider.html)

Envolk 71.102.160.*
envolk[ITS]spider/1.6 (+http://www.envolk.com/envolkspider.html)

FAST 213.3.44.*
FAST Enterprise Crawler 6 used by EPFL ( raul.rabionet@epfl.ch)

FAST 217.114.115.*
FAST Enterprise Crawler 6 used by EPFL ( raul.rabionet@epfl.ch)

FAST 216.255.229.*
FAST Enterprise Crawler 6 used by FAST (iverjor (at) fast.no)

FAST 219.165.118.*
FAST Enterprise Crawler 6 used by FAST (iverjor (at) fast.no)

FAST 12.32.25.*
FAST Enterprise Crawler 6 used by GDSN (e_gfc_it_gdsn-fast@gdsn-home.net )

FAST 193.252.242.*
FAST Enterprise Crawler 6 used by Pages Jaunes (crawladmin@gmail.com )

FAST 81.93.164.*
FAST Enterprise Crawler 6 used by Schibsted (webcrawl@schibstedsok.no )

FAST 202.176.218.*
FAST Enterprise Crawler 6 used by Singapore Press Holdings ( crawler@sphsearch.sg)

FAST 202.176.220.*
FAST Enterprise Crawler 6 used by Singapore Press Holdings ( crawler@sphsearch.sg)

FAST 64.151.114.*
FAST Enterprise Crawler 6 used by TauMed ( crawler@taumed.com)

FAST 131.165.88.*
FAST Enterprise Crawler 6 used by Virk.dk - udvikling ( thomas.bentzen@capgemini.com)

FAST 69.25.71.*
FAST Enterprise Crawler/6.4 (helpdesk at fast.no)

Findexa 80.239.38.*
Findexa Crawler ( http://www.findexa.no/gulesider/article26548.ece)

Gigablast 38.114.104.*
Gigabot/1.0

Gigablast 63.225.19.*
Gigabot/1.0

Gigablast 64.139.94.*
Gigabot/1.0

Gigablast 70.90.210.*
Gigabot/1.0

Gigablast 74.94.177.*
GigaBot/1.0

Gigablast 66.154.102.*
Gigabot/2.0

Gigablast 66.231.188.*
Gigabot/2.0 (http://www.gigablast.com/spider.html)

Gigablast 66.231.189.*
Gigabot/2.0 (http://www.gigablast.com/spider.html)

Gigablast 131.161.247.*
Gigabot/2.0/gigablast.com/spider.html

Gigablast 66.154.103.*
Gigabot/2.0/gigablast.com/spider.html

Google 71.162.124.*
Google Spider

Google 66.249.73.*
Googlebot-Image/1.0

Google 125.177.35.*
Googlebot/2.1 (+http://www.google.com/bot.html)

Google 207.44.196.*
Googlebot/2.1 (+http://www.google.com/bot.html)

Google 222.238.81.*
Googlebot/2.1 (+http://www.google.com/bot.html)

Google 66.249.64.*
Googlebot/2.1 (+http://www.google.com/bot.html)

Google 66.249.71.*
Googlebot/2.1 (+http://www.google.com/bot.html)

Google 65.57.245.*
gsa-crawler (Enterprise; GIX-02530; enterprise-training@google.com )

Google 64.233.178.*
gzip(gfe) (via translate.google.com)

Google 141.51.167.*
Hi! I'm CsCrawler, my homepage: http://www.kde.cs.uni-kassel.de/lehre/ss20...am/crawler.html

Google 84.58.174.*
Hi! I'm CsCrawler, my homepage: http://www.kde.cs.uni-kassel.de/lehre/ss20...am/crawler.html

Google 84.58.96.*
Hi! I'm CsCrawler, my homepage: http://www.kde.cs.uni-kassel.de/lehre/ss20...am/crawler.html

Yahoo! 218.132.56.*
larbin_2.6.3 (ewaina_2003@email.yahoo.com)

Yahoo! 213.91.216.*
larbin_2.6.3 (vesso1221yahoo.com)

SEVENtwentyfour 209.167.50.*
LinkWalker

Lycos 209.202.205.*
Lycos_Spider_(modspider)

Lycos 209.202.229.*
Lycos_Spider_(modspider)

Lycos 209.202.248.*
Lycos_Spider_(modspider)

Google 66.249.65.*
Mediapartners-Google/2.1

Google 66.249.70.*
Mediapartners-Google/2.1

Google 66.249.72.*
Mediapartners-Google/2.1

MIVA 66.150.55.*
Miva (AlgoFeedback@miva.com)

Google 64.178.97.*
Mozilla 4.0(compatible; Googlebot/5.0; +http://www.google.com/googlebot)

Google 24.7.232.*
Mozilla/4.0 (compatible; GoogleToolbar 4.0.1020.2544-big; Windows XP 5.1; MSIE 6.0.2900.2180)

Google 82.47.1.*
Mozilla/4.0 (compatible; GoogleToolbar 4.0.1020.2544-big; Windows XP 5.1; MSIE 6.0.2900.2180)

MSN 68.193.180.*
Mozilla/4.0 (compatible; MSIE 6.0; MSN 2.5; Windows 98; MSIECrawler)

Yahoo! 60.10.2.*
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1) (m_pupil@yahoo.com.cn)

LookSmart 64.242.88.*
Mozilla/4.0 compatible ZyBorg/1.0 Dead Link Checker ( wn.dlc@looksmart.net; http://www.WISEnutbot.com)

Google 66.249.66.*
Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

Google 66.249.67.*
Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

Google 85.114.130.*
Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

Google 66.230.157.*
Mozilla/5.0 (compatible; googlebot/2.1; +http://www.google.com/bot.html) (googlebot@google.com)

Yahoo! 125.22.91.*
Mozilla/5.0 (compatible; heritrix/1.6.0 +http://www.yahoo.com)

Yahoo! 202.160.178.*
Mozilla/5.0 (compatible; Yahoo! Slurp China; http://misc.yahoo.com.cn/help.html)

Yahoo! 202.160.181.*
Mozilla/5.0 (compatible; Yahoo! Slurp China; http://misc.yahoo.com.cn/help.html)

Yahoo! 202.160.185.*
Mozilla/5.0 (compatible; Yahoo! Slurp China; http://misc.yahoo.com.cn/help.html )

Yahoo! 202.160.179.*
Mozilla/5.0 (compatible; Yahoo! Slurp/si-emb; http://help.yahoo.com/help/us/ysearch/slurp)

Yahoo! 66.228.165.*
Mozilla/5.0 (compatible; Yahoo! Slurp/si-emb; http://help.yahoo.com/help/us/ysearch/slurp)

Yahoo! 202.160.180.*
Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

Yahoo! 202.160.183.*
Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

Yahoo! 217.11.235.*
Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

Yahoo! 66.196.101.*
Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

Yahoo! 66.196.65.*
Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

Yahoo! 66.196.77.*
Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

Yahoo! 66.196.90.*
Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

Yahoo! 66.196.91.*
Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

Yahoo! 66.196.92.*
Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

Yahoo! 66.228.167.*
Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

Yahoo! 68.142.246.*
Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

Yahoo! 68.142.249.*
Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

Yahoo! 68.142.250.*
Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

Yahoo! 68.142.251.*
Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

Yahoo! 68.180.251.*
Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

Yahoo! 72.30.101.*
Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

Yahoo! 72.30.102.*
Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

Yahoo! 72.30.103.*
Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

Yahoo! 72.30.104.*
Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

Yahoo! 72.30.107.*
Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

Yahoo! 72.30.110.*
Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

Yahoo! 72.30.111.*
Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

Yahoo! 72.30.128.*
Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

Yahoo! 72.30.129.*
Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

Yahoo! 72.30.131.*
Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

Yahoo! 72.30.132.*
Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

Yahoo! 72.30.133.*
Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

Yahoo! 72.30.134.*
Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

Yahoo! 72.30.135.*
Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

Yahoo! 72.30.177.*
Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

Yahoo! 72.30.214.*
Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

Yahoo! 72.30.215.*
Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

Yahoo! 72.30.216.*
Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

Yahoo! 72.30.221.*
Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

Yahoo! 72.30.226.*
Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

Yahoo! 72.30.252.*
Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

Yahoo! 72.30.54.*
Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

Yahoo! 72.30.56.*
Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

Yahoo! 72.30.61.*
Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

Yahoo! 72.30.81.*
Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

Yahoo! 72.30.97.*
Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

Yahoo! 72.30.98.*
Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

Yahoo! 72.30.99.*
Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

Yahoo! 74.6.130.*
Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

Yahoo! 74.6.131.*
Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

Yahoo! 74.6.17.*
Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

Yahoo! 74.6.18.*
Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

Yahoo! 74.6.19.*
Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

Yahoo! 74.6.20.*
Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

Yahoo! 74.6.22.*
Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

Yahoo! 74.6.23.*
Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

Yahoo! 74.6.24.*
Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

Yahoo! 74.6.25.*
Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

Yahoo! 74.6.26.*
Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

Yahoo! 74.6.27.*
Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

Yahoo! 74.6.28.*
Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

Yahoo! 74.6.29.*
Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

Yahoo! 74.6.65.*
Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

Yahoo! 74.6.66.*
Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

Yahoo! 74.6.67.*
Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

Yahoo! 74.6.68.*
Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

Yahoo! 74.6.69.*
Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

Yahoo! 74.6.70.*
Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

Yahoo! 74.6.71.*
Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

Yahoo! 74.6.72.*
Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

Yahoo! 74.6.73.*
Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

Yahoo! 74.6.74.*
Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

Yahoo! 74.6.75.*
Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

Yahoo! 74.6.76.*
Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

Yahoo! 74.6.8.*
Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

Yahoo! 74.6.85.*
Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

Yahoo! 74.6.86.*
Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

Yahoo! 74.6.87.*
Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

Google 130.203.146.*
Mozilla/5.0 (Windows; Mozilla 4.0(compatible; Googlebot/5.0; +http://www.google.com/googlebot); en-US;)

Google 80.131.221.*
Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.8.0.1) Gecko/20060111 googlebot 2.1

Google 84.188.206.*
Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.8.0.3) Gecko/20060426 Googlebot 2.1

Yahoo! 209.131.61.*
Mozilla/5.0 (Yahoo-Test/4.0 mailto:vertical-crawl-support@yahoo-inc.com )

Yahoo! 202.165.97.*
Mozilla/5.0 (Yahoo-Test/4.0; mailto:vertical-crawl-support@yahoo-inc.com )

Yahoo! 203.209.241.*
Mozilla/5.0 (Yahoo-Test/4.0; mailto:vertical-crawl-support@yahoo-inc.com )

Yahoo! 209.73.172.*
Mozilla/5.0 (Yahoo-Test/4.0; mailto:vertical-crawl-support@yahoo-inc.com )

Yahoo! 209.73.174.*
Mozilla/5.0 (Yahoo-Test/4.0; mailto:vertical-crawl-support@yahoo-inc.com )

MSN 65.54.112.*
msnbot-Products/1.0 (+http://search.msn.com/msnbot.htm)

MSN 65.55.214.*
msnbot-Products/1.0 (+http://search.msn.com/msnbot.htm)

MSN 65.55.216.*
msnbot-Products/1.0 (+http://search.msn.com/msnbot.htm)

MSN 65.54.164.*
msnbot/0.9 (+http://search.msn.com/msnbot.htm)

MSN 65.55.212.*
msnbot/0.9 (+http://search.msn.com/msnbot.htm)

MSN 65.55.213.*
msnbot/0.9 (+http://search.msn.com/msnbot.htm)

MSN 131.107.0.*
msnbot/1.0 (+http://search.msn.com/msnbot.htm)

MSN 202.96.51.*
msnbot/1.0 (+http://search.msn.com/msnbot.htm)

MSN 207.46.89.*
msnbot/1.0 (+http://search.msn.com/msnbot.htm)

MSN 207.46.98.*
msnbot/1.0 (+http://search.msn.com/msnbot.htm)

MSN 207.68.146.*
msnbot/1.0 (+http://search.msn.com/msnbot.htm)

MSN 207.68.154.*
msnbot/1.0 (+http://search.msn.com/msnbot.htm)

MSN 207.68.157.*
msnbot/1.0 (+http://search.msn.com/msnbot.htm)

MSN 207.68.188.*
msnbot/1.0 (+http://search.msn.com/msnbot.htm)

MSN 219.141.219.*
msnbot/1.0 (+http://search.msn.com/msnbot.htm)

MSN 219.142.53.*
msnbot/1.0 (+http://search.msn.com/msnbot.htm)

MSN 64.4.8.*
msnbot/1.0 (+http://search.msn.com/msnbot.htm)

MSN 65.54.165.*
msnbot/1.0 (+http://search.msn.com/msnbot.htm)

MSN 65.54.188.*
msnbot/1.0 (+http://search.msn.com/msnbot.htm)

MSN 65.55.208.*
msnbot/1.0 (+http://search.msn.com/msnbot.htm)

MSN 65.55.209.*
msnbot/1.0 (+http://search.msn.com/msnbot.htm)

MSN 65.55.210.*
msnbot/1.0 (+http://search.msn.com/msnbot.htm)

MSN 65.55.215.*
msnbot/1.0 (+http://search.msn.com/msnbot.htm)

MSN 65.55.233.*
msnbot/1.0 (+http://search.msn.com/msnbot.htm)

MSN 65.55.234.*
msnbot/1.0 (+http://search.msn.com/msnbot.htm)

MSN 65.55.235.*
msnbot/1.0 (+http://search.msn.com/msnbot.htm)

MSN 65.55.246.*
msnbot/1.0 (+http://search.msn.com/msnbot.htm)

MSN 65.55.250.*
msnbot/1.0 (+http://search.msn.com/msnbot.htm)

MSN 65.55.252.*
msnbot/1.0 (+http://search.msn.com/msnbot.htm)

MSN 65.54.158.*
MSNPTC/1.0

Yahoo! 209.191.82.*
Nokia6682/2.0 (3.01.1) (compatible;YahooSeeker/M1A1-R2D2; http://help.yahoo.com/help/us/ysearch/craw...awling-01.html)

Yahoo! 209.191.123.*
Nokia6682/2.0 (3.01.1) (compatible;YahooSeeker/M1A1-R2D2; http://help.yahoo.com/help/us/ysearch/craw...awling-01.html)

Yahoo! 69.147.79.*
Nokia6682/2.0 (3.01.1) (compatible;YahooSeeker/M1A1-R2D2; http://help.yahoo.com/help/us/ysearch/craw...awling-01.html)

NameProtect 12.175.0.*
NPBot/3 (NPBot; http://www.nameprotect.com; npbot@nameprotect.com)

Yahoo! 70.187.130.*
nutchsearch/Nutch-0.9 (Nutch Search 1.0; herceg_novi at yahoo dot com)

PrivacyFinder 128.2.220.*
PrivacyFinder/1.1

AltaVista 216.145.49.*
Scooter/3.3

AltaVista 66.94.232.*
Scooter/3.3

Yahoo! 209.249.86.*
WebCrawler ( pcp071098@yahoo.com)

Yahoo! 66.180.173.*
WebIndexer/1-dev (Web Indexer; mailto://webindexerv1@yahoo.com; webindexerv1@yahoo.com)

Yahoo! 203.141.52.*
Y!J-BSC/1.0 (http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html)

Yahoo! 211.14.11.*
Y!J-PSC/1.0 (http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html)

Yahoo! 211.14.8.*
Y!J/1.0 (http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html )

Yahoo! 209.191.83.*
Yahoo-Blogs/v3.9 (http://help.yahoo.com/help/us/ysearch/crawling/crawling-02.html )

Yahoo! 68.142.195.*
Yahoo-Blogs/v3.9 (http://help.yahoo.com/help/us/ysearch/crawling/crawling-02.html )

Yahoo! 206.190.43.*
Yahoo-MMAudVid/1.0 (mms dash mmaudvidcrawler dash support at yahoo dash inc dot com)

Yahoo! 68.142.210.*
Yahoo-MMAudVid/1.0 (mms dash mmaudvidcrawler dash support at yahoo dash inc dot com)

Yahoo! 209.191.113.*
Yahoo-MMAudVid/2.0(mms dash mm aud vid crawler dash support at yahoo dash inc.com ;)

Yahoo! 216.39.60.*
Yahoo-MMAudVid/2.0(mms dash mm aud vid crawler dash support at yahoo dash inc.com ;)

Yahoo! 66.163.174.*
Yahoo-MMAudVid/2.0(mms dash mm aud vid crawler dash support at yahoo dash inc.com ;)

Yahoo! 66.163.175.*
Yahoo-MMAudVid/2.0(mms dash mm aud vid crawler dash support at yahoo dash inc.com ;)

Yahoo! 69.147.82.*
Yahoo-MMAudVid/2.0(mms dash mm aud vid crawler dash support at yahoo dash inc.com ;)

Yahoo! 209.191.65.*
Yahoo-MMCrawler/3.x (mms dash mmcrawler dash support at yahoo dash inc dot com)

Yahoo! 66.94.238.*
Yahoo-MMCrawler/3.x (mms dash mmcrawler dash support at yahoo dash inc dot com)

Yahoo! 68.142.212.*
Yahoo-MMCrawler/3.x (mms dash mmcrawler dash support at yahoo dash inc dot com)

Yahoo! 209.131.41.*
YahooFeedSeeker Testing/2.0 (compatible; Mozilla 4.0; MSIE 5.5; http://publisher.yahoo.com/rssguide )

Yahoo! 216.39.58.*
YahooFeedSeeker Testing/2.0 (compatible; Mozilla 4.0; MSIE 5.5; http://publisher.yahoo.com/rssguide)

Yahoo! 66.94.237.*
YahooFeedSeeker Testing/2.0 (compatible; Mozilla 4.0; MSIE 5.5 ; http://publisher.yahoo.com/rssguide)

Yahoo! 216.109.121.*
YahooFeedSeeker/1.0 (compatible; Mozilla 4.0; MSIE 5.5; http://my.yahoo.com/s/publishers.html)

Yahoo! 66.218.65.*
YahooFeedSeeker/1.0 (compatible; Mozilla 4.0; MSIE 5.5; http://publisher.yahoo.com/rssguide)

Yahoo! 209.191.126.*
YahooFeedSeeker/2.0 (compatible; Mozilla 4.0; MSIE 5.5; http://publisher.yahoo.com/rssguide)

Yahoo! 216.145.53.*
YahooFeedSeeker/2.0 (compatible; Mozilla 4.0; MSIE 5.5; http://publisher.yahoo.com/rssguide)

Yahoo! 66.163.187.*
YahooFeedSeeker/2.0 (compatible; Mozilla 4.0; MSIE 5.5; http://publisher.yahoo.com/rssguide )

Yahoo! 66.94.229.*
YahooFeedSeeker/2.0 (compatible; Mozilla 4.0; MSIE 5.5; http://publisher.yahoo.com/rssguide)

Yahoo! 216.39.51.*
YahooSeeker/1.2 ( yahooseeker at yahoo-inc dot com ;)

Yahoo! 68.142.230.*
YahooSeeker/1.2 ( yahooseeker at yahoo-inc dot com ; )

Yahoo! 64.157.138.*
YahooSeeker/CafeKelsa-dev

Yahoo! 202.46.19.*
yahooseeker/ms-dev(compatible;Nokia-WAP-Toolkit/2.0;freebsd)

[Arlen Coupland]

Update - I made a note in the original topic about editing and moving ROBOTS.txt to properly work for your server. This can depend on your OpenX installation setup -- but if you have installed OpenX into a subdiretory of a domain, you should move and update ROBOTS.txt to be inside the root directory

[Arlen Coupland]

If you are using OpenX 2.4.4 or below, there might be an issue in the Statistics & Maintenance Settings->Prevent Statistics Logging box. If you enter IPs in a list seperated by commas, there may be extra commas added, causing statistics to be improperly blocked.

This issue is fixed in 2.4.5. Either upgrade to 2.4.5 or add the list of IPs directly to your configuration file at openx/var/www.example.com.conf.php, at the "ignoreHosts=" setting

https://developer.openx.org/jira/browse/OX-179

[Arlen Coupland]

As of 2.4.5, there is a new feature to be able to block by a user-agent string, rather than an IP.

If you go to Settings->Main Settings->Statistics & Maintenance Settings, you will see a box with the description:
Don't log statistics from clients with any of the following strings in their user-agent (one-per-line)

You could put a list like the following:
googlebot
slurp
gigablast

This allows for a more easily maintained list, as well as less strings for the delivery engine to check against - resulting in smoother delivery