Performance question for very large site Options

[swang75]

I'm evaluating whether to switch from my outdated AdJuggler 4.2 ad server to phpAdsNew. I'm really impressed with alot of the features: the hierarchy of advertisers->campaigns->banners, frequency capping, the ability for advertisers to login and get an integrated statistical report, and geotargeting are ALL elements that AdJuggler lacks that I'm EXTREMELY interested in integrating with phpAdsNew.

However, my main question is about performance. My current dedicated ad server answers about 3 million ad calls a day. Does ANYONE have any direct knowledge of phpAdsNew being able to handle this kind of load? Does it require any specific optimizations or hacks? What kind of platform/server specs were you running? What kind of load numbers were you seeing? Finally, how complex were your ad queries (e.g. frequency capping? keyword targeting? etc...)?

My ad server is pretty heavy duty (dual 2GHz Athalon w/ 2 gigs of memory on Linux), and I'd really like to be able to use all of those features that phpAdsNew has over AdJuggler, but I'm not sure if it can handle the number of queries given that it's written in PHP (instead of an executable).

[scott_switzer]

Hello,

I will answer the easy question first (is php fast?):
PHP Accelerator will run PHP as fast as compiled code.

The much more difficult question (how much load?):
I have been in the process of load testing phpAdsNew to deliver 1.6M ads per hour, which is the peak load for 500M ads delivered per month.

I am starting with a server with roughly the same specs (dual 2.8GHz w/ 1Mb memory) as you, and as various bits get tuned, I am moving up to a much more robust hardware configuration.

Currently, it looks like my server is maxed out at about 100K ads per hour, which (given my volume trends in ad serving) turns out to be 1M - 1.2M per day. The CPU, Memory, bandwidth, and disk IO are nowhere near capacity. The problem is that some parts of phpAdsNew are over-using MySQL during ad delivery.

Unfortunately, there is no way to get real time statistics with a load this high. For High Priority campaigns and using Compact Stats, there are 2 Update statements per ad view.

I have changed this to be 0 updates and 1 Insert per adview, but stats are compiled on an hourly or daily basis instead of real time. MySQL can do thousands of inserts per second, but can only update a single row one request at a time.

With this code change, and some time with the web server and DB configuation files, you should be able to scale to the numbers that you are talking about, given your existing hardware.

The unfortunate thing is that you will have to wait until v2.1 to get the changes that I made.

If I were you, I would get familiar with phpAdsNew, install it, and do some load testing. It will take quite a bit of time to get the configuration of Apache/MySQL/phpAdsNew just right for your hardware. Then, hopefully in a few months, v2.1 will be available with some application enhancements to support extremely high volumes.

Good Luck,
Scott

[rcmedia]

Stephen,

I am up to about 1.5 million impressions a day,using compact mode and a machine with 2gigs/Dual Pentium 4 Xeon 1.8 GHz . Everything is running ok, but I am going to have the database set up on its on server, once I reach your load. PHPadsnew is awesome.

[Guest_Kevin Woodsro t_*]

Scott,

Any chance we can see some of your settings for Apache, PHP and MySQL?

I have been trying to 'tune' Apache and MySQL without much success.

I have tried adjusting the MaxClients and Start/SpareServers values in httpd.conf (Apache 1.37). And when calculating my key buffer hit rate (method below) I'm comming up at 98% under load... Seems good. (?)

mysql SHOW STATUS LIKE 'Key%';

+--------------------------+------------+
| Variable_name | Value |
+--------------------------+------------+
| Key_read_requests | 3844786889 |
| Key_reads | 16525182 |
| Key_write_requests | 303516563 |
| Key_writes | 152315649 |
+--------------------------+------------+

100 - ( ( Key_reads / Key_read_requests ) * 100 )


I would be *very* interested to see what successful settings are for high performance / large site deployments of phpAdsNew.

[scott_switzer]

My site is nowhere near fully tuned, but here are some of the things that I have done that make a difference:

1. Use PHP-Accelerator! (best performance enhancer of all...)
2. Turn KeepAlive off.
3. Don't do DNS reverse lookups.
4. I am using InnoDB tables for MySQL, so our parameters will be much different. I am not sure about the performance of MYISAM vs INNODB.
5. When using MySQL, consider turning OFF persistent connections. This is probably more important when you have multiple web servers feeding the same DB.
6. Store your banners on the web server (not in the DB). Use a separate low-end web server (e.g. tux) to serve the images. This server can still be on the same hardware as your other web server.
7. I am currently using file-caching for the banners, but am going to be testing shared memory caching soon.
8. If you have a high priority campaign that will get many views per second, consider splitting it into multiple campaigns. The reason for this is that phpAdsNew does an update on a particular row for the ad view, and multiple updates on the same row will queue behind each other.
9. I am currently using 300 MaxClients. Yours will probably be more because you have double the memory.
10. Make SURE that you are not using swap memory.
11. Start with the configuration file 'my-huge.cnf' for MySQL.
12. Try not to use the admin interface during peak delivery times!

These are the things that I remember when benchmarking on a single machine. I am sure that most of these would change when you move to multiple machines.

As with all benchmarking projects, only change one thing at a time, and document the results. Make sure that your load testing environment is as close to the real world scenario as possible. For instance, I started benchmarking the adview.php page with a single low priority banner. I was able to scale incredibly high with this scenario. But, my customers all use a modified javascript invocation, with zones and many different sources, and average about 3-4 placements per page. About 60-70% of my banners are high priority, which drags the system more. After changing my benchmarking to incorporate these items, the server did not perform nearly as well.

I have changed my method of benchmarking to use a browser pointing to a page that refreshes every few seconds. Then, I replicate this scenario 1000's of times to represent many users. This is the most realistic that I can get.

Good Luck,
Scott

[swang75]

Scott,
Thanks for your very thorough reply. I will take a look at waiting until 2.1 for the high performance changes. I read a previous thread discussing the cron'd hourly processed statistics -- is this the change that you're talking about? Will this definitely be making it into 2.1 as a admin option then, or will it require more code hacking?

Also, you mentioned that you have the ads DB on a dedicated server -- does the Apache server that processes the admin pages and the ad display (not the actualy images themselves, but the PHP pages that determine which ad to show) reside on a separate server from the DB AND the rest of yourr website?

[scott_switzer]

Basically, with my changes (just put into the main branch of CVS), the ads get counted with the cron job every hour. This removes the reliance on update statements during ad delivery.

For your second question, I think that you are asking about the separate server to deliver the advert banners. Basically, what I mean is that you should set up another web server (not physical server) on another IP address on the server which serves the phpAdsNew php pages. The reason for this is as follows:

1. The http processes which are serving php pages use quite a bit of memory, which is overkill for serving static images.
2. If you use pconnect, then each http process has a connection to the MySQL DB, which is not needed to deliver a static image.
3. Usually, apache is compiled with all kinds of options (thus making it slower) which are unneeded for delivering images.

Basically, if you are using Javascript invocation, you are going to get 3-4 hits for every ad view: adjs.php, adlog.php (if you are using beacons), adview.php, and a hit for the banner image. By taking the banner images out of the apache server process, you are freeing up 25% to 33% of your slower apache connections, and replacing them with high speed tux connections.

Scott

[Guest_Phil Hordro tea_*]

I just upgraded this week and my access_log went crazy. It turns out I had "beacons" turned on. Lots of ads per page, and lots more requests generated. (I use local invocation, so I'm not used to all the extra HTTP requests.)

[scott_switzer]

Hi - Use this thread to talk about performance optimisations for high volume sites.

Scott

[md_doc]

Scott--

I hate to differ with you and all but over at devshed I set things up a bit different on our dual 2 GHz Xeon system (setup to run as a quad processor) with 1.5 gigs of ram. The server easily serves 2 million ads a day along with running easystats for 13 sites that get a combination of 200k+ unique users a day. The ad server itself runs 3 web sites and a vBulletin instance. The system rearly has a load over 1.0 and MySQL 4.0.13 averages 150 queries per second. Obviously we have query_caching on and I think about 40% of all queries are cached on the system so that helps a little bit.

I actually do not use PHP-Accelerator as in all my tests I get the exact same numbers in terms of page creation time with it as we do without it.

I have Keep-alive on with a timeout on it of 7 seconds. We use 10 seconds on most of our other servers but for our ad server I found that 7 seconds was the best performance increase. It may be different for each site but for our ad server/sites on that machine 7 sections works best.

I also did not use my.huge.cnf because it actually takes up more resources than are needed just to serve ads from what I could tell. I ended up using the my.large.cnf file instead but added a lot of tweaks to it.

Oh almost forgot about some important stuff. Thread_cache_size=40 I am considering upping that number to about 60 as the creation of a new thread in MySQL is pretty expensive so keeping the varilbed Threads_Created down is a good thing.

I also use a MaxClients of 500. There are a total of 4 web servers, all serving up different sites, but each page of each site has content coming from the other servers so we need to take into account those connections as well. This is why we have double the MaxClients as we do httpd processes.

I hope everyone fines this helpful and good luck with phpAdsNew!

This post has been edited by md_doc: Oct 8 2003, 12:49 PM

[awphoenix]

Pardon my ignorance, but for the purposes of this discussion, how large is large?

Without the ads, our traffic runs about 170G of transfer a month, with an average of 530K or so hits per day (4-5K unique visits). We're not running on a distributed server environment yet (quite frankly, because we haven't delved into paid sponsorship programs until now and couldn't afford a separate web and database server).

How afraid should I be?

[simoon]

We are using php with thttpd http://www.acme.com/software/thttpd/

this is much more faster than apache. up2 10 times in out tests.

Greets

Alex

[md_doc]

Well large as I was talking about it is 200k+ unique users a day as that is what we were doing per day at the last job I was at.

[half-dead]

We managed about 2.1M ads/day on our old setup, which gave about a load of 2.5:

1Ghz intel
256 ram
Apache 1.3.27
php 4.3.2
mysql 4.0.14
SCSI drives

All default configurations, just added MMcache & removed keepalive

WoW


Tho since they switched to blade servers(2 load balanced) & running apache 2, we're having some probs getting it tweaked right.

1ghz intel
512 ram
Apache 2.0.48
php 4.3.4
mysql 4.0.16
ide drives :-(

But in the end we should be doing something like 5M/day (60ads/sec = ~200 queries/sec) so if anyone has some tweaks to share or important points, feel free to post up

Tho i'm surprised that the 1st config above even managed so many ads

[md_doc]

MySQL tweaks

Turn on query_cache, actually I would have to double check, it might not help for phpadsnew stuff and if that is all you are doing then hmmm. Because the cache would expire everytime an ad is loaded because it increments so maybe you do not want to do that because if the cache keeps expiring then things will go slower than without it. You might want to test though.

Make sure you have thread_cache_size set. Do a SHOW STATUS and see what threads_cached, threads_created, threads_running, and threads_connected are. If threads_cached is 0 well then that is not good and you will want to change the thread_cache_size.

Actualy you might want to check out http://jeremy.zawodny.com/blog/archives/000173.html instead of me saying most of the stuff said there. There are a few other things I think he forgets to cover but at least it is a good starting point for you.

You might have already done it all but you really did not say what tweaks you did.

I am not surprised at all that the first setup handled that number of ads to be honest. I was doing 2 million ads a day plus serving up some sites, forum sites at that as well, and the system it was running on had no issues. Sure it was a Dual Xeon system but it literally had a load of .5 so just cutting the system down by about 1/4 should have not been an issue.

Obviously since you are doing hard drive intensive stuff the speed of your drives really do matter so SCSI would certainly be the way to go over IDE and with limited ram it would be hard to get all the data you need from the database into ram so that disk access speed is not an issue.

I would get started with the stuff above though and then from there if things are still slow (they will certainly need to be tweaked) post back with some results and we will see what other kind of advice we can give.