Bug PHP and MySQL High I/O & Load

[Bestrafung]

We briefly tested your vBShout Lite with about 4 users and light chat and everything was fine. Over the weekend we had a large event and had approximately 12-15 users at any one time using the chat. After about 10 minutes we found that the server load skyrocketed to 25+ on a quad core system. I found that PHP and MySQL were both at high CPU load and MySQL also had a fair amount of I/O wait for our user account. We disabled vBShout and restarted apache, once the mysql I/O caught up the load returned to an acceptable level (still higher than usual though).

We're using the first instance on a custom page. I used this article/post from the VB forums to make the custom page: [HOW TO - vB4] Create your own vBulletin page - vBulletin.org Forum. I then disabled it from loading on every page by default. I'm wondering if there might be some PHP and MySQL settings not correct that may cause an issue?

I also get the warning message in the instance settings to increase max_input_vars to 10000 but I don't have that option through the web hosting panel. I tried the .httaccess code but it just produces a HTTP 500 error. I ran "php -i | grep max_input_vars" and get "max_input_vars => 1000 => 1000". I'll look into resolving this but could it be causing high I/O and load? I'd like to be able to use this product again but we can't afford to cripple the server again. Any assistance would be much appreciated.

EDIT: You didn't ask for it but this might be helpful:
httpd: Apache/2.2.26 (Unix)
mysql: 5.5.34-cll MySQL Community Server (GPL)
php: PHP Version => 5.3.28

 

[Fillip H.]

The max_input_vars only affects the ability to save instance configuration, not the performance of the mod.

What kind of CPU/HDD do you have? We frequently have 10-15 users chatting on one out of approx. 15 hosted sites on a server running a quad core and 10k SAS drives, and the load never goes up by much.

 

[Bestrafung]

The max_input_vars only affects the ability to save instance configuration, not the performance of the mod.

What kind of CPU/HDD do you have? We frequently have 10-15 users chatting on one out of approx. 15 hosted sites on a server running a quad core and 10k SAS drives, and the load never goes up by much.

Thanks for the reply. The site is running on a CentOS/cPanel VPS with a quad core (host machine has dual quad Xeons) and 6GB RAM. At the moment we're running off a "datacenter/raid verified" 7200RPM HDD but I plan to upgrade everything in the near future. At the time this happened the only real load on the server was coming from PHP and MySQL both running under this site's account. The day after the event everything was back to normal (about 0.3 to 0.7 load average) and we haven't spiked past 1.5 load since. The issue I have is that I don't know how, or if it's at possible, to see if the chat was causing the sql load or if something else was. The same goes with the PHP. I know that both were using approximately between 45 to 55 percent CPU in addition to the I/O wait but I can't be sure exactly what was eating the resources.

Since I got the code for the custom page directly from one of the VB staff I can only assume there's nothing wrong there. Also, I don't know if it makes any difference or not but it may be worth mentioning that some of the members took it upon themselves to "test" the shoutbox. They were trying to run SQL, XSS, and Javascript exploits. I assume that the shoutbox code just runs everything as plain text and ignores code so I didn't give any further thought to it.

 

[Fillip H.]

Do you have a character limit on the Shoutbox? If there's no limit, then it's plausible that a very long shout (or multiple) could cause extra load due to the server having to send a large amount of data over AJAX.

To be honest though, I think calling a 7200 RPM HDD "datacenter/raid verified" is more of a buzzword, unless it's one of those hybrid drives with a 60GB SSD in there that holds the most frequently accessed files or some such.

If you're ever around during the times in which these issues happen (if they happen again), you can log in to phpMyAdmin and click the Processes tab (or repeatedly execute SHOW PROCESSLIST; in the MySQL CLI tool via SSH, if the former is not an option) to see if there's anything funky going on there.

 

[Bestrafung]

Do you have a character limit on the Shoutbox? If there's no limit, then it's plausible that a very long shout (or multiple) could cause extra load due to the server having to send a large amount of data over AJAX.

To be honest though, I think calling a 7200 RPM HDD "datacenter/raid verified" is more of a buzzword, unless it's one of those hybrid drives with a 60GB SSD in there that holds the most frequently accessed files or some such.

If you're ever around during the times in which these issues happen (if they happen again), you can log in to phpMyAdmin and click the Processes tab (or repeatedly execute SHOW PROCESSLIST; in the MySQL CLI tool via SSH, if the former is not an option) to see if there's anything funky going on there.

Thanks again for the info. I agree with you regarding the HDD, I plan to upgrade it soon but I have to wait for approval
I thought about the shoutbox limit but none of the messages were abnormally large, probably no more than 140-150 characters each. The next time I get a chance to test it I'll definitely try to see what's going on with mysql.