自從把我的一個(gè)網(wǎng)站搬家到linode 1g 的vps上后，總感覺有些不太正常。首先dnspod監(jiān)控上顯示網(wǎng)站有時(shí)能訪問，有時(shí)不行。然后wp supercache插件在執(zhí)行預(yù)緩存任務(wù)時(shí)經(jīng)常失敗后重啟，郵件內(nèi)容如“[http://www.szl724.com]?預(yù)緩存可能已失去響應(yīng)。預(yù)緩存已經(jīng)重新啟動(dòng)。”。還有一個(gè)情況是linode有時(shí)會(huì)發(fā)來磁盤io使用高的報(bào)警郵件。
今天抽空檢查了下服務(wù)器，發(fā)現(xiàn)一個(gè)現(xiàn)象是mysql進(jìn)程頻繁無故重啟。
150424 17:41:14 [note] event scheduler: loaded 0 events150424 17:41:14 [note] /usr/local/mysql/bin/mysqld: ready for connections.version: \'5.5.37\' socket: \'/tmp/mysql.sock\' port: 3306 source distribution150424 19:27:16 mysqld_safe number of processes running now: 0150424 19:27:16 mysqld_safe mysqld restarted150424 19:27:24 [note] plugin \'innodb\' is disabled.150424 19:27:24 [note] server hostname (bind-address): \'0.0.0.0\'; port: 3306150424 19:27:24 [note] - \'0.0.0.0\' resolves to \'0.0.0.0\';150424 19:27:24 [note] server socket created on ip: \'0.0.0.0\'.150424 19:27:24 [warning] \'user\' entry \'root@li676-235\' ignored in --skip-name-resolve mode.150424 19:27:24 [warning] \'proxies_priv\' entry \'@ root@li676-235\' ignored in --skip-name-resolve mode.150424 19:27:25 [note] event scheduler: loaded 0 events150424 19:27:25 [note] /usr/local/mysql/bin/mysqld: ready for connections.version: \'5.5.37\' socket: \'/tmp/mysql.sock\' port: 3306 source distribution通過這個(gè)日志，mysql在提示他運(yùn)行困難，也就是表示服務(wù)器資源不夠用了，接下來開始檢查。
the error log message “mysqld_safe number of processes running now: 0″ indicates scarcity for resources to pursue the operations.
運(yùn)行free -m ，發(fā)現(xiàn)空閑內(nèi)存還有很多，差不多才用一半。
[root@li676-235 ~]# free -m total used free shared buffers cachedmem: 990 903 87 0 114 351-/ buffers/cache: 436 554swap: 255 53 202為了確定服務(wù)器資源是否真的不夠用了，方法是查看系統(tǒng)日志中是否有oom（out of memory） killer運(yùn)行過，果然在日志中發(fā)現(xiàn)有很多類似日志。
[root@li676-235 var]# egrep -i \oom|kill|mysql\ /var/log/messages |moreapr 23 13:36:16 li676-235 kernel: mysqld invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0apr 23 13:36:16 li676-235 kernel: mysqld cpuset=/ mems_allowed=0apr 23 13:36:16 li676-235 kernel: cpu: 0 pid: 16020 comm: mysqld not tainted 3.18.5-x86_64-linode52 #1apr 23 13:36:16 li676-235 kernel: [<ffffffff8112695f>] ? oom_kill_process 0x65/0x32fapr 23 13:36:16 li676-235 kernel: [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj nameapr 23 13:36:16 li676-235 kernel: [12128] 0 12128 26564 1 12 71 0 mysqld_safeapr 23 13:36:16 li676-235 kernel: [12405] 501 12405 155926 2868 120 3676 0 mysqldapr 23 13:36:16 li676-235 kernel: out of memory: kill process 9703 (php-fpm) score 41 or sacrifice childapr 23 13:36:16 li676-235 kernel: killed process 9703 (php-fpm) total-vm:266976kb, anon-rss:38932kb, file-rss:0kbapr 23 13:36:23 li676-235 kernel: mysqld invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0apr 23 13:36:24 li676-235 kernel: mysqld cpuset=/ mems_allowed=0apr 23 13:36:24 li676-235 kernel: cpu: 0 pid: 12405 comm: mysqld not tainted 3.18.5-x86_64-linode52 #1apr 23 13:36:24 li676-235 kernel: [<ffffffff8112695f>] ? oom_kill_process 0x65/0x32fapr 23 13:36:24 li676-235 kernel: [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj nameapr 23 13:36:24 li676-235 kernel: [12128] 0 12128 26564 1 12 71 0 mysqld_safeapr 23 13:36:24 li676-235 kernel: [12405] 501 12405 156056 2873 120 3676 0 mysqldapr 23 13:36:24 li676-235 kernel: out of memory: kill process 18168 (php-fpm) score 38 or sacrifice childapr 23 13:36:24 li676-235 kernel: killed process 18168 (php-fpm) total-vm:263724kb, anon-rss:24872kb, file-rss:0kb通過這個(gè)日志，可以清楚看到oom kill在“apr 23 13:36:16”被php-fpm觸發(fā)，也就是php在那時(shí)被系統(tǒng)強(qiáng)制kill掉了。我設(shè)置php-fpm進(jìn)程最大可以打開的數(shù)量是20。按照20*30 = 600m計(jì)算，php占用的資源最大可能會(huì)大于600m 。
“apr 23 13:36:16”時(shí)nginx日志如下圖所示：
配合nginx日志，可以確定當(dāng)時(shí)php進(jìn)程被打開很多，從而導(dǎo)致系統(tǒng)資源不夠用。
我的解決辦法是減少pm.max_children的數(shù)值。當(dāng)然這個(gè)值調(diào)小后肯定會(huì)犧牲網(wǎng)站的性能，不過我分析日志后發(fā)現(xiàn)正常情況網(wǎng)站是沒那么高并發(fā)的，所以影響應(yīng)該不大。
linux vps服務(wù)器內(nèi)存不夠用的問題暫時(shí)先調(diào)整到這，先觀察幾天。
參考連接：http://www.supportsages.com/blog/tag/mysqld_safe-number-of-processes-running-now-0/