近期由于公司举行大促,访问量巨增,预计流量为平时流量的60倍,在增加了两台服务器后,为了更大程度的压榨服务器性能,从各个层面上进行了优化,现在做下总结以备忘。
1)nginx优化
nginx优化主要涉及多核,IO线程数(需要同步修改系统IO),epoll,gzip等等,可能还会涉及upstream,注意以下标红部分:
worker_processes 8; worker_cpu_affinity 00000001 00000010 00000100 00001000 00010000 00100000 01000000 10000000; error_log /var/log/nginx_error.log error; pid /opt/nginx-1.2.0/nginx.pid;worker_rlimit_nofile 65535; events { use epoll;worker_connections 65535; }http {include mime.types;default_type application/octet-stream;charset utf-8;server_names_hash_bucket_size 128;client_header_buffer_size 32k;large_client_header_buffers 4 32k;client_max_body_size 8m;sendfile on;tcp_nopush on;keepalive_timeout 60;tcp_nodelay on;server_tokens off;fastcgi_connect_timeout 300;fastcgi_send_timeout 300;fastcgi_read_timeout 300;fastcgi_buffer_size 128k;fastcgi_buffers 8 128k;fastcgi_busy_buffers_size 128k;fastcgi_temp_file_write_size 128k;gzip on;gzip_min_length 1k;gzip_buffers 4 16k;gzip_http_version 1.0;gzip_comp_level 2;gzip_types text/plain application/x-javascript text/css application/xml text/javascript application/json;gzip_vary on;add_header Vary Accept-Encoding;#log_format access '$remote_addr - $remote_user [$time_local] "$request" ' '$status $body_bytes_sent "$http_referer" ' '"$http_user_agent" $request_time';log_format access '$http_x_forwarded_for - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent" $request_time ';access_log /var/log/httpd/search_access.log access;server{listen 80;server_name localhost;ssi on;ssi_silent_errors off; location / {root /data/www;index index.html index.htm index.php index.jsp;expires 1d;}location /app { proxy_pass http://localhost/app;proxy_set_header Host $host;proxy_set_header X-Real-IP $remote_addr;proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;client_max_body_size 50m;client_body_buffer_size 256k;proxy_connect_timeout 30;proxy_send_timeout 30;proxy_read_timeout 60;proxy_buffer_size 256k;proxy_buffers 4 256k;proxy_busy_buffers_size 256k;proxy_temp_file_write_size 256k;proxy_next_upstream error timeout invalid_header http_500 http_503 http_404;proxy_max_temp_file_size 128m;}error_page 404 403 /404.html; error_page 500 502 503 504 /50x.html;location = /50x.html {root html;}location = /404.html {root html;}} }
备注:除了以上标红的部分外,还需要注意,如果想在nginx日志中获取访问端ip的话,如果前端有负载均衡设备时,需要使用上面橙色部分配置;否则则使用上面蓝色部分配置。
2)应用服务器优化
应用服务器包含tomcat,resin,主要涉及调整VM参数,IO调整为NIO,线程数,日志配置等等;
2.1)resin.conf配置:
<jvm-arg>-Xmn2500m</jvm-arg><jvm-arg>-Xms4000m</jvm-arg><jvm-arg>-Xmx4000m</jvm-arg><jvm-arg>-XX:PermSize=256M</jvm-arg><jvm-arg>-XX:MaxPermSize=256m</jvm-arg><jvm-arg>-XX:+UseParNewGC</jvm-arg><jvm-arg>-XX:+UseConcMarkSweepGC</jvm-arg><jvm-arg>-XX:+UseCMSCompactAtFullCollection</jvm-arg><jvm-arg>-XX:CMSFullGCsBeforeCompaction=0</jvm-arg><jvm-arg>-XX:+CMSClassUnloadingEnabled</jvm-arg><jvm-arg>-XX:-CMSParallelRemarkEnabled</jvm-arg><jvm-arg>-XX:CMSInitiatingOccupancyFraction=80</jvm-arg><jvm-arg>-Xss512k</jvm-arg><!--<jvm-arg>-XX:-OmitStackTraceInFastThrow</jvm-arg><jvm-arg>-XX:+PrintGCDetails</jvm-arg><jvm-arg>-XX:+PrintGCTimeStamps</jvm-arg><jvm-arg>-XX:+PrintHeapAtGC</jvm-arg><jvm-arg>-Xloggc:serverGC.log</jvm-arg>--><jvm-arg>-Dcom.sun.management.jmxremote</jvm-arg>
其中-XX:-OmitStackTraceInFastThrow用于应用出现了IndexOutOfBoundsException看不到异常堆栈的情况,打开这个参数以关闭虚拟机优化,让堆栈重新抛出,以便定位问题;
所有GC的参数是为了查看GC日志,如果FullGC很频繁则需要进行VM的新生代及老年代的调整,以减少FullGC次数。
2.2)tomcat的server.xml配置
<Connector port="8080" protocol="org.apache.coyote.http11.Http11NioProtocol" connectionTimeout="10000" maxThreads="500"minSpareThreads="100"maxSpareThreads="300"acceptCount="100"redirectPort="8443"URIEncoding="utf-8" />
其中org.apache.coyote.http11.Http11NioProtocol用于将tomcat的处理模式修改为NIO。
2.3)tomcat的catalina.sh配置
JAVA_OPTS="-server -Xmn2500m -Xms4000m -Xmx4000m -XX:PermSize=256M -XX:MaxPermSize=256m -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+UseCMSCompactAtFullCollection -XX:CMSFullGCsBeforeCompaction=0 -XX:+CMSClassUnloadingEnabled -XX:-CMSParallelRemarkEnabled -XX:CMSInitiatingOccupancyFraction=80 -Xss512k -Dclient.encoding.override=UTF-8 -Dfile.encoding=UTF-8 $JAVA_OPTS" echo $JAVA_OPTS
调整VM参数,同resin一样。
2.4)tomcat的logging.properties配置
参考这篇博文:解决Tocmat6.x的catalina.out日志不断增加问题
整个优化过程还包括,尽可能的解决可能出现瓶颈的地方,比如压力测试的瓶颈点,不要去访问数据库等等,都是需要注意的。