一、背景
在使用阿里云k8s集群的时候,内网API网关,刚开始是用的是Nginx,后面又搭建了ingress。
区别于nginx配置,ingress又该怎么设置参数呢?比如http超时时间等等。
本文会先梳理nginx是如何配置,再对比ingress的配置方式。
示例以超时时间的设置。
二、nginx配置
在k8s部署两个节点的Nginx容器
containers:- env:- name: aliyun_logs_nginx-logvalue: /var/log/nginx/*.logimage: nginximagePullPolicy: Alwaysname: xh-nginxports:- containerPort: 80protocol: TCPresources:limits:cpu: '2'memory: 4Girequests:cpu: 250mmemory: 2GiterminationMessagePath: /dev/termination-logterminationMessagePolicy: FilevolumeMounts:- mountPath: /etc/nginx/nginx.confname: nginxsubPath: nginx.conf- mountPath: /etc/nginx/conf.dname: nginx-cm- mountPath: /var/log/nginx/name: volume-k8s-inner-nginx-logvolumes:- configMap:defaultMode: 420items:- key: nginx.confpath: nginx.confname: nginx-confname: nginx- configMap:defaultMode: 420name: nginx-cmname: nginx-cm- hostPath:path: /var/log/nginxtype: Directoryname: volume-k8s-inner-nginx-log- emptyDir: {}name: volumn-sls-16578614717160
这里把/etc/nginx/nginx.conf和下面的/etc/nginx/conf.d/*.conf分别挂载到configMap
1、nginx-conf下的新增了一个子项nginx.conf
对应容器里的文件/etc/nginx/nginx.conf
详情见下:
user nginx;
worker_processes auto;worker_cpu_affinity auto;error_log /var/log/nginx/error.log warn;
pid /var/run/nginx.pid;worker_rlimit_nofile 10240;events {use epoll;worker_connections 10240;
}http {underscores_in_headers on;proxy_set_header X-Real-IP $remote_addr;proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
# 传递http header值include /etc/nginx/mime.types;default_type application/octet-stream;
# 设置log格式log_format access '$proxy_add_x_forwarded_for $time_local $request $request_time "$upstream_response_time" ''$status $body_bytes_sent $host "$http_user_agent" $bytes_sent $request_length "$upstream_addr" ';access_log /var/log/nginx/access.log access;charset utf-8;server_names_hash_bucket_size 128;client_header_buffer_size 32k;large_client_header_buffers 4 32k;client_max_body_size 500m;sendfile on;tcp_nopush on;tcp_nodelay on;keepalive_timeout 600;server {listen 80;server_name nginx_status;location /ngx_status {stub_status;}}fastcgi_connect_timeout 600;fastcgi_send_timeout 600;fastcgi_read_timeout 600;fastcgi_buffer_size 64k;fastcgi_buffers 4 64k;fastcgi_busy_buffers_size 128k;fastcgi_temp_file_write_size 128k;include /etc/nginx/conf.d/*.conf;}
2、nginx-cm
对应容器里的文件/etc/nginx/conf.d/*.conf
下面以常见的用户服务为示例:
upstream user-service-cloud-cluster {server 172.16.17.9:8081 weight=50 max_fails=2 fail_timeout=10s;
}
server
{listen 80;server_name user.xxx.cloud;location / {proxy_next_upstream error timeout invalid_header http_500 http_502 http_503 http_504;proxy_pass http://user-service-cloud-cluster;proxy_redirect off;proxy_set_header Host $host;proxy_set_header X-Real-IP $remote_addr;proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;proxy_set_header HTTP_HOST $host;proxy_set_header HTTP_X_FORWARDED_FOR $remote_addr;proxy_set_header HTTP_X_FORWARDED_HOST $host;proxy_set_header X-Forwarded-Host $host;proxy_set_header X-Forwarded-Server $host;proxy_set_header X-Forwarded-HTTPS 0;}access_log /var/log/nginx/user-service_cloud_access.log access;error_log /var/log/nginx/user-service_cloud_error.log;}
3、小节
当你修改了nginx的配置,别忘记了进入Nginx容器进行reload,以使配置生效。
nginx -s reload
三、ingress配置
除了已知的一些区别,它和Nginx的一个最大不同是,不用手动去reload才能让配置生效。
同样部署两个ingress节点
建议你使用Helm安装ingress,简单方便。具体就不在本文赘述了。
下面再看下它的yaml详情:
apiVersion: apps/v1
kind: Deployment
metadata:name: nginx-ingress-ack-ingress-nginx-v1-controllernamespace: kube-system
spec:progressDeadlineSeconds: 600replicas: 2revisionHistoryLimit: 10selector:matchLabels:app.kubernetes.io/component: controllerapp.kubernetes.io/instance: nginx-ingressapp.kubernetes.io/name: ack-ingress-nginx-v1strategy:rollingUpdate:maxSurge: 25%maxUnavailable: 25%type: RollingUpdatetemplate:metadata:labels:app.kubernetes.io/component: controllerapp.kubernetes.io/instance: nginx-ingressapp.kubernetes.io/name: ack-ingress-nginx-v1spec:containers:- args:- /nginx-ingress-controller- >---publish-service=$(POD_NAMESPACE)/nginx-ingress-ack-ingress-nginx-v1-controller-internal- '--election-id=ingress-controller-leader-ack-nginx'- '--controller-class=k8s.io/ack-ingress-nginx'- '--ingress-class=ack-nginx'- >---configmap=$(POD_NAMESPACE)/nginx-ingress-ack-ingress-nginx-v1-controller- '--validating-webhook=:8443'- '--validating-webhook-certificate=/usr/local/certificates/cert'- '--validating-webhook-key=/usr/local/certificates/key'- '--v=2'env:- name: POD_NAMEvalueFrom:fieldRef:apiVersion: v1fieldPath: metadata.name- name: POD_NAMESPACEvalueFrom:fieldRef:apiVersion: v1fieldPath: metadata.namespace- name: LD_PRELOADvalue: /usr/local/lib/libmimalloc.soimage: >-registry-vpc.cn-hangzhou.aliyuncs.com/acs/aliyun-ingress-controller:v1.8.0-aliyun.1imagePullPolicy: IfNotPresentlifecycle:preStop:exec:command:- /wait-shutdownlivenessProbe:failureThreshold: 5httpGet:path: /healthzport: 10254scheme: HTTPinitialDelaySeconds: 10periodSeconds: 10successThreshold: 1timeoutSeconds: 1name: controllerports:- containerPort: 80name: httpprotocol: TCP- containerPort: 443name: httpsprotocol: TCP- containerPort: 8443name: webhookprotocol: TCPreadinessProbe:failureThreshold: 3httpGet:path: /healthzport: 10254scheme: HTTPinitialDelaySeconds: 10periodSeconds: 10successThreshold: 1timeoutSeconds: 1resources:requests:cpu: 100mmemory: 90MisecurityContext:allowPrivilegeEscalation: truecapabilities:add:- NET_BIND_SERVICEdrop:- ALLrunAsUser: 101terminationMessagePath: /dev/termination-logterminationMessagePolicy: FilevolumeMounts:- mountPath: /usr/local/certificates/name: webhook-certreadOnly: true- mountPath: /etc/localtimename: localtimereadOnly: truednsPolicy: ClusterFirstinitContainers:- command:- /bin/sh- '-c'- |if [ "$POD_IP" != "$HOST_IP" ]; thenmount -o remount rw /proc/syssysctl -w net.core.somaxconn=65535sysctl -w net.ipv4.ip_local_port_range="1024 65535"sysctl -w kernel.core_uses_pid=0fienv:- name: POD_IPvalueFrom:fieldRef:apiVersion: v1fieldPath: status.podIP- name: HOST_IPvalueFrom:fieldRef:apiVersion: v1fieldPath: status.hostIPimage: 'registry-vpc.cn-hangzhou.aliyuncs.com/acs/busybox:v1.29.2'imagePullPolicy: IfNotPresentname: init-sysctlresources: {}securityContext:capabilities:add:- SYS_ADMINdrop:- ALLterminationMessagePath: /dev/termination-logterminationMessagePolicy: FilenodeSelector:kubernetes.io/os: linuxrestartPolicy: AlwaysschedulerName: default-schedulersecurityContext: {}serviceAccount: nginx-ingress-ack-ingress-nginx-v1serviceAccountName: nginx-ingress-ack-ingress-nginx-v1terminationGracePeriodSeconds: 300tolerations:- effect: NoSchedulekey: node-role.alibabacloud.com/addonoperator: Existsvolumes:- name: webhook-certsecret:defaultMode: 420secretName: nginx-ingress-ack-ingress-nginx-v1-admission- hostPath:path: /etc/localtimetype: Filename: localtime
这里使用了一个初始化容器initContainers,它会对系统做一个个性化配置。
sysctl -w net.core.somaxconn=65535
sysctl -w net.ipv4.ip_local_port_range="1024 65535"
sysctl -w kernel.core_uses_pid=0
其次,HOST_IP和POD_IP都从K8s环境变量中读取,因为它们是动态的,非固定不变。
必要的健康检测,配置了livenessProbe和readinessProbe,详情见上。
1、configMap配置
日志格式,见下:
其他的配置这里就不一一列举,总之,它支持你通过变量进行配置就行。
它就对应上文的nginx.conf文件。
2、创建Ingress路由
操作比较简单,下面要切入到本文的重点。
四、Ingress设置超时时间
要说Ingress如何设置超时时间前,先看一看nginx是如何设置。
默认是60秒,现在业务上有需求调整为600秒。
请看下文的具体配置:
1、nginx配置
upstream xxx-cloud-cluster {server 172.16.17.6:8080 weight=9 max_fails=2 fail_timeout=10s;
}
server
{listen 80;server_name image-xxx.xx.cloud;location / {proxy_next_upstream error timeout invalid_header http_500 http_502 http_503 http_504;proxy_pass http://xxx-cloud-cluster;proxy_redirect off;proxy_set_header Host $host;# 增加下面三行proxy_connect_timeout 600;proxy_send_timeout 600;proxy_read_timeout 600;proxy_set_header X-Real-IP $remote_addr;proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;proxy_set_header HTTP_HOST $host;proxy_set_header HTTP_X_FORWARDED_FOR $remote_addr;proxy_set_header HTTP_X_FORWARDED_HOST $host;proxy_set_header X-Forwarded-Host $host;proxy_set_header X-Forwarded-Server $host;proxy_set_header X-Forwarded-HTTPS 0;}access_log /var/log/nginx/xxx_access.log access;error_log /var/log/nginx/xxx_error.log;}
2、ingress配置
参数设置通过注解配置:
proxy_connect_timeout 600;
proxy_send_timeout 600;
proxy_read_timeout 600;
yaml详情见下:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:annotations:nginx.ingress.kubernetes.io/proxy-connect-timeout: '600'nginx.ingress.kubernetes.io/proxy-read-timeout: '600'nginx.ingress.kubernetes.io/proxy-send-timeout: '600'labels:ingress-controller: nginxname: image-xxxnamespace: java-service
spec:ingressClassName: ack-nginxrules:- host: image.xxx.cloudhttp:paths:- backend:service:name: image-xxxport:number: 8080path: /pathType: ImplementationSpecific
五、总结
这里只是以设置超时时间为例,讲述k8s容器部署的Nginx和ingress,如何设置一定自定义的参数配置。
当然,这里没有讲述怎么安装它们,更多的是梳理了一下如何配置,侧重于使用这块。