一.Ingress生产环境常用配置
1.Ingress架构
2.Ingress controller安装
a.安装helm，下载安装包：https://github.com/helm/helm/releases
b.解压：helm-v3.7.2-linux-amd64.tar
c.复制到二进制目录：mv linux-amd64/helm /usr/local/bin/helm
[root@k8s-master01 bin]# which helm #查看 /usr/local/bin/helm
d.添加ingress-nginx仓库：
[root@k8s-master01 ~]# helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx "ingress-nginx" has been added to your repositories
e.下载ingress-nginx并解压
[root@k8s-master01 ~]# helm pull ingress-nginx/ingress-nginx --version 4.0.1 [root@k8s-master01 ~]# tar xf ingress-nginx-4.0.1.tgz
f.修改配置：
[root@k8s-master01 ~]# cd ingress-nginx && vim values.yaml
...12行：修改同步至私有镜像仓库，注释哈希值验证
image:
registry: registry.cn-beijing.aliyuncs.com
image: dotbalo/controller
tag: "v1.0.0"
#digest: sha256:0851b34f69f69352bf168e6ccf30e1e20714a264ab1ecd1933e4d8c0fc3215c6
...58行：
dnsPolicy: ClusterFirstWithHostNet
...72行：
hostNetwork: true
...89行：default设成true
ingressClassResource:
name: nginx
enabled: true
default: true
controllerValue: "k8s.io/ingress-nginx"
...271行：添加ingress: "true"标签，把ingress controller部署到指定节点。
nodeSelector:
kubernetes.io/os: linux
ingress: "true"
...
g.创建命名空间，打标签，最后创建：
[root@k8s-master01 ingress-nginx]# kubectl create ns ingress-nginx namespace/ingress-nginx created [root@k8s-master01 ingress-nginx]# kubectl label node k8s-node02 ingress=true node/k8s-node02 labeled [root@k8s-master01 ingress-nginx]# helm install ingress-nginx -n ingress-nginx .
查看已装在node02上了：
[root@k8s-master01 ingress-nginx]# kubectl get pod -n ingress-nginx -owide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES ingress-nginx-controller-gm5nf 1/1 Running 0 37s 192.168.0.111 k8s-node02 [root@k8s-node02 ~]# ps -ef | grep ingress #在node02节点查看进程 101 26788 26771 0 15:42 ? 00:00:00 /usr/bin/dumb-init -- /nginx-ingress-controller --publish-service=ingress-nginx/ingress-nginx-controller --election-id=ingress-controller-leader --controller-class=k8s.io/ingress-nginx --configmap=ingress-nginx/ingress-nginx-controller --validating-webhook=:8443 --validating-webhook-certificate=/usr/local/certificates/cert --validating-webhook-key=/usr/local/certificates/key
3.Ingress基础使用
创建一个用于学习 Ingress 的 Namespace，之后所有的操作都在此 Namespace 进行：
# kubectl create ns study-ingress namespace/study-ingress created
创建一个简单的 Nginx 模拟 Web 服务：
# kubectl create deploy nginx --image=registry.cn-beijing.aliyuncs.com/dotbalo/nginx:1.15.12 -n study-ingress
然后创建该 Web 容器的 Service：
# kubectl expose deploy nginx --port 80 -n study-ingress
之后创建 Ingress 指向上面创建的 Service：
[root@k8s-master01 ~]# vim web-ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: nginx-ingress
namespace: study-ingress
spec:
rules:
- host: nginx.test.com
http:
paths:
- backend:
service:
name: nginx
port:
number: 80
path: /
pathType: ImplementationSpecific
如果 apiVersion 是 networking.k8s.io/v1beta1，对应的配置如下：
[root@k8s-master01 ~]# vim web-ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: nginx-ingress
namespace: study-ingress
spec:
rules:
- host: nginx.test.com
http:
paths:
- backend:
serviceName: nginx
servicePort: 80
path: /
pathType: ImplementationSpecific
如果 Kubernetes 版本低于1.19 ，可以使用 networking.k8s.io/v1beta1 替代，配置可以参考上述的networking.k8s.io/v1beta1，只有 backend 配置不一样。
创建该 Ingress：
# kubectl create -f web-ingress.yaml
将域名 nginx.test.com即可访问Web服务器，如下图所示：
4.域名重定向Redirect
在 Nginx 作为代理服务器时，Redirect 可用于域名的重定向，比如访问 old.com 被重定向到
new.com。Ingress 可以更简单的实现 Redirect 功能，接下来用 qq.redirect.com 作为旧域名，https://www.qq.com 作为新域名进行演示：
[root@k8s-master01 ~]# vim redirect.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
nginx.ingress.kubernetes.io/permanent-redirect: https://www.qq.com
name: nginx-redirect
namespace: study-ingress
spec:
rules:
- host: qq.redirect.com
http:
paths:
- backend:
service:
name: nginx
port:
number: 80
path: /
pathType: ImplementationSpecific
[root@k8s-master01 ~]# kubectl create -f redirect.yaml
[root@k8s-master01 ~]# kubectl get ingresses.networking -n study-ingress
NAME CLASS HOSTS ADDRESS PORTS AGE
nginx-ingress nginx nginx.test.com 80 92m
nginx-redirect qq.redirect.com 80 85m
通过修改hosts经行访问：
5.前后端分离Rewrite
创建一个应用模拟后端服务：
[root@k8s-master01 ~]# kubectl create deploy backend-api --image=registry.cn-beijing.aliyuncs.com/dotbalo/nginx:backend-api -n study-ingress deployment.apps/backend-api created [root@k8s-master01 ~]# kubectl expose deploy backend-api --port 80 -n study-ingress service/backend-api exposed
nginx模拟前端，backend-api模拟后端：
[root@k8s-master01 ~]# kubectl get svc -n study-ingress NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE backend-api ClusterIP 10.104.51.171 80/TCP 16s nginx ClusterIP 10.98.233.57 80/TCP 23h
查看该 Service 的地址，并且通过/api-a 访问测试是访问不了的，但是直接访问根路径是可以的：
通过 Ingress Nginx 的 Rewrite 功能，将/api-a 重写为“/”，配置示例如下：
[root@k8s-master01 ~]# vim nginx-rewirte.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
nginx.ingress.kubernetes.io/rewrite-target: /$2
name: backend-api
namespace: study-ingress
spec:
rules:
- host: nginx.test.com
http:
paths:
- backend:
service:
name: backend-api
port:
number: 80
path: /api-a(/|$)(.*)
pathType: ImplementationSpecific
[root@k8s-master01 ~]# kubectl create -f nginx-rewirte.yaml
ingress.networking.k8s.io/backend-api created
[root@k8s-master01 ~]# kubectl get ingresses.networking.k8s.io -A
NAMESPACE NAME CLASS HOSTS ADDRESS PORTS AGE
study-ingress backend-api nginx nginx.test.com 80 26s
study-ingress nginx-ingress nginx nginx.test.com 80 102m
study-ingress nginx-redirect none qq.redirect.com 80 95m
再次再web上访问：
6.错误代码重定向
先进入ingress-nginx的安装目录，修改 values.yaml 如下图所示位置：
[root@k8s-master01 ~]# cd /root/ingress-nginx && vim values.yaml
更新 ConfigMap：
...37行 config: apiVersion: v1 client_max_body_size: 20m custom-http-errors: "404,415,503" ...
更新 Release：
更新后 Pod 会自动重启，并且会创建一个 defaultbackend：
# helm upgrade ingress-nginx -n ingress-nginx . # kubectl get po -n ingress-nginx NAME READY STATUS RESTARTS AGE ingress-nginx-controller-pdjvh 1/1 Running 0 95s ingress-nginx-defaultbackend-79d64fb85f-kx1nc 1/1 Running 0 109s
更新完成以后访问一个不存在的页面，比如之前定义的 nginx.test.com。访问一个不存在的
页面 123，就会跳转到 Error Server 中的页面：
7.SSL配置
生产环境对外的服务，一般需要配置 https 协议，使用 Ingress 也可以非常方便的添加 https
的证书。
由于我们是学习环境，并没有权威证书，所以需要使用 OpenSSL 生成一个测试证书。如果
是生产环境，证书为在第三方公司购买的证书，无需自行生成：
# openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout tls.key -out tls.crt -subj "/CN=nginx.test.com" Generating a 2048 bit RSA private key .....+++ ............................+++ writing new private key to 'tls.key' ----- # kubectl create secret tls ca-secret --cert=tls.crt --key=tls.key -n study-ingress secret/ca-secret created
[root@k8s-master01 ~]# vim ingress-ssl.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
creationTimestamp: null
name: nginx-ingress
namespace: study-ingress
# annotations:
# kubernetes.io/ingress.class: nginx
spec:
ingressClassName: nginx # for k8s >= 1.22+
rules:
- host: nginx.test.com
http:
paths:
- backend:
service:
name: nginx
port:
number: 80
path: /
pathType: ImplementationSpecific
tls:
- hosts:
- nginx.test.com
secretName: nginx.test.com
可以看到 Ingress 添加 TLS 配置也非常简单，只需要在 spec 下添加一个 tls 字段即可：
➢ hosts：证书所授权的域名列表
➢ secretName：证书的 Secret 名字
➢ ingressClassName: ingress class 的名字，1.22+需要配置
接下来更新该 Ingress 即可：
[root@k8s-master01 ~]# kubectl apply -f ingress-ssl.yaml Warning: resource ingresses/nginx-ingress is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by kubectl apply. kubectl apply should only be used on resources created declaratively by either kubectl create --save-config or kubectl apply. The missing annotation will be patched automatically. ingress.networking.k8s.io/nginx-ingress configured
使用 curl 进行测试，http重定向到了https：
[root@k8s-master01 ~]# curl -H "Host:nginx.test.com" 192.168.0.111 -I HTTP/1.1 308 Permanent Redirect Date: Sun, 26 Dec 2021 10:43:58 GMT Content-Type: text/html Content-Length: 164 Connection: keep-alive Location: https://nginx.test.com
web访问：能查看到证书，由于不是颁发机构颁发的所以有报错：
8.匹配请求头-区分移动端和PC端
分别部署两pod，用来模拟移动端和PC端：
部署移动端：
[root@k8s-master01 ~]# kubectl create deploy phone --image=registry.cn-beijing.aliyuncs.com/dotbalo/nginx:phone -n study-ingress [root@k8s-master01 ~]# kubectl expose deploy phone --port 80 -n study-ingress Ingress实例也可以通过 kubectl create进行创建，只需要一条命令即可： [root@k8s-master01 ~]# kubectl create ingress phone --rule=m.test.com/*=phone:80 -n study-ingress
部署PC端：
[root@k8s-master01 ~]# kubectl create deploy laptop --image=registry.cn-beijing.aliyuncs.com/dotbalo/nginx:laptop -n study-ingress [root@k8s-master01 ~]# kubectl expose deploy laptop --port 80 -n study-ingress [root@k8s-master01 ~]# kubectl get po -n study-ingress -l app=laptop NAME READY STATUS RESTARTS AGE laptop-664b565969-vw822 1/1 Running 0 22s
之后创建电脑端的 Ingress，注意Ingress annotations的nginx.ingress.kubernetes.io/server-snippet配置。Snippet 配置是专门用于一些复杂的 Nginx 配置，和Nginx配置通用。
匹配移动端
实例如下：
[root@k8s-master01 ~]# vim laptop-ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/server-snippet: |
set $agentflag 0;
if ($http_user_agent ~* "(Android|iPhone|Windows Phone|UC|Kindle)" ){
set $agentflag 1;
}
if ( $agentflag = 1 ) {
return 301 http://m.test.com;
}
name: laptop
namespace: study-ingress
spec:
rules:
- host: test.com
http:
paths:
- backend:
service:
name: laptop
port:
number: 80
path: /
pathType: ImplementationSpecific
在本地hosts文件添加上m.test.com和test.com的dns解析，然后通过浏览器访问：
模拟移动端时，刷新页面会跳转到移动页面：
8.基本认证，添加账号密码
有些网站可能需要通过密码来访问，对于这类网站可以使用 Nginx 的 basic-auth 设置密码访问，具体方法如下，由于需要使用 htpasswd 工具，所以需要安装 httpd：
# yum install httpd -y
使用 htpasswd 创建 foo 用户的密码：
# htpasswd -c auth foo New password: Re-type new password: Adding password for user foo # cat auth foo:$apr1$okma2fx9$hdTJ.KFmi4pY9T6a2MjeS1
基于之前创建的密码文件创建 Secret：
kubectl create secret generic basic-auth --from-file=auth -n study-ingress secret/basic-auth created
创建包含密码认证的 Ingress：
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
# kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/auth-realm: Please Input Your Username and Password
nginx.ingress.kubernetes.io/auth-secret: basic-auth
nginx.ingress.kubernetes.io/auth-type: basic
name: ingress-with-auth
namespace: study-ingress
spec:
ingressClassName: nginx # for k8s >= 1.22+
rules:
- host: auth.test.com
http:
paths:
- backend:
service:
name: nginx
port:
number: 80
path: /
pathType: ImplementationSpecific
➢ nginx.ingress.kubernetes.io/auth-type：认证类型，可以是 basic 和 digest
➢ nginx.ingress.kubernetes.io/auth-secret：密码文件的 Secret 名称
➢ nginx.ingress.kubernetes.io/auth-realm：需要密码认证的消息提醒
创建该 Ingress，把auth.test.com加到本地hosts上，并访问测试：
[root@k8s-master01 ~]# kubectl create -f ingress-with-auth.yaml ingress.networking.k8s.io/ingress-with-auth created
9.黑白名单配置
(1).黑名单配置
配置黑名单禁止某一个或某一段 IP，需要在 Nginx Ingress 的 ConfigMap 中配置，比如将192.168.0.109（多个配置逗号分隔）添加至黑名单：
[root@k8s-master01 ~]# cd ingress-nginx/ && vim values.yaml
...37行左右
config:
block-cidrs: 192.168.0.109 #添加黑名单IP
apiVersion: v1
滚动更新 Nginx Ingress：
# helm upgrade ingress-nginx -n ingress-nginx .
使用 192.168.0.109 主机再次访问，发现该 IP 已经被禁止：
[root@mas-master03 ~]# curl -H "Host:auth.test.com" 192.168.0.111 -I HTTP/1.1 403 Forbidden Date: Mon, 27 Dec 2021 07:49:46 GMT Content-Type: text/html Content-Length: 146 Connection: keep-alive
(2).白名单配置
白名单表示只允许某个 IP 可以访问，直接在 yaml 文件中配置即可（也可以通过 ConfigMap
配置），比如只允许 192.168.10.128 访问，只需要添加一个 nginx.ingress.kubernetes.io/whitelist-source-range 注释即可：
[root@k8s-master01 ~]# vim ingress-with-auth.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
# kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/auth-realm: Please Input Your Username and Password
nginx.ingress.kubernetes.io/auth-secret: basic-auth
nginx.ingress.kubernetes.io/auth-type: basic
nginx.ingress.kubernetes.io/whitelist-source-range: 192.168.0.107
name: ingress-with-auth
namespace: study-ingress
spec:
ingressClassName: nginx # for k8s >= 1.22+
rules:
- host: auth.test.com
http:
paths:
- backend:
service:
name: nginx
port:
number: 80
path: /
pathType: ImplementationSpecific
更新ingress：
[root@k8s-master01 ~]# kubectl apply -f ingress-with-auth.yaml
然后通过加了白名单的IP：192.168.0.107是可以访问的：
[root@k8s-master01 ~]# curl -H "Host:auth.test.com" 192.168.0.111 -I HTTP/1.1 401 Unauthorized Date: Mon, 27 Dec 2021 08:04:51 GMT Content-Type: text/html Content-Length: 172 Connection: keep-alive WWW-Authenticate: Basic realm="Please Input Your Username and Password"
其它 IP 访问被禁止：
[root@k8s-master02 ~]# curl -H "Host:auth.test.com" 192.168.0.111 -I HTTP/1.1 403 Forbidden Date: Mon, 27 Dec 2021 08:04:57 GMT Content-Type: text/html Content-Length: 146 Connection: keep-alive
10.速率限制
有时候可能需要限制速率以降低后端压力，或者限制单个 IP 每秒的访问速率防止攻击。此时可以使用 Nginx 的 rate limit 进行配置。
首先没有加速率限制，使用 ab 进行访问，Failed 为 0：
[root@k8s-master01 ~]# ab -c 10 -n 100 http://www.test.com/ | grep requests Complete requests: 100 Failed requests: 0 Time per request: 0.584 [ms] (mean, across all concurrent requests) Percentage of the requests served within a certain time (ms)
添加速率限制，限制只能有一个连接，只需要添加 nginx.ingress.kubernetes.io/limitconnections 为 1 即可：
[root@k8s-master01 ~]# vim ingress-with-auth.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
# kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/auth-realm: Please Input Your Username and Password
nginx.ingress.kubernetes.io/auth-secret: basic-auth
nginx.ingress.kubernetes.io/auth-type: basic
nginx.ingress.kubernetes.io/limit-connections: "1"
name: ingress-with-auth
namespace: study-ingress
spec:
ingressClassName: nginx # for k8s >= 1.22+
rules:
- host: auth.test.com
http:
paths:
- backend:
service:
name: nginx
port:
number: 80
path: /
pathType: ImplementationSpecific
[root@k8s-master01 ~]# kubectl apply -f ingress-with-auth.yaml
ingress.networking.k8s.io/ingress-with-auth configured
再次使用 ab 测试，Failed 为 26：
[root@k8s-master01 ~]# ab -c 10 -n 100 http://auth.test.com/ | grep requests Complete requests: 100 Failed requests: 26 Time per request: 0.708 [ms] (mean, across all concurrent requests)
11.灰度发布
首先创建模拟 Production（生产）环境的 Namespace 和服务：
# kubectl create ns production namespace/production created # kubectl create deploy canary-v1 --image=registry.cn-beijing.aliyuncs.com/dotbalo/canary:v1 -n production # kubectl expose deploy canary-v1 --port 8080 -n production # kubectl create ingress canary-v1 --rule=canary.com/*=canary-v1:8080 -n production
使用浏览器访问该服务，可以看到 Canary v1 的页面：
接下来创建 v2 版本，充当灰度环境：
# kubectl create ns canary namespace/canary created
创建 v2 版本的应用和 Service：
# kubectl create deploy canary-v2 --image=registry.cnbeijing.aliyuncs.com/dotbalo/canary:v2 -n canary # kubectl expose deploy canary-v2 --port 8080 -n canary
待程序启动完成后，通过 Service 访问该服务，会返回 Canary v2：
[root@k8s-master01 ~]# kubectl get svc -n canary NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE canary-v2 ClusterIP 10.103.118.788080/TCP 85m [root@k8s-master01 ~]# curl 10.103.118.78:8080 Canary v2
创建 v2 版本的 Ingress 时，需要添加两个注释，一个是 nginx.ingress.kubernetes.io/canary，
表明是灰度环境，nginx.ingress.kubernetes.io/canary-weight 表明切多少流量到该环境，本示例为15%：
[root@k8s-master01 ~]# vim canary-v2.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
nginx.ingress.kubernetes.io/canary: “true”
nginx.ingress.kubernetes.io/canary-weight: “15”
name: canary-v2
namespace: canary
spec:
ingressClassName: nginx # for k8s >=1.22+
rules:
– host: canary.com
http:
paths:
– backend:
service:
name: canary-v2
port:
number: 8080
path: /
pathType: ImplementationSpecific
此时通过 nginx.ingress.kubernetes.io/canary-weight: “10”设置的权重是 10，即 v1:v2 为 100:85。
接下来使用 Ruby 脚本进行测试，此脚本会输出 v1 和 v2 的访问次数比值：
[root@k8s-master01 ~]# vim test-canary.rb
counts = Hash.new(0)
100.times do
output = `curl -s canary.com | grep ‘Canary’ | awk ‘{print $2}’ | awk -F”<" '{print $1}'`
counts[output.strip.split.last] += 1
end
puts counts
安装 ruby 并测试，总体偏差值不会太大：
# yum install ruby -y
# ruby test-canary.rb
[root@k8s-master01 ~]# ruby test-canary.rb
{"v1"=>88, "v2"=>12}
[root@k8s-master01 ~]# ruby test-canary.rb
{"v2"=>11, "v1"=>89}
[root@k8s-master01 ~]# ruby test-canary.rb
{"v1"=>85, "v2"=>15}
[root@k8s-master01 ~]# ruby test-canary.rb
{"v1"=>85, "v2"=>15}