一.资源配额ResourceQuota
1.资源配额的重要性
如下三图,假设公司规模比较大,除了集群管理员外,还有很多项目组,开发,测试要使用集群,使用过程中可能不了解集群规模,会过度过量使用K8S资源,或者不及时释放不用的POD,最后照成资源浪费,资源紧缺的现象,所以引出资源配额的概念。
2.ResourceQuota配置
ResourceQuota创建在哪个命名空间下,就对哪个命名空间管理
pods:限制最多启动Pod的个数
requests.cpu:限制最高CPU请求数
requests.memory:限制最高内存的请求数
limits.cpu:限制最高CPU的limit上限
limits.memory:限制最高内存的limit上限
apiVersion: v1 kind: ResourceQuota #ResourceQuota配置标签 metadata: name: resource-test labels: app: resourcequota spec: hard: pods: 50 requests.cpu: 0.5 requests.memory: 512Mi limits.cpu: 5 limits.memory: 16Gi configmaps: 20 requests.storage: 40Gi persistentvolumeclaims: 20 replicationcontrollers: 20 secrets: 20 services: 50 services.loadbalancers: "2" services.nodeports: "10"
演示,设置限制最大2个pod,最大2个configmap
[root@k8s-master01 ~]# vim ResourceQuota.yaml apiVersion: v1 kind: ResourceQuota metadata: name: resource-test labels: app: resourcequota spec: hard: pods: 2 configmaps: 2 [root@k8s-master01 ~]# kubectl create ns rq-test #创建测试的命名空间 [root@k8s-master01 ~]# kubectl create -f ResourceQuota.yaml -n rq-test #在新建的命名空间下创建 [root@k8s-master01 ~]# kubectl get resourcequota -n rq-test #查看使用的资源,configmap已创建1个,pod没使用 NAME AGE REQUEST LIMIT resource-test 2m8s configmaps: 1/2, pods: 0/2 [root@k8s-master01 ~]# kubectl get cm -n rq-test #查看configmap NAME DATA AGE kube-root-ca.crt 1 4m44s
在此命名空间下创建一个副本数为3的deployment,查看,只ready了2个pod。
可以通过describe命令deploy和rs出现的问题。
先describe deploy,显示创建了rs管理pod,所以再describe rs查看。
[root@k8s-master01 ~]# kubectl create deployment test-1 --image=nginx --replicas=3 -n rq-test [root@k8s-master01 ~]# kubectl get deploy -owide -n rq-test #只启动了2个pod NAME READY UP-TO-DATE AVAILABLE AGE CONTAINERS IMAGES SELECTOR test-1 2/3 2 2 25s nginx nginx app=test-1 [root@k8s-master01 ~]# kubectl describe deploy test-1 -n rq-test #先describe deploy,显示创建了rs管理pod ..... Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal ScalingReplicaSet 3m59s deployment-controller Scaled up replica set test-1-595f7df994 to 3 [root@k8s-master01 ~]# kubectl describe rs test-1-595f7df994 -n rq-test #describe查看上面创建的rs ..... Warning FailedCreate 4m35s (x8 over 7m26s) replicaset-controller (combined from similar events): Error creating: pods "test-1-595f7df994-fwqzz" is forbidden: exceeded quota: resource-test, requested: pods=1, used: pods=2, limited: pods=2
再把pod数改为3,等待一会后,pod副本数也会变成3
[root@k8s-master01 ~]# vim ResourceQuota.yaml #把pod数改为3 ... pods: 3 ... [root@k8s-master01 ~]# kubectl replace -f ResourceQuota.yaml -n rq-test 更新查看pod会不会更行 [root@k8s-master01 ~]# kubectl get -f ResourceQuota.yaml -n rq-test #查看ResourceQuota是否更新 [root@k8s-master01 ~]# kubectl get deploy test-1 -n rq-test -owide #或者查看deployment是否更新pod
二.资源限制LimitRange
1.LimitRange的概念
如下两图,如果ResourceQuota只限制了内存和CPU,当创建的pod没有配置resources,他的cpu和内存就为0,然后可以无限创建pod,配置的resources就没有意义。
2.LimitRange的配置
默认的requests和limits
default:默认limits配置
defaultRequest:默认requests配置
apiVersion: v1 kind: LimitRange metadata: name: cpu-mem-limit-range spec: limits: - default: cpu: 1 memory: 512Mi defaultRequest: cpu: 0.5 memory: 256Mi type: Container
根据CPU来限制
max:内存CPU的最大配置
min:内存CPU的最小配置
apiVersion: v1 kind: LimitRange metadata: name: cpu-min-max-demo-lr spec: limits: - max: cpu: "800m" memory: 1Gi min: cpu: "200m" memory: 500Mi type: Container
限制申请存储空间的大小
max:最大PVC的空间
min:最小PVC的空间
apiVersion: v1 kind: LimitRange metadata: name: storagelimits spec: limits: - type: PersistentVolumeClaim max: storage: 2Gi min: storage: 1Gi
3.LimitRange的使用
根据上述三种场景做示例
(1).配置默认的request和limits
先创建配置限制CPUrequest和limits的limitrange-yaml。
[root@k8s-master01 ~]# vim limitrange.yaml #创建 apiVersion: v1 kind: LimitRange metadata: name: cpu-mem-limit-range spec: limits: - default: cpu: 1 memory: 512Mi defaultRequest: cpu: 0.5 memory: 256Mi type: Containe
在之前示例的rq-test空间创建,并在之前创建的deploy的一个pod删除,可以发现在新建的pod中已经有了resources。
[root@k8s-master01 ~]# kubectl create -f limitrange.yaml -n rq-test #再rq测试空间下创建limitrange limitrange/cpu-mem-limit-range created [root@k8s-master01 ~]# kubectl delete -f ResourceQuota.yaml -n rq-test #由于之前创建的ResourceQuota会影响示例,所以先删除 resourcequota "resource-test" deleted [root@k8s-master01 ~]# kubectl get pod -n rq-test #查看之前创建的deploy NAME READY STATUS RESTARTS AGE test-1-595f7df994-96qzk 1/1 Running 1 19h test-1-595f7df994-gnnxg 1/1 Running 1 18h test-1-595f7df994-jgmvg 1/1 Running 1 19h [root@k8s-master01 ~]# kubectl get pod -n rq-test test-1-595f7df994-96qzk -oyaml | grep resources -A 5 #由于删除已经没有了resources f:resources: {} resources: {} [root@k8s-master01 ~]# kubectl delete po test-1-595f7df994-jgmvg -n rq-test #删除一个pod,看新POD又没有resources pod "test-1-595f7df994-jgmvg" deleted [root@k8s-master01 ~]# kubectl get pod test-1-595f7df994-z4rtp -n rq-test -oyaml | grep resources -A 5 #查看新pod,已经有resources -- resources: limits: cpu: "1" memory: 512Mi requests: cpu: 500m
(2).配置requests和limits的范围
由于可以手动更改最低cpu数量,也会影响资源用量,所以也需要限制requests和limits的范围
在上述示例1上修改limitrange.yaml
[root@k8s-master01 ~]# vim limitrange.yaml apiVersion: v1 kind: LimitRange metadata: name: cpu-mem-limit-range spec: limits: - default: cpu: 1 memory: 512Mi defaultRequest: cpu: 0.5 memory: 256Mi type: Container max: cpu: "2" memory: 1Gi min: cpu: "10m" memory: 128Mi type: Container
更新limitrange.yaml,然后修改之前创建的deploy-test-1中pod的最小CPU数和最小内存数,查看并没有更新POD,排查后,发现在2分钟前创建了一个rs-test-1-64457bb565,再describe新建的RS,发现显示新建的pod最小CPU要是10m,最小内存要是128,不满足要求。
[root@k8s-master01 ~]# kubectl replace -f limitrange.yaml -n rq-test #更新limitrange.yaml limitrange/cpu-mem-limit-range replaced [root@k8s-master01 ~]# kubectl edit deployments.apps test-1 -n rq-test #修改最小CPU数和最小内存数 ... resources: requests: cpu: 1m memory: 64Mi [root@k8s-master01 ~]# kubectl get pod -n rq-test #查看并没有更新POD NAME READY STATUS RESTARTS AGE test-1-595f7df994-96qzk 1/1 Running 1 19h test-1-595f7df994-gnnxg 1/1 Running 1 19h test-1-595f7df994-z4rtp 1/1 Running 0 36m [root@k8s-master01 ~]# kubectl describe deployments.apps -n rq-test #排查,在2分钟前创建了一个rs-test-1-64457bb565 Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal ScalingReplicaSet 2m56s deployment-controller Scaled up replica set test-1-64457bb565 to 1 [root@k8s-master01 ~]# kubectl describe rs test-1-64457bb565 -n rq-test #查看新建的RS,发现显示新建的pod最小CPU要是10m,最小内存要是128,不满足要求 ... Warning FailedCreate 82s (x7 over 4m3s) replicaset-controller (combined from similar events): Error creating: pods "test-1-64457bb565-x6xsl" is forbidden: [minimum cpu usage per Container is 10m, but request is 1m, minimum memory usage per Container is 128Mi, but request is 64Mi]
(3).限制申请存储空间的大小
限制pvc存储
在上述示例基础上添加最大最小pvc容量限制
[root@k8s-master01 ~]# vim limitrange.yaml
apiVersion: v1 kind: LimitRange metadata: name: cpu-mem-limit-range spec: limits: - default: cpu: 1 memory: 512Mi defaultRequest: cpu: 0.5 memory: 256Mi type: Container max: cpu: "2" memory: 1Gi min: cpu: "10m" memory: 128Mi type: Container - type: PersistentVolumeClaim #追加一行-,与default同级 max: storage: 2Gi min: storage: 1Gi
更新limitrange并查看
[root@k8s-master01 ~]# kubectl replace -f limitrange.yaml -n rq-test limitrange/cpu-mem-limit-range replaced [root@k8s-master01 ~]# kubectl get limitranges -n rq-test -oyaml
创建8G的pvc,然后报错,显示最大为2G
[root@k8s-master01 ~]# vim limitranges-pvc.yaml #创建8G的PVC apiVersion: v1 kind: PersistentVolumeClaim metadata: name: pvc-nfs spec: accessModes: - ReadWriteOnce resources: requests: storage: 8Gi [root@k8s-master01 ~]# kubectl create -f limitranges-pvc.yaml -n rq-test #创建8G的PVC Error from server (Forbidden): error when creating "limitranges-pvc.yaml": persistentvolumeclaims "pvc-nfs" is forbidden: maximum storage usage per PersistentVolumeClaim is 2Gi, but request is 8Gi
创建500M的pvc,然后报错,显示最小为1G
[root@k8s-master01 ~]# kubectl create -f limitranges-pvc.yaml -n rq-test #创建500M的PVC [root@k8s-master01 ~]# vim limitranges-pvc.yaml #创建500M的PVC apiVersion: v1 kind: PersistentVolumeClaim metadata: name: pvc-nfs spec: accessModes: - ReadWriteOnce resources: requests: storage: 500Mi Error from server (Forbidden): error when creating "limitranges-pvc.yaml": persistentvolumeclaims "pvc-nfs" is forbidden: minimum storage usage per PersistentVolumeClaim is 1Gi, but request is 500Mi
三.服务质量QoS
1.QoS的概念
如下图,假设一台节点是4核8G,创建pod时一些pod的Limit上限较高。当Limit总和超过节点的CPU内存上限,可能刚创建的时候没问题,使用一段时间后pod暂用资源数增大,节点资源就会满载(上图示例超过4核8G)。此时,k8s就会杀掉重启一些pod来保证节点的稳定。
由于pod应用也分重要性,例如mysql服务重要性就大于redis大于nginx和java。
K8S就可以引用了服务质量QoS来保证先杀优先度最低的节点。例如先杀nginx和java,再杀redis,再杀mysql。
2.QoS的三种模式
k8s是通过在配置文件配置resources的limits和requests来区分三种模式,具体如下:
1.Guaranteed:最高服务质量,当宿主机内存不够时,会先kill掉QoS为BestEffort和Burstable的Pod,如果内存还是不够,才会kill掉QoS为Guaranteed,该级别Pod的资源占用量一般比较明确,即requests的cpu和memory和limits的cpu和memory配置的一致。
2.Burstable: 服务质量低于Guaranteed,当宿主机内存不够时,会先kill掉QoS为BestEffort的Pod,如果内存还是不够之后就会kill掉QoS级别为Burstable的Pod,用来保证QoS质量为Guaranteed的Pod,该级别Pod一般知道最小资源使用量,但是当机器资源充足时,还是想尽可能的使用更多的资源,即limits字段的cpu和memory大于requests的cpu和memory的配置。
3.BestEffort:尽力而为,当宿主机内存不够时,首先kill的就是该QoS的Pod,用以保证Burstable和Guaranteed级别的Pod正常运行。
(1).实现QoS为Guaranteed的Pod
Pod中的每个容器必须指定limits.memory和requests.memory,并且两者需要相等;
Pod中的每个容器必须指定limits.cpu和limits.memory,并且两者需要相等。
apiVersion: v1 kind: Pod metadata: name: qos-demo namespace: qos-example spec: containers: - name: qos-demo-ctr image: nginx resources: limits: #limit和requests相等。 memory: "200Mi" cpu: "700m" requests: memory: "200Mi" cpu: "700m"
(2).实现QoS为Burstable的Pod
Pod不符合Guaranteed的配置要求;
Pod中至少有一个容器配置了requests.cpu或requests.memory。
apiVersion: v1 kind: Pod metadata: name: qos-demo-2 namespace: qos-example spec: containers: - name: qos-demo-2-ctr image: nginx resources: limits: memory: "200Mi" requests: memory: "100Mi"
(3).实现QoS为BestEffort的Pod
实现QoS为BestEffort的Pod
apiVersion: v1 kind: Pod metadata: name: qos-demo-3 namespace: qos-example spec: containers: - name: qos-demo-3-ctr image: nginx