kubernetes学习4--资源配额管理(租户配额)

    上一篇博文学习了Namespace，知道通过Namespace可以对集群中的可用资源进行分配和限制。那这篇就学习下，怎样指定资源配额。

    测试环境，2台node服务器，都是CPU为1，内存为1G。查询方法，可以参考另一篇文章“centOS7下实践查询版本/CPU/内存/硬盘容量等硬件信息”。或者kubectl get nodes查询到所有nodes后，bubectl describe node nodeName可查询到node的详细信息，推荐使用这种方法。

1.指定容器配额

    对应指定容器实施配额管理非常简单，只要在Pod或ReplicationController的定义文件中设定resource属性即可为某个容器指定配额。目前容器支持CPU和Memory两类资源的配额限制。

    下面这个RC定义文件中增加资源配额声明。

apiVersion: v1kind: ReplicationControllermetadata:   name: php-controller   labels:      name: php-controllerspec:  replicas: 1  selector:     name: php-test-pod  template:     metadata:     labels:       name: php-test-pod    spec:      containers:      - name: php-test        image: 192.168.174.131:5000/php-base:1.0        env:        - name: ENV_TEST_1          value: env_test_1        - name: ENV_TEST_2          value: env_test_2        ports:        - containerPort: 80        resources:          limits:            cpu: 0.5            memory: 512Mi

[root@localhost k8s]# kubectl create -f php-controller.yaml replicationcontroller "php-controller" created[root@localhost k8s]# kubectl get pods -o wideNAME                   READY     STATUS    RESTARTS   AGE       NODEphp-controller-qx1wg   1/1       Running   0          47s       192.168.174.130[root@localhost k8s]# kubectl describe pod php-controller-qx1wg Name:           php-controller-qx1wgNamespace:      defaultNode:           192.168.174.130/192.168.174.130Start Time:     Fri, 11 Nov 2016 17:27:21 +0800Labels:         name=php-test-podStatus:         RunningIP:             172.17.42.2Controllers:    ReplicationController/php-controllerContainers:  php-test:    Container ID:       docker://1777abd035e7cc3c8dee9eb27487e76424aff61a26143b0f23fb0c411415ed5b    Image:              192.168.174.131:5000/php-base:1.0    Image ID:           docker://sha256:104c7334b9624b054994856318e54b6d1de94c9747ab9f73cf25ae5c240a4de2    Port:               80/TCP    QoS Tier:      memory:   Guaranteed      cpu:      Guaranteed    Limits:      cpu:      500m      memory:   512Mi    Requests:      cpu:              500m      memory:           512Mi    State:              Running      Started:          Fri, 11 Nov 2016 17:27:22 +0800    Ready:              True    Restart Count:      0    Environment Variables:      ENV_TEST_1:       env_test_1      ENV_TEST_2:       env_test_2Conditions:  Type          Status  Ready         True No volumes.Events:  FirstSeen     LastSeen        Count   From                            SubobjectPath                   Type            Reason                  Message  ---------     --------        -----   ----                            -------------                   --------        ------                  -------  1m            1m              2       {kubelet 192.168.174.130}                                       Warning         MissingClusterDNS       kubelet does not have ClusterDNS IP configured and cannot create Pod using "ClusterFirst" policy. Falling back to DNSDefault policy.  1m            1m              1       {kubelet 192.168.174.130}       spec.containers{php-test}       Normal          Pulled                  Container image "192.168.174.131:5000/php-base:1.0" already present on machine  1m            1m              1       {kubelet 192.168.174.130}       spec.containers{php-test}       Normal          Created                 Created container with docker id 1777abd035e7  1m            1m              1       {kubelet 192.168.174.130}       spec.containers{php-test}       Normal          Started                 Started container with docker id 1777abd035e7  1m            1m              1       {default-scheduler }                                            Normal          Scheduled               Successfully assigned php-controller-qx1wg to 192.168.174.130

  

   如果资源配额声明的大小超出了实际物理机的CPU和内存，会怎样呢？是不是无法创建成功？把CPU改为3，memory改为3Gi,测试下。

[root@localhost k8s]# kubectl get pods -o wideNAME                   READY     STATUS    RESTARTS   AGE       NODEphp-controller-47ufd   0/1       Pending   0     

  发现Pod的状态是Pendding。kubectl describe看下Pod的信息

[root@localhost k8s]# kubectl describe pod php-controller-47ufd Name:           php-controller-47ufdNamespace:      defaultNode:           /Labels:         name=php-test-podStatus:         PendingIP:Controllers:    ReplicationController/php-controllerContainers:  php-test:    Image:      192.168.174.131:5000/php-base:1.0    Port:       80/TCP    QoS Tier:      cpu:      Guaranteed      memory:   Guaranteed    Limits:      memory:   3Gi      cpu:      3    Requests:      memory:   3Gi      cpu:      3    Environment Variables:      ENV_TEST_1:       env_test_1      ENV_TEST_2:       env_test_2No volumes.Events:  FirstSeen     LastSeen        Count   From                    SubobjectPath   Type            Reason                  Message  ---------     --------        -----   ----                    -------------   --------        ------                  -------  3m            3m              1       {default-scheduler }                    Warning         FailedScheduling        pod (php-controller-47ufd) failed to fit in any nodefit failure on node (192.168.174.131): Node didn't have enough resource: CPU, requested: 3000, used: 0, capacity: 1000fit failure on node (192.168.174.130): Node didn't have enough resource: CPU, requested: 3000, used: 0, capacity: 1000  3m    15s     13      {default-scheduler }            Warning FailedScheduling        pod (php-controller-47ufd) failed to fit in any nodefit failure on node (192.168.174.130): Node didn't have enough resource: CPU, requested: 3000, used: 0, capacity: 1000fit failure on node (192.168.174.131): Node didn't have enough resource: CPU, requested: 3000, used: 0, capacity: 1000

  可以看到Events信息中，显示Pod调度到192.168.174.131这个Node上，发现CPU实际只有1000，但是申请3000，所以这个Node资源不足。接着调度到192.168.174.130这个Node上，发现也是资源不足。所以Pod无法Running,一直Pandding，一直在所有Node上循环检查，直到某个Node上资源足够。

2.全局默认配额

   除了可以直接在容器（或RC）的定义文件中指定的容器增加资源配额参数，我们还可以通过创建LimitRange对象来定义一个全局默认配额模版。这个默认配额模版会加载到集群中的每个Pod及容器上，这样就不用我们手工为每个Pod和容器重复设置了。

   LimitRange对象可以同时在Pod和Container两个级别上进行对资源配额的设置。当LimitRange创建生效后，之后创建的Pod都将使用LimitRange设置的资源配额进行约束。

   创建个pod-container-limitRnage.yaml文件

apiVersion: v1kind: LimitRangemetadata:  name: limit-range-testspec:  limits:  - type: Pod    max:      cpu: 1      memory: 1Gi    min:      cpu: 0.5      memory: 216Mi  - type: Container    max:      cpu: 1      memory: 1Gi    min:      cpu: 0.5      memory: 216Mi    default:      cpu: 0.5      memory: 512Mi

上述设置表明：

  1）任意Pod内的所有容器的CPU使用都限制在0.5--1，内存使用限制在216Mi--1Gi

  2)任意容器的CPU使用都限制在0.5--1,默认0.5，内存使用限制在216Mi--1Gi，默认512Mi

  

[root@localhost k8s]# kubectl create -f pod-container-limitRange.yaml limitrange "limit-range-test" created[root@localhost k8s]# kubectl get limitrange NAME               AGElimit-range-test   12s[root@localhost k8s]# kubectl describe limitrange limit-range-test Name:           limit-range-testNamespace:      defaultType            Resource        Min     Max     Default Request Default Limit   Max Limit/Request Ratio----            --------        ---     ---     --------------- -------------   -----------------------Pod             memory          216Mi   1Gi     -               -               -Pod             cpu             500m    1       -               -               -Container       cpu             500m    1       500m            500m            -Container       memory          216Mi   1Gi     512Mi           512Mi           -[root@localhost k8s]# 

  写个Pod或RC，不指定resource，然后查看Pod信息，会看到cpu默认500m，memory=512Mi。

  此外，如果我们在Pod或RC定义文件中指定配额参数，则可遵循局部覆盖全局的原则。当然，如果配额超过了全局设定的最大值，那kubectl ceate的时候就会爆粗。在这就不一一演示了。

   LimitRange也是跟Namespace捆绑的，如果定义文件中没指定，则默认是default，kubectl describe limitrange limit-range-test 的时候，也会看到Namespace:default信息。

3.多租户配额管理

    多租户在Kubernetes中以Namespace来体现的，这里的多租户可以是多个用户、多个业务系统或者相互隔离的多种作业环境。一个集群中的资源总是有限的，当这个集群被多个租户的应用同时使用时，为了更好的使用这种有限的共有资源，我们需要将资源配额的管理单元提升到租户级别，只需要在不同组合对应的Namespace上加载对应的ResourceQuota配置即可达到目的。

   创建development Namespace

[root@localhost k8s]# cat namespace-dev.yaml apiVersion: v1kind: Namespacemetadata:   name: development   labels:     name: development

   上面创建了development Namespace，describe查看，会发现，它对资源配额没有任何限制。

[root@localhost k8s]# kubectl describe namespace development Name:   developmentLabels: name=developmentStatus: ActiveNo resource quota.No resource limits.

    现有2台Node服务器，总共cpu 2，memory 2Gi，给development namespce配额cpu 1,memory 1Gi。
    创建dev-resourcequota.yaml,不但可以对CPU,memory做限制，还可以对pods/services/replicationcontrollers/resourcequotas/secrets/configmaps/persistentvolumeclaims/services.nodeports/services.loadbalancers/requests.cpu/requests.memory/limits.cpu/limits.memory/cpu/memory/storage等资源做出限制.注意定义文件中namespace不能漏掉，否则默认是default（文件中如果没指明namespace,则kubectl create -f xxFile --namespace=development也行）

apiVersion: v1kind: ResourceQuotametadata:   name: quota-development   namespace: developmentspec:  hard:    cpu: 1    memory: 1Gi    persistentvolumeclaims: 10    pods: 50    replicationcontrollers: 20    resourcequotas: 1    secrets: 20    services: 20

[root@localhost k8s]# kubectl create -f dev-resourcequota.yaml resourcequota "quota-development" created

[root@localhost k8s]# kubectl get resourcequotas --namespace=developmentNAME                AGEquota-development   1m

查看使用情况

[root@localhost k8s]# kubectl describe resourcequota quota-development --namespace=developmentName:                   quota-developmentNamespace:              developmentResource                Used    Hard--------                ----    ----cpu                     0       1memory                  0       1Gipersistentvolumeclaims  0       10pods                    0       50replicationcontrollers  0       20resourcequotas          1       1secrets                 0       20services                0       20

还可查看这时namespace的信息

[root@localhost k8s]# kubectl describe namespace development Name:   developmentLabels: name=developmentStatus: ActiveResource Quotas Name:                  quota-development Resource               Used    Hard --------               ---     --- cpu                    0       1 memory                 0       1Gi persistentvolumeclaims 0       10 pods                   0       50 replicationcontrollers 0       20 resourcequotas         1       1 secrets                0       20 services               0       20No resource limits.

看到Resource Quoata已经做出了限制，至于resource limit，只要按照上面的全局默认配额操作就行了（定义文件中要指明namespace，文件中如果没指明namespace,则kubectl create -f xxFile --namespace=development也行）

Resource limits限制的是Pod和container的cpu/memory，而resource quota是对namespace的限制，也就是说所有的Pod/container的总额不能超过quota的限制。

  创建完ResourceQuota之后，对于所有需要创建的Pod都必须指定具体的资源配额设置，否则，创建Pod会失败。

apiVersion: v1kind: Podmetadata:   name: php-test   labels:      name: php-test   namespace: developmentspec:  containers:  - name: php-test    image: 192.168.174.131:5000/php-base:1.0    env:    - name: ENV_TEST_1      value: env_test_1    - name: ENV_TEST_2      value: env_test_2    ports:    - containerPort: 80      hostPort: 80

[root@localhost k8s]# kubectl create -f php-pod.yaml Error from server: error when creating "php-pod.yaml": pods "php-test" is forbidden: Failed quota: quota-development: must specify cpu,limits.cpu,limits.memory,memory[root@localhost k8s]#