kubernetes+ prometheus自动伸缩的设计与实现(一)

这篇blog名字起的搞得和我写论文一样，xxxx的设计与实现。其实这个东西原理很简单，kubernetes的hpa使用的是heapster，heapster是k8s那帮家伙在搞的，所以k8s还是喜欢自己搞的东西，所以k8s的hpa默认使用的heapster，但在业内，还有一个比heapster更好的监控方案，那就是prometheus。如果按照写论文的方式，我这边应该分别介绍一下k8s和prometheus，但真的没有那个闲功夫，在此略过，我之前blog也做过它们的源码分析。

上面的图片展示了整个体系结构
下面就分析一下adapter的具体实现，这里需要结合上一篇blog关于api聚合的功能，这个adapter就是通过api聚合的方式注册到apiserver上面。
先看一个hpa的例子

kind: HorizontalPodAutoscalerapiVersion: autoscaling/v2alpha1metadata:  name: wordpress  namespace: defaultspec:  scaleTargetRef:    apiVersion: apps/v1beta1    kind: Deployment    name: wordpress  minReplicas: 1  maxReplicas: 3  metrics:  - type: Pods    pods:      metricName: memory_usage_bytes      targetAverageValue: 100000

上面的代码添加一个内存使用量的hpa例子，通过targetAverageValue去设定阈值还有最大最小副本数，以及管理的deployment。在此额外说一下，hpa支持三种指标，分别是Object：描述k8s对象的某种指标，譬如ingress的hits-per-second。Pods：pod的平均指标，譬如transactions-processed-per-second，描述每个pod的事务每秒事务数，Resource是描述pod资源用量，譬如CPU或者内存。下面举一个object的例子,pod的http请求数。

  - type: Object    object:      target:        kind: Service        name: sample-metrics-app      metricName: http_requests      targetValue: 100

先从hpa代码开始pkg/controller/podautoscaler/horizontal.go，里面的computeReplicasForMetrics方法，它是负责获取监控指标并且计算副本数的方法。针对不同资源调用不同的方法：

case autoscalingv2.ObjectMetricSourceType:GetObjectMetricReplicas...case autoscalingv2.PodsMetricSourceType:GetMetricReplicascase autoscalingv2.ResourceMetricSourceType:GetRawResourceReplicas

先看看GetObjectMetricReplicas这个方法
pkg/controller/podautoscaler/replica_calculator.go

func (c *ReplicaCalculator) GetObjectMetricReplicas(currentReplicas int32, targetUtilization int64, metricName string, namespace string, objectRef *autoscaling.CrossVersionObjectReference) (replicaCount int32, utilization int64, timestamp time.Time, err error) {    utilization, timestamp, err = c.metricsClient.GetObjectMetric(metricName, namespace, objectRef)    if err != nil {        return 0, 0, time.Time{}, fmt.Errorf("unable to get metric %s: %v on %s %s/%s", metricName, objectRef.Kind, namespace, objectRef.Name, err)    }    usageRatio := float64(utilization) / float64(targetUtilization)    if math.Abs(1.0-usageRatio) <= c.tolerance {        // return the current replicas if the change would be too small        return currentReplicas, utilization, timestamp, nil    }    return int32(math.Ceil(usageRatio * float64(currentReplicas))), utilization, timestamp, nil}

GetObjectMetric是一个接口，有两个方法，就是上面图所示的heapster和自定义custom接口。heapster这个就是调用heapster接口去获取性能指标，本blog着重介绍自定义性能指标，在启动controller-manager时候指定–horizontal-pod-autoscaler-use-rest-clients就可以使用自定义的性能指标了
pkg/controller/podautoscaler/metrics/rest_metrics_client.go

func (c *customMetricsClient) GetObjectMetric(metricName string, namespace string, objectRef *autoscaling.CrossVersionObjectReference) (int64, time.Time, error) {    gvk := schema.FromAPIVersionAndKind(objectRef.APIVersion, objectRef.Kind)    var metricValue *customapi.MetricValue    var err error    if gvk.Kind == "Namespace" && gvk.Group == "" {        // handle namespace separately        // NB: we ignore namespace name here, since CrossVersionObjectReference isn't        // supposed to allow you to escape your namespace        metricValue, err = c.client.RootScopedMetrics().GetForObject(gvk.GroupKind(), namespace, metricName)    } else {        metricValue, err = c.client.NamespacedMetrics(namespace).GetForObject(gvk.GroupKind(), objectRef.Name, metricName)    }    if err != nil {        return 0, time.Time{}, fmt.Errorf("unable to fetch metrics from API: %v", err)    }    return metricValue.Value.MilliValue(), metricValue.Timestamp.Time, nil}

上面的objectRef针对本blog只为
{Kind:Service,Name:wordpress,APIVersion:,}，就是我们在yaml文件里面metrics里面定义。

上面通过vendor/k8s.io/metrics/pkg/client/custom_metrics/client.go

func (m *rootScopedMetrics) GetForObject(groupKind schema.GroupKind, name string, metricName string) (*v1alpha1.MetricValue, error) {    // handle namespace separately    if groupKind.Kind == "Namespace" && groupKind.Group == "" {        return m.getForNamespace(name, metricName)    }    resourceName, err := m.client.qualResourceForKind(groupKind)    if err != nil {        return nil, err    }    res := &v1alpha1.MetricValueList{}    err = m.client.client.Get().        Resource(resourceName).        Name(name).        SubResource(metricName).        Do().        Into(res)    if err != nil {        return nil, err    }    if len(res.Items) != 1 {        return nil, fmt.Errorf("the custom metrics API server returned %v results when we asked for exactly one", len(res.Items))    }    return &res.Items[0], nil}

通过client发送https请求获取metrics。具体发送如下所示object：

https://localhost:6443/apis/custom-metrics.metrics.k8s.io/v1alpha1/namespaces/default/services/wordpress/requests-per-second

如果是pod则发送的请求是

https://localhost:6443/apis/custom-metrics.metrics.k8s.io/v1alpha1/namespaces/default/pods/%2A/memory_usage_bytes?labelSelector=app%3Dwordpress%2Ctier%3Dfrontend

至于group为啥是custom-metrics.metrics.k8s.io这个不是别的，是代码里面写死的，vendor/k8s.io/metrics/pkg/apis/custom_metrics/v1alpha1/register.go

// GroupName is the group name use in this packageconst GroupName = "custom-metrics.metrics.k8s.io"// SchemeGroupVersion is group version used to register these objectsvar SchemeGroupVersion = schema.GroupVersion{Group: GroupName, Version: "v1alpha1"}

这样k8s的部分已经讲解完毕。下面就是adapter的部分了。