转载

consul仅当服务发现在k8s无状态部署使用

consul server在机器上的部署已经写完了，但是dba还是希望部署在k8s上，昨天搜了下相关文章扣出来部分步骤自己验证实现了。

先说下需求，三台物理机单独跑mysql主从mha，利用consul client的服务注册+ttl向consul server注册成域名，然后配置k8s内部的coredns把域名 *.service.consul 转发到consul server。架构为下图

Physical server
 +---------------------+                            K8S Cluster
 |                     |                   +--------------------------------+
 |mysql + consul client+-----------+       |     +---------------------+    |
 |                     |           |       |     |    +-------------+  |    |
 +---------------------+           |       |     |    |consul server|  |    |
Physical server                    |       |     | D  +------+------+  |    |
 +---------------------+           |       |     | e         |         |    |
 |                     |           |       |     | p  +------+------+  |    |
 |mysql + consul client+----------->------>+     | l  |consul server|  |    |
 |                     |           |       |     | o  +------+------+  |    |
 +---------------------+           |       |     | y         |         |    |
Physical server                    |       |     |    +------+------+  |    |
 +---------------------+           |       |     |    |consul server|  |    |
 |                     |           |       |     |    +-------------+  |    |
 |mysql + consul client+-----------+       |     +---------------------+    |
 |                     |                   +--------------------------------+
 +---------------------+

consul有k8s client的相关代码，赋予RBAC后能够用selector能够自动join到其他成员

"retry_join": [
    "provider=k8s label_selector=/"app=consul,component=server/""
],

当然其实市面上还有其他的方案的，为啥我不用

官方的helm: https://github.com/hashicorp/consul-helm 太罗嗦，文件也很多，而且不是tls，花里胡哨的
官方的consul-k8s: https://github.com/hashicorp/consul-k8s 这个方案可以见 https://medium.com/wish-engineering/katalog-sync-reliable-integration-of-consul-and-kubernetes-ebe8aae0852a 同步service到k8s，或者k8s域名同步到consul，没这个必要

镜像准备

因为这里我是consul server跑k8s里，consul client在物理机上，通信我要配置成tls，tls我是secret导入的，cm写配置文件。挂载到pod里后因为官方docker镜像的 entrypoint.sh 里chown了配置文件目录，而挂载到pod内部是只读的，pod起来后执行到chown那就报错退出了。所以改造了下官方的entrypoint脚本

FROM consul:1.6.1
LABEL maintainer="zhangguanzhang <zhangguanzhang@qq.com>"

ARG TZ=Asia/Shanghai

RUN set -eux && /
    ln -sf /usr/share/zoneinfo/${TZ} /etc/localtime && /
    echo ${TZ} > /etc/timezone
COPY docker-entrypoint.sh /usr/local/bin/docker-entrypoint.sh

docker-entrypoint.sh 见我github https://raw.githubusercontent.com/zhangguanzhang/Dockerfile/master/consul-k8s-dns/docker-entrypoint.sh

Dockerfile我是打算直接改官方的而不是FROM镜像，但是gpg key出问题所以我写成上面那样构建的镜像。见issue https://github.com/hashicorp/docker-consul/issues/137

自己构建镜像的话 docker-entrypoint.sh 记得加执行权限，我构建的镜像推送到dockerhub上可以直接拉取

tls

证书操作都是在容器里生成的

mkdir ssl
docker run --rm -ti --entrypoint sh --workdir /root -v $PWD/ssl:/root/ zhangguanzhang/consul-k8s-dns

我们先进入容器

step 1: 创建ca

为了简单起见，这里我使用Consul命令行的的内置TLS功能来创建基本的CA。您只需为数据中心创建一个CA。您应该在用于创建CA的同一服务器上生成所有证书。

ca默认五年，其他的证书默认1年，这里需要带参数 -days= 设置长点的日期

consul tls ca create -days=36500
==> Saved consul-agent-ca.pem
==> Saved consul-agent-ca-key.pem

step2: 创建server角色的证书

这里数据中心默认名字为dc1，其他的自行选项赋值。在创建CA的同一台服务器上重复此过程，直到每台服务器都有一个单独的证书。该命令可以反复调用，它将自动增加证书和密钥号。您将需要将证书分发到服务器。

严格来说每个server单独使用，也就是说假如三个server，下面命令应该执行三次。但是我们仅仅当无状态服务使用，如果每个server单独一套证书就不能放secret里，毕竟文件名不一样不能对照到pod上，这里我们仅仅生成一份

consul tls cert create -server -dc=dc1 -days=36500
==> WARNING: Server Certificates grants authority to become a
    server and access all state in the cluster including root keys
    and all ACL tokens. Do not distribute them to production hosts
    that are not server nodes. Store them as securely as CA keys.
==> Using consul-agent-ca.pem and consul-agent-ca-key.pem
==> Saved dc1-server-consul-0.pem
==> Saved dc1-server-consul-0-key.pem

step3: 创建client角色的证书

在Consul 1.5.2中，您可以使用替代过程来自动将证书分发给客户端。要启用此新功能，请设置 auto_encrypt 。

您可以继续使用生成证书 consul tls cert create -client 并手动分发证书。对于需要高度保护的数据中心，仍然需要现有的工作流程。

如果您正在运行Consul 1.5.1或更早版本，则需要使用来为每个客户端创建单独的证书 consul tls cert create -client 。客户端证书也由您的CA签名，但是它们没有特殊性 Subject Alternative Name ，这意味着如果 verify_server_hostname 启用，则它们不能作为server角色启动。

这里我是高于1.5.2的，不需要为每个客户端创建证书，客户端只需要拥有 consul-agent-ca.pem 这个ca下，会自动从server获取证书存在内存中，并且不会持久保存。但是我测试了并没有成功，还是生成了证书

$ consul tls cert create -client -dc=dc1 -days=36500
==> Using consul-agent-ca.pem and consul-agent-ca-key.pem
==> Saved dc1-client-consul-0.pem
==> Saved dc1-client-consul-0-key.pem
$ consul tls cert create -client -dc=dc1 -days=36500
==> Using consul-agent-ca.pem and consul-agent-ca-key.pem
==> Saved dc1-client-consul-1.pem
==> Saved dc1-client-consul-1-key.pem

客户端实体服务跑，可以单独去生成而不是共用一套

step4: 创建cli的证书

$ consul tls cert create -cli -dc=dc1 -days=36500
==> Using consul-agent-ca.pem and consul-agent-ca-key.pem
==> Saved dc1-cli-consul-0.pem
==> Saved dc1-cli-consul-0-key.pem

创建完证书ctrl+d退出容器

k8s相关yaml

consul可以跨数据中心，wan相关配置(hostPort模拟)我没有测试成功，而且考虑到高可用，基本是每台机器一台。所以我用了硬性互斥+hostNetwork，关闭serfwan端口。

nodeSelector 记得自己去打label固定住

apply之前我们先把证书导入成secret

cd ssl
kubectl create secret generic consul /
  --from-file=consul-agent-ca.pem /
  --from-file=dc1-server-consul-0.pem /
  --from-file=dc1-server-consul-0-key.pem /
  --from-file=dc1-cli-consul-0.pem /
  --from-file=dc1-cli-consul-0-key.pem

yaml文件

另外要注意，因为每次pod重启基本相当于node-id变了，参数 leave_on_terminate 应该false，见 https://github.com/hashicorp/consul/issues/6672 和 https://github.com/hashicorp/consul/issues/3938 所以preStop的部分没啥用，忽略即可

因为coredns的转发需要ip或者文件，所以我们得把consul-server的svc的clusterIP给固定住(记住得在svc的cidr内选一个未使用的ip)，或者干脆不创建svc直接使用hostIP，这里按照svc实例

apiVersion: apps/v1
kind: Deployment
metadata:
  name: consul-server
spec:
  selector:
    matchLabels:
      app: consul
      component: server
  replicas: 3
  template:
    metadata:
      labels:
        app: consul
        component: server
    spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            - labelSelector:
                matchExpressions:
                  - key: app
                    operator: In
                    values:
                      - consul
              topologyKey: kubernetes.io/hostname
      serviceAccountName: consul-server
      nodeSelector:
        master: "true"
      terminationGracePeriodSeconds: 7
      hostNetwork: true
      securityContext:
        fsGroup: 1000
      containers:
        - name: consul
          image: "zhangguanzhang/consul-k8s-dns"
          env:
            - name: POD_IP
              valueFrom:
                fieldRef:
                  fieldPath: status.podIP
            - name: NODE
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName
            - name: CONSUL_HTTP_ADDR
              value: https://localhost:8501
            - name: CONSUL_CACERT
              value: /consul/config/ssl/consul-agent-ca.pem
            - name: CONSUL_CLIENT_CERT
              value: /consul/config/ssl/dc1-cli-consul-0.pem
            - name: CONSUL_CLIENT_KEY
              value: /consul/config/ssl/dc1-cli-consul-0-key.pem
            # - name: RETRY_JOIN
            #   value: 172.19.0.5,172.19.0.6,172.19.0.7
          args:
            - agent
            - -advertise=$(POD_IP)
            - -node=$(NODE)
          volumeMounts:
            - name: data
              mountPath: /consul/data
            - name: config
              mountPath: /consul/config/
            - name: tls
              mountPath: /consul/config/ssl/
          lifecycle:
            preStop:
              exec:
                command:
                - consul leave
          readinessProbe:
            exec:
              command:
                - consul
                - members
            failureThreshold: 2
            initialDelaySeconds: 10
            periodSeconds: 5
            successThreshold: 1
            timeoutSeconds: 3
          ports:
            - containerPort: 8300
              name: server
            - containerPort: 8301
              name: serflan
            # - containerPort: 8302
            #   name: serfwan
            - containerPort: 8400
              name: alt-port
            - containerPort: 8501
              name: https
            - containerPort: 8600
              name: dns-udp
              protocol: UDP
            - containerPort: 8600
              name: dns-tcp
              protocol: TCP
      volumes:
        - name: config
          configMap:
            name: consul-server
        - name: tls
          secret:
            secretName: consul  
        - name: data
          emptyDir: {}
---
apiVersion: v1
kind: Service
metadata:
  name: consul-server
  namespace: default
  labels:
    app: consul
spec:
  clusterIP: 10.96.0.11
  ports:
    - name: https
      port: 8501
      targetPort: https
    - name: serflan-tcp
      protocol: "TCP"
      port: 8301
      targetPort: 8301
    - name: serflan-udp
      protocol: "UDP"
      port: 8301
      targetPort: 8301
    # - name: serfwan-tcp
    #   protocol: "TCP"
    #   port: 8302
    #   targetPort: 8302
    # - name: serfwan-udp
    #   protocol: "UDP"
    #   port: 8302
    #   targetPort: 8302
    - name: server
      port: 8300
      targetPort: 8300
    - name: dns-tcp
      protocol: "TCP"
      port: 8600
      targetPort: dns-tcp
    - name: dns-udp
      protocol: "UDP"
      port: 8600
      targetPort: dns-udp
  selector:
    app: consul
    component: server
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: consul-server
  namespace: default
  labels:
    app: consul
    role: server
data:
  server.json: |-
    {
        "client_addr": "0.0.0.0",
        "datacenter": "dc1",
        "bootstrap_expect": 3,
        "domain": "consul",
        "skip_leave_on_interrupt": true,
        "leave_on_terminate" : false,
        "log_level": "INFO",
        "retry_join": [
            "provider=k8s label_selector=/"app=consul,component=server/""
        ],
        "retry_interval": "2s",
        "verify_incoming": true,
        "verify_outgoing": true,
        "verify_server_hostname": true,
        "ca_file": "/consul/config/ssl/consul-agent-ca.pem",
        "cert_file": "/consul/config/ssl/dc1-server-consul-0.pem",
        "key_file": "/consul/config/ssl/dc1-server-consul-0-key.pem",
        "ports": {
          "http": -1,
          "serf_wan": -1,
          "https": 8501
        },
        "server": true,
        "ui": false
    }
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: consul-server
  labels:
    app: consul
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: consul
  labels:
    app: consul
rules:
  - apiGroups: [""]
    resources:
      - pods
    verbs:
      - get
      - list
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: consul
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: consul
subjects:
  - kind: ServiceAccount
    name: consul-server
    namespace: default

apply后

[root@k8s-m1 consul-tls-ansible]# kubectl get pod -o wide
NAME                             READY   STATUS    RESTARTS   AGE     IP           NODE         NOMINATED NODE   READINESS GATES
consul-server-7879498d78-8qjbs   1/1     Running   0          2m34s   172.19.0.6   172.19.0.6   <none>           <none>
consul-server-7879498d78-drm7n   1/1     Running   0          2m34s   172.19.0.7   172.19.0.7   <none>           <none>
consul-server-7879498d78-zjc6z   1/1     Running   0          2m34s   172.19.0.5   172.19.0.5   <none>           <none>

client是实体服务，用我的ansible部署的,client配置服务注册

{
  "services": [
    {
      "name": "r-3306-mysql",
      "tags": [
        "slave-3306"
      ],
      "address": "172.16.0.2",
      "port": 3306,
      "checks": [
        {
          "args": ["echo"], //实际应该写健康检查脚本,并去掉此处注释
          "interval": "5s"
        }
      ]
    }
  ]
}

client的consul reload后看到服务注册上去

2019/10/24 13:32:12 [INFO] agent: (LAN) joined: 3
2019/10/24 13:32:12 [INFO] agent: Join LAN completed. Synced with 3 initial agents
2019/10/24 13:32:12 [INFO] agent: Synced service "r-3306-mysql"
2019/10/24 13:32:21 [INFO] agent: Synced check "service:r-3306-mysql"

查看members

[root@k8s-m1 consul-tls-ansible]# kubectl exec consul-server-7879498d78-8qjbs consul members
Node         Address           Status  Type    Build  Protocol  DC   Segment
172.19.0.5   172.19.0.5:8301   alive   server  1.6.1  2         dc1  <all>
172.19.0.6   172.19.0.6:8301   alive   server  1.6.1  2         dc1  <all>
172.19.0.7   172.19.0.7:8301   alive   server  1.6.1  2         dc1  <all>
172.19.0.13  172.19.0.13:8301  alive   client  1.6.1  2         dc1  <default>
[root@k8s-m1 consul-tls-ansible]# kubectl get svc
NAME            TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)                                                 AGE
consul-server   ClusterIP   10.96.0.11   <none>        8501/TCP,8301/TCP,8301/UDP,8300/TCP,8600/TCP,8600/UDP   2m48s
kubernetes      ClusterIP   10.96.0.1    <none>        443/TCP                                                 2d22h

直接使用svc来测试域名解析

[root@k8s-m1 consul-tls-ansible]# dig -p 8600 @10.96.0.11 r-3306-mysql.service.consul +short
172.16.0.2

配置coredns转发，coredns的configmap添加下面内容，也可以添加上client的ip

...
    service.consul:53 {
        errors
        cache 30
        forward . 10.96.0.11:8600 172.19.0.13:8600
    }

跑一个指定版本带解析的工具pod

$ cat<<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: busybox
  namespace: default
spec:
  containers:
  - name: busybox
    image: busybox:1.28
    command:
      - sleep
      - "3600"
    imagePullPolicy: IfNotPresent
  restartPolicy: Always
EOF

测试解析，不影响集群的内部解析

$ kubectl exec -ti busybox -- nslookup r-3306-mysql.service.consul
Server:    10.96.0.10
Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local

Name:      r-3306-mysql.service.consul
Address 1: 172.16.0.2
$ kubectl exec -ti busybox -- nslookup kubernetes
Server:    10.96.0.10
Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local

Name:      kubernetes
Address 1: 10.96.0.1 kubernetes.default.svc.cluster.local

原文 https://zhangguanzhang.github.io/2019/10/24/deployment-consul-in-k8s/

正文到此结束