转载

Kubernetes技术分析之安全

【编者的话】Docker的流行激活了一直不温不火的PaaS，随着而来的是各类Micro-PaaS的出现，Kubernetes是其中最具代表性的一员，它是Google多年大规模容器管理技术的开源版本。本系列文章将逐一分析Kubernetes，本文说明Kubernetes的安全方案和考虑。

Kubernetes安全

安全永远是一个重大的话题，特别是云计算平台，更需要设计出一套完善的安全方案，以应对复杂的场景。 Kubernetes主要使用Docker作为应用承载环境，Kubernetes首先设计出一套API和敏感信息处理方案，当然也基于Docker提供容器安全控制。以下是Kubernetes的安全设计原则：

1. 保证容器与其运行的宿主机之间有明确的隔离

2. 限制容器对基础设施或者其它容器造成不良影响的能力

3. 最小特权原则——限定每个组件只被赋予了执行操作所必需的最小特权，由此确保可能产生的损失达到最小

4. 允许系统用户明确区别于管理员

5. 允许赋予管理权限给用户

6. 允许应用能够从公开数据中提取敏感信息（keys, certs, passwords）

Authentication & Authorization

Kubernetes API Server是对外暴露的API访问地址，API server提供了认证（Authentication）和授权（Authorization）机制进行安全控制。

- Authentication

支持Client certificate authentication 、Token authentication 、Basic authentication集中方式。

- Authorization

在Authentication的基础上，Authorization可以对HTTP请求设置AlwaysDeny、AlwaysAllow、ABAC三种模式模式，其中ABAC可以设置不同用户的访问权限。

现在使用Basic authentication + ABAC model设置API server，

首先配置Basic authentication,设置用户密码,格式为每行password, user name, user id，

basic_auth.csv：

admin_passwd,admin,admin
 test_passwd,test,test

然后配置ABAC访问策略, 设置admin具有任何权限，test用户只能访问pods，

policy_file.jsonl:

{"user":"admin"}
 {"user":"test", "resource": "pods", "readonly": true}

访问策略配置详情参考：

https://github.com/kubernetes/ ... rithm

然后启动API Server：

$kube-apiserver
 ...
 --basic-auth-file=basic_auth.csv /
 --authorization-mode=ABAC --authorization-policy-file=policy_file.jsonl

访问API，可以看到test用户无法访问Pod之外的资源：

$ curl --basic -u admin:admin_passwd https://192.168.3.146:6443/api/v1/pods -k
 [
 ...
 ]
 
 $ curl --basic -u test:test_passwd https://192.168.3.146:6443/api/v1/pods -k
 [
 ...
 ]
 
 $ curl --basic -u test:test_passwd https://192.168.3.146:6443/api/v1/nodes -k
 Forbidden: "/api/v1/nodes"

Admission Controllers

在认证和授权之外，Admission Controller也可以对Kubernetes API Server的访问控制，任何请求在访问API Server时需要经过一系列的验证，任何一环拒绝了请求，则会返回错误。

实际上Admission Controller是作为Kubernetes API Serve的一部分，并以插件代码的形式存在，在API Server启动的时候，可以配置需要哪些Admission Controller，以及它们的顺序，如：

--admission_control=NamespaceLifecycle,NamespaceExists,LimitRanger,SecurityContextDeny,ServiceAccount,ResourceQuota

Admission Controller支持的插件如下：

AlwaysAdmit
Use this plugin by itself to pass-through all requests.
AlwaysDeny
Rejects all requests. Used for testing.
DenyExecOnPrivileged
This plug-in will intercept all requests to exec a command in a pod if that pod has a privileged container.If your cluster supports privileged containers, and you want to restrict the ability of end-users to exec commands in those containers, we strongly encourage enabling this plug-in.
ServiceAccount
This plug-in implements automation for serviceAccounts. We strongly recommend using this plug-in if you intend to make use of Kubernetes ServiceAccount objects.
SecurityContextDeny
This plug-in will deny any pod with a SecurityContext that defines options that were not available on the Container.
ResourceQuota
This plug-in will observe the incoming request and ensure that it does not violate any of the constraints enumerated in theResourceQuota object in a Namespace. If you are using ResourceQuota objects in your Kubernetes deployment, you MUST use this plug-in to enforce quota constraints.
LimitRanger
This plug-in will observe the incoming request and ensure that it does not violate any of the constraints enumerated in theLimitRange object in a Namespace. If you are using LimitRange objects in your Kubernetes deployment, you MUST use this plug-in to enforce those constraints.
NamespaceExists
This plug-in will observe all incoming requests that attempt to create a resource in a Kubernetes Namespace and reject the request if the Namespace was not previously created. We strongly recommend running this plug-in to ensure integrity of your data.
NamespaceAutoProvision (deprecated) This plug-in will observe all incoming requests that attempt to create a resource in a Kubernetes Namespace and create a new Namespace if one did not already exist previously.
NamespaceLifecycle
This plug-in enforces that a Namespace that is undergoing termination cannot have new content created in it.A Namespace deletion kicks off a sequence of operations that remove all content (pods, services, etc.) in that namespace. In order to enforce integrity of that process, we strongly recommend running this plug-in.Once NamespaceAutoProvision is deprecated, we anticipate NamespaceLifecycle and NamespaceExists will be merged into a single plug-in that enforces the life-cycle of a Namespace in Kubernetes.

ServiceAccount

Service Account概念的引入是基于这样的使用场景：运行在pod里的进程需要调用Kubernetes API以及非Kubernetes API的其它服务（如image repository/被mount到pod上的NFS volumes中的file等）。我们使用Service Account来为pod提供id。

Service Account和User account可能会带来一定程度上的混淆，User account可以认为是与Kubernetes交互的个体，通常可以认为是human, 目前并不作为一个代码中的类型单独出现，比如第一节中配置的用户，它们的区别如下：

user account通常是为human设计的，而service account则是为跑在pod里的process。
user account是global的，即跨namespace使用；而service account是namespaced的，即仅在所属的namespace下使用。
创建一个新的user account通常需要较高的特权并且需要经过比较复杂的business process（即对于集群的访问权限的创建），而service account则不然。

要使用ServiceAccount，在启动Kubernetes的时候，首先生成key:

$ openssl genrsa -out /tmp/kube-serviceaccount.key 2048

启动API Server增加service_account_key_file:

$ kube-apiserver ... --service_account_key_file=/tmp/kube-serviceaccount.key ...

并且API Server设置admission_control包含ServiceAccount:

--admission_control=...,ServiceAccount,...

启动Controller Manager增加service_account_private_key_file:

$ kube-controller-manager ... --service_account_private_key_file=/tmp/kube-serviceaccount.key...

每个namespace都会默认创建一个ServiceAccount:

$ kubectl get serviceaccount --all-namespaces
 NAMESPACE       NAME      SECRETS
 default         default   1
 kube-system     default   3

可以创建ServiceAccount，

serviceaccount.yaml:

apiVersion: v1
 kind: ServiceAccount
 metadata:
 name: build-robot

$ kubectl create -f serviceaccount.yaml
 serviceaccounts/build-robot
 
 $ kubectl get serviceaccounts/build-robot -o yaml
 apiVersion: v1
 kind: ServiceAccount
 metadata:
 creationTimestamp: 2015-08-14T09:59:39Z
 name: build-robot
 namespace: default
 resourceVersion: "538168"
 selfLink: /api/v1/namespaces/default/serviceaccounts/build-robot
 uid: 2d55527f-426b-11e5-91cd-005056817c3e
 secrets:
 - name: build-robot-token-3uazg

可以看到ServiceAccount默认创建一个secret，也可以手动创建secret然后添加到ServiceAccount。

创建Pod使用ServiceAccount,

busybox-pod.yaml:

apiVersion: v1
 kind: Pod
 metadata:
 name: busybox
 spec:
 containers:
 - image: busybox
 command:
   - sleep
   - "3600"
 imagePullPolicy: IfNotPresent
 name: busybox
 serviceAccountName: build-robot

Pod创建成功后，可以查询Pod的容器挂载/var/run/secrets/kubernetes.io/serviceaccount：

"Volumes": {
 "/var/run/secrets/kubernetes.io/serviceaccount": "/var/lib/kubelet/pods/05174b7d-426d-11e5-aacb-005056817c3e/volumes/kubernetes.io~secret/build-robot-token-3uazg"
 },

实际上这个目录是ServiceAccount的Secret，里面包含了一个token,应用通过使用这个token便可以去访问Kubernetes API：

$ kubectl exec busybox ls /var/run/secrets/kubernetes.io/serviceaccount
 token
 $ kubectl exec busybox cat /var/run/secrets/kubernetes.io/serviceaccount/token
 eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJkZWZhdWx0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6ImJ1aWxkLXJvYm90LXRva2VuLTN1YXpnIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQubmFtZSI6ImJ1aWxkLXJvYm90Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiMmQ1NTUyN2YtNDI2Yi0xMWU1LTkxY2QtMDA1MDU2ODE3YzNlIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50OmRlZmF1bHQ6YnVpbGQtcm9ib3QifQ.CjZwP83IqkMjQzGJfLFg4QxXVLmq_wLSIVXV3yrdrzSYT4GGVjgcaRz2HDdy3TFvX2jB0rn3f6X1W_qthyQUkEHoZQkhup05rZSjsvRBQaj2FdhDpZCfaiTm5JrLeQjPEnMWYiilPo9HNk8lTqrZ32fJMHoJzxwnJ7jJDpTme2i7UPix1fW0NhRxuQ2im9iiQuZk49drNFl1oT6CjYBfK0JpyOkeOVBLCX2hDp64Jpiwa3s9Zux0AgDgLkL_shttzreQmx4hzIC22pKlae_-jRR-2EH523ej6w_7YoLCxhmTMnIRkIn5vBRjZ3kFwelnmSiyi8RbfD_oBtgTTWkYsQ

Secrets

Kubernetes提供了Secret来处理敏感信息，目前Secret的类型有3种：

- Opaque(default): 任意字符串

- kubernetes.io/service-account-token: 作用于ServiceAccount

- kubernetes.io/dockercfg: 作用于Docker registry

开发者可以任意定义Secret的格式和内容，现在创建一个假设Opaque Secret，比如应用要使用账号密码：

username: value-1
 password: value-2

首先创建Secret，

secret.yaml:

apiVersion: v1
 kind: Secret
 metadata:
 name: mysecret
 type: Opaque
 data:
 password: dmFsdWUtMg0K
 username: dmFsdWUtMQ0K

注意：其中password和username的值是通过base64 加密。

$ kubectl create -f secret.yaml
 $ kubectl describe secrets mysecret
 Name:       mysecret
 Namespace:  default
 Labels:     <none>
 Annotations:    <none>
 Type:   Opaque

Data

password: 9 bytes

username: 9 bytes

现在创建一个Pod使用该Secret

red-pod.json：

{
 "apiVersion": "v1",
 "kind": "Pod",
 "metadata": {
 "name": "mypod"
 },
 "spec": {
 "containers": [{
   "name": "mypod",
   "image": "redis",
   "volumeMounts": [{
     "name": "foo",
     "mountPath": "/etc/foo",
     "readOnly": true
   }]
 }],
 "volumes": [{
   "name": "foo",
   "secret": {
     "secretName": "mysecret"
   }
 }]
 }
 }

这里将Secret作为一个Volume挂载到Po中容器的/etc/foo下，实际上Secret中的值都会以文件生成到/etc/foo下(文件名是key，文件内容是value)，待Pod运行后查看：

$ kubectl exec mypod ls /etc/foo
 password
 username
 $ kubectl exec mypod cat /etc/foo/password
 value-2
 $ kubectl exec mypod cat /etc/foo/username
 value-1

Secret用于 Docker Registry的安全认证,参考：

https://github.com/GoogleCloud ... a-pod

Security context

Security context是用以对容器进行限制，使得不同的运行容器之前能够实现较为明晰的隔离，以及降低其影响宿主机和其它容器的可能性。通俗而言，容器中的security context用于表征在创建及运行容器时，它能够使用及访问的资源参数。

securityContext目前只实现了capabilities和privileged ，

etcd-discovery-controller.yaml:

kind: ReplicationController
 apiVersion: v1
 metadata:
 name: etcd-discovery
 creationTimestamp: 
 spec:
 strategy:
 type: Recreate
 resources: {}
 triggers:
 - type: ConfigChange
 replicas: 1
 selector:
 name: etcd-discovery
 template:
 metadata:
   creationTimestamp: 
   labels:
     name: etcd-discovery
 spec:
   containers:
   - name: discovery
     image: openshift/etcd-20-centos7
     args:
     - etcd-discovery.sh
     ports:
     - containerPort: 2379
       protocol: TCP
     resources: {}
     terminationMessagePath: "/dev/termination-log"
     imagePullPolicy: IfNotPresent
     capabilities: {}
     securityContext:
       capabilities: {}
       privileged: false
   restartPolicy: Always
   dnsPolicy: ClusterFirst
   serviceAccount: ''
 status: {}