first

deploy.md

+# kpl-launcher部署文档
+
+作者：刘弘也
+
+## 依赖安装
+
+本服务包含如下依赖
+*  Kubernetes 1.15+，支持CRD
+*  volcano v0.4.0
+
+其中volcano可以使用本项目提供的部署脚本`k8s/volcano/*.yaml`进行安装，和官方版本的差异主要是把其镜像被搬运到了金山云镜像仓库，以加速部署速度：
+
+* 安装volcano：`kubectl apply -f k8s/volcano`
+* 检查`volcano-system`命名空间及其内部的Pod是否全部成功启动，如下所示：
+  ```shell
+  $ kubectl get pods -n volcano-system
+  NAME                                   READY   STATUS      RESTARTS   AGE
+  volcano-admission-7b498d4d56-7djqg     1/1     Running     0          4d23h
+  volcano-admission-init-7sqzw           0/1     Completed   0          4d23h
+  volcano-controllers-68d55f9444-sq4vt   1/1     Running     0          4d23h
+  volcano-scheduler-7cc766767b-hvbvb     1/1     Running     0          4d23h
+  ```
+
+## kpl-launcher安装
+
+目前kpl-launcher服务进行部署的过程包含如下内容
+  * （事先准备好的）命名空间："kpl"
+  * `k8s/kpl-launcher/rbac.yaml`
+    * ServiceAccount："kpl-launcher"
+    * ClusterRole："kpl-launcher"
+    * ClusterRoleBinding："kpl-launcher"
+  * `k8s/kpl-launcher/deployment.yaml`
+    * Deployment："kpl-launcher"
+        * 服务启动命令的可选参数
+          ```shell
+          $ kpl_launcher --help
+          Usage of ./build/bin/kpl_launcher:
+            -address string
+                  service listening address (default "[::]")
+            -port int
+                  service listening port (default 8000)
+            -private-key string
+                  private key for ssl/tls secured service
+            -cert-chain string
+                  certificate chain for ssl/tls secured service
+            -incluster
+                  if use incluster config
+            -local-config string
+                  (optional) absolute path to the kubeconfig file (default "~/.kube/config")
+          ```
+    * Service (默认为ClusterIP类型)："kpl-launcher-service"
+
+具体部署步骤如下：
+
+* 联系相关开发负责人（刘弘也）确认当前要部署的镜像版本，如：`hub.kce.ksyun.com/aivc-kpl/kpl-launcher:launcher-9efebf5`
+* 修改`k8s/kpl-launcher/deployment.yaml`中的部署镜像：
+  ```shell
+  export IMAGE_NAME=hub.kce.ksyun.com/aivc-kpl/kpl-launcher:launcher-f2d3958
+  sed -i "s/image: .*/image: ${IMAGE_NAME}/" k8s/kpl-launcher/deployment.yaml
+  ```
+* 创建命名空间kpl
+  ```shell
+  kubectl create ns kpl
+  ```
+* 准备好ssl/tls的证书文件
+    * 使用openssl生成自签名证书（注意指定CN）：
+    ```shell
+    mkdir certs
+    openssl req -newkey rsa:2048 -nodes -keyout certs/server.key -x509 -days 3650 -out certs/server.crt -subj "/CN=KPL"
+    ```
+    * 生成一个叫kpl-ssl的ConfigMap，其包含刚才生成的两个ssl证书文件：
+      ```shell
+      kubectl -n kpl create configmap kpl-ssl --from-file=./certs
+      ```
+    * 注意保留`certs`里的证书文件，客户端程序也需要使用。
+* 安装kpl-launcher
+  ```shell
+  kubectl apply -f k8s/kpl-launcher/rbac.yaml
+  kubectl apply -f k8s/kpl-launcher/deployment.yaml
+  ```
+* 检查kpl-launcher服务是否已经启动
+  ```shell
+  $ kubectl logs -n kpl kpl-launcher-5b9b6d74bc-swhmm
+  I0526 13:01:17.015731       6 main.go:82] Start in secured mode ...
+  I0526 13:01:17.020704       6 launcher.go:44] new launcher with the following backends:
+  I0526 13:01:17.020714       6 launcher.go:46]   volcano: &{0xc00036b680 0xc000402940 map[]}
+  I0526 13:01:17.020727       6 launcher.go:46]   simple_job: &{0xc00036b680 map[]}
+  I0526 13:01:17.020742       6 main.go:96] try to start launcher server at [::]:8000
+  ```

deploy/kpl-launcher/deployment.yaml

+kind: Deployment
+apiVersion: apps/v1
+metadata:
+  name: kpl-launcher
+  namespace: kpl
+  labels:
+    app: kpl-launcher
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app: kpl-launcher
+  template:
+    metadata:
+      labels:
+        app: kpl-launcher
+    spec:
+      serviceAccount: kpl-launcher
+      containers:
+        - name: launcher
+          image: hub.kce.ksyun.com/aivc-kpl/kpl-launcher:launcher-f2d3958
+          command:
+            - /bin/bash
+            - -c
+            - kpl_launcher --incluster --private-key /etc/kpl/ssl/server.key --cert-chain /etc/kpl/ssl/server.crt --port 8000 2>&1
+          ports:
+            - containerPort: 8000
+              name: launcher-port
+          imagePullPolicy: "IfNotPresent"
+          resources:
+              limits:
+                cpu: 8
+                memory: 100Mi
+          env:
+            - name: KPL_IMAGE_SECRET_NAME
+              value: kpl-regcred
+          volumeMounts:
+            - name: kpl-ssl
+              mountPath: /etc/kpl/ssl
+              readOnly: true
+      volumes:
+        - name: kpl-ssl
+          configMap:
+            name: kpl-ssl
+      imagePullSecrets:
+        - name: kpl-regcred
+---
+apiVersion: v1
+kind: Service
+metadata:
+  labels:
+    app: kpl-launcher
+  name: kpl-launcher-service
+  namespace: kpl
+spec:
+  ports:
+    - port: 8000
+      protocol: TCP
+      targetPort: 8000
+  # type: NodePort
+  selector:
+    app: kpl-launcher

deploy/kpl-launcher/rbac.yaml

+apiVersion: v1
+kind: ServiceAccount
+metadata:
+  name: kpl-launcher
+  namespace: kpl
+---
+kind: ClusterRole
+apiVersion: rbac.authorization.k8s.io/v1
+metadata:
+  name: kpl-launcher
+rules:
+  - apiGroups: ["apiextensions.k8s.io"]
+    resources: ["customresourcedefinitions"]
+    verbs: ["get", "list", "watch"]
+  - apiGroups: ["batch.volcano.sh"]
+    resources: ["jobs"]
+    verbs: ["get", "create", "list", "watch", "update", "delete"]
+  - apiGroups: [""]
+    resources: ["pods", "pods/status"]
+    verbs: ["create", "get", "list", "watch", "update", "bind", "updateStatus", "delete"]
+  - apiGroups: [""]
+    resources: ["persistentvolumeclaims"]
+    verbs: ["list", "watch"]
+  - apiGroups: [""]
+    resources: ["persistentvolumes"]
+    verbs: ["list", "watch"]
+  - apiGroups: [""]
+    resources: ["services"]
+    verbs: ["list", "watch"]
+  - apiGroups: ["apps"]
+    resources: ["deployments"]
+    verbs: ["list", "watch"]
+  - apiGroups: [""]
+    resources: ["namespaces"]
+    verbs: ["list", "watch"]
+  - apiGroups: ["storage.k8s.io"]
+    resources: ["storageclasses"]
+    verbs: ["list", "watch"]
+  - apiGroups: [""]
+    resources: ["nodes"]
+    verbs: ["list", "watch"]
+---
+kind: ClusterRoleBinding
+apiVersion: rbac.authorization.k8s.io/v1
+metadata:
+  name: kpl-launcher
+subjects:
+- kind: ServiceAccount
+  name: kpl-launcher
+  namespace: kpl
+roleRef:
+  kind: ClusterRole
+  name: kpl-launcher
+  apiGroup: rbac.authorization.k8s.io

deploy/kpl-ssl.yaml

+apiVersion: v1
+data:
+  server.crt: |
+    -----BEGIN CERTIFICATE-----
+    MIIC7zCCAdegAwIBAgIJALlcgWTrwOS9MA0GCSqGSIb3DQEBCwUAMA4xDDAKBgNV
+    BAMMA0tQTDAeFw0yMDA1MjYwNzA4MzBaFw0zMDA1MjQwNzA4MzBaMA4xDDAKBgNV
+    BAMMA0tQTDCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAOC+N7eY/ePy
+    LPC0frnxzC1Fs4GEN2QQW6crfcAHxWkdkq7umaND+qQsKwuUYaICUyaFOjI3N8gq
+    OJcuX04hs6P7hoPVeXrqtEw9WaJeP0afK/LRIcjJ604v1LF3Jq513iSe/3f7iaz9
+    3iSuK2ItP4tHuyzRuQLFdTxqMjxy74e83nI9yuYvSvra4wnpr3qyOOVyNGYowEvz
+    laeebEMS7g2KMXk7B0uh6l8lt0TR9iWFJHPdTY0a+ralFVrD+YlGlvi0NwpvVFID
+    gFxWOv18mAk65qkPp0L1Qc2dbEQkkTZYPSTzQqEWd02woZI1UjyYsyGcGE4iQyI+
+    YM9Y99vBQKUCAwEAAaNQME4wHQYDVR0OBBYEFEnyHBiqic8i7SPbCbyLmxVa60Vr
+    MB8GA1UdIwQYMBaAFEnyHBiqic8i7SPbCbyLmxVa60VrMAwGA1UdEwQFMAMBAf8w
+    DQYJKoZIhvcNAQELBQADggEBABOnbugVrr6SpAp9SuSlt3omfh6D4I1xxC6puQAp
+    zfzT8Y14s0UE94bSfH99PWXKrADdfuunMDXftUKB2hbTLaRVI225XMmb1dVulzei
+    ZA0VP4FTPuPwsl1C1g0ySLb7CCX+KjQV+bDegD7n2apJfgTTDF/g/HU9I5iEkJhh
+    5aFtsx54ex5nbvq7wLc8iVaN1ygqseq4Peb3UG99x16jFELwhXfNx+2+mraD1tSK
+    PPWX1gYcfTI1QVQyxAX6T3Q72Bufqtv0mvLakjWkkkv0pDUNei8xloTaFT6rFwQg
+    AxgXn/VTuUUsEjLILfH0bkbGwXh8Gz6gmzjOPvYClXzy6Sc=
+    -----END CERTIFICATE-----
+  server.key: |
+    -----BEGIN PRIVATE KEY-----
+    MIIEvgIBADANBgkqhkiG9w0BAQEFAASCBKgwggSkAgEAAoIBAQDgvje3mP3j8izw
+    tH658cwtRbOBhDdkEFunK33AB8VpHZKu7pmjQ/qkLCsLlGGiAlMmhToyNzfIKjiX
+    Ll9OIbOj+4aD1Xl66rRMPVmiXj9Gnyvy0SHIyetOL9Sxdyaudd4knv93+4ms/d4k
+    ritiLT+LR7ss0bkCxXU8ajI8cu+HvN5yPcrmL0r62uMJ6a96sjjlcjRmKMBL85Wn
+    nmxDEu4NijF5OwdLoepfJbdE0fYlhSRz3U2NGvq2pRVaw/mJRpb4tDcKb1RSA4Bc
+    Vjr9fJgJOuapD6dC9UHNnWxEJJE2WD0k80KhFndNsKGSNVI8mLMhnBhOIkMiPmDP
+    WPfbwUClAgMBAAECggEBAMwyyLkeXwDvl3mU5dfZJKS1weWZ1iIq4ru/PmBuLyXU
+    4uPikuHO8DqejvXUeekTp61GO5xIo3xRMF3cmmeJI4mJDa1XFytheTQUQg7PBzsJ
+    gIr9DW88V3oQ53XfRtwY6B/p/0Bq4aq2d1Jola58YlIQeLa+TxV/h4e/Dany3EQ8
+    2OHgsHDFjmkpNUQoIqBQqptL8UhLZiXp8FkwfvEi+SAd/f6KoRyOhEOqOG1FFqp4
+    JYCPCj5tc2fMlqLU4vvN4H5wAU8tXM2mKmKXxlnF7xcxWOrmmLbl0qhEcFU0aeFc
+    PTQW4VLoPSe4R1+TFzeW5iLgXlRbbkLO0hHNx/ukbzkCgYEA92qzcMrRRU7IqYdW
+    xZdpXmHR6WY+vRW5EIrci6ygNfez4JZYRmyb9TSErxiRuXm5DR5n5d42vCNEXn2Z
+    obcLwnSukRn5R6AQM0IkQ6/VRN91h42uzK4G7xMraXuJNWL44XxOe+DfNfJx/pXS
+    7u/+65I8ltejNsdA5eBIOClJyqsCgYEA6Iom4mcB4qFvxfnleNfB1m4aHM7drjg7
+    TSHGcUZChiHuKKki4bh2CMLwXD3WM63XFHwWwByBVnuCOGseTCioPB3NXTwbXvo3
+    OnUuxZ+uWsj8TmHK7N9qoc6uZ0ZLhJd1V3qBGqnacadeYsYQ916HIXunn7V7BYTI
+    RQK41MY3Ie8CgYBwEUeoHmX+A7qGd4QJShTzKFHBa4udcBp8cLmHFrEW4NT8NH1V
+    oNpfjeNPwEw5e/YjCBO9nhKi00KjnjaZdmUANoPfu6kbFVhhL2NnuVxbru/4b2eu
+    1GDVp5QxWkCDCwp6cCjImcnhEKkrzMdXbSPFpl6FYLVQGZ6+wQFVVvZSswKBgQDi
+    4G9BsyhmugU4jjDx7uei03dMphQo95n8DjjzqY4cGhyntSVgPr2CnNcv1/EWKMTQ
+    qqTzuJwECaqqavB7c6t4Q8dq8Mrvpoad1VjV3Y6Z6sAnpUf8RZpe1izYfAAQFEmb
+    GN2/avrCqn3vNrm075g99cw8iS7G2p8enaJBjkaaJwKBgCqYvGa4foM2yPFLejSk
+    zSmkeUvqxUdxQ9ttfC1zEViCibyKJtC9u+btlv7h+XoxS9d946rrrxB8aua2YK/P
+    QZzI4IkArs66UN+CI41y0NmAiOlHVNwfTo47h1rxBoHwqRrX0olIEESlmTu2b6yY
+    6axih4ba/X3xqTGGyk/N5krB
+    -----END PRIVATE KEY-----
+kind: ConfigMap
+metadata:
+  creationTimestamp: "2020-05-26T07:08:41Z"
+  name: kpl-ssl
+  namespace: kpl
+  resourceVersion: "60214464"
+  selfLink: /api/v1/namespaces/kpl/configmaps/kpl-ssl
+  uid: 84ec63bc-c722-4e2e-aa50-78e18ba3796c

deploy/volcano/namespace.yaml

+apiVersion: v1
+kind: Namespace
+metadata:
+  name: volcano-system