[Kubernetes] jenkins Kubernetes 에 설치 및 배포하기

jenkins 를 kunernetes cluster 내부에 설치하고 pod 배포가 되도록 설정을 해본다.

 

Kubernetes 상에서 Jenkins  설치하기

 

젠킨스 설치를 위한 yaml은 아래 git 에서 기본적으로 제공하고 있다. 아래 내용을 다운로드 받는다.

git clone https://github.com/scriptcamp/kubernetes-jenkins

 

 

namespace 를 아래와 같이 생성한다.

kubectl create namespace devops-tools

 

 

serviceAccount.yaml 실행

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: jenkins-admin
rules:
  - apiGroups: [""]
    resources: ["*"]
    verbs: ["*"]

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: jenkins-admin
  namespace: devops-tools

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: jenkins-admin
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: jenkins-admin
subjects:
- kind: ServiceAccount
  name: jenkins-admin
  namespace: devops-tools

 

 

volume.yaml

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: local-storage
provisioner: kubernetes.io/no-provisioner
volumeBindingMode: WaitForFirstConsumer

---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: jenkins-pv-volume
  labels:
    type: local
spec:
  storageClassName: local-storage
  claimRef:
    name: jenkins-pv-claim
    namespace: devops-tools
  capacity:
    storage: 10Gi
  accessModes:
    - ReadWriteOnce
  local:
    path: /mnt
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - worker-node01

---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: jenkins-pv-claim
  namespace: devops-tools
spec:
  storageClassName: local-storage
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 3Gi

 

 

여기서 worker-node01은 각자 환경에 맞게 바꿔어 준다. 아래 get nodes  명령어로 hostname을 알 수 있다.

kubectl get nodes

 

 

명령어를 실행하면 아래와 본인의  cluster 의 각 노드 명을 알수 있고 하나의 노드를 정해서 설정하도록한다.

 

root@master:~/01-jenkins/kubernetes-jenkins# kubectl get nodes
NAME      STATUS   ROLES                  AGE   VERSION
master    Ready    control-plane,master   9d    v1.23.1
worker1   Ready    <none>                 9d    v1.23.1
worker2   Ready    <none>                 9d    v1.23.1

 

단, 위와 같이 임의의 하나의 노드로 volumn 을 설정하는것은 데모용으로 쉽게 설정을 한 것이고.. 해당 node가 failure되거나 delete 되었을때는 모든 data가 지워지므로 궁극적으로는 제대로 된 persistent provider를 설정하는게 필요할 것이다.

 

아래는 젠킨스 문서에 언급된 원문 내용이다.

For volume, we are using the 'local' storage class for the purpose of demonstration. Meaning, it creates a 'PersistentVolume' volume in a specific node under the '/mnt' location.

As the 'local' storage class requires the node selector, you need to specify the worker node name correctly for the Jenkins pod to get scheduled in the specific node.

If the pod gets deleted or restarted, the data will get persisted in the node volume. However, if the node gets deleted, you will lose all the data.

Ideally, you should use a persistent volume using the available storage class with the cloud provider, or the one provided by the cluster administrator to persist data on node failures.

 

위 노드 변경이 완료 되었으면 yaml을 적용한다.

kubectl create -f volume.yaml

 

 

deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: jenkins
  namespace: devops-tools
spec:
  replicas: 1
  selector:
    matchLabels:
      app: jenkins-server
  template:
    metadata:
      labels:
        app: jenkins-server
    spec:
      securityContext:
            fsGroup: 1000
            runAsUser: 1000
      serviceAccountName: jenkins-admin
      containers:
        - name: jenkins
          image: jenkins/jenkins:lts
          resources:
            limits:
              memory: "2Gi"
              cpu: "1000m"
            requests:
              memory: "500Mi"
              cpu: "500m"
          ports:
            - name: httpport
              containerPort: 8080
            - name: jnlpport
              containerPort: 50000
          livenessProbe:
            httpGet:
              path: "/login"
              port: 8080
            initialDelaySeconds: 90
            periodSeconds: 10
            timeoutSeconds: 5
            failureThreshold: 5
          readinessProbe:
            httpGet:
              path: "/login"
              port: 8080
            initialDelaySeconds: 60
            periodSeconds: 10
            timeoutSeconds: 5
            failureThreshold: 3
          volumeMounts:
            - name: jenkins-data
              mountPath: /var/jenkins_home
      volumes:
        - name: jenkins-data
          persistentVolumeClaim:
              claimName: jenkins-pv-claim

 

위  deployment.yaml을 적용한다.

kubectl apply -f deployment.yaml

 

deployment 상태를 체크한다.

kubectl get deployments -n devops-tools

root@master:~/01-jenkins/kubernetes-jenkins# kubectl get deploy -n devops-tools -o wide
NAME      READY   UP-TO-DATE   AVAILABLE   AGE   CONTAINERS   IMAGES                SELECTOR
jenkins   0/1     1            0           32s   jenkins      jenkins/jenkins:lts   app=jenkins-server

 

 

describe 명령으로 상세 내용을 확인한다.

kubectl describe deployments --namespace=devops-tools

 

root@master:~/01-jenkins/kubernetes-jenkins# kubectl describe deploy -n devops-tools
Name:                   jenkins
Namespace:              devops-tools
CreationTimestamp:      Sun, 31 Dec 2023 07:26:07 +0900
Labels:                 <none>
Annotations:            deployment.kubernetes.io/revision: 1
Selector:               app=jenkins-server
Replicas:               1 desired | 1 updated | 1 total | 0 available | 1 unavailable
StrategyType:           RollingUpdate
MinReadySeconds:        0
RollingUpdateStrategy:  25% max unavailable, 25% max surge
Pod Template:
  Labels:           app=jenkins-server
  Service Account:  jenkins-admin
  Containers:
   jenkins:
    Image:       jenkins/jenkins:lts
    Ports:       8080/TCP, 50000/TCP
    Host Ports:  0/TCP, 0/TCP
    Limits:
      cpu:     1
      memory:  2Gi
    Requests:
      cpu:        500m
      memory:     500Mi
    Liveness:     http-get http://:8080/login delay=90s timeout=5s period=10s #success=1 #failure=5
    Readiness:    http-get http://:8080/login delay=60s timeout=5s period=10s #success=1 #failure=3
    Environment:  <none>
    Mounts:
      /var/jenkins_home from jenkins-data (rw)
  Volumes:
   jenkins-data:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  jenkins-pv-claim
    ReadOnly:   false
Conditions:
  Type           Status  Reason
  ----           ------  ------
  Available      False   MinimumReplicasUnavailable
  Progressing    True    ReplicaSetUpdated
OldReplicaSets:  <none>
NewReplicaSet:   jenkins-b96f7764f (1/1 replicas created)
Events:
  Type    Reason             Age   From                   Message
  ----    ------             ----  ----                   -------
  Normal  ScalingReplicaSet  92s   deployment-controller  Scaled up replica set jenkins-b96f7764f to 1

 

 

Kubernetes service에서 jenkins에 접근 하기 위해서 아래 service.yaml 설정을 적용한다.

 

service.yaml

apiVersion: v1
kind: Service
metadata:
  name: jenkins-service
  namespace: devops-tools
  annotations:
      prometheus.io/scrape: 'true'
      prometheus.io/path:   /
      prometheus.io/port:   '8080'
spec:
  selector:
    app: jenkins-server
  type: NodePort
  ports:
    - port: 8080
      name: tcp
      targetPort: 8080
      nodePort: 32000
    - name: jnlpport
      port: 50000
      targetPort: 50000

 

jenkins 웹 포트는 32000으로 외부로 설정되도록 하였고 jenkins 빌드시 사용할 jnlpport는 50000으로 맵핑 되엉 있다.

 

service.yaml을 적용한다.

kubectl apply -f service.yaml

 

서비스 적용이 완료되면 아래 주소로 jenkins portal에 접속이 가능해진다.

 

http://<node-ip>:32000

 

단, 최초 설치 후 웹 접속을 하면 젠킨스 연동을 위한 passowrd 입력을 요구하게 된다. 연동 password는 아래 pod log 확인으로 발췌할 수 있다.

root@master:~/01-jenkins/kubernetes-jenkins# kubectl get pods -n devops-tools
NAME                      READY   STATUS    RESTARTS   AGE
jenkins-b96f7764f-5wf5w   1/1     Running   0          6m22s

 

위에서 확인한 pod명으로 아래 설치히 log를 확인하면 설치 후 적용할 password 를 확인할 수 잉ㅆ다.

kubectl logs jenkins-deployment-2539456353-j00w5 --namespace=devops-tools

 

 

또는, 아래 명령으로 찾을 수도 있다.

kubectl exec -it jenkins-559d8cd85c-cfcgk cat /var/jenkins_home/secrets/initialAdminPassword -n devops-tools

 

최종적으로 발췌된 password를 입력하고 초기 admin 계정을 설정하면 jenkins 사용이 가능해진다.

 

 

 

Jenkins pipeline 설정하기

 

pipeline 을 설정하기 앞서 기본적인 jenkins system, cloud 설정을 하도록 한다.

 

Dashboard -> Managed Jenkins -> System 에서 Jenkins Url을 입력 해준다.

 

 

Dashboard -> Managed Jenkins -> Clouds에서 Kunernetes 설정을 생성한다.

 

앞서 위에서 serviceAccount.yaml을 수행하였으므로 Credential 부분은 생성이 완료되었고 jenkins 에서  kubernetes cluster를 인식하고 접근권한도 정상 부여되었는지 확인하기 위해서  Test Connection 버튼을 입력한다.

 

정상 설정 되었을 시에 현재 연동된 cluster  버전이 출력된다.

 

 

Jenkins tunnel 부분이 중요한데 pipeline등으로 pod 빌드 시 jnlp 통신을 위해서 꼭 정확하게 입력을 해야하는 주소이다.

 

내부 IP는 Service.yaml에서 50000으로 설정해둔 상태 이므로 Cluster IP : 50000 으로 입력한다. Cluster IP 는 아래 get svc 명령으로 알 수 있다.

 

root@master:~/01-jenkins/kubernetes-jenkins# kubectl get pods,svc,deploy -n devops-tools -o wide
NAME                          READY   STATUS        RESTARTS   AGE     IP                NODE      NOMINATED NODE   READINESS GATES
pod/jenkins-b96f7764f-5wf5w   1/1     Running       0          54m     192.168.189.98    worker2   <none>           <none>
pod/pod-golang-kngdl-6mdj9    2/2     Terminating   0          8m45s   192.168.235.189   worker1   <none>           <none>
pod/pod-golang-r5msj-8fzt5    0/2     Pending       0          94s     <none>            <none>    <none>           <none>

NAME                      TYPE       CLUSTER-IP      EXTERNAL-IP   PORT(S)                          AGE   SELECTOR
service/jenkins-service   NodePort   10.106.241.17   <none>        8080:32000/TCP,50000:32544/TCP   49m   app=jenkins-server

NAME                      READY   UP-TO-DATE   AVAILABLE   AGE   CONTAINERS   IMAGES                SELECTOR
deployment.apps/jenkins   1/1     1            1           54m   jenkins      jenkins/jenkins:lts   app=jenkins-server

 

 service/jenkins-service항목의  CLUSTER-IP가 10.106.241.17로 할당되었음을 알 수 있다.

 

Test Pipeline 수행

 

테스트로 Build pipeline을 생성해 본다.

 

Dashboard -> New Item 을 누르고 Pipeline 을 선택한다.

 

 

Pipeline script에 다음 테스트 script를 입력한다.

podTemplate(label: 'pod-golang', 
    containers: [
        containerTemplate(
            name: 'golang',
            image: 'golang',
            ttyEnabled: true,
            command: 'cat'
        )
    ]
) {
    node ('pod-golang') {

        stage 'Switch to Utility Container'
        container('golang') {

          sh ("go version")

        }
    }
}

 

설정 저장 후에 pipeline을 수행해 본다.

 

Started by user kms
[Pipeline] Start of Pipeline
[Pipeline] podTemplate
[Pipeline] {
[Pipeline] node
Still waiting to schedule task
‘pod-golang-kngdl-6mdj9’ is offline
Created Pod: kubernetes devops-tools/pod-golang-r5msj-8fzt5
Agent pod-golang-r5msj-8fzt5 is provisioned from template pod-golang-r5msj
---
apiVersion: "v1"
kind: "Pod"
metadata:
  annotations:
    buildUrl: "http://172.30.1.25:32000/job/Test%20Pipeline/2/"
    runUrl: "job/Test%20Pipeline/2/"
  labels:
    jenkins: "slave"
    jenkins/label-digest: "ff34b1c153af20a06bce43415e5abcde07bffa4f"
    jenkins/label: "pod-golang"
  name: "pod-golang-r5msj-8fzt5"
  namespace: "devops-tools"
spec:
  containers:
  - command:
    - "cat"
    image: "golang"
    imagePullPolicy: "IfNotPresent"
    name: "golang"
    resources: {}
    tty: true
    volumeMounts:
    - mountPath: "/home/jenkins/agent"
      name: "workspace-volume"
      readOnly: false
  - env:
    - name: "JENKINS_SECRET"
      value: "********"
    - name: "JENKINS_TUNNEL"
      value: "10.106.241.17:50000"
    - name: "JENKINS_AGENT_NAME"
      value: "pod-golang-r5msj-8fzt5"
    - name: "JENKINS_NAME"
      value: "pod-golang-r5msj-8fzt5"
    - name: "JENKINS_AGENT_WORKDIR"
      value: "/home/jenkins/agent"
    - name: "JENKINS_URL"
      value: "http://172.30.1.25:32000/"
    image: "jenkins/inbound-agent:3192.v713e3b_039fb_e-1"
    name: "jnlp"
    resources:
      requests:
        memory: "256Mi"
        cpu: "100m"
    volumeMounts:
    - mountPath: "/home/jenkins/agent"
      name: "workspace-volume"
      readOnly: false
  nodeSelector:
    kubernetes.io/os: "linux"
  restartPolicy: "Never"
  volumes:
  - emptyDir:
      medium: ""
    name: "workspace-volume"

Running on pod-golang-r5msj-8fzt5 in /home/jenkins/agent/workspace/Test Pipeline
[Pipeline] {
[Pipeline] stage (Switch to Utility Container)
Using the ‘stage’ step without a block argument is deprecated
[Pipeline] container
[Pipeline] {
[Pipeline] sh
+ go version
go version go1.21.5 linux/amd64
[Pipeline] }
[Pipeline] // container
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // podTemplate
[Pipeline] End of Pipeline
Finished: SUCCESS

 

 

  참고사이트 : https://www.jenkins.io/doc/book/installing/kubernetes/

 

 

-- The end --