Harbor 로컬 컨테이터 서버가 구축되었으면 로컬에서 도커 이미지를 빌드해서 Push하고 쿠버네티스에서 해당 Image를 pull 해보도록 테스트 해본다.
Harbor 로컬 컨테이너 저장소 구성 관련 내용은 아래 post에 나와 있다.
https://kindloveit.tistory.com/129
Harbor 신규 프로젝트 생성
Harbor 로 로그인 (초기 비번은 admin/Harbor12345) 후 신규 프로젝트를 생성한다.
생성팝업에서 Project 명을 입력한다.
Push하고자 하는 도커 이미지를 빌드한다. 빌드시 tag명 앞에 push 대상 서버 주소(172.17.0.1)을 적어준다.
root@DESKTOP-GH94F8C:/home/kindlove/test# docker build -t 172.17.0.1:443/library/quartz .
[+] Building 1.6s (7/7) FINISHED docker:default
=> [internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 229B 0.0s
=> [internal] load metadata for docker.io/library/openjdk:17-alpine 1.5s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [internal] load build context 0.0s
=> => transferring context: 88B 0.0s
=> [1/2] FROM docker.io/library/openjdk:17-alpine@sha256:4b6abae565492dbe9e7a894137c966a7485154238902f2f25e9dbd9784383d81 0.0s
=> CACHED [2/2] COPY target/test-1.0.0-RELEASE-exec.jar app.jar 0.0s
=> exporting to image 0.0s
=> => exporting layers 0.0s
=> => writing image sha256:b509576b225acd4a14a10206281d18c3c9f5e56c518b7b21df8459380ee342b9 0.0s
=> => naming to 172.17.0.1:443/library/quartz 0.0s
root@DESKTOP-GH94F8C:/home/kindlove/test#
root@DESKTOP-GH94F8C:/home/kindlove/test# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
172.17.0.1:443/library/quartz latest b509576b225a 2 months ago 403MB
nginx latest d453dd892d93 4 months ago 187MB
goharbor/redis-photon v2.8.2 6f4498a430ca 9 months ago 121MB
goharbor/harbor-registryctl v2.8.2 fa61a236a6d6 9 months ago 142MB
goharbor/registry-photon v2.8.2 f80e71363231 9 months ago 79.3MB
goharbor/nginx-photon v2.8.2 3d009028f260 9 months ago 120MB
goharbor/harbor-log v2.8.2 2914d282d9bf 9 months ago 127MB
goharbor/harbor-jobservice v2.8.2 40118f1568a8 9 months ago 141MB
goharbor/harbor-core v2.8.2 0bbbd1f379fc 9 months ago 165MB
goharbor/harbor-portal v2.8.2 3e74e0758aa4 9 months ago 127MB
goharbor/harbor-db v2.8.2 5126635ae9f0 9 months ago 174MB
goharbor/prepare v2.8.2 eb3cf3cdd17a 9 months ago 163MB
kindest/node <none> 36d37c652064 11 months ago 936MB
생성한 Docker 를 push한다.
root@DESKTOP-GH94F8C:/home/kindlove/ia-batch# docker push 172.17.0.1:443/library/quartz
Using default tag: latest
The push refers to repository [172.17.0.1:443/library/quartz]
83a8285d2cd2: Layer already exists
34f7184834b2: Layer already exists
5836ece05bfd: Layer already exists
72e830a4dff5: Layer already exists
latest: digest: sha256:74acc62d61c84fb57d6dfd8353440246147ebca83618e93ab66a05677c6e951c size: 1163
Harbor의 신규생성한 library 프로젝트에 들어가보면 방금 푸쉬한 Artifacts가 적재되어 있음을 볼 수 있다.
Kubernetes 에서 Harbor 저장소 활용하기
Kubernetes에서 Harbor에 올라간 Docker image를 pull 하기 위해서는 몇가지 설정이 필요하다. 로컬 이미지 저장소 인증을 위한 secret 생성이 필요하다.
Secrect 생성 및 확인
kubectl create secret docker-registry harbor --docker-server=172.17.0.1:443 --docker-username=admin --docker-password=Harbor12345 -n default
root@DESKTOP-GH94F8C:/home/kindlove/k8s/service# kubectl get deploy,svc,rs,pods,secret -o wide
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
service/kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 69d <none>
service/nodeport-hostname-service NodePort 10.96.252.140 <none> 8080:30080/TCP 69d app=hostname-server
NAME TYPE DATA AGE
secret/harbor kubernetes.io/dockerconfigjson 1 12d
Kubernetes 매니페스트 yml에 secret 명시
root@DESKTOP-GH94F8C:/home/kindlove/k8s/service# cat hostname-server_batch_harbor.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: test
labels:
app: test
spec:
replicas: 3
selector:
matchLabels:
app: test
template:
metadata:
labels:
app: test
spec:
imagePullSecrets:
- name: harbor -> 위 생성한 secret을 imagePullScrests 항목 이하에 명시한다.
containers:
- name: test
image: 172.17.0.1:443/library/quartz
ports:
- containerPort: 6099
Harbor image pull 확인을 위해 위에서 작성한 메니페스트를 실행해본다.
실행 후 pod를 확인해보면 ImagePullBackOff 에러 Status가 보인다.
root@DESKTOP-GH94F8C:/home/kindlove/k8s/service# kubectl apply -f hostname-server_batch_harbor.yaml
deployment.apps/ia-batch created
root@DESKTOP-GH94F8C:/home/kindlove/k8s/service# kubectl get deploy,svc,rs,pods,secret -o wide
NAME READY UP-TO-DATE AVAILABLE AGE CONTAINERS IMAGES SELECTOR
deployment.apps/test 0/3 3 0 15s ia-batch 172.17.0.1:443/library/quartz app=ia-batch
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
service/kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 69d <none>
service/nodeport-hostname-service NodePort 10.96.252.140 <none> 8080:30080/TCP 69d app=hostname-server
NAME DESIRED CURRENT READY AGE CONTAINERS IMAGES SELECTOR
replicaset.apps/test-745654577d 3 3 0 15s ia-batch 172.17.0.1:443/library/quartz app=ia-batch,pod-template-hash=745654577d
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
pod/test-745654577d-qklxn 0/1 ErrImagePull 0 15s 10.244.1.4 kubernetes-worker2 <none> <none>
pod/test-745654577d-rjqrr 0/1 ImagePullBackOff 0 15s 10.244.1.5 kubernetes-worker2 <none> <none>
pod/test-745654577d-wh5bq 0/1 ImagePullBackOff 0 15s 10.244.2.5 kubernetes-worker <none> <none>
좀더 pod의 구체적인 상황을 살펴보자
아래가 직접적인 원인으로 보인다.
failed to pull and unpack image "172.17.0.1:443/library/quartz:latest": failed to resolve reference "172.17.0.1:443/library/quartz:latest": failed to do request: Head "https://172.17.0.1:443/v2/library/quartz/manifests/latest": x509: certificate signed by unknown authority
root@DESKTOP-GH94F8C:/home/kindlove/k8s/service# kubectl describe pod ia-batch-745654577d-qklxn
Name: test-745654577d-qklxn
Namespace: default
Priority: 0
Service Account: default
Node: kubernetes-worker2/172.18.0.2
Start Time: Sat, 16 Mar 2024 07:37:49 +0900
Labels: app=ia-batch
pod-template-hash=745654577d
Annotations: <none>
Status: Pending
IP: 10.244.1.4
IPs:
IP: 10.244.1.4
Controlled By: ReplicaSet/ia-batch-745654577d
Containers:
ia-batch:
Container ID:
Image: 172.17.0.1:443/library/quartz
Image ID:
Port: 6099/TCP
Host Port: 0/TCP
State: Waiting
Reason: ErrImagePull
Ready: False
Restart Count: 0
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-5h9vr (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
kube-api-access-5h9vr:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 73s default-scheduler Successfully assigned default/ia-batch-745654577d-qklxn to kubernetes-worker2
Normal Pulling 27s (x3 over 73s) kubelet Pulling image "172.17.0.1:443/library/quartz"
Warning Failed 27s (x3 over 73s) kubelet Failed to pull image "172.17.0.1:443/library/quartz": rpc error: code = Unknown desc = failed to pull and unpack image "172.17.0.1:443/library/quartz:latest": failed to resolve reference "172.17.0.1:443/library/quartz:latest": failed to do request: Head "https://172.17.0.1:443/v2/library/quartz/manifests/latest": x509: certificate signed by unknown authority
Warning Failed 27s (x3 over 73s) kubelet Error: ErrImagePull
Normal BackOff 12s (x3 over 73s) kubelet Back-off pulling image "172.17.0.1:443/library/quartz"
Warning Failed 12s (x3 over 73s) kubelet Error: ImagePullBackOff
이는 Harbor 서버에 접근하려고 했으나 인증서가 로컬에 없어서 인증에러가 난것이다.
Harbor 서버 설치 시 사용한 인증서를 kind 노드의 로컬 Docker image에 설치한다.
root@DESKTOP-GH94F8C:/home/kindlove/temp# docker cp ./skkxxx@naver.com.crt kubernetes-worker:/usr/local/share/ca-certificates
Successfully copied 3.58kB to kubernetes-worker:/usr/local/share/ca-certificates
root@DESKTOP-GH94F8C:/home/kindlove/temp# docker cp ./skkxxx@naver.com.crt kubernetes-worker2:/usr/local/share/ca-certificates
Successfully copied 3.58kB to kubernetes-worker2:/usr/local/share/ca-certificates
kind 노드 도커에 콘솔 접속해서 update-ca-certificates 로 인증서 인스톨한다.
root@DESKTOP-GH94F8C:/home/kindlove/temp# docker exec -it kubernetes-worker /bin/bash
root@kubernetes-worker:/# update-ca-certificates
Updating certificates in /etc/ssl/certs...
rehash: warning: skipping ca-certificates.crt,it does not contain exactly one certificate or CRL
1 added, 0 removed; done.
Running hooks in /etc/ca-certificates/update.d...
done.
인증서 적용 완료 후 다시 deployment를 생성하면 정상적으로 harbor에서 image pull이 되고 pod 생성이 되는것을 볼 수 있다.
-- The End --
'Devops > Docker' 카테고리의 다른 글
| [Docker] Docker로 Postgresql 설치하기 (0) | 2024.05.25 |
|---|---|
| [Docker] 도커로 몽고db 설치하기 (1) | 2024.03.23 |
| [Docker] Harbor 로컬 컨테이너 저장소 구축하기 (0) | 2024.02.14 |
| [Synology Docker] Oracle 12c password expired 처리 (1) | 2023.12.11 |
| [Docker] windows wsl 2 설치하기 (1) | 2022.10.31 |
