Harbor 私有镜像仓库部署

Harbor 私有镜像仓库部署一、方案概述1.1 目标在 K8s 高可用集群上部署一套 Harbor 私有镜像仓库具备镜像存储存储 Docker 镜像和 Helm Chart访问控制用户认证、项目权限HTTPSLet’s Encrypt 泛域名证书持久化Longhorn 分布式存储高可用核心组件多副本外部访问FRP Ingress 从公网访问1.2 技术选型组件用途Harbor Helm Chart部署 HarborNginx Ingress Controller外部访问入口Longhorn持久化存储Let’s EncryptTLS 证书FRP内网穿透1.3 架构text外网用户 ↓ https://harbor.域名:40443 ↓ frpsvhost_https_port 40443 ↓ frpc ↓ 内网 VIP:443 ↓ HAProxy ↓ Ingress ControllerTLS 终止 ↓ Harbor Service ↓ Harbor Podcore、portal、registry、jobservice二、前置条件2.1 集群状态K8s 集群kubectl get nodes全部ReadyNginx Ingress Controller 已部署Longhorn 已部署longhornStorageClass 就绪FRP 已配置2.2 存储要求Harbor 需要 PVC 持久化数据。本文档使用Longhorn作为存储后端。关键约束Harbor registry 组件不支持多副本共享 RWO 卷所以registry和jobservice必须单副本。Longhorn 卷默认 3 副本如果节点磁盘空间紧张可以降到 2 副本。Longhorn 的storage-over-provisioning-percentage和storage-minimal-available-percentage需要提前调整否则卷会因insufficient storage调度失败。2.3 TLS 证书需要泛域名证书*.域名。本文档使用acme.sh Let’s Encrypt申请。2.4 FRP 配置frpc 需要加 HTTP 和 HTTPS 的 vhost 映射[harbor-http] type http local_ip 内网VIP local_port 80 custom_domains harbor.域名 [harbor-https] type https local_ip 内网VIP local_port 443 custom_domains harbor.域名三、申请 TLS 证书3.1 安装 acme.shcurl https://get.acme.sh | sh -s email邮箱 source ~/.bashrc3.2 配置 DNS API以阿里云为例export Ali_KeyAccessKeyID export Ali_SecretAccessKeySecret3.3 申请泛域名证书acme.sh --issue --server letsencrypt --dns dns_ali --dnssleep 60 -d 域名 -d *.域名解析--server letsencrypt指定 Let’s Encrypt不用 ZeroSSL。--dnssleep 60加完 TXT 记录后等 60 秒再验证。-d 域名 -d *.域名同时申请根域名和泛域名。证书路径~/.acme.sh/域名_ecc/fullchain.cer ~/.acme.sh/域名_ecc/域名.key3.4 配置自动续期脚本创建/root/update-tls-secrets.sh#!/bin/bash CERT_DIR/root/.acme.sh/域名_ecc NAMESPACES(harbor argocd jenkins gitlab default) for ns in ${NAMESPACES[]}; do kubectl create secret tls 域名-tls \ --cert$CERT_DIR/fullchain.cer \ --key$CERT_DIR/域名.key \ --namespace $ns \ --dry-runclient -o yaml | kubectl apply -f - done加执行权限chmod x /root/update-tls-secrets.sh配置 acme.sh 续期回调acme.sh --install-cert -d 域名 --ecc --reloadcmd /root/update-tls-secrets.sh四、安装 Harbor4.1 添加 Helm 仓库helm repo add harbor https://helm.goharbor.io helm repo update helm search repo harbor/harbor如果helm.goharbor.io访问超时从 GitHub 代理下载 chartcurl -fsSL https://ghproxy.net/https://github.com/goharbor/harbor-helm/releases/download/v1.19.2/harbor-1.19.2.tgz -o harbor-1.19.2.tgz4.2 创建命名空间kubectl create namespace harbor4.3 创建 TLS Secretkubectl create secret tls 域名-tls \ --cert/root/.acme.sh/域名_ecc/fullchain.cer \ --key/root/.acme.sh/域名_ecc/域名.key \ --namespace harbor4.4 创建 values.yamlexpose: type: ingress tls: enabled: true certSource: secret secret: secretName: 域名-tls notarySecretName: 域名-tls ingress: hosts: core: harbor.域名 className: nginx annotations: nginx.ingress.kubernetes.io/proxy-body-size: 0 nginx.ingress.kubernetes.io/proxy-read-timeout: 600 nginx.ingress.kubernetes.io/proxy-send-timeout: 600 nginx.ingress.kubernetes.io/ssl-redirect: true nginx.ingress.kubernetes.io/force-ssl-redirect: true externalURL: https://harbor.域名:40443 persistence: enabled: true resourcePolicy: keep persistentVolumeClaim: registry: storageClass: longhorn size: 15Gi database: storageClass: longhorn size: 20Gi redis: storageClass: longhorn size: 10Gi jobservice: storageClass: longhorn size: 10Gi trivy: storageClass: longhorn size: 10Gi harborAdminPassword: 强密码 core: replicas: 2 resources: requests: cpu: 500m memory: 1Gi limits: cpu: 2 memory: 2Gi portal: replicas: 2 resources: requests: cpu: 200m memory: 256Mi limits: cpu: 1 memory: 1Gi jobservice: replicas: 1 resources: requests: cpu: 500m memory: 512Mi limits: cpu: 2 memory: 2Gi registry: replicas: 1 resources: requests: cpu: 500m memory: 512Mi limits: cpu: 2 memory: 2Gi database: type: internal internal: resources: requests: cpu: 500m memory: 1Gi limits: cpu: 2 memory: 2Gi redis: type: internal internal: resources: requests: cpu: 200m memory: 256Mi limits: cpu: 1 memory: 1Gi trivy: enabled: true replicas: 1 resources: requests: cpu: 200m memory: 512Mi limits: cpu: 1 memory: 1Gi4.5 逐项解析字段含义expose.type: ingress用 Ingress 暴露tls.enabled: true开启 HTTPStls.certSource: secret证书从 K8s Secret 读tls.secret.secretNameTLS Secret 名字ingress.hosts.coreHarbor 域名proxy-body-size: 0不限上传大小proxy-read-timeout: 600读超时 10 分钟ssl-redirect: trueHTTP 自动跳转 HTTPSexternalURL外部访问地址带端口persistence各组件 PVC 配置harborAdminPasswordadmin 初始密码core.replicas: 2core 双副本portal.replicas: 2portal 双副本jobservice.replicas: 1jobservice 单副本registry.replicas: 1registry 单副本关键点registry和jobservice必须单副本因为它们的卷是 RWO多副本会报Multi-Attach error。externalURL必须带端口因为外网访问是https://harbor.域名:40443。4.6 安装 Harborhelm install harbor harbor/harbor \ --namespace harbor \ -f /root/harbor-values.yaml如果helm.goharbor.io不通用本地 charthelm install harbor ./harbor-1.19.2.tgz \ --namespace harbor \ -f /root/harbor-values.yaml4.7 观察 Pod 启动kubectl get pods -n harbor -w按CtrlC退出。预期harbor-core-xxx 1/1 Running (2个) harbor-database-0 1/1 Running harbor-jobservice-xxx 1/1 Running harbor-portal-xxx 1/1 Running (2个) harbor-redis-0 1/1 Running harbor-registry-xxx 2/2 Running harbor-trivy-0 1/1 Running五、验证 Harbor5.1 查看 Pod 和 PVCkubectl get pods -n harbor kubectl get pvc -n harbor kubectl get ingress -n harbor预期所有 PodRunning。PVC 全部BoundSTORAGECLASS是longhorn。Ingress 有harbor-ingress域名harbor.域名。5.2 测试 HTTPS 访问curl -v https://harbor.域名:40443 21 | grep SSL certificate预期SSL certificate verify ok.5.3 浏览器访问https://harbor.域名:40443用户名admin密码你设置的强密码第一次登录会要求修改密码。5.4 创建项目登录后创建项目比如library。解析项目是 Harbor 里镜像的组织单位。推送镜像时路径是harbor.域名:40443/library/image:tag。六、配置 K8s 拉取 Harbor 镜像6.1 配置 containerd 信任 Harbor在每台节点上编辑/etc/containerd/config.toml在registry.mirrors段加[plugins.io.containerd.grpc.v1.cri.registry.mirrors.harbor.域名:40443] endpoint [https://harbor.域名:40443]用 Ansible 批量推ansible k8s_cluster -i inventory.ini -m shell -a grep -q registry.mirrors.\harbor.域名:40443\ /etc/containerd/config.toml || sed -i /\[plugins.\io.containerd.grpc.v1.cri\.registry.mirrors\]/a\ [plugins.\io.containerd.grpc.v1.cri\.registry.mirrors\.\harbor.域名:40443\]\n endpoint [\https://harbor.域名:40443\] /etc/containerd/config.toml systemctl restart containerd -b6.2 创建镜像拉取 Secretkubectl create secret docker-registry harbor-secret \ --docker-serverharbor.域名:40443 \ --docker-usernameadmin \ --docker-password强密码 \ --namespace default6.3 在 Deployment 里引用spec: template: spec: imagePullSecrets: - name: harbor-secret containers: - name: myapp image: harbor.域名:40443/library/myapp:1.06.4 验证拉取crictl pull harbor.域名:40443/library/nginx:latest七、常见问题7.1 Pod 卡在ImagePullBackOffHarbor 镜像在docker.io/goharbor上containerd 镜像加速应该能覆盖。如果拉不下来手动拉取并打标签crictl pull docker.m.daocloud.io/goharbor/harbor-jobservice:v2.15.2 ctr -n k8s.io images tag docker.m.daocloud.io/goharbor/harbor-jobservice:v2.15.2 docker.m.daocloud.io/goharbor/harbor-jobservice:v2.15.2如果某个节点拉不下来从其他节点导出ctr -n k8s.io images export /tmp/jobservice.tar docker.m.daocloud.io/goharbor/harbor-jobservice:v2.15.2 scp /tmp/jobservice.tar root目标节点:/tmp/ ctr -n k8s.io images import /tmp/jobservice.tar7.2 registry 多副本时Multi-Attach error原因registry 用 RWO 卷多副本无法共享。解决方案registry.replicas: 1。或者改 Deployment 的更新策略为Recreatekubectl patch deployment -n harbor harbor-registry -p {spec:{strategy:{type:Recreate}}}7.3 PVC 绑不上卷faulted或degraded原因Longhorn 卷调度失败。排查kubectl get volumes.longhorn.io -n longhorn-system kubectl get volume.longhorn.io volume-name -n longhorn-system -o yaml | grep -A5 type: Scheduled如果报insufficient storagekubectl patch settings.longhorn.io storage-over-provisioning-percentage -n longhorn-system --typemerge -p {value:200} kubectl patch settings.longhorn.io storage-minimal-available-percentage -n longhorn-system --typemerge -p {value:10}然后触发副本重建kubectl get engines.longhorn.io -n longhorn-system | grep volume-name kubectl delete engine.longhorn.io engine-name -n longhorn-system7.4 Harbor 登不上报relation properties does not exist原因database 重建后表没建。解决方案删掉 Harbor 和所有 PVC重建。helm uninstall harbor -n harbor kubectl delete pvc -n harbor --all kubectl delete namespace harbor kubectl create namespace harbor kubectl create secret tls 域名-tls --cert/root/.acme.sh/域名_ecc/fullchain.cer --key/root/.acme.sh/域名_ecc/域名.key --namespace harbor helm install harbor harbor/harbor -n harbor -f /root/harbor-values.yaml7.5 413 Request Entity Too Large原因Ingress 的proxy-body-size太小。解决方案设为0不限大小。7.6 HTTPS 访问报证书错误原因证书链不完整或 Secret 没配对。排查kubectl get secret 域名-tls -n harbor kubectl describe ingress -n harbor harbor-ingress | grep TLS7.7 镜像拉取超时Harbor 镜像在docker.io/goharbor上。如果拉不下来配置 containerd 镜像加速或手动导入。八、常用操作8.1 查看 Harbor 状态kubectl get pods -n harbor kubectl get svc -n harbor kubectl get ingress -n harbor kubectl get pvc -n harbor8.2 查看日志kubectl logs -n harbor -l componentcore --tail100 kubectl logs -n harbor -l componentregistry --tail100 kubectl logs -n harbor -l componentdatabase --tail1008.3 取 admin 密码kubectl get secret -n harbor harbor-core -o jsonpath{.data.HARBOR_ADMIN_PASSWORD} | base64 -d echo8.4 升级 Harborhelm upgrade harbor harbor/harbor -n harbor -f /root/harbor-values.yaml8.5 删除 Harborhelm uninstall harbor -n harbor kubectl delete pvc -n harbor --all kubectl delete namespace harbor警告删除 PVC 会删除所有镜像数据操作前先备份。九、架构总结9.1 组件清单组件副本数作用harbor-core2API 服务、认证harbor-portal2Web UIharbor-registry1镜像存储harbor-jobservice1镜像复制、扫描任务harbor-database1PostgreSQL 数据库harbor-redis1缓存harbor-trivy1镜像扫描9.2 端口速查端口用途40443外网 HTTPS 入口40080外网 HTTP 入口443内网 HAProxy HTTPS80内网 HAProxy HTTP30443Ingress Controller HTTPS NodePort30080Ingress Controller HTTP NodePort9.3 已知隐患隐患影响解决方案registry/jobservice 单副本节点故障服务中断需要 RWX 存储才能多副本Longhorn 空间调度保守卷报insufficient storage改 over-provisioning 和 minimal-available证书手动更新续期后 Secret 需更新配 acme.sh 续期回调脚本database 重建后表没建Harbor 登不上删掉 Harbor 和 PVC重建十、总结这套 Harbor 基于官方 Helm Chart 部署提供了镜像存储私有 Docker 镜像仓库访问控制用户认证和项目权限外部访问FRP Ingress 从公网访问HTTPSLet’s Encrypt 泛域名证书持久化Longhorn 分布式存储高可用core、portal 多副本部署中最容易踩的坑chart 拉取超时用 GitHub 代理下载本地安装。registry 多副本 Multi-Attachregistry 和 jobservice 必须单副本。卷容量超过节点可用空间registry 卷不要超过 20Gi。Longhorn 空间调度保守改storage-over-provisioning-percentage200和storage-minimal-available-percentage10。Harbor database 表没建删掉 Harbor 和所有 PVC重建。HTTPS 证书链不完整用fullchain.cer。externalURL 不带端口外网访问带 40443externalURL必须包含端口。解决了这七个问题Harbor 就能稳定运行。