深入理解kubernetes-第一章 calico IPAM

我们都知道在物理主机或者虚拟机上一台正常的主机是可以通过后DHCP动态分配IP的但是在kubernetes中不存在DHCP来分配每个Pod的ip地址。在 Kubernetes 中每个 Pod 都需要一个唯一的 IP 地址来与其他 Pod 和服务进行通信。IP 地址管理非常重要因为动态创建 PodPod 是临时性的需要频繁地创建和销毁。每个新的 Pod 都需要从集群的地址空间中获取一个可用的 IP 地址。避免 IP 冲突如果没有适当的 IP 管理工具多个 Pod 可能会被分配相同的 IP 地址从而导致网络冲突和通信故障。高效的地址分配IPAM 系统将 IP 地址按块为单位分配给各个节点从而减少了单个 IP 请求所带来的负担并提升了系统的可扩展性。跨节点通信集群中不同节点上的容器需要可路由的 IP 地址以便能够直接进行通信而无需通过 NAT 层。IPAM 通过合理的子网分配机制帮助解决了这一问题。Calico 的 IPAM 技术通过预先将 IP 块分配给各个节点从而实现高效的 IP 分配。这一机制能够确保节点快速启动同时在整个集群范围内保持正确的 IP 地址管理。使用clab实现实验环境的搭建前提条件完成containerlab的安装 - 采用宿主机安装方式完成kind镜像的下载 - 采用容器安装完成docker的安装清单文件calico-ipam.clab.yamlContainerLab 拓扑结构name:calico-ipam# 生命使用和calico 同样的网桥避免和现有的docker网络有冲突mgmt:network:bridgetopology:nodes:calico-ipam:kind:k8s-kindimage:kindest/node:v1.28.0startup-config:calico-ipam-no-cni.yamlextras:k8s_kind:deploy:wait:0scalico-ipam-no-cni.yamlcalico 无需使用 CNI 的集群配置文件这里的配置是给kind使用的andrewandrew ~/k8-networking-calico-containerlab/containerlab/01-calico-ipam(master*?)$catcalico-ipam-no-cni.yamlapiVersion:kind.x-k8s.io/v1alpha4kind:Clusternetworking:disableDefaultCNI:truepodSubnet:192.168.0.0/16serviceSubnet:10.96.0.0/16nodes:-role:control-plane-role:worker-role:workercalico-cni-config/custom-resources.yaml基于 IPAM 和 IP 池配置的定制型 Calico 安装资源apiVersion:operator.tigera.io/v1kind:Installationmetadata:name:defaultspec:# Configures Calico networking.calicoNetwork:# Note: The ipPools section cannot be modified post-install.ipPools:-blockSize:26cidr:192.168.0.0/16encapsulation:VXLANCrossSubnetnatOutgoing:EnablednodeSelector:all()---apiVersion:operator.tigera.io/v1kind:APIServermetadata:name:defaultspec:{}通过一下命令完成实验环境的部署chmodx deploy.sh ./deploy.shcalico CNI ConfigurationCalico 的部署方式使用了一种自定义的安装资源custom-resources.yaml▸该资源用于定义容器网络的配置。在选择 CIDR 范围时需要注意以下几个关键点默认IP地址池cidr: 192.168.0.0/16可以提供65536个IP地址足够支持大型集群的规模块大小根据需要为每个节点分配了26个块每个块包含了64个IP地址封装机制采用 VXLAN CrossSubnet 技术来实现节点间的通信。NAT 输出功能已启用允许容器实例访问外部网络CIDR 地址选择直接影响了集群的可扩展性和网络策略的有效性。/16 范围可以支持大约 1,024 个节点而/26 范围的地址则适合用于大多数实验室和生产环境。使用deploy.sh脚本能自动化完成以上实验#!/bin/bashset-e# Exit immediately if a command exits with a non-zero status# 如果以前安装过这里先清理echo Destroying existing ContainerLab topology sudocontainerlab destroy-tcalico-ipam.clab.yaml||{echoFailed to destroy existing topology;exit1;}# 按照配置部署集群环境echo Deploying ContainerLab topology sudocontainerlab deploy-tcalico-ipam.clab.yaml||{echoFailed to deploy topology;exit1;}echo Waiting for Kind cluster to be ready (30 seconds) sleep30echo Setting up kubeconfig # Create kubeconfig directory if it doesnt existmkdir-p~/.kube# Export kubeconfig to a specific file to avoid conflictssudokind get kubeconfig--namecalico-ipamcalico-ipam.kubeconfigsudochmod644calico-ipam.kubeconfig# 保证kueclt 指向 kind集群# 导出配置以便通过kubectl命令能访问kind集群# Use the specific kubeconfig file for all kubectl commandsexportKUBECONFIG$(pwd)/calico-ipam.kubeconfigecho Installing calicoctl curl-Lhttps://github.com/projectcalico/calico/releases/download/v3.30.0/calicoctl-linux-amd64-ocalicoctl||{echoFailed to download calicoctl;exit1;}chmodx calicoctlsudomvcalicoctl /usr/local/bin/||{echoFailed to move calicoctl to /usr/local/bin;exit1;}echocalicoctl version:$(calicoctl version)||{echoWarning: calicoctl may not be installed correctly;}echo Waiting for Kubernetes API to be available untilkubectl get nodes/dev/null;doechoWaiting for Kubernetes API...sleep5done# 部署calico环境echo Installing Calico 3.30.0 kubectl create-fhttps://raw.githubusercontent.com/projectcalico/calico/v3.30.0/manifests/operator-crds.yaml||{echoFailed to install Calico CRDs;exit1;}kubectl create-fhttps://raw.githubusercontent.com/projectcalico/calico/v3.30.0/manifests/tigera-operator.yaml||{echoFailed to install Tigera operator;exit1;}# 使用calico 自定义资源进行配置echo Applying custom Calico resources kubectl apply-fcalico-cni-config/custom-resources.yaml||{echoFailed to apply custom resources;exit1;}echo Waiting for TigeraStatus to be ready echoInitial check, expect resources to be unavailable...kubectl get tigerastatusechoWaiting for TigeraStatus to become available (may take several minutes)...whiletrue;do# More precise check using conditionsAPI_AVAILABLE$(kubectl get tigerastatus apiserver-ojsonpath{.status.conditions[?(.typeAvailable)].status}2/dev/null||echoFalse)CALICO_AVAILABLE$(kubectl get tigerastatus calico-ojsonpath{.status.conditions[?(.typeAvailable)].status}2/dev/null||echoFalse)if[[$API_AVAILABLETrue$CALICO_AVAILABLETrue]];thenechoCalico API server and core components are ready!breakfiechoStill waiting for Calico components to be ready...sleep15doneecho TigeraStatus final check kubectl get tigerastatusechoKubernetes nodes:kubectl get nodes-owideechoechoTo use this cluster with kubectl, run:echoexport KUBECONFIG$(pwd)/calico-ipam.kubeconfig%环境检查检查ContainerLab的topo结构andrewandrew ~/k8-networking-calico-containerlab/containerlab/01-calico-ipam(master*?)$ containerlab inspect-tcalico-ipam.clab.yaml14:10:38 INFO Parsingchecking topologyfilecalico-ipam.clab.yaml ╭───────────────────────────┬──────────────────────┬─────────┬───────────────────────╮ │ Name │ Kind/Image │ State │ IPv4/6 Address │ ├───────────────────────────┼──────────────────────┼─────────┼───────────────────────┤ │ calico-ipam-control-plane │ k8s-kind │ running │192.168.48.3 │ │ │ kindest/node:v1.28.0 │ │ fc00:f853:ccd:e793::3 │ ├───────────────────────────┼──────────────────────┼─────────┼───────────────────────┤ │ calico-ipam-worker │ k8s-kind │ running │192.168.48.2 │ │ │ kindest/node:v1.28.0 │ │ fc00:f853:ccd:e793::2 │ ├───────────────────────────┼──────────────────────┼─────────┼───────────────────────┤ │ calico-ipam-worker2 │ k8s-kind │ running │192.168.48.4 │ │ │ kindest/node:v1.28.0 │ │ fc00:f853:ccd:e793::4 │ ╰───────────────────────────┴──────────────────────┴─────────┴───────────────────────╯检查calico的安装状态andrewandrew ~/k8-networking-calico-containerlab/containerlab/01-calico-ipam(master*?)$ kubectl get tigerastatus NAME AVAILABLE PROGRESSING DEGRADED SINCE apiserver True False False 124m calico True False False 124m ippools True False False 158m检查节点状态andrewandrew ~/k8-networking-calico-containerlab/containerlab/01-calico-ipam(master*?)$ kubectl get nodes NAME STATUS ROLES AGE VERSION calico-ipam-control-plane Ready control-plane 166m v1.28.0 calico-ipam-worker Readynone166m v1.28.0 calico-ipam-worker2 Readynone166m v1.28.0ipam的整体状态andrewandrew ~/k8-networking-calico-containerlab/containerlab/01-calico-ipam(master*?)$ calicoctl ipam show ---------------------------------------------------------------|GROUPING|CIDR|IPS TOTAL|IPS IN USE|IPS FREE|---------------------------------------------------------------|IP Pool|192.168.0.0/16|65536|12(0%)|65524(100%)|---------------------------------------------------------------CIDR整个子网网络的范围192.168.0.0/16总 IP 数该池中可用的 IP 地址总数65,536 个正在使用的 IP 地址当前已将 IP 地址分配给了相应的容器。免费 IP 地址可供新节点分配使用的 IP 地址阻塞关联列表andrewandrew ~/k8-networking-calico-containerlab/containerlab/01-calico-ipam(master*?)$ kubectl get blockaffinities NAME CREATED AT calico-ipam-control-plane-192-168-18-64-262026-09-23T04:00:09Z calico-ipam-worker-192-168-131-128-262026-09-23T04:03:36Z calico-ipam-worker2-192-168-112-128-262026-09-23T04:06:00ZBlockAffinity 资源代表了分配给节点的 IPAM 块分配信息每个条目都显示了哪个 IP 地址范围被分配给了哪个节点命名规范是使用calico-ipam-block-cidr-with-dashes-prefix-length来表示Calico 根据需求将/26 的块每个块包含 64 个 IP 地址分配给各个节点详细区块关联性数据andrewandrew ~/k8-networking-calico-containerlab/containerlab/01-calico-ipam(master*?)$ kubectl get blockaffinities-oyaml apiVersion: v1 items: - apiVersion: projectcalico.org/v3 kind: BlockAffinity metadata: creationTimestamp:2026-09-23T04:00:09Zname: calico-ipam-control-plane-192-168-18-64-26 resourceVersion:4698uid: a4c00304-a475-4212-a05d-6f390e886ed5 spec:# 分配给某个节点的特定 IP 地址范围例如192.168.0.64/26cidr:192.168.18.64/26# 拥有该 IP 地址的 Kubernetes 节点node: calico-ipam-control-plane# confirmed 这个区块正在被积极分配并投入使用中# pending 赋值操作正在进行中state: confirmed - apiVersion: projectcalico.org/v3 kind: BlockAffinity metadata: creationTimestamp:2026-09-23T04:03:36Zname: calico-ipam-worker-192-168-131-128-26 resourceVersion:5163uid: e5227720-b462-4482-97b9-c016e1dc1d5e spec: cidr:192.168.131.128/26 node: calico-ipam-worker state: confirmed - apiVersion: projectcalico.org/v3 kind: BlockAffinity metadata: creationTimestamp:2026-09-23T04:06:00Zname: calico-ipam-worker2-192-168-112-128-26 resourceVersion:5474uid: 35c56f39-4eff-4b2e-b825-02e92bb24077 spec: cidr:192.168.112.128/26 node: calico-ipam-worker2 state: confirmed kind: List metadata: resourceVersion:Formatted Block Affinitiesandrewandrew ~/k8-networking-calico-containerlab/containerlab/01-calico-ipam(master*?)$ kubectl get blockaffinities-ojsonpath{range .items[*]}{.metadata.name}{\t}{.spec.node}{\t}{.spec.cidr}{\n}{end}calico-ipam-control-plane-192-168-18-64-26 calico-ipam-control-plane192.168.18.64/26 calico-ipam-worker-192-168-131-128-26 calico-ipam-worker192.168.131.128/26 calico-ipam-worker2-192-168-112-128-26 calico-ipam-worker2192.168.112.128/26总结IP 地址池较大的 CIDR 范围如 192.168.0.0/16用于定义整体地址空间IP 块从 IP 池中划分出较小的子网如/26 的块并将其分配给各个节点。块关联IP 块与节点之间的分配关系IPAM当创建 Pod 时Calico 会自动管理分配给 Pod 的 IP 地址分配使其位于指定的 IP 块内。Troubleshooting检查这些节点是否被分配了足够的IP地址块请确认calico相关的组件在正常运行kubectl get pods -n calico-system查看calico node的日志kubectl logs -n calico-system -l k8s-appcalico-node清理环境andrewandrew ~/k8-networking-calico-containerlab/containerlab/01-calico-ipam(master*?)$catdestroy.sh#!/bin/bash# Lab-specific cleanup script# This script removes only resources created by this specific labset-euopipefail# Colors for outputGREEN\033[0;32mYELLOW\033[1;33mNC\033[0m# No Color# Get the lab directory nameLAB_DIR$(basename$(pwd))# Detect topology name from .clab.yaml file# Try common names first, then any .clab.yaml fileTOPOLOGY_FILEif[-ftopology.clab.yaml];thenTOPOLOGY_FILEtopology.clab.yamlelseTOPOLOGY_FILE$(find.-maxdepth1-name*.clab.yaml-typef|head-1)fiif[-z$TOPOLOGY_FILE];thenechoError: No .clab.yaml file found in current directoryexit1fiTOPOLOGY_NAME$(grep^name:$TOPOLOGY_FILE|head-1|awk{print $2}|tr-d|tr-d)if[-z$TOPOLOGY_NAME];thenechoError: Could not detect topology name from$TOPOLOGY_FILEexit1fi# Detect Kind cluster name from deploy.shKIND_CLUSTER$(grepkind get kubeconfig --namedeploy.sh2/dev/null|head-1|grep-oname[^ ]*|cut-d-f2|tr-d|tr-d ||echo)if[-z$KIND_CLUSTER];then# Try to get from topology file (k8s-kind node name)KIND_CLUSTER$(grep-A5kind: k8s-kind$TOPOLOGY_FILE|grep-E^\s[a-zA-Z0-9-]:|head-1|awk-F:{print $1}|tr-d ||echo)fi# Detect kubeconfig filename from deploy.shKUBECONFIG_FILE$(grep\.kubeconfigdeploy.sh2/dev/null|head-1|grep-o[^ ]*\.kubeconfig|head-1||echo)if[-z$KUBECONFIG_FILE][-n$KIND_CLUSTER];thenKUBECONFIG_FILE${KIND_CLUSTER}.kubeconfigfiecho-e${YELLOW}${NC}echo-e${YELLOW}Cleaning up lab:$LAB_DIR${NC}echo-e${YELLOW}${NC}echoechoDetected resources:echo Topology name:$TOPOLOGY_NAMEecho Kind cluster:${KIND_CLUSTER:-N/A}echo Kubeconfig file:${KUBECONFIG_FILE:-N/A}echoread-pContinue with cleanup? (yes/no): -rechoif[[!$REPLY~^[Yy][Ee][Ss]$]];thenechoCleanup cancelled.exit0fi# Step 1: Destroy ContainerLab topologyecho-e${GREEN}[1/5] Destroying ContainerLab topology:$TOPOLOGY_NAME${NC}ifcommand-vcontainerlab/dev/null;thensudocontainerlab destroy-t$TOPOLOGY_FILE2/dev/null||echo Topology may not exist or already destroyedelseecho containerlab command not found, skippingfi# Step 2: Delete Kind clusterif[-n$KIND_CLUSTER];thenecho-e${GREEN}[2/5] Deleting Kind cluster:$KIND_CLUSTER${NC}ifcommand-vkind/dev/null;thenkind delete cluster--name$KIND_CLUSTER2/dev/null||echo Cluster may not exist or already deletedelseecho kind command not found, skippingfielseecho-e${GREEN}[2/5] No Kind cluster detected, skipping${NC}fi# Step 3: Remove lab-specific containersecho-e${GREEN}[3/5] Removing lab-specific containers...${NC}CONTAINERS$(dockerps-a--format{{.Names}}2/dev/null|grep-Eclab-${TOPOLOGY_NAME}-|${KIND_CLUSTER}-||true)if[-n$CONTAINERS];thenecho$CONTAINERS|whileread-rcontainer;doif[-n$container];thenecho Removing container:$containerdockerrm-f$container2/dev/null||truefidoneelseecho No lab-specific containers foundfi# Step 4: Remove lab-specific networksecho-e${GREEN}[4/5] Removing lab-specific networks...${NC}NETWORKS$(dockernetworkls--format{{.Name}}2/dev/null|grep-Eclab-${TOPOLOGY_NAME}||true)if[-n$NETWORKS];thenecho$NETWORKS|whileread-rnetwork;doif[-n$network];thenecho Removing network:$networkdockernetworkrm$network2/dev/null||truefidoneelseecho No lab-specific networks foundfi# Step 5: Clean up Kubernetes resources and kubeconfigif[-n$KUBECONFIG_FILE][-f$KUBECONFIG_FILE];thenecho-e${GREEN}[5/5] Cleaning up Kubernetes resources...${NC}exportKUBECONFIG$(pwd)/$KUBECONFIG_FILEifcommand-vkubectl/dev/nullkubectl cluster-info/dev/null21;then# Force delete stuck terminating podskubectl get pods --all-namespaces --field-selectorstatus.phase!Running,status.phase!Succeeded-ojson2/dev/null|\jq-r.items[] | select(.metadata.deletionTimestamp!null) | \(.metadata.namespace) \(.metadata.name)2/dev/null|\whileread-rnamespace name;doif[-n$namespace][-n$name];thenecho Force deleting stuck pod:$namespace/$namekubectl delete pod$name-n$namespace--force--grace-period02/dev/null||truefidone||true# Delete lab-related namespaces (excluding system namespaces)kubectl get namespaces-ojsonpath{range .items[*]}{.metadata.name}{\n}{end}2/dev/null|\grep-vE^(default|kube-system|kube-public|kube-node-lease|local-path-storage)$|\whileread-rnamespace;doif[-n$namespace];thenecho Deleting namespace:$namespacekubectl delete namespace$namespace--timeout30s2/dev/null||truefidone||truefiunsetKUBECONFIG# Remove kubeconfig fileecho Removing kubeconfig file:$KUBECONFIG_FILErm-f$KUBECONFIG_FILEelseecho-e${GREEN}[5/5] No kubeconfig file found, skipping${NC}fiechoecho-e${GREEN}${NC}echo-e${GREEN}Cleanup Complete!${NC}echo-e${GREEN}${NC}echo