MetalLB hooks into your Kubernetes cluster, and provides a network load-balancer implementation. In short, it allows you to create Kubernetes services of type LoadBalancer in clusters that don’t run on a cloud provider, and thus cannot simply hook into paid products to provide load balancers.
It has two features that work together to provide this service: address allocation, and external announcement.
The short version is: cloud providers expose proprietary APIs instead of standard protocols to control their network layer, and MetalLB doesn’t work with those APIs.
# 创建namespace $ kubectl apply -f namespace.yaml namespace/metallb-system created $ kubectl get ns NAME STATUS AGE default Active 8d kube-node-lease Active 8d kube-public Active 8d kube-system Active 8d metallb-system Active 8s nginx-quic Active 8d
# 部署deployment和daemonset，以及相关所需的其他资源 $ kubectl apply -f metallb.yaml Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+ podsecuritypolicy.policy/controller created podsecuritypolicy.policy/speaker created serviceaccount/controller created serviceaccount/speaker created clusterrole.rbac.authorization.k8s.io/metallb-system:controller created clusterrole.rbac.authorization.k8s.io/metallb-system:speaker created role.rbac.authorization.k8s.io/config-watcher created role.rbac.authorization.k8s.io/pod-lister created role.rbac.authorization.k8s.io/controller created clusterrolebinding.rbac.authorization.k8s.io/metallb-system:controller created clusterrolebinding.rbac.authorization.k8s.io/metallb-system:speaker created rolebinding.rbac.authorization.k8s.io/config-watcher created rolebinding.rbac.authorization.k8s.io/pod-lister created rolebinding.rbac.authorization.k8s.io/controller created daemonset.apps/speaker created deployment.apps/controller created
# 这里主要就是部署了controller这个deployment来检查service的状态 $ kubectl get deploy -n metallb-system NAME READY UP-TO-DATE AVAILABLE AGE controller 1/1 1 1 86s # speaker则是使用ds部署到每个节点上面用来协商VIP、收发ARP、NDP等数据包 $ kubectl get ds -n metallb-system NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE speaker 3 3 3 3 3 kubernetes.io/os=linux 64s $ kubectl get pod -n metallb-system -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES controller-57fd9c5bb-svtjw 1/1 Running 0 117s 10.8.65.4 tiny-flannel-worker-8-11.k8s.tcinternal <none> <none> speaker-bf79q 1/1 Running 0 117s 10.31.8.11 tiny-flannel-worker-8-11.k8s.tcinternal <none> <none> speaker-fl5l8 1/1 Running 0 117s 10.31.8.12 tiny-flannel-worker-8-12.k8s.tcinternal <none> <none> speaker-nw2fm 1/1 Running 0 117s 10.31.8.1 tiny-flannel-master-8-1.k8s.tcinternal <none> <none>
$ kubectl apply -f configmap-layer2.yaml configmap/config created
apiVersion: v1 kind: Service metadata: name: nginx-lb-service namespace: nginx-quic spec: externalTrafficPolicy: Cluster internalTrafficPolicy: Cluster selector: app: nginx-lb ports: - protocol: TCP port: 80 # match for service access port targetPort: 80 # match for pod access port type: LoadBalancer loadBalancerIP: 10.31.8.100 EOF
# 查看服务状态，这时候TYPE已经变成LoadBalancer $ kubectl get svc -n nginx-quic NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE nginx-lb-service LoadBalancer 10.8.32.221 10.31.8.100 80:30181/TCP 25h
$ arp -a | grep 10.31.8.100 ? (10.31.8.100) at 52:54:00:5c:9c:97 [ether] on eth0
$ arp -a | grep 52:54:00:5c:9c:97 tiny-flannel-worker-8-12.k8s.tcinternal (10.31.8.12) at 52:54:00:5c:9c:97 [ether] on eth0 ? (10.31.8.100) at 52:54:00:5c:9c:97 [ether] on eth0
$ ip a | grep 52:54:00:5c:9c:97 link/ether 52:54:00:5c:9c:97 brd ff:ff:ff:ff:ff:ff
又或者我们可以查看speaker的pod日志，我们可以找到对应的服务IP被宣告的日志记录
1 2 3 4 5 6
$ kubectl logs -f -n metallb-system speaker-fl5l8 {"caller":"level.go:63","event":"serviceAnnounced","ips":["10.31.8.100"],"level":"info","msg":"service has IP, announcing","pool":"default","protocol":"layer2","service":"nginx-quic/nginx-lb-service","ts":"2022-05-16T06:11:34.099204376Z"} {"caller":"level.go:63","event":"serviceAnnounced","ips":["10.31.8.100"],"level":"info","msg":"service has IP, announcing","pool":"default","protocol":"layer2","service":"nginx-quic/nginx-lb-service","ts":"2022-05-16T06:12:09.527334808Z"} {"caller":"level.go:63","event":"serviceAnnounced","ips":["10.31.8.100"],"level":"info","msg":"service has IP, announcing","pool":"default","protocol":"layer2","service":"nginx-quic/nginx-lb-service","ts":"2022-05-16T06:12:09.547734268Z"} {"caller":"level.go:63","event":"serviceAnnounced","ips":["10.31.8.100"],"level":"info","msg":"service has IP, announcing","pool":"default","protocol":"layer2","service":"nginx-quic/nginx-lb-service","ts":"2022-05-16T06:12:34.267651651Z"} {"caller":"level.go:63","event":"serviceAnnounced","ips":["10.31.8.100"],"level":"info","msg":"service has IP, announcing","pool":"default","protocol":"layer2","service":"nginx-quic/nginx-lb-service","ts":"2022-05-16T06:12:34.286130424Z"}
apiVersion:v1 kind:Service metadata: name:nginx-lb-service namespace:nginx-quic spec: allocateLoadBalancerNodePorts:false externalTrafficPolicy:Cluster internalTrafficPolicy:Cluster selector: app:nginx-lb ports: -protocol:TCP port:80# match for service access port targetPort:80# match for pod access port type:LoadBalancer loadBalancerIP:10.31.8.100
IPv4 Unicast Summary (VRF default): BGP router identifier 10.31.254.251, local AS number 64512 vrf-id 0 BGP table version 0 RIB entries 0, using 0 bytes of memory Peers 3, using 2149 KiB of memory
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd PfxSnt Desc 10.31.8.1 4 64513 0 0 0 0 0 never Active 0 10-31-8-1 10.31.8.11 4 64513 0 0 0 0 0 never Active 0 10-31-8-11 10.31.8.12 4 64513 0 0 0 0 0 never Active 0 10-31-8-12
IPv4 Unicast Summary (VRF default): BGP router identifier 10.31.254.251, local AS number 64512 vrf-id 0 BGP table version 3 RIB entries 5, using 920 bytes of memory Peers 3, using 2149 KiB of memory
$ kubectl get svc -n nginx-quic NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE nginx-lb-service LoadBalancer 10.8.4.92 <pending> 80/TCP 18h
重新修改loadBalancerIP为10.9.1.1，此时可以看到服务已经正常
1 2 3
$ kubectl get svc -n nginx-quic NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE nginx-lb-service LoadBalancer 10.8.4.92 10.9.1.1 80/TCP 18h
再查看controller的日志可以看到
1 2 3 4 5 6 7 8 9 10 11 12 13
$ kubectl logs controller-57fd9c5bb-d6jsl -n metallb-system {"branch":"HEAD","caller":"level.go:63","commit":"v0.12.1","goversion":"gc / go1.16.14 / amd64","level":"info","msg":"MetalLB controller starting version 0.12.1 (commit v0.12.1, branch HEAD)","ts":"2022-05-18T03:45:45.440872105Z","version":"0.12.1"} {"caller":"level.go:63","configmap":"metallb-system/config","event":"configLoaded","level":"info","msg":"config (re)loaded","ts":"2022-05-18T03:45:45.610395481Z"} {"caller":"level.go:63","error":"[\"10.31.8.100\"] is not allowed in config","event":"clearAssignment","level":"info","msg":"current IP not allowed by config, clearing","service":"nginx-quic/nginx-lb-service","ts":"2022-05-18T03:45:45.611009691Z"} {"caller":"level.go:63","event":"clearAssignment","level":"info","msg":"user requested a different IP than the one currently assigned","reason":"differentIPRequested","service":"nginx-quic/nginx-lb-service","ts":"2022-05-18T03:45:45.611062419Z"} {"caller":"level.go:63","error":"controller not synced","level":"error","msg":"controller not synced yet, cannot allocate IP; will retry after sync","op":"allocateIPs","service":"nginx-quic/nginx-lb-service","ts":"2022-05-18T03:45:45.611080525Z"} {"caller":"level.go:63","event":"stateSynced","level":"info","msg":"controller synced, can allocate IPs now","ts":"2022-05-18T03:45:45.611117023Z"} {"caller":"level.go:63","error":"[\"10.31.8.100\"] is not allowed in config","event":"clearAssignment","level":"info","msg":"current IP not allowed by config, clearing","service":"nginx-quic/nginx-lb-service","ts":"2022-05-18T03:45:45.617013146Z"} {"caller":"level.go:63","event":"clearAssignment","level":"info","msg":"user requested a different IP than the one currently assigned","reason":"differentIPRequested","service":"nginx-quic/nginx-lb-service","ts":"2022-05-18T03:45:45.617089367Z"} {"caller":"level.go:63","error":"[\"10.31.8.100\"] is not allowed in config","level":"error","msg":"IP allocation failed","op":"allocateIPs","service":"nginx-quic/nginx-lb-service","ts":"2022-05-18T03:45:45.617122976Z"} {"caller":"level.go:63","event":"serviceUpdated","level":"info","msg":"updated service object","service":"nginx-quic/nginx-lb-service","ts":"2022-05-18T03:45:45.626039403Z"} {"caller":"level.go:63","error":"[\"10.31.8.100\"] is not allowed in config","level":"error","msg":"IP allocation failed","op":"allocateIPs","service":"nginx-quic/nginx-lb-service","ts":"2022-05-18T03:45:45.626361986Z"} {"caller":"level.go:63","event":"ipAllocated","ip":["10.9.1.1"],"level":"info","msg":"IP address assigned by controller","service":"nginx-quic/nginx-lb-service","ts":"2022-05-18T03:47:19.943434144Z"}
{"caller":"level.go:63","configmap":"metallb-system/config","event":"peerAdded","level":"info","msg":"peer configured, starting BGP session","peer":"10.31.254.251","ts":"2022-05-18T03:41:55.046091105Z"} {"caller":"level.go:63","configmap":"metallb-system/config","event":"configLoaded","level":"info","msg":"config (re)loaded","ts":"2022-05-18T03:41:55.046268735Z"} {"caller":"level.go:63","error":"assigned IP not allowed by config","ips":["10.31.8.100"],"level":"error","msg":"IP allocated by controller not allowed by config","op":"setBalancer","service":"nginx-quic/nginx-lb-service","ts":"2022-05-18T03:41:55.051955069Z"} struct { Version uint8; ASN16 uint16; HoldTime uint16; RouterID uint32; OptsLen uint8 }{Version:0x4, ASN16:0xfc00, HoldTime:0xb4, RouterID:0xa1ffefd, OptsLen:0x1e} {"caller":"level.go:63","event":"sessionUp","level":"info","localASN":64513,"msg":"BGP session established","peer":"10.31.254.251:179","peerASN":64512,"ts":"2022-05-18T03:41:55.052734174Z"} {"caller":"level.go:63","level":"info","msg":"triggering discovery","op":"memberDiscovery","ts":"2022-05-18T03:42:40.183574415Z"} {"caller":"level.go:63","level":"info","msg":"node event - forcing sync","node addr":"10.31.8.12","node event":"NodeLeave","node name":"tiny-flannel-worker-8-12.k8s.tcinternal","ts":"2022-05-18T03:44:03.649494062Z"} {"caller":"level.go:63","error":"assigned IP not allowed by config","ips":["10.31.8.100"],"level":"error","msg":"IP allocated by controller not allowed by config","op":"setBalancer","service":"nginx-quic/nginx-lb-service","ts":"2022-05-18T03:44:03.655003303Z"} {"caller":"level.go:63","level":"info","msg":"node event - forcing sync","node addr":"10.31.8.12","node event":"NodeJoin","node name":"tiny-flannel-worker-8-12.k8s.tcinternal","ts":"2022-05-18T03:44:06.247929645Z"} {"caller":"level.go:63","error":"assigned IP not allowed by config","ips":["10.31.8.100"],"level":"error","msg":"IP allocated by controller not allowed by config","op":"setBalancer","service":"nginx-quic/nginx-lb-service","ts":"2022-05-18T03:44:06.25369106Z"} {"caller":"level.go:63","event":"updatedAdvertisements","ips":["10.9.1.1"],"level":"info","msg":"making advertisements using BGP","numAds":1,"pool":"default","protocol":"bgp","service":"nginx-quic/nginx-lb-service","ts":"2022-05-18T03:47:19.953729779Z"} {"caller":"level.go:63","event":"serviceAnnounced","ips":["10.9.1.1"],"level":"info","msg":"service has IP, announcing","pool":"default","protocol":"bgp","service":"nginx-quic/nginx-lb-service","ts":"2022-05-18T03:47:19.953912236Z"}
我们在集群外的任意一个机器进行测试
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
$ curl -v 10.9.1.1 * About to connect() to 10.9.1.1 port 80 (#0) * Trying 10.9.1.1... * Connected to 10.9.1.1 (10.9.1.1) port 80 (#0) > GET / HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 10.9.1.1 > Accept: */* > < HTTP/1.1 200 OK < Server: nginx < Date: Wed, 18 May 2022 04:17:41 GMT < Content-Type: text/plain < Content-Length: 16 < Connection: keep-alive < 10.8.64.0:43939 * Connection #0 to host 10.9.1.1 left intact
tiny-openwrt-plus# show ip route Codes: K - kernel route, C - connected, S - static, R - RIP, O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP, T - Table, v - VNC, V - VNC-Direct, A - Babel, F - PBR, f - OpenFabric, > - selected route, * - FIB route, q - queued, r - rejected, b - backup t - trapped, o - offload failure
K>* 0.0.0.0/0 [0/0] via 10.31.254.254, eth0, 00:04:52 B>* 10.9.1.1/32 [20/0] via 10.31.8.1, eth0, weight 1, 00:01:40 * via 10.31.8.11, eth0, weight 1, 00:01:40 * via 10.31.8.12, eth0, weight 1, 00:01:40 C>* 10.31.0.0/16 is directly connected, eth0, 00:04:52
我们再创建多几个服务进行测试
1 2 3 4 5
# kubectl get svc -n nginx-quic NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE nginx-lb-service LoadBalancer 10.8.4.92 10.9.1.1 80/TCP 23h nginx-lb2-service LoadBalancer 10.8.10.48 10.9.1.2 80/TCP 64m nginx-lb3-service LoadBalancer 10.8.6.116 10.9.1.3 80/TCP 64m
tiny-openwrt-plus# show ip bgp BGP table version is 3, local router ID is 10.31.254.251, vrf id 0 Default local pref 100, local AS 64512 Status codes: s suppressed, d damped, h history, * valid, > best, = multipath, i internal, r RIB-failure, S Stale, R Removed Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self Origin codes: i - IGP, e - EGP, ? - incomplete RPKI validation codes: V valid, I invalid, N Not found
tiny-openwrt-plus# show ip route Codes: K - kernel route, C - connected, S - static, R - RIP, O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP, T - Table, v - VNC, V - VNC-Direct, A - Babel, F - PBR, f - OpenFabric, > - selected route, * - FIB route, q - queued, r - rejected, b - backup t - trapped, o - offload failure
K>* 0.0.0.0/0 [0/0] via 10.31.254.254, eth0, 00:06:12 B>* 10.9.1.1/32 [20/0] via 10.31.8.1, eth0, weight 1, 00:03:00 * via 10.31.8.11, eth0, weight 1, 00:03:00 * via 10.31.8.12, eth0, weight 1, 00:03:00 B>* 10.9.1.2/32 [20/0] via 10.31.8.1, eth0, weight 1, 00:03:00 * via 10.31.8.11, eth0, weight 1, 00:03:00 * via 10.31.8.12, eth0, weight 1, 00:03:00 B>* 10.9.1.3/32 [20/0] via 10.31.8.1, eth0, weight 1, 00:03:00 * via 10.31.8.11, eth0, weight 1, 00:03:00 * via 10.31.8.12, eth0, weight 1, 00:03:00 C>* 10.31.0.0/16 is directly connected, eth0, 00:06:12