526互联

办公网络访问k8s集群内部ip和域名

发布时间 2023-08-24 18:22:00作者: 小憩清风

本地k8s集群pod和办公网互通并实现域名解析

背景

在微服务场景下，研发团队在进行开发联调测试过程中，需要连接到k8s集群的注册中心中等中间件并和集群内的其他业务服务pod互相通信。

网络基本情况

网络类型 ip段
办公网络 172.16.0.0/16
Pod网络 10.233.64.0/18
Service网络 10.233.0.0/18

实施方案

  • 添加一台主机，打上污点，禁止调度，专门用于路由转发
# 开启该主机的路由转发功能
[root@nfs bin]# sysctl -a |grep ip_forward
net.ipv4.ip_forward = 1
# 如果该值不为1则需修改
vim /etc/sysctl.conf
net.ipv4.ip_forward = 1
# 生效，如需稳妥可以重启服务器
sysctl -p

# 查看node状态
[root@master ~]# kubectl describe node node4
Name:               node4
Roles:              worker
...
Taints:             <none>
Unschedulable:      false
...

# 打污点
kubectl taint nodes node4 node-role.kubernetes.io/master=:NoSchedule
# 再次查看node状态
[root@master ~]# kubectl describe node node4
Name:               node4
Roles:              worker
...
                    node.alpha.kubernetes.io/ttl: 0
                    projectcalico.org/IPv4Address: 172.16.50.146/23
                    projectcalico.org/IPv4IPIPTunnelAddr: 10.233.105.0
                    volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp:  Wed, 23 Aug 2023 11:20:08 +0800
Taints:             node-role.kubernetes.io/master:NoSchedule
Unschedulable:      false
...

# node4配置iptables进行转发
iptables -t nat -A POSTROUTING -s 172.16.0.0/16 -d 10.233.64.0/18 -j MASQUERADE
iptables -t nat -A POSTROUTING -s 172.16.0.0/16 -d 10.233.0.0/18 -j MASQUERADE

# 检查是iptable链情况
[root@node4 bin]#  iptables -t nat -L -n --line-numbers | grep -A 10 "Chain POSTROUTING"
Chain POSTROUTING (policy ACCEPT)
num  target     prot opt source               destination         
1    cali-POSTROUTING  all  --  0.0.0.0/0            0.0.0.0/0            /* cali:O3lYWMrLQYEMJtB5 */
2    KUBE-POSTROUTING  all  --  0.0.0.0/0            0.0.0.0/0            /* kubernetes postrouting rules */
3    MASQUERADE  all  --  172.17.0.0/16        0.0.0.0/0           
4    MASQUERADE  all  --  172.16.0.0/16        10.233.64.0/18      
5    MASQUERADE  all  --  172.16.0.0/16        10.233.0.0/18  

# 如有错误可以删除
 iptables -t nat -D POSTROUTING 3
  • 三层交换机配置路由,将pod和service下一跳指向到node3节点
# 添加路由
sys
ip route-static 10.233.0.0 18 172.16.50.146
ip route-static 10.233.64.0 18 172.16.50.146

# 查看路由
display ip routing-table

...

Destination/Mask   Proto   Pre Cost        NextHop         Interface
0.0.0.0/0          Static  60  0           172.16.0.1      ...
0.0.0.0/32         Direct  0   0           127.0.0.1       ...
10.233.0.0/18      Static  60  0           172.16.50.146   ...  # pod路由
10.233.64.0/18     Static  60  0           172.16.50.146   ...  # service路由

# 保存配置
save

# 本地电脑测试连通性
xxxdeiMac:~ xxx$ ping 10.233.0.3
PING 10.233.0.3 (10.233.0.3): 56 data bytes
64 bytes from 10.233.0.3: icmp_seq=0 ttl=63 time=0.460 ms
64 bytes from 10.233.0.3: icmp_seq=1 ttl=63 time=0.481 ms
64 bytes from 10.233.0.3: icmp_seq=2 ttl=63 time=0.453 ms
64 bytes from 10.233.0.3: icmp_seq=3 ttl=63 time=0.464 ms
  • 集群内域名解析集群外使用

这里只需要拿到coredns的service IP，配置到个人电脑上即可以进行正常解析。如果公司内部有搭建dns服务器，可以直接将cluster.local forword到coredns service ip上，也能实现。

kubectl get svc coredns -n kube-system

NAME      TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)                  AGE
coredns   ClusterIP   10.233.0.3   <none>        53/UDP,53/TCP,9153/TCP   279d


# 测试解析
iMac:~ xxx$ dig A default-http-backend.kubesphere-controls-system.svc.cluster.local

; <<>> DiG 9.10.6 <<>> A default-http-backend.kubesphere-controls-system.svc.cluster.local
;; global options: +cmd
;; Got answer:
;; WARNING: .local is reserved for Multicast DNS
;; You are currently testing what happens when an mDNS query is leaked to DNS
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 14176
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;default-http-backend.kubesphere-controls-system.svc.cluster.local. IN A

;; ANSWER SECTION:
default-http-backend.kubesphere-controls-system.svc.cluster.local. 30 IN A 10.233.21.176 # 此处可以看到，已经解析成功

;; Query time: 1 msec
;; SERVER: 10.233.0.3#53(10.233.0.3)
;; WHEN: Thu Aug 24 18:03:27 CST 2023
;; MSG SIZE  rcvd: 175