目录
kubernetes 持久化存储
k8s 存储介绍
特地对象存储
容器内部的存储在生命周期是短暂的，会随着容器环境的销毁而销毁，具有不稳定性。在 k8s 里将对容器应用所需的存储资源抽象为存储卷 (Volume) 概念来解决这些问题。
ConfigMap: 应用配置
Secret: 加密数据
ServiceAccountToken: token数据
本地存储
hostPath
emptyDir
网络共享存储
CephFS: 开源共享存储系统
GlusterFS: 开源共享存储系统
NFS: 开源共享存储
PersistentVolume: 简称PV，划分资源
## 申请磁盘空间资源（限制pod对磁盘的使用）
PersistentVolumeClaim: 简称PVC，持久化存储的申请空间
EmptyDir 类型
cat >emptyDir.yaml <<EOF
apiVersion: v1
kind: Pod
metadata:
name: busybox-empty
spec:
containers:
- name: busybox-pod
image: busybox
volumeMounts:
- mountPath: /data/busybox/
name: cache-volume
command: ["/bin/sh","-c","while true;do echo $(date) >> /data/busybox/index.html;sleep 3;done"]
volumes:
- name: cache-volume
emptyDir: {}
EOF
hostPath 类型
type 类型说明
https://kubernetes.io/docs/concepts/storage/volumes/#hostpath
DirectoryOrCreate 目录不存在就自动创建
Directory 目录必须存在
FileOrCreate 文件不存在则创建
File 文件必须存在
持久化存储 PV 和 PVC
PV 和 PVC 生命周期
PV 和 PVC 需要注意的地方
# 在PV的整个生命周期中，可能会处于4种不同的阶段:
Avaliable（可用）：表示可用状态，还未被任何PVC绑定
Bound（可绑定）：表示PV已经被PVC绑定
Released（已释放）：PVC被删除，但是资源还未被集群重新声明
Failed（失败）：表示该PV的自动回收失败
创建 PVC 之后，k8s 就会去查找满足我们声明要求的 PV, 比如 storageClassName , accessModes 以及容量这些是否满足要求，如果满足要求就将 PV 和 PVC 绑定在一起。
pv 资源清单
[root@master-1 tmp]# vim pv.yml
apiVersion: v1
kind: PersistentVolume
metadata:
name: pv01
spec:
capacity:
storage: 5Gi
accessModes:
- ReadWriteOnce
persistentVolumeReclaimPolicy: Recycle
storageClassName: nfs01
nfs:
path: /data/wordpress/
server: 172.16.1.91
capacity: PV存储的容量
accessModes: 访问模式,k8s支持的访问模式如下
------------------------------------------------------
ReadWriteOnce(RWO): 读写权限，并且只能被单个Node挂载
ReadOnlyMany(ROX): 只读权限，允许被多个Node挂载
ReadWriteMany(RWX): 读写权限，允许被多个Node挂载
persistentVolumeReclaimPolicy: 回收策略
------------------------------------------------------
Retain: 保留数据，需要手工处理
Recycle: 简单清除文件的操作(例如运行rm -rf /dada/* 命令)
Delete: 与PV相连的后端存储完成Volume的删除操作
目前只有NFS和HostPath两种类型的PV支持Recycle策略。
storageClassName: 存储类别
具有特定类别的PV只能与请求了该类别的PVC绑定。未指定类型的PV则只能对与不请求任何类别的PVC绑定。
[root@master-1 tmp]# kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
pv01 5Gi RWO Recycle Available nfs01 16s
PVC 资源清单
[root@master-1 tmp]# vim pvc.yml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: wp-pvc
namespace: blog
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 4Gi
storageClassName: nfs01
[root@master-1 tmp]# kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
wp-pvc Bound pv01 5Gi RWO nfs01 4s
[root@master-1 tmp]# kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
pv01 5Gi RWO Recycle Bound default/wp-pvc nfs01 3m57s
## PVC绑定PV，匹配规则
ACCESS MODES：挂载模式
STORAGECLASS：存储名字
CAPACITY：存储大小
POD 要挂载 PVC
apiVersion: apps/v1
kind: Deployment
metadata:
name: wp-dp
namespace: blog
spec:
replicas: 2
selector:
matchLabels:
app: wordpress
template:
metadata:
name: wp
labels:
app: wordpress
spec:
-----------添加部分--------------------
volumes:
- name: wp-data
persistentVolumeClaim:
claimName: wp-pvc
--------------------------------------
containers:
- name: wp-container
image: wordpress:latest
imagePullPolicy: IfNotPresent
env:
- name: WORDPRESS_DB_HOST
value: "10.1.77.209:3306"
- name: WORDPRESS_DB_NAME
value: wordpress
- name: WORDPRESS_DB_USER
value: wordpress
- name: WORDPRESS_DB_PASSWORD
value: "123"
volumeMounts:
- name: wp-data
mountPath: /var/www/html/
wordpress 完整综合实践
包含模块：
就绪探针
存活探针
nfs/GlusterFS
PV
PVC
mysql
apiVersion: v1
kind: Namespace
metadata:
name: blog
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: mysql-dp
namespace: blog
spec:
replicas: 1
selector:
matchLabels:
app: mysql
template:
metadata:
name: mysql
labels:
app: mysql
spec:
volumes:
- name: mysql-data
hostPath:
path: /data/mysql/data
containers:
- name: mysql-container
image: mysql:5.7
imagePullPolicy: IfNotPresent
env:
- name: MYSQL_ROOT_PASSWORD
value: "123"
- name: MYSQL_DATABASE
value: "wordpress"
- name: MYSQL_USER
value: "wordpress"
- name: MYSQL_PASSWORD
value: "123"
args:
- --character-set-server=utf8
- --collation-server=utf8_general_ci
volumeMounts:
- name: mysql-data
mountPath: /var/lib/mysql
readinessProbe:
tcpSocket:
port: 3306
initialDelaySeconds: 5
timeoutSeconds: 3
periodSeconds: 3
successThreshold: 3
failureThreshold: 3
livenessProbe:
tcpSocket:
port: 3306
initialDelaySeconds: 30
periodSeconds: 10
---
apiVersion: v1
kind: Service
metadata:
name: mysql-svc
namespace: blog
spec:
type: ClusterIP
selector:
app: mysql
ports:
- name: mysql-port
protocol: TCP
port: 3306
targetPort: 3306
wordpress
---
apiVersion: v1
kind: PersistentVolume
metadata:
name: pv01
spec:
capacity:
storage: 5Gi
accessModes:
- ReadWriteOnce
persistentVolumeReclaimPolicy: Recycle
storageClassName: nfs01
nfs:
path: /data/wordpress/
server: 172.16.1.91
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: wp-pvc
namespace: blog
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 4Gi
storageClassName: nfs01
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: wp-dp
namespace: blog
spec:
replicas: 2
selector:
matchLabels:
app: wordpress
template:
metadata:
name: wp
labels:
app: wordpress
spec:
volumes:
- name: wp-data
persistentVolumeClaim:
claimName: wp-pvc
containers:
- name: wp-container
image: wordpress:latest
imagePullPolicy: IfNotPresent
env:
- name: WORDPRESS_DB_HOST
value: "10.1.103.228:3306"
- name: WORDPRESS_DB_NAME
value: wordpress
- name: WORDPRESS_DB_USER
value: wordpress
- name: WORDPRESS_DB_PASSWORD
value: "123"
volumeMounts:
- name: wp-data
mountPath: /var/www/html/
readinessProbe:
httpGet:
path: /
port: 80
initialDelaySeconds: 5
timeoutSeconds: 3
periodSeconds: 3
successThreshold: 3
failureThreshold: 3
livenessProbe:
httpGet:
path: /
port: 80
initialDelaySeconds: 10
periodSeconds: 6
---
apiVersion: v1
kind: Service
metadata:
name: wp-svc
namespace: blog
spec:
type: ClusterIP
selector:
app: wordpress
ports:
- name: wp-port
protocol: TCP
port: 80
targetPort: 80
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: wp-ingress
namespace: blog
spec:
rules:
- host: blog.ljy.com
http:
paths:
- path: /
pathType: ImplementationSpecific
backend:
service:
name: wp-svc
port:
number: 80
mysql 主从复制实践 (失败)
ConfigMap
apiVersion: v1
kind: ConfigMap
metadata:
name: mysql
labels:
app: mysql
data:
master.cnf: |
# 这个配置会被挂在到master节点.
[mysqld]
log-bin
slave.cnf: |
# 这个配置会被挂在到slave节点.
[mysqld]
super-read-only
Service
因为 mysql 是有状态的，所以使用 statefulset 去调度 mysql 的 pod, 同时我们是主从架构的，所以我们需要直接链接到某个 pod 上面去，所以先用 headlessService 为每个 pod 副本创建一个可预知的 DNS 主机名.
同时内部约定一个名为 mysql-read 的 sevice 为每个 pod 做读操作的负载均衡。如果有 mysql 的写操作可以通过 mysql-0.mysql 去访问 master
apiVersion: v1
kind: Service
metadata:
name: mysql
labels:
app: mysql
spec:
ports:
- name: mysql
port: 3306
clusterIP: None
selector:
app: mysql
---
apiVersion: v1
kind: Service
metadata:
name: mysql-read
labels:
app: mysql
spec:
ports:
- name: mysql
port: 3306
selector:
app: mysql
StatefulSet
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: mysql
spec:
selector:
matchLabels:
# 适用于所有label包括app=mysql的pod
app: mysql
serviceName: mysql
replicas: 3
# 定义pod
template:
metadata:
labels:
app: mysql
spec:
# 在init容器中为pod中的mysql容器做初始化工作
initContainers:
# init-mysql容器会分配pod的角色是master还是slave, 然后生成配置文件
- name: init-mysql
image: mysql:5.7
command:
- bash
- "-c"
- |
set -ex
# 生成server-id
[[ `hostname` =~ -([0-9]+)$ ]] || exit 1
ordinal=${BASH_REMATCH[1]}
echo [mysqld] > /mnt/conf.d/server-id.cnf
# 写入server-id
echo server-id=$((100 + $ordinal)) >> /mnt/conf.d/server-id.cnf
# server-id尾号为0作为master, 否则作为slave
# 这里cp到pod中的cnf会与server-id.cnf一块被mysql.cnf include进去
# 这里指定了序号为0的pod会作为master节点提供写, 其他pod作为slave节点提供读
if [[ $ordinal -eq 0 ]]; then
cp /mnt/config-map/master.cnf /mnt/conf.d/
else
cp /mnt/config-map/slave.cnf /mnt/conf.d/
fi
volumeMounts:
# 将conf临时卷挂载到了pod的/mnt/conf.d路径下
- name: conf
mountPath: /mnt/conf.d
# 这里把ConfigMap中的配置哉到了pod的/mnt/config-map路径下
- name: config-map
mountPath: /mnt/config-map
# 这一个init容器会正在pod启动时假定之前已经存在数据, 并将之前的数据复制过来, 以确保新pod中有数据可以提供使用
- name: clone-mysql
# xtrabackup是一个开源工具, 用于克隆mysql的数据
image: ist0ne/xtrabackup:latest
command:
- bash
- "-c"
- |
set -ex
# Skip the clone if data already exists.
[[ -d /var/lib/mysql/mysql ]] && exit 0
# Skip the clone on master (ordinal index 0).
[[ `hostname` =~ -([0-9]+)$ ]] || exit 1
ordinal=${BASH_REMATCH[1]}
[[ $ordinal -eq 0 ]] && exit 0
# Clone data from previous peer.
ncat --recv-only mysql-$(($ordinal-1)).mysql 3307 | xbstream -x -C /var/lib/mysql
# Prepare the backup.
xtrabackup --prepare --target-dir=/var/lib/mysql
volumeMounts:
- name: data
mountPath: /var/lib/mysql
subPath: mysql
- name: conf
mountPath: /etc/mysql/conf.d
containers:
# 实际运行mysqld服务的mysql容器
- name: mysql
image: mysql:5.7
env:
- name: MYSQL_ROOT_PASSWORD
value: "123"
ports:
- name: mysql
containerPort: 3306
volumeMounts:
# 将data卷的mysql目录挂在到容器的/var/lib/mysql
- name: data
mountPath: /var/lib/mysql
subPath: mysql
- name: conf
mountPath: /etc/mysql/conf.d
resources:
requests:
cpu: 500m
memory: 1Gi
# 启动存活探针, 如果失败会重启pod
livenessProbe:
exec:
command: ["mysqladmin", "ping"]
initialDelaySeconds: 30
periodSeconds: 10
timeoutSeconds: 5
# 启动就绪探针确保容器的运行正常, 如果有失败会将pod从service关联的endpoint中剔除
readinessProbe:
exec:
# Check we can execute queries over TCP (skip-networking is off).
command: ["mysql", "-h", "127.0.0.1", "-e", "SELECT 1"]
initialDelaySeconds: 5
periodSeconds: 2
timeoutSeconds: 1
# init结束后还会在启动一个xtrabackup容器作为mysqld容器的sidecar运行
- name: xtrabackup
image: ist0ne/xtrabackup:latest
ports:
- name: xtrabackup
containerPort: 3307
command:
- bash
- "-c"
- |
set -ex
cd /var/lib/mysql
# 他会在启动时查看之前是否有数据克隆文件存在, 如果有那就去其他从节点复制数据, 如果没有就去主节点复制数据
# Determine binlog position of cloned data, if any.
if [[ -f xtrabackup_slave_info && "x$(<xtrabackup_slave_info)" != "x" ]]; then
# XtraBackup already generated a partial "CHANGE MASTER TO" query
# because we're cloning from an existing slave. (Need to remove the tailing semicolon!)
cat xtrabackup_slave_info | sed -E 's/;$//g' > change_master_to.sql.in
# Ignore xtrabackup_binlog_info in this case (it's useless).
rm -f xtrabackup_slave_info xtrabackup_binlog_info
elif [[ -f xtrabackup_binlog_info ]]; then
# We're cloning directly from master. Parse binlog position.
[[ `cat xtrabackup_binlog_info` =~ ^(.*?)[[:space:]]+(.*?)$ ]] || exit 1
rm -f xtrabackup_binlog_info xtrabackup_slave_info
echo "CHANGE MASTER TO MASTER_LOG_FILE='${BASH_REMATCH[1]}',\
MASTER_LOG_POS=${BASH_REMATCH[2]}" > change_master_to.sql.in
fi
# Check if we need to complete a clone by starting replication.
if [[ -f change_master_to.sql.in ]]; then
echo "Waiting for mysqld to be ready (accepting connections)"
until mysql -h 127.0.0.1 -e "SELECT 1"; do sleep 1; done
echo "Initializing replication from clone position"
mysql -h 127.0.0.1 \
-e "$(<change_master_to.sql.in), \
MASTER_HOST='mysql-0.mysql', \
MASTER_USER='root', \
MASTER_PASSWORD='', \
MASTER_CONNECT_RETRY=10; \
START SLAVE;" || exit 1
# In case of container restart, attempt this at-most-once.
mv change_master_to.sql.in change_master_to.sql.orig
fi
# Start a server to send backups when requested by peers.
exec ncat --listen --keep-open --send-only --max-conns=1 3307 -c \
"xtrabackup --backup --slave-info --stream=xbstream --host=127.0.0.1 --user=root"
volumeMounts:
# 将data卷的mysql目录挂在到容器的/var/lib/mysql
- name: data
mountPath: /var/lib/mysql
subPath: mysql
- name: conf
mountPath: /etc/mysql/conf.d
resources:
requests:
cpu: 100m
memory: 100Mi
volumes:
- name: conf
# pod在节点上被移除时, emptyDir会同时被删除
# emptyDir一般被用作缓存目录, 这里用在config
emptyDir: {}
- name: config-map
# ConfigMap对象中存储的数据可以被configMap类型的卷引用, 然后被Pod中运行的容器使用
# 这里引用了前面定义了名称为mysql的ConfigMap对象
configMap:
name: mysql
volumeClaimTemplates:
# 这里面定义的是对PVC的模板, 这里没有单独为mysql创建pvc, 而是动态创建的
- metadata:
name: data
spec:
accessModes: ["ReadWriteOnce"]
# 如果没有配置默认的storageClass的话, 需要指定storageClassName
# storageClassName: your-sc-name
resources:
requests:
storage: 10Gi