测试k8s基于containerd做容器从harbor拉取镜像

发表于 分类于 ，
CKS学习笔记Containerd拉取镜像部分：

测试k8s基于containerd做容器从harbor拉取镜像

软件版本
docker最新版
containerd1.6.6
kubernetes1.25.0
calico3.25
harbor2.3.0
docker-compose1.26.2
节点IP系统功能CPU内存硬盘
node110.80.10.1centos7.9k8s-master4核心8GB20GB
node210.80.10.2centos7.9k8s-node4核心8GB20GB
node310.80.10.3centos7.9k8s-node4核心8GB20GB
node410.80.10.4centos7.9harbor4核心8GB20GB

node4

修改主机名：

1
# hostnamectl set-hostname harbor && bash

生成ca证书：

1
2
3
4
5
6
7
8
9
10
# mkdir -p /data/ssl && cd /data/ssl
# openssl genrsa -out ca.key 3072
# openssl req -new -x509 -days 3650 -key ca.key -out ca.pem
CH
BJ
BJ
回车
回车
回车
回车

生成域名的证书：

1
# openssl genrsa -out harbor.key 3072

生成一个3072位的key，也就是私钥：

1
2
3
4
5
6
7
8
9
10
# openssl req -new -key harbor.key -out harbor.csr
CH
BJ
BJ
回车
回车
harbor
回车
回车
回车

签发证书：

1
# openssl x509 -req -in harbor.csr -CA ca.pem -CAkey ca.key -CAcreateserial -out harbor.pem -days 3650

下载安装docker：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
# yum install -y yum-utils device-mapper-persistent-data lvm2
# yum-config-manager --add-repo https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
# yum makecache fast
# yum install -y docker-ce
# systemctl start docker
# systemctl enable docker
# systemctl status docker
# modprobe br_netfilter
# cat > /etc/sysctl.d/docker.conf << EOF
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
EOF
# sysctl -p /etc/sysctl.d/docker.conf
# vim /etc/docker/daemon.json
{
  "registry-mirrors": ["https://pmn1o05g.mirror.aliyuncs.com","https://registry.docker-cn.com","https://docker.mirrors.ustc.edu.cn","https://dockerhub.azk8s.cn","http://hub-mirror.c.163.com"]
}
1
2
# systemctl daemon-reload
# systemctl restart docker

下载安装docker-compose：

下载地址：https://github.com/docker/compose/releases/tag/1.26.2

1
2
3
4
5
6
# cd /usr/local/src/
# wget https://github.com/docker/compose/releases/download/1.26.2/docker-compose-Linux-x86_64
# mv docker-compose-Linux-x86_64 /usr/bin/docker-compose
# chmod +x /usr/bin/docker-compose
# docker-compose --version
docker-compose version 1.26.2, build eefe0d31

下载安装harbor：

下载地址：https://github.com/goharbor/harbor/releases/tag/v2.3.0-rc3

1
2
3
4
5
6
7
8
9
10
11
# cd /usr/local/src/
# wget https://github.com/goharbor/harbor/releases/download/v2.3.0-rc3/harbor-offline-installer-v2.3.0-rc3.tgz
# tar -xzvf harbor-offline-installer-v2.3.0-rc3.tgz
# cd harbor
# cp harbor.yml.tmpl harbor.yml
# vim harbor.yml
# 5行，修改配置
hostname: harbor
# 17~18行，修改配置
  certificate: /data/ssl/harbor.pem
  private_key: /data/ssl/harbor.key

启动harbor：

1
# /usr/local/src/harbor/install.sh

浏览器访问：https://10.80.10.4

1
2
用户名：admin
密码：Harbor12345

进入主页：

node1

查看容器运行时为containerd：

1
2
3
4
5
# kubectl get nodes -o wide
NAME          STATUS   ROLES           AGE   VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE                KERNEL-VERSION           CONTAINER-RUNTIME
k8s-master1   Ready    control-plane   18h   v1.25.0   10.80.10.1    <none>        CentOS Linux 7 (Core)   3.10.0-1160.el7.x86_64   containerd://1.6.6
k8s-node1     Ready    work            18h   v1.25.0   10.80.10.2    <none>        CentOS Linux 7 (Core)   3.10.0-1160.el7.x86_64   containerd://1.6.6
k8s-node2     Ready    work            18h   v1.25.0   10.80.10.3    <none>        CentOS Linux 7 (Core)   3.10.0-1160.el7.x86_64   containerd://1.6.6

node1、node2、node3

升级containerd：

1
2
# yum remove -y containerd.io
# yum install -y containerd.io-1.6.22

node1

修改配置文件：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
# containerd config default > /etc/containerd/config.toml
# vim /etc/containerd/config.toml
# 125行，修改配置
            SystemdCgroup = true
# 61行，修改配置
    sandbox_image = "registry.aliyuncs.com/google_containers/pause:3.7"
# 150行，添加配置
        [plugins."io.containerd.grpc.v1.cri".registry.configs."10.80.10.4".tls]
          insecure_skip_verify = true
        [plugins."io.containerd.grpc.v1.cri".registry.configs."10.80.10.4".auth]
          username = "admin"
          password = "Harbor12345"
# 158行，添加配置
      [plugins."io.containerd.grpc.v1.cri".registry.mirrors]
        [plugins."io.containerd.grpc.v1.cri".registry.mirrors."10.80.10.4"]
          endpoint = ["https://10.80.10.4:443"]
        [plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker.io"]
          endpoint = ["https://pmn1o05g.mirror.aliyuncs.com","https://registry.docker-cn.com"]

拷贝配置文件：

1
2
# scp /etc/containerd/config.toml k8s-node1:/etc/containerd/config.toml
# scp /etc/containerd/config.toml k8s-node2:/etc/containerd/config.toml

node1、node2、node3

重启containerd：

1
2
# systemctl restart containerd
# systemctl status containerd

重新安装docker：

1
2
3
4
# yum install -y docker-ce
# systemctl start docker
# systemctl enable docker
# systemctl status docker

node2

拉取镜像：

1
# docker pull nginx:latest

镜像打tag：

1
# docker tag docker.io/library/nginx:latest 10.80.10.4/library/nginx:v1.6

添加harbor配置：

1
2
3
4
5
6
# vim /etc/docker/daemon.json
{
  "registry-mirrors": ["https://pmn1o05g.mirror.aliyuncs.com","https://registry.docker-cn.com","https://docker.mirrors.ustc.edu.cn","https://dockerhub.azk8s.cn","http://hub-mirror.c.163.com"],
  "exec-opts": ["native.cgroupdriver=systemd"],
  "insecure-registries": ["10.80.10.4","harbor"]
}
1
# systemctl restart docker

修改hosts文件：

1
2
3
# vim /etc/hosts
# 尾行，添加配置
10.80.10.4 harbor

登录harbor：

1
2
3
# docker login 10.80.10.4
admin
Harbor12345

上传镜像：

1
# docker push 10.80.10.4/library/nginx:v1.6

harbor上有nginx镜像：

node1

创建pod：

1
2
3
4
5
6
7
8
9
10
11
# vim pod.yaml
apiVersion: v1
kind: Pod
metadata:
  name: nginx
  namespace: default
spec:
  containers:
  - name: nginx
    image: 10.80.10.4/library/nginx:v1.6
    imagePullPolicy: Always
1
2
3
4
5
# kubectl apply -f pod.yaml 
pod/nginx created
# kubectl get pods
NAME    READY   STATUS    RESTARTS   AGE
nginx   1/1     Running   0          10s

查看镜像详细信息：

1
# kubectl describe pods nginx