Datehoer's Blog It's now or never

麒麟系统离线安装kubernetes，kubesphere, Flowerfine

运维部署
KubernetesKubeSphereFlowerfineKylin

一、安装前准备

1. 关闭防火墙

systemctl disable firewalld
systemctl stop firewalld
systemctl status firewalld

2. 关闭 SELinux

getenforce
setenforce 0
sed -i 's/^SELINUX=.*/SELINUX=disabled/' /etc/selinux/config

3. 关闭 Swap 分区

swapoff -a
echo "vm.swappiness=0" >> /etc/sysctl.conf
sysctl -p /etc/sysctl.conf
sed -i 's/.*swap.*/#&/' /etc/fstab
free -m

4. 时间同步

设置系统时区为上海

timedatectl set-timezone Asia/Shanghai
timedatectl set-local-rtc 0
systemctl restart rsyslog
systemctl restart crond

安装和配置 Chrony

yum install -y chrony

  • 客户端配置

    编辑 /etc/chrony.conf，添加 NTP 服务器：

    server ntp1.aliyun.com iburst

  • 启动并设置开机自启

    systemctl start chronyd
systemctl enable chronyd

  • 验证同步状态

    chronyc sources
chronyc tracking

二、主机名和 Hosts 配置

1. 设置主机名

hostnamectl set-hostname master

2. 配置 Hosts 文件

cat >> /etc/hosts <<EOF
192.168.56.102 master
192.168.56.103 slave1
192.168.56.104 slave2
EOF

三、配置免密登录

1. 生成 SSH 密钥

ssh-keygen -t rsa

2. 分发公钥到其他节点

ssh-copy-id -i ~/.ssh/id_rsa.pub root@slave1
ssh-copy-id -i ~/.ssh/id_rsa.pub root@slave2

四、内核参数设置

cat > /etc/sysctl.d/k8s.conf <<EOF
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
EOF

modprobe br_netfilter
sysctl -p /etc/sysctl.d/k8s.conf

五、安装 IPVS

1. 加载 IPVS 模块

cat > /etc/sysconfig/modules/ipvs.modules <<EOF
#!/bin/bash
modprobe -- ip_vs
modprobe -- ip_vs_rr
modprobe -- ip_vs_wrr
modprobe -- ip_vs_sh
#modprobe -- nf_conntrack_ipv4
modprobe -- nf_conntrack
EOF

chmod 755 /etc/sysconfig/modules/ipvs.modules
bash /etc/sysconfig/modules/ipvs.modules
lsmod | grep -e ip_vs -e nf_conntrack_ipv4

2. 安装 ipvsadm

yum install -y ipvsadm

六、安装依赖组件

yum install -y curl ebtables socat ipset conntrack

七、安装 Docker

1. 二进制安装包下载

下载地址: https://download.docker.com/linux/static/stable/ 选择对应的架构，然后下载对应的版本即可；如果服务器可以联网，也可以直接复制命令下载

cd /home
wget https://download.docker.com/linux/static/stable/x86_64/docker-24.0.6.tgz

2. 安装

  1. 确保/home下有docker-24.0.6.tgz安装包后，执行命令完成解压
tar -xzf docker-24.0.6.tgz
  1. 移动解压后的全部内容到/usr/bin/下
mv docker/* /usr/bin/

3. 编译配置文件

  1. 编辑docker.service文件
vi /usr/lib/systemd/system/docker.service
  1. 复制如下内容
[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
After=network-online.target firewalld.service
Wants=network-online.target
[Service]
Type=notify
ExecStart=/usr/bin/dockerd
ExecReload=/bin/kill -s HUP $MAINPID
LimitNOFILE=infinity
LimitNPROC=infinity
TimeoutStartSec=0
Delegate=yes
KillMode=process
Restart=on-failure
StartLimitBurst=3
StartLimitInterval=60s
[Install]
WantedBy=multi-user.target
  1. 添加docker.service文件的权限
chmod +x /usr/lib/systemd/system/docker.service
systemctl daemon-reload
  1. 创建daemon.json文件
mkdir -p /etc/docker
vim /etc/docker/daemon.json
  1. 复制下面的内容
{
  "insecure-registries": ["dockerhub.kubekey.local"]
}
  1. reload内容、启动docker、设置开机启动
systemctl daemon-reload
systemctl start docker
systemctl enable docker

4. 验证docker安装是否成功

docker -v

输出对应版本，即是成功

5. 迁移存储位置

首先停止docker

systemctl stop docker

然后在对应磁盘目录下创建docker目录

mkdir -p /data/docker

移动文件

mv /var/lib/docker/* /data/docker/

软连接

ln -s /data/docker /var/lib/docker

启动docker

systemctl start docker

6. Docker-compose安装

docker-compose下载地址：https://github.com/docker/compose/releases 选择对应的架构，如果可以联网，也可以直接执行命令下载

cd /home
wget https://github.com/docker/compose/releases/download/v2.25.0/docker-compose-linux-x86_64

安装并分配权限

mv docker-compose-linux-x86_64 /usr/bin/docker-compose
chmod +x /usr/bin/docker-compose

验证

docker-compose -v

输出对应版本，即是成功

八、安装Kubernetes

解压其他服务器打包好的压缩包

搭建本地镜像私服

在一台服务器创建本地镜像仓库 从联网服务器拉取docker.io/registry或从本地导入

从本地导入
docker load -i registry.tar

运行

docker run 
-itd --name=registry  \
-v /registry-data:/var/lib/registry  \
-p 5000:5000 docker.io/registry

上传镜像

  1. 解压镜像包 tar -xzvf images.tar.gz -C /root/images
  2. 导入镜像 for image in /root/images/*.tar; do docker load -i $image done
  3. 重新打标签 docker tag docker.io/calico/kube-controllers:v3.26.1 dockerhub.kubekey.local/calico/kube-controllers:v3.26.1
  4. 上传到私服 docker push dockerhub.kubekey.local/calico/kube-controllers:v3.26.1

镜像列表

docker.io/calico/kube-controllers:v3.26.1 docker.io/calico/cni:v3.26.1 docker.io/calico/pod2daemon-flexvol:v3.26.1 docker.io/calico/node:v3.26.1 docker.io/kubesphere/kube-apiserver:v1.23.10 docker.io/kubesphere/kube-scheduler:v1.23.10 docker.io/kubesphere/kube-proxy:v1.23.10 docker.io/kubesphere/kube-controller-manager:v1.23.10 docker.io/coredns/coredns:1.8.6 docker.io/kubesphere/pause:3.6 docker.io/kubesphere/k8s-dns-node-cache:1.15.12

清理数据

删除现有集群 ./kk delete cluster -y -f config.yaml

删除etcd数据 rm -rf ~/kube/kubekey/pki/etcd/*

手动配置本地依赖

注意：本地安装依赖，不要移动，用cp，kubekey会检测本地是否有对应的文件，如果没有会从网络上下载

安装 containerd

  1. 解压二进制文件增加权限
cd containerd/1.6.4/amd64/
tar -zxvf containerd-1.6.4-linux-amd64.tar.gz
cp bin/* /usr/local/bin/
chmod +x /usr/local/bin/ctr
chmod +x /usr/local/bin/containerd*
  1. 配置并启动 containerd
sudo mkdir -p /etc/containerd
containerd config default | sudo tee /etc/containerd/config.toml

编辑 /etc/systemd/system/containerd.service 文件

[root@master amd64]# sudo vim /etc/systemd/system/containerd.service

添加以下内容：

[Unit]
Description=containerd container runtime
Documentation=https://containerd.io
After=network.target

[Service]
ExecStart=/usr/local/bin/containerd
Restart=always
RestartSec=5
Delegate=yes
KillMode=process
OOMScoreAdjust=-999
LimitNOFILE=1048576
LimitNPROC=infinity
LimitCORE=infinity

[Install]
WantedBy=multi-user.target

启动并启用 containerd 服务

[root@master amd64]# sudo systemctl daemon-reload
[root@master amd64]# sudo systemctl start containerd
[root@master amd64]# sudo systemctl enable containerd

安装 crictl

cd crictl/v1.24.0/amd64/
tar -zxvf crictl-v1.24.0-linux-amd64.tar.gz 
chmod +x crictl
cp crictl /usr/local/bin/

安装 etcd

cd etcd/v3.4.13/amd64/
tar -zxvf etcd-v3.4.13-linux-amd64.tar.gz 
cd etcd-v3.4.13-linux-amd64
cp etcd etcdctl /usr/local/bin/
chmod +x /usr/local/bin/etcd*

安装 Helm

cd helm/v3.9.0/amd64/
chmod +x helm
cp helm /usr/local/bin/

安装 Kubernetes 二进制文件

cd kube/v1.23.10/amd64/
chmod +x *
cp * /usr/local/bin/

安装 runc

cd runc/v1.1.1/amd64/
chmod +x runc.amd64 
cp runc.amd64 /usr/local/bin/runc

安装 CNI

mkdir -p /opt/cni/bin/
tar -zxvf /root/kube/kubekey/cni/v1.2.0/amd64/cni-plugins-linux-amd64-v1.2.0.tgz -C /opt/cni/bin/
chmod +x /opt/cni/bin/*
cp /root/kube/kubekey/cni/v3.26.1/amd64/calicoctl /usr/local/bin/
chmod +x /usr/local/bin/calicoctl

创建 CNI 配置目录并赋予权限

mkdir -p /etc/cni/net.d
chmod 755 /etc/cni/net.d
sudo systemctl daemon-reload
sudo systemctl restart containerd

自动配置

保证kubekey文件夹下有对应的依赖文件，各种压缩包。

安装kubernetes

对config.yaml进行修改，增加本地镜像

  registry:
    privateRegistry: ""
    namespaceOverride: ""
    registryMirrors: []
    insecureRegistries: []
  registry:
    privateRegistry: "dockerhub.kubekey.local"
    namespaceOverride: ""
    registryMirrors: []
    insecureRegistries: []

使用以下命令安装，等待即可

./kk create cluster -f config.yaml -y

如果需要安装kubesphere，则启动的时候要指定存储空间

./kk create cluster -f config.yaml -y --with-local-storage

如果自动安装出现下载，那么证明有些依赖不在本地，从已有的服务器的kubekey文件夹下拷贝过来即可

离线安装kubesphere

首先拉取镜像到本地，然后打包上传 下载镜像列表

curl -L -O https://github.com/kubesphere/ks-installer/releases/download/v3.4.1/images-list.txt

下载脚本

curl -L -O https://github.com/kubesphere/ks-installer/releases/download/v3.4.1/offline-installation-tool.sh

给权限

chmod +x offline-installation-tool.sh

拉取镜像

./offline-installation-tool.sh -s -l images-list.txt -d ./kubesphere-images

推送镜像，最后的参数是镜像仓库地址

./offline-installation-tool.sh -l images-list.txt -d ./kubesphere-images -r dockerhub.kubekey.local

下载安装文件

curl -L -O https://github.com/kubesphere/ks-installer/releases/download/v3.4.1/cluster-configuration.yaml

curl -L -O https://github.com/kubesphere/ks-installer/releases/download/v3.4.1/kubesphere-installer.yaml

修改cluster-configuration.yaml，增加镜像仓库地址

spec:
  persistence:
    storageClass: ""
  authentication:
    jwtSecret: ""
  local_registry: dockerhub.kubekey.local

修改kubesphere-installer.yaml，修改镜像地址

spec:
  serviceAccountName: ks-installer
  containers:
  - name: installer
    image: dockerhub.kubekey.local/kubesphere/ks-installer:v3.4.1
    imagePullPolicy: "Always"

安装

kubectl apply -f kubesphere-installer.yaml
kubectl apply -f cluster-configuration.yaml

安装 Flowerfine

  1. 将 flowerfine 及 flowerfine_images.zip 上传到服务器。
  2. 导入镜像。

安装 Nginx Ingress Controller

cd nginx-ingress
kubectl apply -f ingress-nginx-deploy.yaml

  1. 安装证书管理器：

    cd flink-kubernetes
kubectl create -f cert-manager.yaml
kubectl get pods -n ingress-nginx
kubectl get services -n ingress-nginx

  2. 手动解压并安装：

    cd flink-kubernetes-operator-1.8.0/
tar -xvf flink-kubernetes-operator-1.8.0-helm.tgz
cd flink-kubernetes-operator
cd ../../
helm install flink-kubernetes-operator flink-kubernetes-operator-1.8.0/flink-kubernetes-operator --values values.yaml
## 卸载： helm uninstall flink-kubernetes-operator

## https://github.com/flowerfine/scaleph/blob/dev/tools/kubernetes/flink/values.yaml

kubectl get deployment
kubectl describe deployment flink-kubernetes-operator

安装 Doris Operator

  1. 应用 dorisclusters 配置：

    cd doris
kubectl apply -f doris.selectdb.com_dorisclusters_modified.yaml

  2. 安装 Operator：

    kubectl apply -f operator.yaml
kubectl -n doris get pods

windows拉取镜像脚本

拉取镜像并推送

# Define private registry address
$privateRegistry = "dockerhub.kubekey.local"

# Define image list
$images = @(
    "kubesphere/ks-installer:v3.4.1",
    "kubesphere/ks-apiserver:v3.4.1",
    "kubesphere/ks-console:v3.4.1",
    "kubesphere/ks-controller-manager:v3.4.1",
    "kubesphere/kubectl:v1.20.0",
    "kubesphere/kubefed:v0.8.1",
    "kubesphere/tower:v0.2.1",
    "minio/minio:RELEASE.2019-08-07T01-59-21Z",
    "minio/mc:RELEASE.2019-08-07T23-14-43Z",
    "csiplugin/snapshot-controller:v4.0.0",
    "kubesphere/nginx-ingress-controller:v1.3.1",
    "mirrorgooglecontainers/defaultbackend-amd64:1.4",
    "kubesphere/metrics-server:v0.4.2",
    "redis:5.0.14-alpine",
    "haproxy:2.0.25-alpine",
    "alpine:3.14",
    "osixia/openldap:1.3.0",
    "kubesphere/netshoot:v1.0",
    "kubeedge/cloudcore:v1.13.0",
    "kubesphere/iptables-manager:v1.13.0",
    "kubesphere/edgeservice:v0.3.0",
    "openpolicyagent/gatekeeper:v3.5.2",
    "kubesphere/openpitrix-jobs:v3.3.2",
    "kubesphere/devops-apiserver:ks-v3.4.1",
    "kubesphere/devops-controller:ks-v3.4.1",
    "kubesphere/devops-tools:ks-v3.4.1",
    "kubesphere/ks-jenkins:v3.4.0-2.319.3-1",
    "jenkins/inbound-agent:4.10-2",
    "kubesphere/builder-base:v3.2.2",
    "kubesphere/builder-nodejs:v3.2.0",
    "kubesphere/builder-maven:v3.2.0",
    "kubesphere/builder-maven:v3.2.1-jdk11",
    "kubesphere/builder-python:v3.2.0",
    "kubesphere/builder-go:v3.2.0",
    "kubesphere/builder-go:v3.2.2-1.16",
    "kubesphere/builder-go:v3.2.2-1.17",
    "kubesphere/builder-go:v3.2.2-1.18",
    "kubesphere/builder-base:v3.2.2-podman",
    "kubesphere/builder-nodejs:v3.2.0-podman",
    "kubesphere/builder-maven:v3.2.0-podman",
    "kubesphere/builder-maven:v3.2.1-jdk11-podman",
    "kubesphere/builder-python:v3.2.0-podman",
    "kubesphere/builder-go:v3.2.0-podman",
    "kubesphere/builder-go:v3.2.2-1.16-podman",
    "kubesphere/builder-go:v3.2.2-1.17-podman",
    "kubesphere/builder-go:v3.2.2-1.18-podman",
    "kubesphere/s2ioperator:v3.2.1",
    "kubesphere/s2irun:v3.2.0",
    "kubesphere/s2i-binary:v3.2.0",
    "kubesphere/tomcat85-java11-centos7:v3.2.0",
    "kubesphere/tomcat85-java11-runtime:v3.2.0",
    "kubesphere/tomcat85-java8-centos7:v3.2.0",
    "kubesphere/tomcat85-java8-runtime:v3.2.0",
    "kubesphere/java-11-centos7:v3.2.0",
    "kubesphere/java-8-centos7:v3.2.0",
    "kubesphere/java-8-runtime:v3.2.0",
    "kubesphere/java-11-runtime:v3.2.0",
    "kubesphere/nodejs-8-centos7:v3.2.0",
    "kubesphere/nodejs-6-centos7:v3.2.0",
    "kubesphere/nodejs-4-centos7:v3.2.0",
    "kubesphere/python-36-centos7:v3.2.0",
    "kubesphere/python-35-centos7:v3.2.0",
    "kubesphere/python-34-centos7:v3.2.0",
    "kubesphere/python-27-centos7:v3.2.0",
    "quay.io/argoproj/argocd:v2.3.3",
    "quay.io/argoproj/argocd-applicationset:v0.4.1",
    "ghcr.io/dexidp/dex:v2.30.2",
    "redis:6.2.6-alpine",
    "jimmidyson/configmap-reload:v0.7.1",
    "prom/prometheus:v2.39.1",
    "kubesphere/prometheus-config-reloader:v0.55.1",
    "kubesphere/prometheus-operator:v0.55.1",
    "kubesphere/kube-rbac-proxy:v0.11.0",
    "kubesphere/kube-state-metrics:v2.6.0",
    "prom/node-exporter:v1.3.1",
    "prom/alertmanager:v0.23.0",
    "thanosio/thanos:v0.31.0",
    "grafana/grafana:8.3.3",
    "kubesphere/kube-rbac-proxy:v0.11.0",
    "kubesphere/notification-manager-operator:v2.3.0",
    "kubesphere/notification-manager:v2.3.0",
    "kubesphere/notification-tenant-sidecar:v3.2.0",
    "kubesphere/elasticsearch-curator:v5.7.6",
    "kubesphere/opensearch-curator:v0.0.5",
    "kubesphere/elasticsearch-oss:6.8.22",
    "opensearchproject/opensearch:2.6.0",
    "opensearchproject/opensearch-dashboards:2.6.0",
    "kubesphere/fluentbit-operator:v0.14.0",
    "docker:19.03",
    "kubesphere/fluent-bit:v1.9.4",
    "kubesphere/log-sidecar-injector:v1.2.0",
    "elastic/filebeat:6.7.0",
    "kubesphere/kube-events-operator:v0.6.0",
    "kubesphere/kube-events-exporter:v0.6.0",
    "kubesphere/kube-events-ruler:v0.6.0",
    "kubesphere/kube-auditing-operator:v0.2.0",
    "kubesphere/kube-auditing-webhook:v0.2.0",
    "istio/pilot:1.14.6",
    "istio/proxyv2:1.14.6",
    "jaegertracing/jaeger-operator:1.29",
    "jaegertracing/jaeger-agent:1.29",
    "jaegertracing/jaeger-collector:1.29",
    "jaegertracing/jaeger-query:1.29",
    "jaegertracing/jaeger-es-index-cleaner:1.29",
    "kubesphere/kiali-operator:v1.50.1",
    "kubesphere/kiali:v1.50",
    "busybox:1.31.1",
    "nginx:1.14-alpine",
    "joosthofman/wget:1.0",
    "nginxdemos/hello:plain-text",
    "wordpress:4.8-apache",
    "mirrorgooglecontainers/hpa-example:latest",
    "fluent/fluentd:v1.4.2-2.0",
    "perl:latest",
    "kubesphere/examples-bookinfo-productpage-v1:1.16.2",
    "kubesphere/examples-bookinfo-reviews-v1:1.16.2",
    "kubesphere/examples-bookinfo-reviews-v2:1.16.2",
    "kubesphere/examples-bookinfo-details-v1:1.16.2",
    "kubesphere/examples-bookinfo-ratings-v1:1.16.3",
    "weaveworks/scope:1.13.0"
)

foreach ($image in $images) {
    try {
        Write-Host "Pulling image: ${image}" -ForegroundColor Cyan
        docker pull $image

        # Build new tag
        $newImage = "$privateRegistry/$image"
        
        Write-Host "Tagging image: ${image} as $newImage" -ForegroundColor Cyan
        docker tag $image $newImage

        Write-Host "Removing old image: ${image}" -ForegroundColor Yellow
        docker rmi $image
		
		    #$fileName = ($newImage.Split("/")[-1]).Replace(":", "_")
        #$tarFile = "$fileName.tar"
		    #Write-Host "save ${newImage} to ${tarFile}"  -ForegroundColor Green
		    #docker save -o $tarFile $newImage
        Write-Host "Pushing image to private registry: ${newImage}" -ForegroundColor Cyan
        docker push $newImage

        Write-Host "Removing tagged image: ${newImage}" -ForegroundColor Yellow
        docker rmi $newImage

        Write-Host "Completed: ${image}`n" -ForegroundColor Green
    }
    catch {
	Write-Host "Error processing image ${image}: $_" -ForegroundColor Red
    }
}

Write-Host "All images have been processed." -ForegroundColor Green