Home | 简体中文 | 繁体中文 | 杂文 | Github | 知乎专栏 | 51CTO学院 | CSDN程序员研修院 | OSChina 博客 | 腾讯云社区 | 阿里云栖社区 | Facebook | Linkedin | Youtube | 打赏(Donations) | About
知乎专栏多维度架构

第 2 章 Kubernetes

目录

2.1. Minikube
2.1.1. CentOS 8 安装 minikube
2.1.2. Quickstart
2.1.3. minikube 命令
2.1.4. Minikube 案例演示
2.1.5. FAQ
2.2. microk8s
2.2.1. 安装 microk8s
2.2.2. 组件管理
2.2.3. kubectl
2.2.4. Kubernetes Addons
2.3. kubectl - controls the Kubernetes cluster manager.
2.3.1. 配置
2.3.2. 如何从 docker 过渡到 kubectl 命令
2.3.3. 查看信息
2.3.4. namespace 命名空间
2.3.5. label 标签
2.3.6. Pod 管理
2.3.7. expose
2.3.8. 服务管理
2.3.9. 部署管理
2.3.10. 资源管理
2.3.11. 查看 pod 日志
2.3.12. endpoints
2.3.13. 执行 Shell
2.3.14. edit
2.3.15. rollout
2.3.16. port-forward 端口映射
2.3.17. secret 密钥管理
2.3.18. ConfigMap
2.3.19. Job/CronJob
2.3.20. explain
2.3.21. 操作系统资源配置
2.3.22. 端口转发
2.3.23. 更新镜像
2.3.24. 复制文件
2.3.25. describe
2.3.26. clusterrolebinding
2.3.27. kubectl example
2.4. Kubernetes YAML
2.4.1. ServiceAccount
2.4.2. 创建命名空间
2.4.3. Pod
2.4.4. Service
2.4.5. ConfigMap
2.4.6. Volume
2.4.7. Job
2.4.8. Ingress
2.5. istio
2.5.1. 启动 istio
2.5.2. 禁用 istio
2.6. Kubeapps

2.1. Minikube

2.1.1. CentOS 8 安装 minikube

2.1.1.1. CentOS

执行下面命令检查服务器是否开启虚拟化技术

			
egrep --color 'vmx|svm' /proc/cpuinfo
			
			

如果没有任何输出,请重启服务器进入 BIOS 启用 VT-X 或 AMD-v

			
curl -LO https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64 \
  && install minikube-linux-amd64 /usr/local/bin/minikube		 	
			
			

尝试运行 minikube 如果输出帮助信息表示安装成功

			
[root@localhost ~]# minikube version
minikube version: v1.13.0
commit: 0c5e9de4ca6f9c55147ae7f90af97eff5befef5f-dirty			
			
			

			
echo "1" > /proc/sys/net/bridge/bridge-nf-call-iptables			
			
			

dnf 安装 kubectl

			
cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
EOF
			
			
			
[root@localhost ~]# dnf install kubectl
			
			

二进制安装 kubectl

						
curl -LO "https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/linux/amd64/kubectl" \
	&& install kubectl /usr/local/bin/kubectl
			
			
无虚拟机

如果你不想安装虚拟机

			
adduser docker
su - docker
sudo usermod -aG docker $USER && newgrp docker			
			
				

				
[docker@localhost ~]$ minikube start --driver=docker
* minikube v1.13.0 on Centos 8.2.2004
* Using the docker driver based on user configuration

X Requested memory allocation (1694MB) is less than the recommended minimum 2000MB. Deployments may fail.


X The requested memory allocation of 1694MiB does not leave room for system overhead (total system memory: 1694MiB). You may face stability issues.
* Suggestion: Start minikube with less memory allocated: 'minikube start --memory=1694mb'

* Starting control plane node minikube in cluster minikube
* Pulling base image ...
* Downloading Kubernetes v1.19.0 preload ...
    > preloaded-images-k8s-v6-v1.19.0-docker-overlay2-amd64.tar.lz4: 486.28 MiB				
				
				

2.1.1.2. Mac OS

检查硬件是否支持虚拟化

			
iMac:Linux neo$ sysctl -a | grep -E --color 'machdep.cpu.features|VMX'
machdep.cpu.features: FPU VME DE PSE TSC MSR PAE MCE CX8 APIC SEP MTRR PGE MCA CMOV PAT PSE36 CLFSH DS ACPI MMX FXSR SSE SSE2 SS HTT TM PBE SSE3 PCLMULQDQ DTES64 MON DSCPL VMX SMX EST TM2 SSSE3 CX16 TPR PDCM SSE4.1 SSE4.2 x2APIC POPCNT AES PCID XSAVE OSXSAVE TSCTMR AVX1.0			
			
			
			
$ brew install hyperkit			
$ brew install minikube
$ brew install kubectl
$ brew install kubernetes-helm
			
			
			
neo@MacBook-Pro-Neo ~ % minikube start                 
😄  minikube v1.13.1 on Darwin 11.0
🆕  Kubernetes 1.19.2 is now available. If you would like to upgrade, specify: --kubernetes-version=v1.19.2
✨  Using the hyperkit driver based on existing profile
👍  Starting control plane node minikube in cluster minikube
🔄  Restarting existing hyperkit VM for "minikube" ...
❗  This VM is having trouble accessing https://k8s.gcr.io
💡  To pull new external images, you may need to configure a proxy: https://minikube.sigs.k8s.io/docs/reference/networking/proxy/
🐳  Preparing Kubernetes v1.19.0 on Docker 19.03.12 ...
🔎  Verifying Kubernetes components...
🌟  Enabled addons: dashboard, default-storageclass, storage-provisioner
🏄  Done! kubectl is now configured to use "minikube" by default			
			
			

有些老系统可能不支持 hyperkit,需要virtualbox。

			
$ brew cask install virtualbox
$ minikube start —vm-driver=virtualbox
$ minikube dashboard			
			
			

检查 minikube 启动状态

			
Neo-iMac:~ neo$ docker container inspect minikube --format={{.State.Status}}
running			
			
			

2.1.2. Quickstart

启动

		
minikube start
		
		

运行一个 echoserver 镜像

		
kubectl run hello-minikube --image=k8s.gcr.io/echoserver:1.4 --port=8080
kubectl expose deployment hello-minikube --type=NodePort
minikube service hello-minikube
		
		

查询 echoserver 访问地址

		
minikube service hello-minikube --url		
		
		

在浏览器中访问查询到的网址

停止并删除镜像

		
minikube stop
minikube delete		
		
		

例 2.1. minikube 操作演示

快速开始使用 minikube 运行一个镜像

			
[root@localhost ~]# kubectl run hello-minikube --image=k8s.gcr.io/echoserver:1.4 --port=8080
kubectl run --generator=deployment/apps.v1 is DEPRECATED and will be removed in a future version. Use kubectl run --generator=run-pod/v1 or kubectl create instead.
deployment.apps/hello-minikube created

[root@localhost ~]# kubectl expose deployment hello-minikube --type=NodePort
service/hello-minikube exposed

[root@localhost ~]# minikube service hello-minikube
Opening kubernetes service default/hello-minikube in default browser...		

[root@localhost ~]# kubectl get pod
NAME                              READY   STATUS    RESTARTS   AGE
hello-minikube-5c856cbf98-6vfvp   1/1     Running   0          6m59s

[root@localhost ~]# minikube service hello-minikube --url
http://172.16.0.121:30436

[root@localhost ~]# curl http://172.16.0.121:30436
CLIENT VALUES:
client_address=172.17.0.1
command=GET
real path=/
query=nil
request_version=1.1
request_uri=http://172.16.0.121:8080/

SERVER VALUES:
server_version=nginx: 1.10.0 - lua: 10001

HEADERS RECEIVED:
accept=*/*
host=172.16.0.121:30436
user-agent=curl/7.29.0
BODY:
-no body in request-
			
			

2.1.3. minikube 命令

			
[root@localhost ~]# minikube
Minikube is a CLI tool that provisions and manages single-node Kubernetes clusters optimized for development workflows.

Usage:
  minikube [command]

Available Commands:
  addons         Modify minikube's kubernetes addons
  cache          Add or delete an image from the local cache.
  completion     Outputs minikube shell completion for the given shell (bash or zsh)
  config         Modify minikube config
  dashboard      Access the kubernetes dashboard running within the minikube cluster
  delete         Deletes a local kubernetes cluster
  docker-env     Sets up docker env variables; similar to '$(docker-machine env)'
  help           Help about any command
  ip             Retrieves the IP address of the running cluster
  logs           Gets the logs of the running instance, used for debugging minikube, not user code
  mount          Mounts the specified directory into minikube
  profile        Profile sets the current minikube profile
  service        Gets the kubernetes URL(s) for the specified service in your local cluster
  ssh            Log into or run a command on a machine with SSH; similar to 'docker-machine ssh'
  ssh-key        Retrieve the ssh identity key path of the specified cluster
  start          Starts a local kubernetes cluster
  status         Gets the status of a local kubernetes cluster
  stop           Stops a running local kubernetes cluster
  tunnel         tunnel makes services of type LoadBalancer accessible on localhost
  update-check   Print current and latest version number
  update-context Verify the IP address of the running cluster in kubeconfig.
  version        Print the version of minikube

Flags:
      --alsologtostderr                  log to standard error as well as files
  -b, --bootstrapper string              The name of the cluster bootstrapper that will set up the kubernetes cluster. (default "kubeadm")
  -h, --help                             help for minikube
      --log_backtrace_at traceLocation   when logging hits line file:N, emit a stack trace (default :0)
      --log_dir string                   If non-empty, write log files in this directory
      --logtostderr                      log to standard error instead of files
  -p, --profile string                   The name of the minikube VM being used.  
                                         	This can be modified to allow for multiple minikube instances to be run independently (default "minikube")
      --stderrthreshold severity         logs at or above this threshold go to stderr (default 2)
  -v, --v Level                          log level for V logs
      --vmodule moduleSpec               comma-separated list of pattern=N settings for file-filtered logging

Use "minikube [command] --help" for more information about a command.			
			
		

2.1.3.1. minikube ip 地址

			
[docker@localhost ~]$ minikube ip
192.168.58.2
			
			

			
kubectl get nodes -o jsonpath='{.items[*].status.addresses[].address}'			
			
			

2.1.3.2. 启动 minikube

虚拟机驱动

--vm-driver=none

				
minikube start --vm-driver=none
				
				
开启GPU
				
minikube start --vm-driver kvm2 --gpu				
				
				
日志输出级别

指定日志输出级别

				
minikube start --v=7			
				
				
CPU 和 内存分配
				
minikube start --memory 8000 --cpus 2			
				
				
指定 kubernetes 版本
				
minikube start --memory 8000 --cpus 2 --kubernetes-version v1.6.0					
				
				
配置启动项
				
minikube start --extra-config=apiserver.v=10 --extra-config=kubelet.max-pods=100				
				
				
指定 registry-mirror 镜像
				
minikube start --registry-mirror=https://registry.docker-cn.com

minikube start --image-mirror-country=cn --registry-mirror="https://docker.mirrors.ustc.edu.cn" --insecure-registry="127.0.0.1:5000"				

minikube start --image-mirror-country=cn --registry-mirror="https://docker.mirrors.ustc.edu.cn" --insecure-registry="192.168.0.0/24"
				
				
指定下载镜像
				
minikube start --image-mirror-country=cn --image-repository=registry.cn-hangzhou.aliyuncs.com/google_containers				
				
				
				
# 从阿里云下载 virtualbox 镜像
minikube start --vm-driver='virtualbox' --image-mirror-country cn \
    --iso-url=https://kubernetes.oss-cn-hangzhou.aliyuncs.com/minikube/iso/minikube-v1.9.0.iso \
    --registry-mirror=https://docker.mirrors.ustc.edu.cn

minikube start --vm-driver=virtualbox \
--image-mirror-country cn \
--registry-mirror=https://docker.mirrors.ustc.edu.cn \
--image-repository=registry.aliyuncs.com/google_containers \
--insecure-registry=192.168.0.10:5000 //访问宿主机的私有docker仓库				
				
				
Enabling Unsafe Sysctls
				
minikube start --extra-config="kubelet.allowed-unsafe-sysctls=kernel.msg*,net.core.somaxconn".
				
				
使用 CRI-O 容易
				
minikube start --container-runtime=cri-o --vm-driver=none				
				
				

启动演示

				
iMac:~ neo$ minikube start --container-runtime=cri-o
😄  Darwin 10.13.6 上的 minikube v1.15.0
🆕  Kubernetes 1.19.4 is now available. If you would like to upgrade, specify: --kubernetes-version=v1.19.4
✨  根据现有的配置文件使用 hyperkit 驱动程序
👍  Starting control plane node minikube in cluster minikube
🔄  Restarting existing hyperkit VM for "minikube" ...
🎁  正在 CRI-O 1.17.3 中准备 Kubernetes v1.19.2…
🔗  Configuring bridge CNI (Container Networking Interface) ...
🔎  Verifying Kubernetes components...
🌟  Enabled addons: storage-provisioner, dashboard, default-storageclass
🏄  Done! kubectl is now configured to use "minikube" cluster and "" namespace by default
				
				

2.1.3.3. 停止 minikube

			
[root@localhost ~]# minikube stop
Stopping local Kubernetes cluster...
Machine stopped.			
			
			

2.1.3.4. Docker 环境变量

			
neo@MacBook-Pro-Neo ~ % minikube docker-env
export DOCKER_TLS_VERIFY="1"
export DOCKER_HOST="tcp://192.168.64.3:2376"
export DOCKER_CERT_PATH="/Users/neo/.minikube/certs"
export MINIKUBE_ACTIVE_DOCKERD="minikube"

# To point your shell to minikube's docker-daemon, run:
# eval $(minikube -p minikube docker-env)			
			
			

设置环境变量

			
# eval $(minikube docker-env)	
# eval $(minikube -p minikube docker-env)	
			
			

2.1.3.5. SSH

			
neo@MacBook-Pro-Neo ~ % minikube ssh                                                           
                         _             _            
            _         _ ( )           ( )           
  ___ ___  (_)  ___  (_)| |/')  _   _ | |_      __  
/' _ ` _ `\| |/' _ `\| || , <  ( ) ( )| '_`\  /'__`\
| ( ) ( ) || || ( ) || || |\`\ | (_) || |_) )(  ___/
(_) (_) (_)(_)(_) (_)(_)(_) (_)`\___/'(_,__/'`\____)

$ 			
			
			

2.1.3.6. 缓存镜像

			
# cache a image into $HOME/.minikube/cache/images

$ minikube cache add ubuntu:16.04
$ minikube cache add redis:3

# list cached images
$ minikube cache list
redis:3
ubuntu:16.04

# delete cached images
$ minikube cache delete ubuntu:16.04
$ minikube cache delete $(minikube cache list)
			
			

2.1.3.7. 清理 minikube

			
minikube delete
rm ~/.minikube 
minikube start
			
			

2.1.3.8. Kubernetes 控制面板

Dashboard是基于Web的Kubernetes管理界面。使用下面的命令启动:

			
minikube dashboard
			
			

查询控制面板访问地址

			
$ minikube dashboard --url
http://192.168.3.14:30000			
			
			

2.1.3.9. service

列出所有服务

			
Neo-iMac:~ neo$ minikube service list
|----------------------|------------------------------------|--------------|-----|
|      NAMESPACE       |                NAME                | TARGET PORT  | URL |
|----------------------|------------------------------------|--------------|-----|
| default              | kubernetes                         | No node port |
| default              | nginx                              |           80 |     |
| ingress-nginx        | ingress-nginx-controller           | http/80      |     |
|                      |                                    | https/443    |     |
| ingress-nginx        | ingress-nginx-controller-admission | No node port |
| kube-system          | kube-dns                           | No node port |
| kubernetes-dashboard | dashboard-metrics-scraper          | No node port |
| kubernetes-dashboard | kubernetes-dashboard               | No node port |
|----------------------|------------------------------------|--------------|-----|			
			
			

查看指定服务

			
Neo-iMac:~ neo$ minikube service nginx
|-----------|-------|-------------|---------------------------|
| NAMESPACE | NAME  | TARGET PORT |            URL            |
|-----------|-------|-------------|---------------------------|
| default   | nginx |          80 | http://192.168.49.2:30330 |
|-----------|-------|-------------|---------------------------|
🏃  Starting tunnel for service nginx.
|-----------|-------|-------------|------------------------|
| NAMESPACE | NAME  | TARGET PORT |          URL           |
|-----------|-------|-------------|------------------------|
| default   | nginx |             | http://127.0.0.1:55815 |
|-----------|-------|-------------|------------------------|
🎉  Opening service default/nginx in default browser...
❗  Because you are using a Docker driver on darwin, the terminal needs to be open to run it.			
			
			

查看服务的网址

			
[root@localhost ~]# minikube service hello-minikube --url
http://172.16.0.121:30436			
			
			

2.1.3.10. 查看日志

			
minikube logs -v10			
			
			

2.1.3.11. 查看 Docker 环境变量

minikube docker-env

			
Neo-iMac:~ neo$ minikube docker-env
export DOCKER_TLS_VERIFY="1"
export DOCKER_HOST="tcp://127.0.0.1:54734"
export DOCKER_CERT_PATH="/Users/neo/.minikube/certs"
export MINIKUBE_ACTIVE_DOCKERD="minikube"

# To point your shell to minikube's docker-daemon, run:
# eval $(minikube -p minikube docker-env)
			
			

2.1.3.12. profile

			
minikube profile demo
minikube start -p demo --memory=8192 --cpus=6 --disk-size=50g			
			
			

2.1.3.13. addons

查看所有插件
				
iMac:registry neo$ minikube addons list
|-----------------------------|----------|--------------|
|         ADDON NAME          | PROFILE  |    STATUS    |
|-----------------------------|----------|--------------|
| ambassador                  | minikube | disabled     |
| dashboard                   | minikube | enabled ✅   |
| default-storageclass        | minikube | enabled ✅   |
| efk                         | minikube | disabled     |
| freshpod                    | minikube | disabled     |
| gcp-auth                    | minikube | disabled     |
| gvisor                      | minikube | disabled     |
| helm-tiller                 | minikube | disabled     |
| ingress                     | minikube | disabled     |
| ingress-dns                 | minikube | disabled     |
| istio                       | minikube | disabled     |
| istio-provisioner           | minikube | disabled     |
| kubevirt                    | minikube | disabled     |
| logviewer                   | minikube | disabled     |
| metallb                     | minikube | disabled     |
| metrics-server              | minikube | disabled     |
| nvidia-driver-installer     | minikube | disabled     |
| nvidia-gpu-device-plugin    | minikube | disabled     |
| olm                         | minikube | disabled     |
| pod-security-policy         | minikube | disabled     |
| registry                    | minikube | disabled     |
| registry-aliases            | minikube | disabled     |
| registry-creds              | minikube | disabled     |
| storage-provisioner         | minikube | enabled ✅   |
| storage-provisioner-gluster | minikube | disabled     |
|-----------------------------|----------|--------------|				
				
				
启用 addons
				
minikube addons enable heapster
minikube addons enable ingress		
				
				

启用 WebUI

				
[root@localhost ~]# minikube addons enable dashboard
dashboard was successfully enabled
[root@localhost ~]# minikube addons list | grep dashboard
- dashboard: enabled				
				
				
查看 addons 列表
				
[root@localhost ~]# minikube addons list
- addon-manager: enabled
- dashboard: enabled
- default-storageclass: enabled
- efk: disabled
- freshpod: disabled
- gvisor: disabled
- heapster: disabled
- ingress: disabled
- kube-dns: disabled
- metrics-server: disabled
- nvidia-driver-installer: disabled
- nvidia-gpu-device-plugin: disabled
- registry: disabled
- registry-creds: disabled
- storage-provisioner: enabled
- storage-provisioner-gluster: disabled			
				
				
dashboard
				
Neo-iMac:~ neo$ minikube addons enable dashboard
    ▪ Using image registry.cn-hangzhou.aliyuncs.com/google_containers/metrics-scraper:v1.0.7
    ▪ Using image registry.cn-hangzhou.aliyuncs.com/google_containers/dashboard:v2.3.1
💡  Some dashboard features require the metrics-server addon. To enable all features please run:

	minikube addons enable metrics-server	


🌟  The 'dashboard' addon is enabled				
				
				
				
Neo-iMac:~ neo$ minikube dashboard
🤔  Verifying dashboard health ...
🚀  Launching proxy ...
🤔  Verifying proxy health ...
🎉  Opening http://127.0.0.1:62433/api/v1/namespaces/kubernetes-dashboard/services/http:kubernetes-dashboard:/proxy/ in your default browser...				
				
				
开启 registry 私有库
		
# enable the registry addon
$ minikube addons enable registry

$ minikube start

# use the minikube docker daemon from the host
$ eval $(minikube docker-env)

# get the ip of the registry endpoint
$ kubectl -n kube-system get svc registry -o jsonpath="{.spec.clusterIP}"
10.0.0.240	
		
				
		
{
  "insecure-registries" : ["10.0.0.240"]
}		
		
				

		
$ minikube ssh
$ docker pull busybox
$ docker tag busybox 10.0.0.240/busybox

# or

# build and push to insecure registry
$ docker build -t 10.0.0.240/busybox .
$ docker push 10.0.0.240/busybox
		
				
启用 ingress
				
Neo-iMac:~ neo$ minikube addons enable ingress
💡  After the addon is enabled, please run "minikube tunnel" and your ingress resources would be available at "127.0.0.1"
    ▪ Using image registry.cn-hangzhou.aliyuncs.com/google_containers/kube-webhook-certgen:v1.1.1
    ▪ Using image registry.cn-hangzhou.aliyuncs.com/google_containers/kube-webhook-certgen:v1.1.1
    ▪ Using image registry.cn-hangzhou.aliyuncs.com/google_containers/nginx-ingress-controller:v1.0.4
🔎  Verifying ingress addon...
🌟  The 'ingress' addon is enabled				
				
				

运行一个简单的demo

				
运行 nginx 服务
kubectl run nginx --image=nginx --port=80
暴露服务
kubectl expose deployment nginx --port=80 --target-port=80

创建ingress
yaml 定义 ingress.yaml

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: nginx
spec:
  rules:
  - host: www.netkiller.cn
    http:
      paths:
      - path: /
        backend:
          serviceName: nginx
          servicePort: 80

运行 
kubectl apply -f ingress.yaml

配置本机host获取minikube ip 
 
[docker@localhost ~]$ minikube ip
192.168.58.2

配置 /etc/hosts 文件 
192.168.58.2 www.netkiller.cn			
				
				

访问 http://www.netkiller.cn

2.1.3.14. SSH

--vm-driver=none 不支持 ssh

			
[root@localhost ~]# minikube ssh
'none' driver does not support 'minikube ssh' command			
			
			

2.1.3.15. 查看IP地址

			
[root@localhost ~]# minikube ip
172.16.0.121			
			
			

2.1.3.16. 镜像管理

			
neo@MacBook-Pro-Neo ~ % minikube image ls
registry.cn-hangzhou.aliyuncs.com/google_containers/storage-provisioner:v5
registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.2
registry.cn-hangzhou.aliyuncs.com/google_containers/metrics-scraper:v1.0.4
registry.cn-hangzhou.aliyuncs.com/google_containers/kube-scheduler:v1.20.7
registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy:v1.20.7
registry.cn-hangzhou.aliyuncs.com/google_containers/kube-controller-manager:v1.20.7
registry.cn-hangzhou.aliyuncs.com/google_containers/kube-apiserver:v1.20.7
registry.cn-hangzhou.aliyuncs.com/google_containers/etcd:3.4.13-0
registry.cn-hangzhou.aliyuncs.com/google_containers/dashboard:v2.1.0
registry.cn-hangzhou.aliyuncs.com/google_containers/coredns:1.7.0
docker.io/netkiller/flask:latest			
			
			

2.1.3.17. kubectl

			
neo@MacBook-Pro-Neo ~ % minikube kubectl -- get pods -A
    > kubectl.sha256: 64 B / 64 B [--------------------------] 100.00% ? p/s 0s
    > kubectl: 44.08 MiB / 44.08 MiB [--------------] 100.00% 5.30 MiB p/s 8.5s
NAMESPACE       NAME                                        READY   STATUS              RESTARTS   AGE
ingress-nginx   ingress-nginx-admission-create-vzk2b        0/1     ImagePullBackOff    0          118d
ingress-nginx   ingress-nginx-admission-patch-65b85         0/1     ImagePullBackOff    0          118d
ingress-nginx   ingress-nginx-controller-7f79776f95-ncqkn   0/1     ContainerCreating   0          118d
kube-system     coredns-54d67798b7-cnjgw                    1/1     Running             2          121d
kube-system     etcd-minikube                               1/1     Running             2          121d
kube-system     kube-apiserver-minikube                     1/1     Running             2          121d
kube-system     kube-controller-manager-minikube            1/1     Running             2          121d
kube-system     kube-proxy-tr8fd                            1/1     Running             2          121d
kube-system     kube-scheduler-minikube                     1/1     Running             2          121d
kube-system     storage-provisioner                         1/1     Running             2          121d			
			
			

2.1.4. Minikube 案例演示

2.1.5. FAQ

2.1.5.1. This computer doesn't have VT-X/AMD-v enabled. Enabling it in the BIOS is mandatory

检查一下 BIOS 是否开启 VT-X/AMD-v

如果在虚拟机安装 Minikube 也会遇到这个问题。 可以使用 --vm-driver=none 参数启动。

			
neo@ubuntu:~$ sudo minikube start --vm-driver=none
			
			

2.1.5.2. ERROR FileContent--proc-sys-net-bridge-bridge-nf-call-iptables

解决方法

			
echo "1" > /proc/sys/net/bridge/bridge-nf-call-iptables
			
			

然后在 minikube start

2.1.5.3. ERROR ImagePull

[ERROR ImagePull]: failed to pull image k8s.gcr.io/pause:3.1: output: 3.1: Pulling from pause Get https://k8s.gcr.io/v2/pause/manifests/sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610: net/http: TLS handshake timeout

更换镜像再重试

			
[root@localhost ~]# minikube start --vm-driver=none --registry-mirror=https://registry.docker-cn.com			
			
			

2.1.5.4. 证书已存在错误

启动提示如下错误,一般出现这种错误是因为 minikube stop, minikube delete 后再重启 minikube start

			
error execution phase kubeconfig/admin: a kubeconfig file "/etc/kubernetes/admin.conf" exists already but has got the wrong CA cert	
error execution phase kubeconfig/kubelet: a kubeconfig file "/etc/kubernetes/kubelet.conf" exists already but has got the wrong CA cert
error execution phase kubeconfig/controller-manager: a kubeconfig file "/etc/kubernetes/controller-manager.conf" exists already but has got the wrong CA cert
error execution phase kubeconfig/scheduler: a kubeconfig file "/etc/kubernetes/scheduler.conf" exists already but has got the wrong CA cert
			
			

解决方法

			
[root@localhost ~]# mv /etc/kubernetes/admin.conf /etc/kubernetes/admin.conf.backup
[root@localhost ~]# mv /etc/kubernetes/kubelet.conf /etc/kubernetes/kubelet.conf.backup
[root@localhost ~]# mv /etc/kubernetes/controller-manager.conf /etc/kubernetes/controller-manager.conf.backup
[root@localhost ~]# mv /etc/kubernetes/scheduler.conf /etc/kubernetes/scheduler.conf.backup
			
			

现在启动 minikube start 不会再出错

			
[root@localhost ~]# minikube start --vm-driver=none
Starting local Kubernetes v1.13.2 cluster...
Starting VM...
Getting VM IP address...
Moving files into cluster...
Setting up certs...
Connecting to cluster...
Setting up kubeconfig...
Stopping extra container runtimes...
Starting cluster components...
Verifying kubelet health ...
Verifying apiserver health ...
Kubectl is now configured to use the cluster.
===================
WARNING: IT IS RECOMMENDED NOT TO RUN THE NONE DRIVER ON PERSONAL WORKSTATIONS
	The 'none' driver will run an insecure kubernetes apiserver as root that may leave the host vulnerable to CSRF attacks

When using the none driver, the kubectl config and credentials generated will be root owned and will appear in the root home directory.
You will need to move the files to the appropriate location and then set the correct permissions.  An example of this is below:

	sudo mv /root/.kube $HOME/.kube # this will write over any previous configuration
	sudo chown -R $USER $HOME/.kube
	sudo chgrp -R $USER $HOME/.kube

	sudo mv /root/.minikube $HOME/.minikube # this will write over any previous configuration
	sudo chown -R $USER $HOME/.minikube
	sudo chgrp -R $USER $HOME/.minikube

This can also be done automatically by setting the env var CHANGE_MINIKUBE_NONE_USER=true
Loading cached images from config file.


Everything looks great. Please enjoy minikube!
			
			

2.1.5.5. http: server gave HTTP response to HTTPS client

问题原因,使用私有 registry 由于没有 HTTPS 导致 kubectl 使用 https 去访问私有 registry.

			
Failed to pull image "192.168.3.85:5000/netkiller/config:latest": rpc error: code = Unknown desc = Error response from daemon: Get https://192.168.3.85:5000/v2/: http: server gave HTTP response to HTTPS client			
			
			

minikube 并不会使用 docker 配置文件中的 insecure-registry 配置项

解决办法

					
minikube start --insecure-registry=127.0.0.1:5000		
			
			

或指定网段

			
minikube start --insecure-registry "10.0.0.0/24"			
			
			

2.1.5.6. provided port is not in the valid range. The range of valid ports is 30000-32767

			
iMac:kubernetes neo$ kubectl create -f redis/redis.yml 
configmap/redis-config created
deployment.apps/redis created
The Service "redis" is invalid: spec.ports[0].nodePort: Invalid value: 6379: provided port is not in the valid range. The range of valid ports is 30000-32767			
			
			

编辑kube-apiserver.yaml文件

			
$ minikube ssh
$ sudo vi /etc/kubernetes/manifests/kube-apiserver.yaml
			
			

增加kube-apiserver的启动配置项

			
--service-node-port-range=1024-65535
			
			

			
$ sudo cat /etc/kubernetes/manifests/kube-apiserver.yaml
apiVersion: v1
kind: Pod
metadata:
  annotations:
    kubeadm.kubernetes.io/kube-apiserver.advertise-address.endpoint: 192.168.64.5:8443
  creationTimestamp: null
  labels:
    component: kube-apiserver
    tier: control-plane
  name: kube-apiserver
  namespace: kube-system
spec:
  containers:
  - command:
    - kube-apiserver
    - --advertise-address=192.168.64.5
    - --allow-privileged=true
    - --authorization-mode=Node,RBAC
    - --client-ca-file=/var/lib/minikube/certs/ca.crt
    - --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota
    - --enable-bootstrap-token-auth=true
    - --etcd-cafile=/var/lib/minikube/certs/etcd/ca.crt
    - --etcd-certfile=/var/lib/minikube/certs/apiserver-etcd-client.crt
    - --etcd-keyfile=/var/lib/minikube/certs/apiserver-etcd-client.key
    - --etcd-servers=https://127.0.0.1:2379
    - --insecure-port=0
    - --kubelet-client-certificate=/var/lib/minikube/certs/apiserver-kubelet-client.crt
    - --kubelet-client-key=/var/lib/minikube/certs/apiserver-kubelet-client.key
    - --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
    - --proxy-client-cert-file=/var/lib/minikube/certs/front-proxy-client.crt
    - --proxy-client-key-file=/var/lib/minikube/certs/front-proxy-client.key
    - --requestheader-allowed-names=front-proxy-client
    - --requestheader-client-ca-file=/var/lib/minikube/certs/front-proxy-ca.crt
    - --requestheader-extra-headers-prefix=X-Remote-Extra-
    - --requestheader-group-headers=X-Remote-Group
    - --requestheader-username-headers=X-Remote-User
    - --secure-port=8443
    - --service-account-key-file=/var/lib/minikube/certs/sa.pub
    - --service-cluster-ip-range=10.10.0.0/24
    - --service-node-port-range=1024-65535
    - --tls-cert-file=/var/lib/minikube/certs/apiserver.crt
    - --tls-private-key-file=/var/lib/minikube/certs/apiserver.key
    image: registry.cn-hangzhou.aliyuncs.com/google_containers/kube-apiserver:v1.19.2
    imagePullPolicy: IfNotPresent
    livenessProbe:
      failureThreshold: 8
      httpGet:
        host: 192.168.64.5
        path: /livez
        port: 8443
        scheme: HTTPS
      initialDelaySeconds: 10
      periodSeconds: 10
      timeoutSeconds: 15
    name: kube-apiserver
    readinessProbe:
      failureThreshold: 3
      httpGet:
        host: 192.168.64.5
        path: /readyz
        port: 8443
        scheme: HTTPS
      periodSeconds: 1
      timeoutSeconds: 15
    resources:
      requests:
        cpu: 250m
    startupProbe:
      failureThreshold: 24
      httpGet:
        host: 192.168.64.5
        path: /livez
        port: 8443
        scheme: HTTPS
      initialDelaySeconds: 10
      periodSeconds: 10
      timeoutSeconds: 15
    volumeMounts:
    - mountPath: /etc/ssl/certs
      name: ca-certs
      readOnly: true
    - mountPath: /var/lib/minikube/certs
      name: k8s-certs
      readOnly: true
    - mountPath: /usr/share/ca-certificates
      name: usr-share-ca-certificates
      readOnly: true
  hostNetwork: true
  priorityClassName: system-node-critical
  volumes:
  - hostPath:
      path: /etc/ssl/certs
      type: DirectoryOrCreate
    name: ca-certs
  - hostPath:
      path: /var/lib/minikube/certs
      type: DirectoryOrCreate
    name: k8s-certs
  - hostPath:
      path: /usr/share/ca-certificates
      type: DirectoryOrCreate
    name: usr-share-ca-certificates
status: {}			
			
			

			
sudo systemctl restart kubelet			
			
			

2.1.5.7. Exiting due to MK_ENABLE: run callbacks: running callbacks: [verifying registry addon pods : timed out waiting for the condition: timed out waiting for the condition]

			
iMac:~ neo$ minikube addons enable registry
🔎  Verifying registry addon...

❌  Exiting due to MK_ENABLE: run callbacks: running callbacks: [verifying registry addon pods : timed out waiting for the condition: timed out waiting for the condition]

😿  If the above advice does not help, please let us know: 
👉  https://github.com/kubernetes/minikube/issues/new/choose			
			
			

2.1.5.8.  Exiting due to SVC_URL_TIMEOUT: http://127.0.0.1:11068/api/v1/namespaces/kubernetes-dashboard/services/http:kubernetes-dashboard:/proxy/ is not accessible: Temporary Error: unexpected response code: 503

			
minikube dashboard --alsologtostderr -v=1
			
			

			
[docker@localhost ~]$ kubectl get pods --all-namespaces | grep dashboard
kubernetes-dashboard   dashboard-metrics-scraper-6f7955cd98-xjzkq   0/1     ImagePullBackOff    0          11d
kubernetes-dashboard   kubernetes-dashboard-7bf64fd654-ckr7v        0/1     ImagePullBackOff    0          11d			
			
			

			
[docker@localhost ~]$ kubectl logs --namespace=kubernetes-dashboard kubernetes-dashboard-7bf64fd654-ckr7v
Error from server (BadRequest): container "kubernetes-dashboard" in pod "kubernetes-dashboard-7bf64fd654-ckr7v" is waiting to start: trying and failing to pull image			
			
			

2.1.5.9. Mac minikube ip 不通,ingress 不工作

			
minikube start --image-mirror-country=cn --insecure-registry="registry.netkiller.cn" --cache-images=true			
			
			
			
Neo-iMac:~ neo$ kubectl get pods -n ingress-nginx
NAME                                        READY   STATUS      RESTARTS   AGE
ingress-nginx-admission-create--1-qpckk     0/1     Completed   0          18h
ingress-nginx-admission-patch--1-5x94l      0/1     Completed   0          18h
ingress-nginx-controller-78d858bdc7-nrszs   1/1     Running     1          18h

Neo-iMac:~ neo$ kubectl create deployment web --image=nginx:latest
deployment.apps/web created

Neo-iMac:~ neo$ kubectl expose deployment web --type=NodePort --port=80
service/web exposed

Neo-iMac:~ neo$ kubectl get service web
NAME   TYPE       CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGE
web    NodePort   10.109.55.204   <none>        8080:30857/TCP   19s

Neo-iMac:~ neo$ minikube service web --url
🏃  Starting tunnel for service web.
|-----------|------|-------------|------------------------|
| NAMESPACE | NAME | TARGET PORT |          URL           |
|-----------|------|-------------|------------------------|
| default   | web  |             | http://127.0.0.1:62956 |
|-----------|------|-------------|------------------------|
http://127.0.0.1:62956
❗  Because you are using a Docker driver on darwin, the terminal needs to be open to run it.			
			
			

ingress.yaml

			
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: nginx
  annotations:
    kubernetes.io/ingress.class: nginx  
spec:
  rules:
    - host: www.netkiller.cn
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: web
                port:
                  number: 80			
			
			

http://www.netkiller.cn 无法访问,解决方案 minikube tunnel

			
Neo-iMac:~ neo$ minikube tunnel
❗  The service/ingress example-ingress requires privileged ports to be exposed: [80 443]
🔑  sudo permission will be asked for it.
🏃  Starting tunnel for service example-ingress.
Password:			
			
			

如果注意观察,在启动的时候系统已经提示:After the addon is enabled, please run "minikube tunnel" and your ingress resources would be available at "127.0.0.1"

			
Neo-iMac:nginx neo$ minikube start --image-mirror-country=cn --insecure-registry="registry.netkiller.cn" --cache-images=true
😄  minikube v1.24.0 on Darwin 12.0.1
✨  Using the docker driver based on existing profile
👍  Starting control plane node minikube in cluster minikube
🚜  Pulling base image ...
🔄  Restarting existing docker container for "minikube" ...
🐳  Preparing Kubernetes v1.22.3 on Docker 20.10.8 ...
🔎  Verifying Kubernetes components...
💡  After the addon is enabled, please run "minikube tunnel" and your ingress resources would be available at "127.0.0.1"
    ▪ Using image registry.cn-hangzhou.aliyuncs.com/google_containers/dashboard:v2.3.1
    ▪ Using image registry.cn-hangzhou.aliyuncs.com/google_containers/storage-provisioner:v5
    ▪ Using image registry.cn-hangzhou.aliyuncs.com/google_containers/nginx-ingress-controller:v1.0.4
    ▪ Using image registry.cn-hangzhou.aliyuncs.com/google_containers/metrics-scraper:v1.0.7
    ▪ Using image registry.cn-hangzhou.aliyuncs.com/google_containers/kube-webhook-certgen:v1.1.1
    ▪ Using image registry.cn-hangzhou.aliyuncs.com/google_containers/kube-webhook-certgen:v1.1.1
🔎  Verifying ingress addon...
🌟  Enabled addons: dashboard, storage-provisioner, default-storageclass, ingress
🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default