解决了

Karbon 2.3和Calico CNI的部署问题

  • 2021年10月17日
  • 5个答复
  • 160次观看
一世
徽章

你好呀，

看来我在部署的尽头接近失败。这是来自karbon_core.out的输出

2021-10-17T11：57：31.052Z kube_prometheus.go：1016：[debug] [k8s_cluster = rgs-pa-k8-cluster阶段]期望5个节点在kube-system namespace中运行Calico-node daemon daemon pod。当前正在运行：4
2021-10-17T11：57：33.093Z kube_prometheus.go：1016：[debug] [k8s_cluster = rgs-pa-k8-cluster阶段]期望5个节点在kube-system namesspace中运行Calico-node daemon Pod。当前正在运行：4
2021-10-17T11：57：35.135Z kube_prometheus.go：1016：[debug] [k8s_cluster = rgs-pa-k8-cluster阶段]期望5个节点在kube-system namesspace中运行Calico-node daemon Pod。当前正在运行：4
2021-10-17T11：57：36.806Z CALICO.GO：552：[错误] [k8s_cluster = rgs-pa-k8 cluster阶段]无法验证Calico addon
2021-10-17T11：57：36.806Z k8s_deploy.go：1478：[错误] [k8s_cluster = rgs-pa-k8-cluster阶段阶段]未能部署卡利科/绒布/绒布：无法部署calico：失败的calico：Operative calico：Operative califify califify califify califify califify califify：时机：期望5个节点将在Kube-System名称空间中运行Calico节点守护程序POD。当前正在运行：4
2021-10-17T11：57：36.806Z k8s_deploy.go：155：[错误] [k8s_cluster = rgs-pa-k8-cluster阶段]无法部署群集插件：失败无法部署K8S群集Addon：无法部署Calico Calico：无法验证Calico：Operation Outed Out：期望在Kube-System名称空间中运行5个节点。当前正在运行：4
2021-10-17T11:57:36.832Z k8s_lib_deploy_task.go:112: [ERROR] [k8s_cluster=RGS-PA-K8-CLUSTER-STAGING] failed to deploy K8s cluster: failed to deploy cluster addons: failed to deploy K8s cluster addon：无法部署印花布：无法验证卡利科：操作定时：期望在Kube-System名称空间中运行5个节点。当前正在运行：4
2021-10-17T11：57：36.832Z k8s_lib_deploy_task.go：78：[info] [info] [k8s_cluster = rgs-pa-k8-cluster阶段] token refresher refersher收到了pocrefresh
2021-10-17T11：57：36.844Z deploy_k8s_task.go：364：[错误] [k8s_cluster = rgs-pa-k8-cluster staging]群集RGS-pa-k8-cluster staging staging：失败K8S cluster：失败：失败部署群集插件：无法部署K8S群集插件：无法部署卡利科：无法验证卡利科：操作时机计时：期望5个节点在Kube-System名称空间中运行Calico-Node Daemon Pod。当前正在运行：4
2021-10-17T11：57：36.844Z deploy_k8s_task.go：370：[info] [k8s_cluster = rgs-pa-k8-cluster阶段]

如果有人有任何洞察力如何解决这个问题，请非常感谢！

图标

最好的答案乔森纳克斯2021年10月18日，10：28

Hi Igor,<\/p>

I suggest you open a ticket with support, so they can investigate why this pod is crashing (saw calico-node-bvct7 crashing once too). Two of three containers in this pod are alive, with calico-node crashing not finding \/var\/lib\/calico\/nodename. Usually this is sort of issues are related to network\/performance issues.\u00a0<\/p>","className":"post__content__best_answer"}">

查看原件
Hi there,<\/p>

It seems that I have a failure near the very end of deployment. Here is the output from karbon_core.out<\/p>

2021-10-17T11:57:31.052Z kube_prometheus.go:1016: [DEBUG] [k8s_cluster=RGS-PA-K8-CLUSTER-STAGING] expecting 5 nodes to be running calico-node daemon pod in kube-system namespace. Currently running: 4
2021-10-17T11:57:33.093Z kube_prometheus.go:1016: [DEBUG] [k8s_cluster=RGS-PA-K8-CLUSTER-STAGING] expecting 5 nodes to be running calico-node daemon pod in kube-system namespace. Currently running: 4
2021-10-17T11:57:35.135Z kube_prometheus.go:1016: [DEBUG] [k8s_cluster=RGS-PA-K8-CLUSTER-STAGING] expecting 5 nodes to be running calico-node daemon pod in kube-system namespace. Currently running: 4
2021-10-17T11:57:36.806Z calico.go:552: [ERROR] [k8s_cluster=RGS-PA-K8-CLUSTER-STAGING] Failed to verify calico addon
2021-10-17T11:57:36.806Z k8s_deploy.go:1478: [ERROR] [k8s_cluster=RGS-PA-K8-CLUSTER-STAGING] Failed to deploy calico\/flannel: Failed to deploy calico: Failed to verify calico: Operation timed out: expecting 5 nodes to be running calico-node daemon pod in kube-system namespace. Currently running: 4
2021-10-17T11:57:36.806Z k8s_deploy.go:155: [ERROR] [k8s_cluster=RGS-PA-K8-CLUSTER-STAGING] failed to deploy cluster addons: failed to deploy K8s cluster addon: Failed to deploy calico: Failed to verify calico: Operation timed out: expecting 5 nodes to be running calico-node daemon pod in kube-system namespace. Currently running: 4
2021-10-17T11:57:36.832Z k8s_lib_deploy_task.go:112: [ERROR] [k8s_cluster=RGS-PA-K8-CLUSTER-STAGING] failed to deploy K8s cluster: failed to deploy cluster addons: failed to deploy K8s cluster addon: Failed to deploy calico: Failed to verify calico: Operation timed out: expecting 5 nodes to be running calico-node daemon pod in kube-system namespace. Currently running: 4
2021-10-17T11:57:36.832Z k8s_lib_deploy_task.go:78: [INFO] [k8s_cluster=RGS-PA-K8-CLUSTER-STAGING] token refresher received stopRefresh
2021-10-17T11:57:36.844Z deploy_k8s_task.go:364: [ERROR] [k8s_cluster=RGS-PA-K8-CLUSTER-STAGING] Cluster RGS-PA-K8-CLUSTER-STAGING:failed to deploy K8s cluster: failed to deploy cluster addons: failed to deploy K8s cluster addon: Failed to deploy calico: Failed to verify calico: Operation timed out: expecting 5 nodes to be running calico-node daemon pod in kube-system namespace. Currently running: 4
2021-10-17T11:57:36.844Z deploy_k8s_task.go:370: [INFO] [k8s_cluster=RGS-PA-K8-CLUSTER-STAGING] Wait for subtasks to finish before completing parent task<\/p><\/blockquote>

\u00a0<\/p>

If anyone would have any insight how to resolve this, much appreciated!<\/p>","quoteUsername":"IgorStankovic","translations":{"Common":{"like":"Like","unlike":"Unlike"},"Forum":{"Quote":"Quote","Share":"Share"}}}">

该主题已关闭以供评论

5个答复

一世
徽章

忘了发布，豆荚状态：

名称空间名称准备状态重新启动年龄
Kube-System Calico-kube-controllers-7F667666F7F-ND8SX 1/1运行1 74m
Kube-System Calico-Node-2ctb4 1/1运行0 74m
Kube-System Calico-Node-7FX7N 1/1运行0 74m
Kube-System Calico-node-bvct7 1/1运行1 74m
Kube-System Calico-node-fjwjp 0/1 crashloopbackoff 23 74m
Kube-System Calico-node-XTH2K 1/1运行0 74m
Kube-System Calico-Typha-6bfd555555df7-ptc7d 1/1运行0 74m
Kube-System Kube-apiserver-karbon-rgs-pa-k8-cluster strusting-e77682-k8s-master-0 3/3运行0 77m
Kube-System Kube-apiserver-karbon-rgs-pa-k8-cluster Stiging-e77682-k8s-Master-1 3/3运行0 77m
Kube-System Kube-Proxy-DS-DSD5V 1/1运行0 74m
Kube-System Kube-Proxy-ds-gnng4 1/1运行0 74m
Kube-System Kube-Proxy-DS-PH68Q 1/1运行0 74m
Kube-System Kube-Proxy-DS-TF4ML 1/1运行0 74m
Kube-System Kube-Proxy-DS-WHBPL

Forgot to post, the pods status:<\/p>
NAMESPACE     NAME                                                                  READY   STATUS             RESTARTS   AGE
kube-system   calico-kube-controllers-7f66766f7f-nd8sx                              1\/1     Running            1          74m
kube-system   calico-node-2ctb4                                                     1\/1     Running            0          74m
kube-system   calico-node-7fx7n                                                     1\/1     Running            0          74m
kube-system   calico-node-bvct7                                                     1\/1     Running            1          74m
kube-system   calico-node-fjwjp                                                     0\/1     CrashLoopBackOff   23         74m
kube-system   calico-node-xth2k                                                     1\/1     Running            0          74m
kube-system   calico-typha-6bfd55df7-ptc7d                                          1\/1     Running            0          74m
kube-system   kube-apiserver-karbon-rgs-pa-k8-cluster-staging-e77682-k8s-master-0   3\/3     Running            0          77m
kube-system   kube-apiserver-karbon-rgs-pa-k8-cluster-staging-e77682-k8s-master-1   3\/3     Running            0          77m
kube-system   kube-proxy-ds-dsd5v                                                   1\/1     Running            0          74m
kube-system   kube-proxy-ds-gnng4                                                   1\/1     Running            0          74m
kube-system   kube-proxy-ds-ph68q                                                   1\/1     Running            0          74m
kube-system   kube-proxy-ds-tf4ml                                                   1\/1     Running            0          74m
kube-system   kube-proxy-ds-whbpl <\/code><\/pre>
\u00a0<\/p>","quoteUsername":"IgorStankovic","translations":{"Common":{"like":"Like","unlike":"Unlike"},"Forum":{"Quote":"Quote","Share":"Share"}}}">
乔森纳克斯
秩
UserLevel 4
徽章 +5

嗨，伊戈尔，

操作正时。您必须检查站点之间是否有足够的带宽来拉图像。

另外，您可以检查吊舱的日志Calico-node-fjwjp看看它是否下载了图像，如果确实如此，那么卡利科为什么要崩溃。

何塞·戈麦斯（Jose Gomez）|技术营销工程师|jose.gomez@nutanix.com |@pipoe2H
Hi Igor,<\/p>

The operation is timing out. You\u2019ll have to check if there is enough bandwidth between sites to pull the images.<\/p>

Also, you can check the logs for the pod\u00a0calico-node-fjwjp<\/strong> and see if it downloaded the image, and if it did, then why Calico is crashing.\u00a0<\/p>","quoteUsername":"JoseNutanix","translations":{"Common":{"like":"Like","unlike":"Unlike"},"Forum":{"Quote":"Quote","Share":"Share"}}}">

一世
徽章

你好，

是的，带宽很好……进行了一些基本的测试，所有基于K8的VM都很好。很奇怪的是，他的特殊豆荚无法初始化印花布网络，因此karbon部署失败了。不过（自动）并未删除Karvon群集，因此有机会环顾四周。

对于豆荚Calico-node-fjwjp

Kube-System Calico-node-fjwjp 0/1 crashloopbackoff 327 19h

它一直在重新启动，因为人们期望的是未达到准备状态。

事件：
从消息输入原因年龄
---------------------------------------------------------------------------------------------
警告不健康的12m（X2224超过19H）kubelet准备就绪探针失败：Calico/Node还没有准备好：bird尚未准备就绪：未准备好：失败stat（）nodeName文件：stat/var/var/lib/lib/calico/nodeName：没有这样的文件或目录
警告向后2M46S（X3945超过19H）库贝特退出重新启动失败容器

POD的全部输出描述：

名称：Calico-node-fjwjp
名称空间：Kube-System
优先级：2000001000
优先类名称：系统节点至关重要
节点：karbon-rgs-pa-k8群集阶段-E77682-k8s-worker-0/10.20.25.73
开始时间：2021年10月17日太阳11:47:36 +0000
标签：Controller-Revision-Hash = 547955649b
k8s-app =卡利科节点
POD-Template生成= 1
注释：scheduler.alpha.kubernetes.io/critical-pod：
状态：运行
IP：10.20.25.73
IPS：
IP：10.20.25.73
控制者：Daemonset/Calico节点
初始容器：
升级 -  ipam：
容器ID：Docker：// 025878DE4F3AB420BDC8D572C1037FF591C892F32B1607C1F60F60F60F523C398DB8DE
图片：quay.io/karbon/cni:v3.14.0
图片ID：docker-pullable：//quay.io/karbon/cni@sha256：CC951CCD15AA8C94B1B3EEC673E434853BF3BF3BF8C2DEB8C2DEB83B83BDB4A3B4A3F934C68E0E8EAEAE
端口：
主机端口：
命令：
/opt/cni/bin/calico-ipam
-升级
状态：终止
原因：完成
退出代码：0
开始：2021年10月17日太阳11:47:45 +0000
完成：2021年10月17日太阳11:47:45 +0000
准备好：是的
重新启动计数：0
环境：
kubernetes_node_name ：（ v1：spec.nodeName）
calico_networking_backend：<设置为键'calico_backend'配置映射'calico-config'>可选：false
坐骑：
/host/opt/opt/cni/bin，来自CNI-BIN-DIR（RW）
/var/lib/cni/网络来自主机 - 本地网络dir（RW）
/var/run/secrets/kubernetes.io/serviceaccount来自Calico-node-token-x5lvc（ro）
install-cni：
容器ID：Docker：// 455ED002C1D8450E362FCA773854F54000022229A11401C3943D00D691060827
图片：quay.io/karbon/cni:v3.14.0
图片ID：docker-pullable：//quay.io/karbon/cni@sha256：CC951CCD15AA8C94B1B3EEC673E434853BF3BF3BF8C2DEB8C2DEB83B83BDB4A3B4A3F934C68E0E8EAEAE
端口：
主机端口：
命令：
/innstall-cni.sh
状态：终止
原因：完成
退出代码：0
开始：2021年10月17日太阳11:47:47 +0000
完成：2021年10月17日太阳11:47:47 +0000
准备好：是的
重新启动计数：0
环境：
cni_conf_name：10-calico.conflist
cni_network_config：<设置为键'cni_network_config'config map'calico-config'>可选：false
kubernetes_node_name ：（ v1：spec.nodeName）
cni_mtu：<设置为“配置映射”键'veth_mtu'calico-config'>可选：false
睡眠：错误
坐骑：
来自cni-net-dir（RW）的/host/etc/cni/net.d
/host/opt/opt/cni/bin，来自CNI-BIN-DIR（RW）
/var/run/secrets/kubernetes.io/serviceaccount来自Calico-node-token-x5lvc（ro）
Flexvol-Driver：
容器ID：Docker：// 68F392F6D3BDE62F14185FB50C6B41099982BD63AC060CCBADC18522E84FDC60B
图片：quay.io/karbon/pod2daemon-flexvol:v3.14.0
图片ID：docker-pullable：//quay.io/karbon/pod2daemon-flexvol@sha256：e5f2c2b9e67ec463ef5b538bf538bf10453cc6a6538f72888a4760ee9255149251498E798E7DD
端口：
主机端口：
状态：终止
原因：完成
退出代码：0
开始：2021年10月17日太阳11:47:51 +0000
完成：2021年10月17日太阳11:47:51 +0000
准备好：是的
重新启动计数：0
环境：<无>
坐骑：
/flexvol-driver-host（RW）的主机/驱动程序
/var/run/secrets/kubernetes.io/serviceaccount来自Calico-node-token-x5lvc（ro）
容器：
Calico节点：
容器ID：docker：// 96FA1881578BD5BAE774A6F25FFC108888888888888888888888888882413EF4ACB6C8E450CF6B38345A8D
图像：quay.io/karbon/node:v3.14.0
图片ID：docker-pullable：//quay.io/karbon/node@sha256：1a643541c41c46ea412de19454bfada5a7e03e7e03e7cbbbbbbbbbbbb51dddf76def9baf9baf9baf84baf84baf84bdad7c
端口：
主机端口：
状态：等待
原因：crashloopbackoff
最后状态：终止
原因：错误
退出代码：137
开始：2021年10月18日星期一07:41:15 +0000
完成：2021年10月18日星期一07：42：24 +0000
准备就绪：false
重新启动计数：327
要求：
CPU：250m
livices：exec [/bin/calico节点-felix-live] delay = 10s timeout = 1s oferout = 1s perique = 10s＃成功= 1＃＃失败= 6
准备时间：exec [/bin/calico节点-felix -ready-bird-Ready] delay = 0s timeout = 1S oferout = 1S oferet = 10s＃成功= 1＃＃＃fafer = 3
环境：
datastore_type：kubernetes
felix_typhak8sserviceName：<设置为“ typha_service_name”配置映射'calico-config'>可选：false
WAIT_FOR_DATASTORE：正确
nodename ：（ v1：spec.nodeName）
calico_networking_backend：<设置为键'calico_backend'配置映射'calico-config'>可选：false
cluster_type：k8s，bgp
IP：自动检索
Calico_ipv4pool_ipip：从不
ip_autodetection_method：接口= eth。*
felix_ipinipmtu：<设置为“配置映射”键'veth_mtu'calico-config'>可选：false
Calico_ipv4pool_cidr：172.20.0.0/16
Calico_Advertise_cluster_ips：172.19.0.0/16
Calico_disable_file_logging：true
felix_defaultendpointTohostaction：接受
felix_ipv6support：false
felix_logseverityscreen：info
Felix_HealthEnabled：true
felix_prometheusgometricsenabled：false
felix_prometheusmetricsEnabled：true
坐骑：
/lib/lib-modules（RO）的LIB/模块
/run/xtables.lock from xtables-lock（rw）
/var/lib/calico来自var-lib-calico（RW）
/var/run/calico来自var-run-calico（RW）
/var/run/nodeagent来自policySync（RW）
/var/run/secrets/kubernetes.io/serviceaccount来自Calico-node-token-x5lvc（ro）
条件：
类型状态
初始化的true
准备好了
容器准备的false
podscheduled true
卷：
lib模型：
类型：主机（裸机目录卷）
路径： /lib /模块
HOSTPATHTYPE：
var-run-calico：
类型：主机（裸机目录卷）
路径：/var/run/calico
HOSTPATHTYPE：
var-lib-calico：
类型：主机（裸机目录卷）
路径：/var/lib/calico
HOSTPATHTYPE：
Xtables-Lock：
类型：主机（裸机目录卷）
路径：/run/xtables.lock
主机路径：FileOrcreate
CNI-BIN-DIR：
类型：主机（裸机目录卷）
路径：/var/lib/hyperkube/opt/cni/bin
HOSTPATHTYPE：
CNI-NET-DIR：
类型：主机（裸机目录卷）
路径：/etc/cni/net.d
HOSTPATHTYPE：
主机 - 本地网络dir：
类型：主机（裸机目录卷）
路径：/var/lib/cni/网络
HOSTPATHTYPE：
policySync：
类型：主机（裸机目录卷）
路径：/var/run/nodeagent
主机路径：目录
Flexvol-Driver-Host：
类型：主机（裸机目录卷）
路径：/usr/libexec/kubernetes/kubelet-plugins/卷/exec/nodeagent〜uds
主机路径：目录
Calico-node-token-X5LVC：
类型：秘密（由秘密填充的卷）
SecretName：Calico-node-token-X5LVC
可选：错误
QoS类：可爆裂
节点选择器：kubernetes.io/os=linux
公差：： noschedule op =存在
：noexecute op =存在
Criticaladdonsonly op =存在
node.kubernetes.io/disk-pressure：noschedule op =存在
node.kubernetes.io/memory-pressure：noschedule op =存在
node.kubernetes.io/network-unavailable：noschedule op =存在
node.kubernetes.io/not-ready:noexecute op =存在
node.kubernetes.io/pid-pressure：noschedule op =存在
node.kubernetes.io/unreachable:noexecute op =存在
node.kubernetes.io/unschedulable：noschedule op =存在
事件：
从消息输入原因年龄
---------------------------------------------------------------------------------------------
警告不健康的14m（X2224超过19H）kubelet准备就绪探针失败：卡利科/节点还没有准备就绪：伯德还没有准备好：无法使用stat（）nodeName file：stat/var/var/lib/calico/node
名称：没有这样的文件或目录
警告向后4M24S（19H上方x3945）库贝特退出重新启动失败容器

Hi,<\/p>

Yes, bandwidth is just fine \u2026\u00a0did some basic testing and all K8 based VMs initialised just fine. It\u2019s just weird that his particular pod can\u2019t initialise Calico network hence the Karbon deployment fails. The Karvon cluster is not removed though (automatically) so there is a chance to look around.<\/p>

For the pod\u00a0calico-node-fjwjp<\/strong>\u00a0<\/p>

kube-system \u00a0 calico-node-fjwjp \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 0\/1 \u00a0 \u00a0 CrashLoopBackOff \u00a0 327 \u00a0 \u00a0 \u00a0 \u00a019h
\u00a0<\/p>

It\u2019s constantly restarting as one would expect as readiness\u00a0state is not reached.<\/p>

Events:
\u00a0 Type \u00a0 \u00a0 Reason \u00a0 \u00a0 Age \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 From \u00a0 \u00a0 Message
\u00a0 ---- \u00a0 \u00a0 ------ \u00a0 \u00a0 ---- \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0---- \u00a0 \u00a0 -------
\u00a0 Warning \u00a0Unhealthy \u00a012m (x2224 over 19h) \u00a0 \u00a0kubelet \u00a0Readiness probe failed: calico\/node is not ready: BIRD is not ready: Failed to stat() nodename file: stat \/var\/lib\/calico\/nodename: no such file or directory
\u00a0 Warning \u00a0BackOff \u00a0 \u00a02m46s (x3945 over 19h) \u00a0kubelet \u00a0Back-off restarting failed container<\/p><\/blockquote>

\u00a0<\/p>

Full output from pod describe:<\/p>

Name:                 calico-node-fjwjp
Namespace:            kube-system
Priority:             2000001000
Priority Class Name:  system-node-critical
Node:                 karbon-rgs-pa-k8-cluster-staging-e77682-k8s-worker-0\/10.20.25.73
Start Time:           Sun, 17 Oct 2021 11:47:36 +0000
Labels:               controller-revision-hash=547955649b
                      k8s-app=calico-node
                      pod-template-generation=1
Annotations:          scheduler.alpha.kubernetes.io\/critical-pod: 
Status:               Running
IP:                   10.20.25.73
IPs:
  IP:           10.20.25.73
Controlled By:  DaemonSet\/calico-node
Init Containers:
  upgrade-ipam:
    Container ID:  docker:\/\/025878de4f3ab420bdc8d572c1037ff591c892f32b1607c1f60f523c398db8de
    Image:         quay.io\/karbon\/cni:v3.14.0
    Image ID:      docker-pullable:\/\/quay.io\/karbon\/cni@sha256:cc951ccd15aa8c94b1b3eec673e434853f3bf8c2deb83bdb4a3f934c68e0e8ae
    Port:          <none>
    Host Port:     <none>
    Command:
      \/opt\/cni\/bin\/calico-ipam
      -upgrade
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Sun, 17 Oct 2021 11:47:45 +0000
      Finished:     Sun, 17 Oct 2021 11:47:45 +0000
    Ready:          True
    Restart Count:  0
    Environment:
      KUBERNETES_NODE_NAME:        (v1:spec.nodeName)
      CALICO_NETWORKING_BACKEND:  <set to the key 'calico_backend' of config map 'calico-config'>  Optional: false
    Mounts:
      \/host\/opt\/cni\/bin from cni-bin-dir (rw)
      \/var\/lib\/cni\/networks from host-local-net-dir (rw)
      \/var\/run\/secrets\/kubernetes.io\/serviceaccount from calico-node-token-x5lvc (ro)
  install-cni:
    Container ID:  docker:\/\/455ed002c1d8450e362fca773854f54000022d29a11401c3943d00d691060827
    Image:         quay.io\/karbon\/cni:v3.14.0
    Image ID:      docker-pullable:\/\/quay.io\/karbon\/cni@sha256:cc951ccd15aa8c94b1b3eec673e434853f3bf8c2deb83bdb4a3f934c68e0e8ae
    Port:          <none>
    Host Port:     <none>
    Command:
      \/install-cni.sh
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Sun, 17 Oct 2021 11:47:47 +0000
      Finished:     Sun, 17 Oct 2021 11:47:47 +0000
    Ready:          True
    Restart Count:  0
    Environment:
      CNI_CONF_NAME:         10-calico.conflist
      CNI_NETWORK_CONFIG:    <set to the key 'cni_network_config' of config map 'calico-config'>  Optional: false
      KUBERNETES_NODE_NAME:   (v1:spec.nodeName)
      CNI_MTU:               <set to the key 'veth_mtu' of config map 'calico-config'>  Optional: false
      SLEEP:                 false
    Mounts:
      \/host\/etc\/cni\/net.d from cni-net-dir (rw)
      \/host\/opt\/cni\/bin from cni-bin-dir (rw)
      \/var\/run\/secrets\/kubernetes.io\/serviceaccount from calico-node-token-x5lvc (ro)
  flexvol-driver:
    Container ID:   docker:\/\/68f392f6d3bde62f14185fb50c6b4109982bd63ac060ccbadc18522e84fdc60b
    Image:          quay.io\/karbon\/pod2daemon-flexvol:v3.14.0
    Image ID:       docker-pullable:\/\/quay.io\/karbon\/pod2daemon-flexvol@sha256:e5f2c2b9e67ec463ef5b538b8bf10453cc6a6538f7288a4760ee925c51498e7d
    Port:           <none>
    Host Port:      <none>
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Sun, 17 Oct 2021 11:47:51 +0000
      Finished:     Sun, 17 Oct 2021 11:47:51 +0000
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      \/host\/driver from flexvol-driver-host (rw)
      \/var\/run\/secrets\/kubernetes.io\/serviceaccount from calico-node-token-x5lvc (ro)
Containers:
  calico-node:
    Container ID:   docker:\/\/96fa1881578bd5bae774a6f25ffc108882413ef44acb6c8e450cf6b38345aa8d
    Image:          quay.io\/karbon\/node:v3.14.0
    Image ID:       docker-pullable:\/\/quay.io\/karbon\/node@sha256:1a643541c4d76ea412dde19454bfada5a7e03e7cbb51ddf76def9baf84bdad7c
    Port:           <none>
    Host Port:      <none>
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    137
      Started:      Mon, 18 Oct 2021 07:41:15 +0000
      Finished:     Mon, 18 Oct 2021 07:42:24 +0000
    Ready:          False
    Restart Count:  327
    Requests:
      cpu:      250m
    Liveness:   exec [\/bin\/calico-node -felix-live] delay=10s timeout=1s period=10s #success=1 #failure=6
    Readiness:  exec [\/bin\/calico-node -felix-ready -bird-ready] delay=0s timeout=1s period=10s #success=1 #failure=3
    Environment:
      DATASTORE_TYPE:                     kubernetes
      FELIX_TYPHAK8SSERVICENAME:          <set to the key 'typha_service_name' of config map 'calico-config'>  Optional: false
      WAIT_FOR_DATASTORE:                 true
      NODENAME:                            (v1:spec.nodeName)
      CALICO_NETWORKING_BACKEND:          <set to the key 'calico_backend' of config map 'calico-config'>  Optional: false
      CLUSTER_TYPE:                       k8s,bgp
      IP:                                 autodetect
      CALICO_IPV4POOL_IPIP:               Never
      IP_AUTODETECTION_METHOD:            interface=eth.*
      FELIX_IPINIPMTU:                    <set to the key 'veth_mtu' of config map 'calico-config'>  Optional: false
      CALICO_IPV4POOL_CIDR:               172.20.0.0\/16
      CALICO_ADVERTISE_CLUSTER_IPS:       172.19.0.0\/16
      CALICO_DISABLE_FILE_LOGGING:        true
      FELIX_DEFAULTENDPOINTTOHOSTACTION:  ACCEPT
      FELIX_IPV6SUPPORT:                  false
      FELIX_LOGSEVERITYSCREEN:            info
      FELIX_HEALTHENABLED:                true
      FELIX_PROMETHEUSGOMETRICSENABLED:   false
      FELIX_PROMETHEUSMETRICSENABLED:     true
    Mounts:
      \/lib\/modules from lib-modules (ro)
      \/run\/xtables.lock from xtables-lock (rw)
      \/var\/lib\/calico from var-lib-calico (rw)
      \/var\/run\/calico from var-run-calico (rw)
      \/var\/run\/nodeagent from policysync (rw)
      \/var\/run\/secrets\/kubernetes.io\/serviceaccount from calico-node-token-x5lvc (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  lib-modules:
    Type:          HostPath (bare host directory volume)
    Path:          \/lib\/modules
    HostPathType:  
  var-run-calico:
    Type:          HostPath (bare host directory volume)
    Path:          \/var\/run\/calico
    HostPathType:  
  var-lib-calico:
    Type:          HostPath (bare host directory volume)
    Path:          \/var\/lib\/calico
    HostPathType:  
  xtables-lock:
    Type:          HostPath (bare host directory volume)
    Path:          \/run\/xtables.lock
    HostPathType:  FileOrCreate
  cni-bin-dir:
    Type:          HostPath (bare host directory volume)
    Path:          \/var\/lib\/hyperkube\/opt\/cni\/bin
    HostPathType:  
  cni-net-dir:
    Type:          HostPath (bare host directory volume)
    Path:          \/etc\/cni\/net.d
    HostPathType:  
  host-local-net-dir:
    Type:          HostPath (bare host directory volume)
    Path:          \/var\/lib\/cni\/networks
    HostPathType:  
  policysync:
    Type:          HostPath (bare host directory volume)
    Path:          \/var\/run\/nodeagent
    HostPathType:  DirectoryOrCreate
  flexvol-driver-host:
    Type:          HostPath (bare host directory volume)
    Path:          \/usr\/libexec\/kubernetes\/kubelet-plugins\/volume\/exec\/nodeagent~uds
    HostPathType:  DirectoryOrCreate
  calico-node-token-x5lvc:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  calico-node-token-x5lvc
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  kubernetes.io\/os=linux
Tolerations:     :NoSchedule op=Exists
                 :NoExecute op=Exists
                 CriticalAddonsOnly op=Exists
                 node.kubernetes.io\/disk-pressure:NoSchedule op=Exists
                 node.kubernetes.io\/memory-pressure:NoSchedule op=Exists
                 node.kubernetes.io\/network-unavailable:NoSchedule op=Exists
                 node.kubernetes.io\/not-ready:NoExecute op=Exists
                 node.kubernetes.io\/pid-pressure:NoSchedule op=Exists
                 node.kubernetes.io\/unreachable:NoExecute op=Exists
                 node.kubernetes.io\/unschedulable:NoSchedule op=Exists
Events:
  Type     Reason     Age                     From     Message
  ----     ------     ----                    ----     -------
  Warning  Unhealthy  14m (x2224 over 19h)    kubelet  Readiness probe failed: calico\/node is not ready: BIRD is not ready: Failed to stat() nodename file: stat \/var\/lib\/calico\/node
name: no such file or directory
  Warning  BackOff    4m24s (x3945 over 19h)  kubelet  Back-off restarting failed container
<\/code><\/pre>
\u00a0<\/p>","quoteUsername":"IgorStankovic","translations":{"Common":{"like":"Like","unlike":"Unlike"},"Forum":{"Quote":"Quote","Share":"Share"}}}">
乔森纳克斯
秩
UserLevel 4
徽章 +5

嗨，伊戈尔，

我建议您打开一张支持票，以便他们可以调查为什么这个吊舱崩溃了（看到Calico-node-bvct7也一次崩溃了）。该吊舱中的三个容器中有两个还活着，Calico节点崩溃没有找到/var/lib/calico/nodename。通常，这是与网络/性能问题有关的问题。

何塞·戈麦斯（Jose Gomez）|技术营销工程师|jose.gomez@nutanix.com |@pipoe2H
Hi Igor,<\/p>

I suggest you open a ticket with support, so they can investigate why this pod is crashing (saw calico-node-bvct7 crashing once too). Two of three containers in this pod are alive, with calico-node crashing not finding \/var\/lib\/calico\/nodename. Usually this is sort of issues are related to network\/performance issues.\u00a0<\/p>","quoteUsername":"JoseNutanix","translations":{"Common":{"like":"Like","unlike":"Unlike"},"Forum":{"Quote":"Quote","Share":"Share"}}}">

一世
徽章

嗨，何塞，

是的，这很好 - 只是想弄清楚如何提高支持票，因为我从来没有很高兴使用它：脸红：

是的，似乎有些特定的工人节点（10.20.25.73）和属于那里的豆荚并通过kubelet进行通信，而不是严格来自卡利科节点：

igor.stankovic@rgs-pa-bastion-1：〜$ kubectl -n kube-system logs -f kube-proxy-ds-whbpl
服务器中的错误：获取“ https：//10.25.73:10250/containerlogs/kube-system/kube-proxy-ds-whbpl/kube-proxy?follow=true”：dial TCP 10.20.25.73:10250：I/O超时
igor.stankovic@rgs-pa-bastion-1：〜$

我们试图重新启动Kubelet，Docker然后完全回收VM节点，但仍然相同。

从支持中听到会很有趣。

Hi Jose,<\/p>

Yes, that\u2019s fine - just to figure out how to raise the ticket for support, as I never had a pleasure to use it in the past\u00a0\u00a0\":blush:\"<\/p>

Yes, it seems something is off with the particular\u00a0worker node (10.20.25.73) and pods belonging there and communicating via kubelet and not strictly from calico nodes:<\/p>

igor.stankovic@rgs-pa-bastion-1:~$ kubectl -n kube-system logs -f kube-proxy-ds-whbpl\u00a0
Error from server: Get \"https:\/\/10.20.25.73:10250\/containerLogs\/kube-system\/kube-proxy-ds-whbpl\/kube-proxy?follow=true\": dial tcp 10.20.25.73:10250: i\/o timeout
igor.stankovic@rgs-pa-bastion-1:~$\u00a0<\/p><\/blockquote>

\u00a0<\/p>

We tried to reboot the\u00a0kubelet, docker then full recycle for the VM node but still the same.\u00a0<\/p>

It would be interesting to hear from the support.\u00a0<\/p>","quoteUsername":"IgorStankovic","translations":{"Common":{"like":"Like","unlike":"Unlike"},"Forum":{"Quote":"Quote","Share":"Share"}}}">

Learn more about our cookies.<\/a>","cookiepolicy.button":"Accept cookies","cookiepolicy.button.deny":"Deny all","cookiepolicy.link":"Cookie settings","cookiepolicy.modal.title":"Cookie settings","cookiepolicy.modal.content":"We use 3 different kinds of cookies. You can choose which cookies you want to accept. We need basic cookies to make this site work, therefore these are the minimum you can select. Learn more about our cookies.<\/a>","cookiepolicy.modal.level1":"Basic
Functional","cookiepolicy.modal.level2":"Normal
Functional + analytics","cookiepolicy.modal.level3":"Complete
Functional + analytics + social media + embedded videos"}}}">
Baidu