问题

Karbon 2.3 DVP部署问题（集中PRISM VPN SITE2SITE）

  • 2021年10月14日
  • 4个答复
  • 61次观点
我
徽章

你好呀，

我们目前正在为所有Nutanix平台（全球）部署（和测试）Karbon作为K8编排平台。我的安装尝试失败了。

附带说明，我们正在使用集中棱镜控制的VPN Site2Site，我可以从遥远的Nutanix部署到达远程K8 VLAN。REACHOUT测试是使用属于K8 VLAN测试节点的小VM和通过私有CIDR（双向测试）进行的，但使用相同的加密VPN通道。

这就是我从karbon_core.out（PCVM）所拥有的：

2021-10-13T21：17：12.687Z ssh.go：153：[debug] [k8s_cluster = rgs-pa-k8阶段] 10.20.25.130:22执行：docker插件：
2021-10-13T21：17：12.825Z SSH.GO：166：[WARN] [K8S_CLUSTER = RGS-PA-K8阶段]运行CMD失败：无法运行命令：ON HOST：ON HOST（10.20.25.130:22）CMD（CMD）CMD（CMD）Docker插件检查NUTANIX）错误：“以状态1退出的过程：“错误：”错误：无插件：Nutanix \ n [] \ n”
2021-10-13T21：17：12.825Z sshutils.go：44：[错误] [k8s_cluster = rgs-pa-k8阶段]无法在10.20.25.130上运行命令[docker插件检查nutanix]：SSH：无法运行命令：在主机上（1​​0.20.25.130：22）CMD（Docker插件检查NUTANIX）错误：\“以状态1 \“”的过程退出的过程
2021-10-13T21：17：12.825Z install.go：55：[info] [info] [k8s_cluster = rgs-pa-k8阶段]无法检查是否启用了DVP，请访问：10.20.25.130未能检查DVP状态：无法使用SSH配置：无法运行命令：在主机上（1​​0.20.25.130：22）CMD（Docker插件检查NUTANIX）错误：“随着状态1的流程1”的流程
2021-10-13T21：17：12.825Z install.go：67：[info] [k8s_cluster = rgs-pa-k8阶段]主机上安装ntnx dvp：10.20.25.130
2021-10-13T21：17：12.825Z install.go：150：[debug] [k8s_cluster = rgs-pa-k8阶段]/create_plugin_from_tar.sh'/home/nutanix/docker_plugin/dvp.tar.gz''nutanix''nutanix''10 .20.1.10''10 .20.10.1.50'''''''
2021-10-13T21：17：12.993Z ssh.go：138：[debug] [k8s_cluster = rgs-pa-k8阶段]复制/etc/etc/docker-plugin-certs/key至10.20.25.130:22
2021-10-13T21：17：13.157Z ssh.go：138：[debug] [k8s_cluster = rgs-pa-k8阶段]复制/etc/etc/docker-plugin-certs/cert至10.20.25.130:22
2021-10-13T21：17：13.262Z ssh.go：138：[debug] [k8s_cluster = rgs-pa-k8阶段]复制/etc/docker-plugin-certs/ca.pem至10.20.25.130:22
2021-10-13T21：17：13.367Z ssh.go：138：[debug] [k8s_cluster = rgs-pa-k8 staging]复制/var/var/nutanix/hentanix/host_upgrade/preupgrade-docker-docker-plugin-plugin-certs.s.s.sh to 10.20。25.130：22
I1013 14:21:57.379644 1 forwarder.go:328] Forwarding MetricDataSampleList to CFS: [timestamp_usecs:1634159700000000 entity_type_name:"acs_stats_table" metric_list:"pc_cluster_uuid" metric_list:"k8s_cluster_uuid" metric_list:"last_val_karbon_version" metric_list:"last_val_cluster_name" metric_list:"last_val_cluster_prefix" metric_list:"last_val_k8s_version" metric_list:"last_val_os_flavor" metric_list:"last_val_etcd_cluster_uuid" metric_list:"last_val_etcd_members_count" metric_list:"last_val_per_etcd_cpu" metric_list:"last_val_per_etcd_mem" metric_list:"last_val_master_deploy_type" metric_list:"last_val_masters_count" metric_list:"last_val_per_master_cpu" metric_list:"last_val_per_master_mem" metric_list:"last_val_workers_count" metric_list:"last_val_per_worker_cpu" metric_list:"last_val_per_worker_mem" metric_list:"last_val_logging_state" metric_list:"last_val_logging_version" metric_list:"last_val_fluentbit_version" metric_list:"last_val_elasticsearch_version" metric_list:"last_val_elasticsearh_image“ metric_list：“ last_val_kibana_version” metric_list：“ last_val_kibana_image” metric_list：“ last_val_proxy_used” num_dimensions：2]
i1013 14：21：57.385927 1转发器。GO：339]收到了Putmetricdataarg的回复：
2021-10-13T21：22：08.925Z sshutils.go：44：[错误] [k8s_cluster = rgs-pa-k8阶段]无法运行命令[mkdir -p /docker-docker-plugin-certs && home /home /home /home /home /home /home /home /home /home /home /home /home /home /home /home /home /home /home /home /home /home /home /home /home /home /home /home /home /home /home /home /home /nutanix/docker_plugin/create_plugin_from_tar.sh '/home/nutanix/docker_plugin/dvp.tar.gz' 'nutanix' '10.20.1.10' '10.20.1.50' '' '' 'RGS-PA-K8'] on 10.20.25.15：“操作超时”
2021-10-13T21：22：08.925Z install.go：177：[error] [k8s_cluster = rgs-pa-k8阶段]错误安装ntnx dvp with err：erry：操作时间：
2021-10-13T21：22：08.925Z etcd_scale.go：40：[error] [k8s_cluster = rgs-pa-k8阶段]无法安装ntnx dvp on etcd node：10.20.25.15 with erry：失败：失败了NTNX DVP：操作计时
2021-10-13T21：22：08.925Z node_pool_create.go：337：[error] [k8s_cluster = rgs-pa-k8阶段] init vm sallback中的失败：未能部署ntnx dvp：操作定时出局。
2021-10-13T21：22：08.925Z node_pool_create.go：251：[debug] [k8s_cluster = rgs-pa-k8阶段]清理失败的VM”它的实体

对我来说，失败在哪里发生，这有点不清楚。如果不存在DVP插件，则下一个命令序列是安装的一部分，但尚不清楚安装失败的是什么。

值得注意的是，我确实尝试使用测试VM中的DVP插件，并且可以将存储容器安装为“卷”。当安装失败时，该节点属于同一K8 VLAN。

感谢您的任何反馈！！

伊戈尔

Hi there,<\/p>

We are currently deploying (and testing) Karbon as K8 orchestration platform for all our Nutanix platforms (worldwide). I have failed installation attempts from Prism Central.<\/p>

As a side note, we are using VPN Site2Site from centralised Prism Control and I can reach remote K8 VLAN from distant Nutanix deployment. The reachout tests were done with small VM belonging\u00a0the the K8 VLAN testing node and via private CIDR (bidirectional tests), but utilizing the same encrypted VPN channel.<\/p>

This is what I\u2019m having from karbon_core.out (PCVM):<\/p>

2021-10-13T21:17:12.687Z ssh.go:153: [DEBUG] [k8s_cluster=RGS-PA-K8-STAGING] On 10.20.25.130:22 executing: docker plugin inspect nutanix
2021-10-13T21:17:12.825Z ssh.go:166: [WARN] [k8s_cluster=RGS-PA-K8-STAGING] Run cmd failed: Failed to run command: on host(10.20.25.130:22) cmd(docker plugin inspect nutanix) error: \"Process exited with status 1\", output: \"Error: No such plugin: nutanix\\n[]\\n\"
2021-10-13T21:17:12.825Z sshutils.go:44: [ERROR] [k8s_cluster=RGS-PA-K8-STAGING] Unable to run commands [docker plugin inspect nutanix] on 10.20.25.130: \"Failed to configure with SSH: Failed to run command: on host(10.20.25.130:22) cmd(docker plugin inspect nutanix) error: \\\"Process exited with status 1\\\"//www.jhbzcj.com/next/installation-configuration-23/\"
2021-10-13T21:17:12.825Z install.go:55: [INFO] [k8s_cluster=RGS-PA-K8-STAGING] Unable to check if the dvp is enabled on: 10.20.25.130 failed to check the dvp status: Failed to configure with SSH: Failed to run command: on host(10.20.25.130:22) cmd(docker plugin inspect nutanix) error: \"Process exited with status 1\"
2021-10-13T21:17:12.825Z install.go:67: [INFO] [k8s_cluster=RGS-PA-K8-STAGING] Installing ntnx dvp on host: 10.20.25.130
2021-10-13T21:17:12.825Z install.go:150: [DEBUG] [k8s_cluster=RGS-PA-K8-STAGING] Command: mkdir -p \/etc\/docker-plugin-certs && \/home\/nutanix\/docker_plugin\/create_plugin_from_tar.sh '\/home\/nutanix\/docker_plugin\/dvp.tar.gz' 'nutanix' '10.20.1.10' '10.20.1.50' '' '' 'RGS-PA-K8'
2021-10-13T21:17:12.993Z ssh.go:138: [DEBUG] [k8s_cluster=RGS-PA-K8-STAGING] Copying \/etc\/docker-plugin-certs\/key to 10.20.25.130:22
2021-10-13T21:17:13.157Z ssh.go:138: [DEBUG] [k8s_cluster=RGS-PA-K8-STAGING] Copying \/etc\/docker-plugin-certs\/cert to 10.20.25.130:22
2021-10-13T21:17:13.262Z ssh.go:138: [DEBUG] [k8s_cluster=RGS-PA-K8-STAGING] Copying \/etc\/docker-plugin-certs\/ca.pem to 10.20.25.130:22
2021-10-13T21:17:13.367Z ssh.go:138: [DEBUG] [k8s_cluster=RGS-PA-K8-STAGING] Copying \/var\/nutanix\/host_upgrade\/preupgrade-docker-plugin-certs.sh to 10.20.25.130:22
I1013 14:21:57.379644 \u00a0 \u00a0 \u00a0 1 forwarder.go:328] Forwarding MetricDataSampleList to CFS: [timestamp_usecs:1634159700000000 entity_type_name:\"acs_stats_table\" metric_list:\"pc_cluster_uuid\" metric_list:\"k8s_cluster_uuid\" metric_list:\"last_val_karbon_version\" metric_list:\"last_val_cluster_name\" metric_list:\"last_val_cluster_prefix\" metric_list:\"last_val_k8s_version\" metric_list:\"last_val_os_flavor\" metric_list:\"last_val_etcd_cluster_uuid\" metric_list:\"last_val_etcd_members_count\" metric_list:\"last_val_per_etcd_cpu\" metric_list:\"last_val_per_etcd_mem\" metric_list:\"last_val_master_deploy_type\" metric_list:\"last_val_masters_count\" metric_list:\"last_val_per_master_cpu\" metric_list:\"last_val_per_master_mem\" metric_list:\"last_val_workers_count\" metric_list:\"last_val_per_worker_cpu\" metric_list:\"last_val_per_worker_mem\" metric_list:\"last_val_logging_state\" metric_list:\"last_val_logging_version\" metric_list:\"last_val_fluentbit_version\" metric_list:\"last_val_elasticsearch_version\" metric_list:\"last_val_elasticsearh_image\" metric_list:\"last_val_kibana_version\" metric_list:\"last_val_kibana_image\" metric_list:\"last_val_proxy_used\" num_dimensions:2 ]
I1013 14:21:57.385927 \u00a0 \u00a0 \u00a0 1 forwarder.go:339] Received response from PutMetricDataArg:\u00a0
2021-10-13T21:22:08.925Z sshutils.go:44: [ERROR] [k8s_cluster=RGS-PA-K8-STAGING] Unable to run commands [mkdir -p \/etc\/docker-plugin-certs && \/home\/nutanix\/docker_plugin\/create_plugin_from_tar.sh '\/home\/nutanix\/docker_plugin\/dvp.tar.gz' 'nutanix' '10.20.1.10' '10.20.1.50' '' '' 'RGS-PA-K8'] on 10.20.25.15: \"Operation timed out\"
2021-10-13T21:22:08.925Z install.go:177: [ERROR] [k8s_cluster=RGS-PA-K8-STAGING] Error installing ntnx dvp with err: Operation timed out []
2021-10-13T21:22:08.925Z etcd_scale.go:40: [ERROR] [k8s_cluster=RGS-PA-K8-STAGING] Failed to install the ntnx dvp on etcd node: 10.20.25.15 with err: failed to deploy the ntnx dvp: Operation timed out
2021-10-13T21:22:08.925Z node_pool_create.go:337: [ERROR] [k8s_cluster=RGS-PA-K8-STAGING] Failure in init vm callback: failed to deploy the ntnx dvp: Operation timed out
2021-10-13T21:22:08.925Z node_pool_create.go:251: [DEBUG] [k8s_cluster=RGS-PA-K8-STAGING] Cleaning up failed VM \"rgs-pa-k8-staging-d2d032-etcd-2\" and its entities<\/p><\/blockquote>

\u00a0<\/p>

It\u2019s a bit unclear to me where the failure happened. If the\u00a0\u00a0DVP plug-in is not present, the next command sequence is part of the installation, but its not clear what failed the installation.<\/p>

As a note, I did try to use DVP plugin from a testing VM and I could mount the storage container as a volume OK. That node is belonging to the same K8 VLAN when installation failed.<\/p>

Thanks for any feedback!!<\/p>

Igor<\/p>

\u00a0<\/p>

\u00a0<\/p>

\u00a0<\/p>","quoteUsername":"IgorStankovic","translations":{"Common":{"like":"Like","unlike":"Unlike"},"Forum":{"Quote":"Quote","Share":"Share"}}}">

4个答复

乔森纳克斯
秩
UserLevel 4
徽章 +5

嗨，伊戈尔，

您是否在PC和远程K8S网络之间有任何防火墙规则，它不从您连接到测试VM的计算机上应用以安装DVP？

另外，如果可能的话，我建议在本地VLAN中进行测试，以确保部署成功并集中在PC和远程站点之间的连接中。

何塞·戈麦斯（Jose Gomez）|技术营销工程师|jose.gomez@nutanix.com |@pipoe2H
Hi Igor,<\/p>

Do you have any firewall rules between PC and the remote K8s network that its not applying from the computer you are connecting to the test VM to install the DVP?<\/p>

Also, I\u2019d suggest if possible, to run a test in a local VLAN to make sure the deployment is successful and focus then in the connection between PC and the remote site.<\/p>","quoteUsername":"JoseNutanix","translations":{"Common":{"like":"Like","unlike":"Unlike"},"Forum":{"Quote":"Quote","Share":"Share"}}}">

我
徽章

嗨，何塞！非常感谢您的到来。

我在这里找到了罪魁祸首。从本质上讲，我们正在运行集中的棱镜控制，并且它与基于Site2Site FW的VPN隧道联系到其他Nutanix平台 - 因此管理是中心的。

但是，我不得不在加密的S2S通道的接口之间完全豁免完全私有的CIDR，尤其是在Nutanix接口之间进行路由。返回的网络数据包没有原始的源IP地址，因此发生了故障，现在已修复。

奇怪的是，您是否打算很快为Karbon提供支持Kubernetes 2.1+？

因此，Karbon部署几乎发展到最后，现在我对印花布有不同的问题，因此再次失败了：

2021-10-16T11：51：32.407Z calico.go：552：[错误] [K8S_CLUSTER = RGS-PA-K8阶段]无法验证Calico Addon
2021-10-16T11：51：32.407Z k8s_deploy.go：1478：[错误] [k8s_cluster = rgs-pa-k8阶段]无法部署卡利科/绒布：失败的卡利科：失败：无法验证Calico：[操作时间：出局：期望5个节点在Kube-System名称空间中运行Calico-Node Daemon Pod。当前运行：2，操作定时：期望1个在Kube-System名称空间中部署Calico-kube-controllers的可用副本。当前运行：0]
2021-10-16T11：51：32.407Z k8s_deploy.go：155：[错误] [k8s_cluster = rgs-pa-k8阶段]无法部署群集插件：无法部署k8s cluster addon：无法部署calico：失败calico：失败验证印花布：[操作定时：期望5个节点在Kube-System名称空间中运行Calico-Node Daemon Pod。当前运行：2，操作定时：期望1个在Kube-System名称空间中部署Calico-kube-controllers的可用副本。当前运行：0]
2021-10-16T11:51:32.432Z k8s_lib_deploy_task.go:112: [ERROR] [k8s_cluster=RGS-PA-K8-STAGING] failed to deploy K8s cluster: failed to deploy cluster addons: failed to deploy K8s cluster addon: Failed要部署印花布：无法验证卡利科：[操作定时：期望5个节点在Kube-System名称空间中运行Calico-Node Daemon Pod。当前运行：2，操作定时：期望1个在Kube-System名称空间中部署Calico-kube-controllers的可用副本。当前运行：0]
2021-10-16T11：51：32.432Z k8s_lib_deploy_task.go：78：[info] [info] [k8s_cluster = rgs-pa-k8阶段]代币恢复pofresher
2021-10-16T11：51：32.444Z deploy_k8s_task.go：364：[error] [k8s_cluster = rgs-pa-k8阶段] clustr rgs-pa-k8阶段：失败k8s群集：无法部署K8S群集插件：无法部署印花布：无法验证卡利科：[操作计时：期望5个节点在kube-system namepace中运行Calico-node Daemon Pod。当前运行：2，操作定时：期望1个在Kube-System名称空间中部署Calico-kube-controllers的可用副本。当前运行：0]

Hi Jose!\u00a0many thanks for the reach.<\/p>

I found the culprit\u00a0here. In essence, we are running centralised Prism control and it\u2019s linked with site2site\u00a0 FW based VPN tunnels to other Nutanix platforms - so that management is central.<\/p>

However, I had to NAT exempt completely private CIDR between the interfaces for the encrypted S2S channel and in particular for inter routing between the Nutanix interfaces. The returned network packets didn\u2019t have originated source IP addresses, so there was a breakdown and this is now fixed.<\/p>

Curiously, are you planning to support Kubernetes 2.1+ anytime soon for Karbon?<\/p>

So,\u00a0Karbon deployment progressed\u00a0almost to the end, and now I have different problem with Calico, so failed again:<\/p>

\u00a0<\/p>

2021-10-16T11:51:32.407Z calico.go:552: [ERROR] [k8s_cluster=RGS-PA-K8-STAGING] Failed to verify calico addon
2021-10-16T11:51:32.407Z k8s_deploy.go:1478: [ERROR] [k8s_cluster=RGS-PA-K8-STAGING] Failed to deploy calico\/flannel: Failed to deploy calico: Failed to verify calico: [ Operation timed out: expecting 5 nodes to be running calico-node daemon pod in kube-system namespace. Currently running: 2, Operation timed out: expecting 1 available replica of calico-kube-controllers deployment in kube-system namespace. Currently running: 0 ]
2021-10-16T11:51:32.407Z k8s_deploy.go:155: [ERROR] [k8s_cluster=RGS-PA-K8-STAGING] failed to deploy cluster addons: failed to deploy K8s cluster addon: Failed to deploy calico: Failed to verify calico: [ Operation timed out: expecting 5 nodes to be running calico-node daemon pod in kube-system namespace. Currently running: 2, Operation timed out: expecting 1 available replica of calico-kube-controllers deployment in kube-system namespace. Currently running: 0 ]
2021-10-16T11:51:32.432Z k8s_lib_deploy_task.go:112: [ERROR] [k8s_cluster=RGS-PA-K8-STAGING] failed to deploy K8s cluster: failed to deploy cluster addons: failed to deploy K8s cluster addon: Failed to deploy calico: Failed to verify calico: [ Operation timed out: expecting 5 nodes to be running calico-node daemon pod in kube-system namespace. Currently running: 2, Operation timed out: expecting 1 available replica of calico-kube-controllers deployment in kube-system namespace. Currently running: 0 ]
2021-10-16T11:51:32.432Z k8s_lib_deploy_task.go:78: [INFO] [k8s_cluster=RGS-PA-K8-STAGING] token refresher received stopRefresh
2021-10-16T11:51:32.444Z deploy_k8s_task.go:364: [ERROR] [k8s_cluster=RGS-PA-K8-STAGING] Cluster RGS-PA-K8-STAGING:failed to deploy K8s cluster: failed to deploy cluster addons: failed to deploy K8s cluster addon: Failed to deploy calico: Failed to verify calico: [ Operation timed out: expecting 5 nodes to be running calico-node daemon pod in kube-system namespace. Currently running: 2, Operation timed out: expecting 1 available replica of calico-kube-controllers deployment in kube-system namespace. Currently running: 0 ]
\u00a0<\/p><\/blockquote>","quoteUsername":"IgorStankovic","translations":{"Common":{"like":"Like","unlike":"Unlike"},"Forum":{"Quote":"Quote","Share":"Share"}}}">

乔森纳克斯
秩
UserLevel 4
徽章 +5

嗨，伊戈尔，

我想你的意思是kubernetes 1.21+。有计划支持这一点。如果您想了解有关时间表的更多信息，请与您的帐户团队联系。

关于印花布问题，让我们继续您打开的另一篇文章。

何塞·戈麦斯（Jose Gomez）|技术营销工程师|jose.gomez@nutanix.com |@pipoe2H
Hi Igor,<\/p>

I guess you mean Kubernetes 1.21+. There is plan for supporting this. If you want to know more about timeframe, please reach out to your account team.\u00a0<\/p>

About Calico issue, let\u2019s continue in the other post you opened.<\/p>","quoteUsername":"JoseNutanix","translations":{"Common":{"like":"Like","unlike":"Unlike"},"Forum":{"Quote":"Quote","Share":"Share"}}}">

我
徽章

是的，1.21+对不起，这是漫长的一天：liticle_smile：

我现在将标记为解决。谢谢！！

Yes, 1.21+ sorry it was long day\u00a0\":slight_smile:\"<\/p>

I will mark this now as resolved. Thanks!!<\/p>","quoteUsername":"IgorStankovic","translations":{"Common":{"like":"Like","unlike":"Unlike"},"Forum":{"Quote":"Quote","Share":"Share"}}}">

回复


Baidu