解决了

ioError [错误28]设备上没有剩余的空间

  • 2021年7月13日
  • 4个答复
  • 778意见
j

大家好，

我们集群的一个节点突然断开了连接。我最初无法ping，但是在重新启动节点并检查是否启用了维护模式之后。连通性恢复了，所有IP都可以彼此ping。

即使这样，即使所有ping都很好，群集也无法识别先前断开的节点。尝试重新启动群集和本地CVM，但以下错误出现：

nutanix@ntnx-a-cvm：192.168.50.182：〜$ cluster状态
2021-07-13 07：34：46,652Z警告Genesis_utils.py:1304未能到达创世纪启动的节点。重试...（打Ctrl-C中止
Trackback（最近的最新电话）：
文件“/usr/lib64/python2.7/logging/__init__.py”，第875行，在EMIT中
self.flush（）
文件“/usr/lib64/python2.7/logging/__init__.py”，第835行，
self.stream.flush（）
IOERROR：[ERRNO 28]设备上没有剩余的空间
从文件log.py登录，第191行
2021-07-13 07：34：47,654Z警告Genesis_utils.py:1304未能到达创世纪启动的节点。重试...（打Ctrl-C中止
Trackback（最近的最新电话）：
文件“/usr/lib64/python2.7/logging/__init__.py”，第875行，在EMIT中
self.flush（）
文件“/usr/lib64/python2.7/logging/__init__.py”，第835行，
self.stream.flush（）
IOERROR：[ERRNO 28]设备上没有剩余的空间
从文件log.py登录，第191行
2021-07-13 07：34：48,656Z警告Genesis_utils.py:1304未能到达创世纪启动的节点。重试...（打Ctrl-C中止
Trackback（最近的最新电话）：
文件“/usr/lib64/python2.7/logging/__init__.py”，第875行，在EMIT中
self.flush（）
文件“/usr/lib64/python2.7/logging/__init__.py”，第835行，
self.stream.flush（）
IOERROR：[ERRNO 28]设备上没有剩余的空间
从文件log.py登录，第191行
2021-07-13 07：34：49,659Z警告Genesis_utils.py:1304未能到达创世纪启动的节点。重试...（打Ctrl-C中止
Trackback（最近的最新电话）：
文件“/usr/lib64/python2.7/logging/__init__.py”，第875行，在EMIT中
self.flush（）
文件“/usr/lib64/python2.7/logging/__init__.py”，第835行，
self.stream.flush（）
IOERROR：[ERRNO 28]设备上没有剩余的空间
从文件log.py登录，第191行

有人遇到过吗？不幸的是，几个月前的支持任期结束了。

图标

最好的答案拉阿吉2021年7月22日，21：54

Hi John,<\/p>

Let me respond to your query.<\/p>

If \/home is full, we will need to carefully check and clean files from approved directories on the CVM<\/p>

Could you please ssh into CVM (192.168.50.182) and execute the following command?<\/p>

du -h ~\/data\/logs<\/span><\/p>

This shows the space utilized by the logs in the CVM\u00a0<\/p>

And then please refer to this KB to clean up space: [AOS Only] What to do when \/home partition or \/home\/nutanix directory on a Controller VM (CVM) is full<\/a><\/p>

查看原件
Hi Everyone,<\/p>

One of the nodes of our cluster suddenly got disconnected. I was not able to ping initially, but after restarting the node and checking if\u00a0maintenance mode is enabled. Connectivity was regained and all the IPs can be pinged from each other.\u00a0<\/p>

\u00a0<\/p>

Even so, the cluster can\u2019t recognize the previously disconnected node even when all pings are good. Tried restarting the cluster and the local CVM but the error below appears:<\/p>

nutanix@NTNX-A-CVM:192.168.50.182:~$ cluster status
2021-07-13 07:34:46,652Z WARNING genesis_utils.py:1304 Failed to reach a node where Genesis is up. Retrying... (Hit Ctrl-C to abort
Traceback (most recent call last):
\u00a0 File \"\/usr\/lib64\/python2.7\/logging\/__init__.py\", line 875, in emit
\u00a0 \u00a0 self.flush()
\u00a0 File \"\/usr\/lib64\/python2.7\/logging\/__init__.py\", line 835, in flush
\u00a0 \u00a0 self.stream.flush()
IOError: [Errno 28] No space left on device
Logged from file log.py, line 191
2021-07-13 07:34:47,654Z WARNING genesis_utils.py:1304 Failed to reach a node where Genesis is up. Retrying... (Hit Ctrl-C to abort
Traceback (most recent call last):
\u00a0 File \"\/usr\/lib64\/python2.7\/logging\/__init__.py\", line 875, in emit
\u00a0 \u00a0 self.flush()
\u00a0 File \"\/usr\/lib64\/python2.7\/logging\/__init__.py\", line 835, in flush
\u00a0 \u00a0 self.stream.flush()
IOError: [Errno 28] No space left on device
Logged from file log.py, line 191
2021-07-13 07:34:48,656Z WARNING genesis_utils.py:1304 Failed to reach a node where Genesis is up. Retrying... (Hit Ctrl-C to abort
Traceback (most recent call last):
\u00a0 File \"\/usr\/lib64\/python2.7\/logging\/__init__.py\", line 875, in emit
\u00a0 \u00a0 self.flush()
\u00a0 File \"\/usr\/lib64\/python2.7\/logging\/__init__.py\", line 835, in flush
\u00a0 \u00a0 self.stream.flush()
IOError: [Errno 28] No space left on device
Logged from file log.py, line 191
2021-07-13 07:34:49,659Z WARNING genesis_utils.py:1304 Failed to reach a node where Genesis is up. Retrying... (Hit Ctrl-C to abort
Traceback (most recent call last):
\u00a0 File \"\/usr\/lib64\/python2.7\/logging\/__init__.py\", line 875, in emit
\u00a0 \u00a0 self.flush()
\u00a0 File \"\/usr\/lib64\/python2.7\/logging\/__init__.py\", line 835, in flush
\u00a0 \u00a0 self.stream.flush()
IOError: [Errno 28] No space left on device
Logged from file log.py, line 191<\/em>
\u00a0<\/p>

Has anyone encountered this? Unfortunately, the support term ended months ago.<\/p>","quoteUsername":"John Renz","translations":{"Common":{"like":"Like","unlike":"Unlike"},"Forum":{"Quote":"Quote","Share":"Share"}}}">

该主题已关闭以供评论

4个答复

H
秩
Userlevel 1
徽章 +3

你好 @John Renz请验证：

  1. 从IPMI查看任何硬件问题
  2. 从PE那里，您是否看到有关CVM/主机的任何警报？从PE中的硬件图中，您是否在主机上看到任何未安装的磁盘/错误磁盘/错误？
  3. 请检查portal.nutanix.com/kb/8086与该问题相匹配。

谢谢，最好

Hello @John Renz<\/user-mention>\u00a0please verify:<\/p>
  1. Check from IPMI for any hardware issues<\/li>\t
  2. From PE, do you see any alerts with regards to the CVM\/host concerned? From hardware diagram in the PE, do you see any unmounted disks\/bad disks\/errors on the host?<\/li>\t
  3. Please check portal.nutanix.com\/kb\/8086 is a match to the issue.\u00a0<\/li><\/ol>

    Thanks and Best<\/p>","quoteUsername":"Haritha Andal","translations":{"Common":{"like":"Like","unlike":"Unlike"},"Forum":{"Quote":"Quote","Share":"Share"}}}">

j

你好 @Haritha Andal，，，，

棱镜有一个警报：

手术失败。原因：Prechecks失败：未能在192.168.50.182到达LCM。请检查KB 7781在节点192.168.50.182中预计找到远程版本，而2.4.2.25804。这可能是一个缓存问题。请确保清除所有本地缓存​​，并等待几分钟，以便在重试之前任何远程缓存都不会无效。请检查KB 7784

我尝试了KB 7784和7781，但无济于事。

我尝试了您给我的KB，我看到 /家为100％

================== 192.168.50.182 =======================
使用的文件系统大小可用使用％安装在
DEVTMPFS 9.8G 0 9.8G 0％ /dev
TMPFS 512M 0 512M 0％ /dev /shm
TMPFS 9.9g 624K 9.9g 1％ /跑步
TMPFS 9.9g 0 9.9g 0％/sys/fs/cgroup
/dev /sda2 9.8g 4.0g 5.8g 41％ /
/dev /loop0 240m 2.1m 222m 1％ /TMP
/dev /sda3 40g 39g 0 100％ /家庭
tmpfs 2.0g 0 2.0g 0％/run/user/1000
/dev/sdb1 3.6t 1.3g 3.6t 1％/home/nutanix/data/stargate-store/disk/zc1b0wky
/dev/sdc1 3.6t 1.4g 3.6t 1％/home/nutanix/data/stargate-storage/disk/zc1arfyn
/dev/sda4 1.7t 269g 1.4t 16％/home/nutanix/data/stargate-storage/disk/s47pne0m400683

试图删除其内容，但它说：


nutanix@ntnx-19fm6j260111-a-cvm：192.168.50.182：〜/Foundation $ rm/isos/*
/usr/bin/rm：无法删除“/isos/*”：没有此类文件或目录

我的语法有问题吗？

Hi @Haritha Andal<\/user-mention>,\u00a0<\/p>

There\u2019s an alert in Prism:<\/p>

Operation failed. Reason: Prechecks failed: Failed to reach lcm on 192.168.50.182. Please check KB 7781 Found remote version None while 2.4.2.25804 was expected on node 192.168.50.182. This may be a caching issue. Please ensure all local caches are cleared and wait a few minutes for any remote caches to get invalidated before retrying. Please check KB 7784<\/p>

I\u2019ve tried KB 7784 and 7781 but to no avail.<\/p>

I\u2019ve tried the KB you gave me and I saw that \/home is at 100%<\/p>

================== 192.168.50.182 =================
Filesystem \u00a0 \u00a0 \u00a0Size \u00a0Used Avail Use% Mounted on
devtmpfs \u00a0 \u00a0 \u00a0 \u00a09.8G \u00a0 \u00a0 0 \u00a09.8G \u00a0 0% \/dev
tmpfs \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 512M \u00a0 \u00a0 0 \u00a0512M \u00a0 0% \/dev\/shm
tmpfs \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 9.9G \u00a0624K \u00a09.9G \u00a0 1% \/run
tmpfs \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 9.9G \u00a0 \u00a0 0 \u00a09.9G \u00a0 0% \/sys\/fs\/cgroup
\/dev\/sda2 \u00a0 \u00a0 \u00a0 9.8G \u00a04.0G \u00a05.8G \u00a041% \/
\/dev\/loop0 \u00a0 \u00a0 \u00a0240M \u00a02.1M \u00a0222M \u00a0 1% \/tmp
\/dev\/sda3 \u00a0 \u00a0 \u00a0 \u00a040G \u00a0 39G \u00a0 \u00a0 0 100% \/home
tmpfs \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 2.0G \u00a0 \u00a0 0 \u00a02.0G \u00a0 0% \/run\/user\/1000
\/dev\/sdb1 \u00a0 \u00a0 \u00a0 3.6T \u00a01.3G \u00a03.6T \u00a0 1% \/home\/nutanix\/data\/stargate-storage\/disks\/ZC1B0WKY
\/dev\/sdc1 \u00a0 \u00a0 \u00a0 3.6T \u00a01.4G \u00a03.6T \u00a0 1% \/home\/nutanix\/data\/stargate-storage\/disks\/ZC1ARFYN
\/dev\/sda4 \u00a0 \u00a0 \u00a0 1.7T \u00a0269G \u00a01.4T \u00a016% \/home\/nutanix\/data\/stargate-storage\/disks\/S47PNE0M400683
\u00a0<\/p>

Tried deleting its contents\u00a0but it says:<\/p>


nutanix@NTNX-19FM6J260111-A-CVM:192.168.50.182:~\/foundation$ rm \/isos\/*
\/usr\/bin\/rm: cannot remove \u2018\/isos\/*\u2019: No such file or directory<\/p>

\u00a0<\/p>

Is there something wrong with my syntax ?
\u00a0<\/p>","quoteUsername":"John Renz","translations":{"Common":{"like":"Like","unlike":"Unlike"},"Forum":{"Quote":"Quote","Share":"Share"}}}">

r
秩
UserLevel 2
徽章 +4

你好，约翰，

让我回复您的查询。

如果 /家庭已满，我们将需要仔细检查和清洁CVM上批准的目录的文件

您能否请SSH进入CVM（192.168.50.182）并执行以下命令？

du -h〜/数据/日志

这显示了CVM中日志使用的空间

然后，请参考此KB以清理空间：[仅AOS]在控制器VM（CVM）上的 /家庭分区或 /家庭 /nutanix目录时该怎么办

  • 不使用RM -RF在任何情况下，除非有说明。这将导致数据丢失方案。

如果您仍然需要帮助，请与Nutanix打开支持票，以解决问题

拉阿吉
Hi John,<\/p>

Let me respond to your query.<\/p>

If \/home is full, we will need to carefully check and clean files from approved directories on the CVM<\/p>

Could you please ssh into CVM (192.168.50.182) and execute the following command?<\/p>

du -h ~\/data\/logs<\/span><\/p>

This shows the space utilized by the logs in the CVM\u00a0<\/p>

And then please refer to this KB to clean up space: [AOS Only] What to do when \/home partition or \/home\/nutanix directory on a Controller VM (CVM) is full<\/a><\/p>

j

你好 @raaji，，，，

谢谢！

我昨天实际上尝试过。方法2似乎最有效。

我刚刚删除了一些文件/home/nutanix/data/如KB中所示

足以将利用率从100％降低到97％。开始集群后，利用率降至约30％。

Hi\u00a0@raaji<\/user-mention>,<\/p>

Thanks!\u00a0<\/p>

I actually tried this yesterday. Method 2 seems to work best.\u00a0<\/p>

I just deleted some files at\u00a0\/home\/nutanix\/data\/\u00a0<\/em>as indicated in the KB<\/p>

Just enough to reduce the utilization from 100% to 97%. After starting the cluster, the utilization came down to around 30%.<\/p>","quoteUsername":"John Renz","translations":{"Common":{"like":"Like","unlike":"Unlike"},"Forum":{"Quote":"Quote","Share":"Share"}}}">

Learn more about our cookies.<\/a>","cookiepolicy.button":"Accept cookies","cookiepolicy.button.deny":"Deny all","cookiepolicy.link":"Cookie settings","cookiepolicy.modal.title":"Cookie settings","cookiepolicy.modal.content":"We use 3 different kinds of cookies. You can choose which cookies you want to accept. We need basic cookies to make this site work, therefore these are the minimum you can select. Learn more about our cookies.<\/a>","cookiepolicy.modal.level1":"Basic
Functional","cookiepolicy.modal.level2":"Normal
Functional + analytics","cookiepolicy.modal.level3":"Complete
Functional + analytics + social media + embedded videos"}}}">
Baidu