解决了

IOError [Error 28]设备上没有剩余空间

  • 2021年7月13日
  • 4回复
  • 250的浏览量
J

大家好！

集群中的一个节点突然断开。我最初无法ping通，但在重新启动节点并检查维护模式是否启用后。网络连接恢复，所有ip都能ping通。

即便如此，即使所有的ping都正常，集群也无法识别之前断开的节点。尝试重新启动集群和本地CVM，但出现以下错误:

nutanix@NTNX-A-CVM: 192.168.50.182: ~ $集群状态
2019-07-13 07:34:46,652Z WARNING genesis_utils.py:1304无法到达Genesis节点。重试…(按Ctrl-C终止
回溯(最近的电话last):
文件"/usr/lib64/python2.7/logging/__init__.py"，第875行，在emit
self.flush ()
文件"/usr/lib64/python2.7/logging/__init__.py"，第835行，齐平
self.stream.flush ()
IOError: [Errno 28]设备上没有剩余空间
从log.py文件中记录，第191行
2019-07-13 07:34:47 7654z WARNING genesis_utils.py:1304无法到达Genesis节点。重试…(按Ctrl-C终止
回溯(最近的电话last):
文件"/usr/lib64/python2.7/logging/__init__.py"，第875行，在emit
self.flush ()
文件"/usr/lib64/python2.7/logging/__init__.py"，第835行，齐平
self.stream.flush ()
IOError: [Errno 28]设备上没有剩余空间
从log.py文件中记录，第191行
2019-07-13 07:34:48,656Z WARNING genesis_utils.py:1304无法到达Genesis节点。重试…(按Ctrl-C终止
回溯(最近的电话last):
文件"/usr/lib64/python2.7/logging/__init__.py"，第875行，在emit
self.flush ()
文件"/usr/lib64/python2.7/logging/__init__.py"，第835行，齐平
self.stream.flush ()
IOError: [Errno 28]设备上没有剩余空间
从log.py文件中记录，第191行
2019-07-13 07:34:49 659z WARNING genesis_utils.py:1304无法到达Genesis节点。重试…(按Ctrl-C终止
回溯(最近的电话last):
文件"/usr/lib64/python2.7/logging/__init__.py"，第875行，在emit
self.flush ()
文件"/usr/lib64/python2.7/logging/__init__.py"，第835行，齐平
self.stream.flush ()
IOError: [Errno 28]设备上没有剩余空间
从log.py文件中记录，第191行

有人遇到过吗?不幸的是，支持期限几个月前就结束了。

图标

最佳答案raaji2021年7月22日21:54

Hi John,<\/p>

Let me respond to your query.<\/p>

If \/home is full, we will need to carefully check and clean files from approved directories on the CVM<\/p>

Could you please ssh into CVM (192.168.50.182) and execute the following command?<\/p>

du -h ~\/data\/logs<\/span><\/p>

This shows the space utilized by the logs in the CVM\u00a0<\/p>

And then please refer to this KB to clean up space: [AOS Only] What to do when \/home partition or \/home\/nutanix directory on a Controller VM (CVM) is full<\/a><\/p>

查看原始
Hi Everyone,<\/p>

One of the nodes of our cluster suddenly got disconnected. I was not able to ping initially, but after restarting the node and checking if\u00a0maintenance mode is enabled. Connectivity was regained and all the IPs can be pinged from each other.\u00a0<\/p>

\u00a0<\/p>

Even so, the cluster can\u2019t recognize the previously disconnected node even when all pings are good. Tried restarting the cluster and the local CVM but the error below appears:<\/p>

nutanix@NTNX-A-CVM:192.168.50.182:~$ cluster status
2021-07-13 07:34:46,652Z WARNING genesis_utils.py:1304 Failed to reach a node where Genesis is up. Retrying... (Hit Ctrl-C to abort
Traceback (most recent call last):
\u00a0 File \"\/usr\/lib64\/python2.7\/logging\/__init__.py\", line 875, in emit
\u00a0 \u00a0 self.flush()
\u00a0 File \"\/usr\/lib64\/python2.7\/logging\/__init__.py\", line 835, in flush
\u00a0 \u00a0 self.stream.flush()
IOError: [Errno 28] No space left on device
Logged from file log.py, line 191
2021-07-13 07:34:47,654Z WARNING genesis_utils.py:1304 Failed to reach a node where Genesis is up. Retrying... (Hit Ctrl-C to abort
Traceback (most recent call last):
\u00a0 File \"\/usr\/lib64\/python2.7\/logging\/__init__.py\", line 875, in emit
\u00a0 \u00a0 self.flush()
\u00a0 File \"\/usr\/lib64\/python2.7\/logging\/__init__.py\", line 835, in flush
\u00a0 \u00a0 self.stream.flush()
IOError: [Errno 28] No space left on device
Logged from file log.py, line 191
2021-07-13 07:34:48,656Z WARNING genesis_utils.py:1304 Failed to reach a node where Genesis is up. Retrying... (Hit Ctrl-C to abort
Traceback (most recent call last):
\u00a0 File \"\/usr\/lib64\/python2.7\/logging\/__init__.py\", line 875, in emit
\u00a0 \u00a0 self.flush()
\u00a0 File \"\/usr\/lib64\/python2.7\/logging\/__init__.py\", line 835, in flush
\u00a0 \u00a0 self.stream.flush()
IOError: [Errno 28] No space left on device
Logged from file log.py, line 191
2021-07-13 07:34:49,659Z WARNING genesis_utils.py:1304 Failed to reach a node where Genesis is up. Retrying... (Hit Ctrl-C to abort
Traceback (most recent call last):
\u00a0 File \"\/usr\/lib64\/python2.7\/logging\/__init__.py\", line 875, in emit
\u00a0 \u00a0 self.flush()
\u00a0 File \"\/usr\/lib64\/python2.7\/logging\/__init__.py\", line 835, in flush
\u00a0 \u00a0 self.stream.flush()
IOError: [Errno 28] No space left on device
Logged from file log.py, line 191<\/em>
\u00a0<\/p>

Has anyone encountered this? Unfortunately, the support term ended months ago.<\/p>","quoteUsername":"John Renz","translations":{"Common":{"like":"Like","unlike":"Unlike"},"Forum":{"Quote":"Quote","Share":"Share"}}}">

本主题已关闭供评论

4回复

H
排名
徽章 ＋３

你好 @John还建议请核实:

  1. 从IPMI检查任何硬件问题
  2. 从PE，你看到任何关于CVM/主机的警报吗?从PE的硬件图中，您看到主机上有未挂载的磁盘/坏磁盘/错误吗?
  3. 请查看portal.nutanix.com/kb/8086是否匹配问题。

感谢和最好的

Hello @John Renz<\/user-mention>\u00a0please verify:<\/p>
  1. Check from IPMI for any hardware issues<\/li>\t
  2. From PE, do you see any alerts with regards to the CVM\/host concerned? From hardware diagram in the PE, do you see any unmounted disks\/bad disks\/errors on the host?<\/li>\t
  3. Please check portal.nutanix.com\/kb\/8086 is a match to the issue.\u00a0<\/li><\/ol>

    Thanks and Best<\/p>","quoteUsername":"Haritha Andal","translations":{"Common":{"like":"Like","unlike":"Unlike"},"Forum":{"Quote":"Quote","Share":"Share"}}}">

J

嗨 @Haritha安达，

棱镜城有一个警告:

操作失败。原因:Prechecks failed: 192.168.50.182访问lcm失败。请检查KB 7781发现远程版本无，而2.4.2.25804预期节点192.168.50.182。这可能是缓存问题。请确保所有本地缓存都已清除，并等待几分钟，等待远程缓存失效后再重试。请查阅KB 7784

我试过KB 7784和7781，但没有用。

我试过你给我的KB，我看到/家是100%

================== 192.168.50.182 =================
Filesystem Size Used Avail Use% Mounted on
devtmpfs 9.8G 0 9.8G 0% /dev
tmpfs 512M 0 512M 0% /dev/shm
tmpfs 9.9G 624K 9.9G 1% /运行
/sys/fs/cgroup . tmpfs 9.9G 0 9.9G 0%
/dev/sda2 9.8G 4.0G 5.8G 41% /
/dev/loop0 240M 2.1M 222M 1% /tmp
/dev/sda3 40G 39G 0 100% /home
tmpfs 2.0G 0 2.0G 0% /run/user/1000
/dev/sdb1 3.6T 1.3G 3.6T 1% /home/nutanix/data/stargate-storage/disks/ZC1B0WKY
/dev/sdc1 3.6T 1.4G 3.6T 1% /home/nutanix/data/stargate-storage/disks/ZC1ARFYN
/dev/sda4 1.7T 269G 1.4T 16% /home/nutanix/data/stargate-storage/disks/S47PNE0M400683

试图删除它的内容，但它说:


nutanix@NTNX-19FM6J260111-A-CVM: 192.168.50.182: $ rm / iso / * ~ /基础
/usr/bin/rm: cannot remove ' /isos/* ':没有这样的文件或目录

我的语法有问题吗?

Hi @Haritha Andal<\/user-mention>,\u00a0<\/p>

There\u2019s an alert in Prism:<\/p>

Operation failed. Reason: Prechecks failed: Failed to reach lcm on 192.168.50.182. Please check KB 7781 Found remote version None while 2.4.2.25804 was expected on node 192.168.50.182. This may be a caching issue. Please ensure all local caches are cleared and wait a few minutes for any remote caches to get invalidated before retrying. Please check KB 7784<\/p>

I\u2019ve tried KB 7784 and 7781 but to no avail.<\/p>

I\u2019ve tried the KB you gave me and I saw that \/home is at 100%<\/p>

================== 192.168.50.182 =================
Filesystem \u00a0 \u00a0 \u00a0Size \u00a0Used Avail Use% Mounted on
devtmpfs \u00a0 \u00a0 \u00a0 \u00a09.8G \u00a0 \u00a0 0 \u00a09.8G \u00a0 0% \/dev
tmpfs \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 512M \u00a0 \u00a0 0 \u00a0512M \u00a0 0% \/dev\/shm
tmpfs \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 9.9G \u00a0624K \u00a09.9G \u00a0 1% \/run
tmpfs \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 9.9G \u00a0 \u00a0 0 \u00a09.9G \u00a0 0% \/sys\/fs\/cgroup
\/dev\/sda2 \u00a0 \u00a0 \u00a0 9.8G \u00a04.0G \u00a05.8G \u00a041% \/
\/dev\/loop0 \u00a0 \u00a0 \u00a0240M \u00a02.1M \u00a0222M \u00a0 1% \/tmp
\/dev\/sda3 \u00a0 \u00a0 \u00a0 \u00a040G \u00a0 39G \u00a0 \u00a0 0 100% \/home
tmpfs \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 2.0G \u00a0 \u00a0 0 \u00a02.0G \u00a0 0% \/run\/user\/1000
\/dev\/sdb1 \u00a0 \u00a0 \u00a0 3.6T \u00a01.3G \u00a03.6T \u00a0 1% \/home\/nutanix\/data\/stargate-storage\/disks\/ZC1B0WKY
\/dev\/sdc1 \u00a0 \u00a0 \u00a0 3.6T \u00a01.4G \u00a03.6T \u00a0 1% \/home\/nutanix\/data\/stargate-storage\/disks\/ZC1ARFYN
\/dev\/sda4 \u00a0 \u00a0 \u00a0 1.7T \u00a0269G \u00a01.4T \u00a016% \/home\/nutanix\/data\/stargate-storage\/disks\/S47PNE0M400683
\u00a0<\/p>

Tried deleting its contents\u00a0but it says:<\/p>


nutanix@NTNX-19FM6J260111-A-CVM:192.168.50.182:~\/foundation$ rm \/isos\/*
\/usr\/bin\/rm: cannot remove \u2018\/isos\/*\u2019: No such file or directory<\/p>

\u00a0<\/p>

Is there something wrong with my syntax ?
\u00a0<\/p>","quoteUsername":"John Renz","translations":{"Common":{"like":"Like","unlike":"Unlike"},"Forum":{"Quote":"Quote","Share":"Share"}}}">

R
排名
Userlevel 2
徽章 ＋４

嗨,约翰,

让我来回答你的问题。

如果/home是满的，我们需要仔细检查和清理CVM上批准的目录中的文件

您可以ssh到CVM(192.168.50.182)并执行以下命令吗?

du - h ~ /数据/日志

显示CVM中日志所占用的空间

然后请参考此KB来清理空间:[AOS Only] Controller VM (CVM)的/home分区或/home/nutanix目录已满时的处理方法

  • 不要使用rm射频在任何情况下，除非另有说明。会导致数据丢失的场景。

如果您仍然需要这方面的帮助，请使用Nutanix打开支持票来解决该问题

Raaji
Hi John,<\/p>

Let me respond to your query.<\/p>

If \/home is full, we will need to carefully check and clean files from approved directories on the CVM<\/p>

Could you please ssh into CVM (192.168.50.182) and execute the following command?<\/p>

du -h ~\/data\/logs<\/span><\/p>

This shows the space utilized by the logs in the CVM\u00a0<\/p>

And then please refer to this KB to clean up space: [AOS Only] What to do when \/home partition or \/home\/nutanix directory on a Controller VM (CVM) is full<\/a><\/p>

J

嗨 @raaji，

谢谢!

我昨天就试过了。方法2似乎是最有效的。

我刚删除了一些文件/home/nutanix/data/如知识库所示

刚好能把利用率从100%降低到97%。启动集群后，利用率下降到30%左右。

Hi\u00a0@raaji<\/user-mention>,<\/p>

Thanks!\u00a0<\/p>

I actually tried this yesterday. Method 2 seems to work best.\u00a0<\/p>

I just deleted some files at\u00a0\/home\/nutanix\/data\/\u00a0<\/em>as indicated in the KB<\/p>

Just enough to reduce the utilization from 100% to 97%. After starting the cluster, the utilization came down to around 30%.<\/p>","quoteUsername":"John Renz","translations":{"Common":{"like":"Like","unlike":"Unlike"},"Forum":{"Quote":"Quote","Share":"Share"}}}">

Baidu