解决了

X射线测试完成但未结束

r
UserLevel 2
徽章 +3
有没有人经历过X射线测试，但要保持“开放”的时间为0％？

在过去的几天中，我们的X射线测试一直在完成，但没有完成。结果，其他测试无法进行。

居里日志指示“退出_cluster_results_loop”，但测试以0％的速度悬挂起来，必须取消它们。
图标

最好的答案加里2019年4月30日，15:26

Thanks for reporting this\u00a0@Roberteo<\/user-mention> We will probably need to engage the X-Ray developers. In the meantime, can you tell us which version of X-Ray you are running, and against what sort of targets (vCenter or Prism) as well as which tests are having issues. In particular are they customized or one of the standard -Xray scenarios?<\/p>","className":"post__content__best_answer"}">
查看原件
\r\n
\r\nFor the last few days our X-Ray tests have been running to completion but not finishing. As a result the additional tests do not get run.
\r\n
\r\nThe curie log indicates \"exiting _cluster_results_loop\" but the tests hang open at 0% and they have to be cancelled.","quoteUsername":"Roberteo","translations":{"Common":{"like":"Like","unlike":"Unlike"},"Forum":{"Quote":"Quote","Share":"Share"}}}">
该主题已关闭以供评论

8个答复

加里
秩
Userlevel 1
徽章 +4

感谢您的报告 @Roberteo我们可能需要参与X射线开发人员。同时，您可以告诉我们您正在运行哪种版本的X射线，以及哪种目标（vCenter或Prism）以及哪些测试存在问题。特别是它们是定制的还是标准XARE场景之一？

Thanks for reporting this\u00a0@Roberteo<\/user-mention> We will probably need to engage the X-Ray developers. In the meantime, can you tell us which version of X-Ray you are running, and against what sort of targets (vCenter or Prism) as well as which tests are having issues. In particular are they customized or one of the standard -Xray scenarios?<\/p>","quoteUsername":"gary","translations":{"Common":{"like":"Like","unlike":"Unlike"},"Forum":{"Quote":"Quote","Share":"Share"}}}">
r
UserLevel 2
徽章 +3
我们当前使用标准（提供的）测试用例运行X射线V3.3。

我们的测试实验室目前有四（4）个集群。两（2）个是ESXI，两（2）为AHV。我们安装X射线的开发集群（ESXI）当前是三（3）个节点。从那里开始，我们针对其他四（4）个节点的其他三（3）个簇，当我们需要定位开发群集时，我们将X射线安装在另一个ESXI群集上。

我们目前已经进行了约400次测试，并且直到现在才进入IPMI相关的测试。

最初，我们遇到了我们从那以后的问题，并取得了良好的成功。从那时起，我们将VLAN标记为开关界面，并开始遇到大部分已解决的问题（Cross Rack流量仍然需要工作）。

昨晚“悬挂”的三个测试是VDI模拟器：Power Wormer（50）；任务工作者50;和知识工作者（100）。它们针对两个（2）个AHV簇。过去，我们在两个（2）小时内成功地进行了三（3）个小时的测试。大约15个小时过夜后，我们今天早上被迫取消这些测试。
\r\n
\r\nOur test lab currently has four (4) clusters. Two (2) are ESXi and two (2) are AHV. Our development cluster (ESXi) where we installed X-Ray is currently three (3) nodes. From there we target the other three (3) clusters of four (4) nodes and when we need to target the development cluster we have X-Ray installed on the other ESXi cluster.
\r\n
\r\nWe have currently run ~400 tests and are only now getting into the IPMI related tests.
\r\n
\r\nInitially we had issues that we have since smoothed over and were having fairly good success. Since then we vLan tagged the switch interfaces and started having issues that for the most part have been addressed (cross rack traffic still needs work).
\r\n
\r\nThe three tests from last night that 'hung' were VDI Simulator: Power Worker (50); Task Worker 50; and Knowledge Worker (100). They were targeted to the two (2) AHV clusters. In the past we had successfully run the three (3) hour tests in two (2) hours. After ~15 hours running over night we were forced to cancel these tests this morning.","quoteUsername":"Roberteo","translations":{"Common":{"like":"Like","unlike":"Unlike"},"Forum":{"Quote":"Quote","Share":"Share"}}}">
rthardin
秩
徽章
你好 @Roberteo。得知您遇到了X射线3.3的问题，我感到很遗憾。我是开发团队的成员，希望提供帮助。当您有机会时，您能否在直接消息中将测试和服务器日志发送给我？

可以从悬挂测试的操作下拉列表中导出测试日志，并且服务器日志是从X射线UI顶部的“ Gear Icon下拉菜单”菜单中导出的。

我想重申一下，以确保我正确理解这个问题；这是否与已成功运行许多测试的X射线3.3 VM相同，但是最近它们开始挂起（特别是VDI模拟器测试）？

非常感谢！

瑞安
@Roberteo<\/user-mention>. I'm sorry to hear you're running into issues with X-Ray 3.3. I'm a member of the development team and would like to help. When you have a chance, would you please send the test and server logs to me in a direct message?
\r\n
\r\nThe test logs can be exported from the Actions dropdown for the hung tests, and the server logs are exported from the gear icon dropdown menu in the top-right of the X-Ray UI.
\r\n
\r\nI'd like to reiterate just to make sure I understand the issue correctly; Is this the same X-Ray 3.3 VM that has been used to run many tests successfully, but recently they have started to hang (specifically the VDI Simulator tests)?
\r\n
\r\nThank you very much!
\r\n
\r\nRyan","quoteUsername":"rthardin","translations":{"Common":{"like":"Like","unlike":"Unlike"},"Forum":{"Quote":"Quote","Share":"Share"}}}">
r
UserLevel 2
徽章 +3
你好。

感谢你的回复。

我将在几分钟内将日志转发（它们被导出，然后在审查后的早晨删除）。

是的，这是以前成功运行它们的VM。我们已经切换到其他X射线VM，以继续测试...
\r\n
\r\nThank you for your reply.
\r\n
\r\nI will forward you the logs in a few minutes (they are exported then deleted in the morning after review).
\r\n
\r\nYes, this is the same VM that successfully ran them before. We have switched over to the other X-Ray VM for now to continue testing...","quoteUsername":"Roberteo","translations":{"Common":{"like":"Like","unlike":"Unlike"},"Forum":{"Quote":"Quote","Share":"Share"}}}">
rthardin
秩
徽章
听起来不错， @Roberteo- 谢谢你！

瑞安
@Roberteo<\/user-mention> - thank you!
\r\n
\r\nRyan","quoteUsername":"rthardin","translations":{"Common":{"like":"Like","unlike":"Unlike"},"Forum":{"Quote":"Quote","Share":"Share"}}}">
r
UserLevel 2
徽章 +3
NCC和Curie日志在前往您的路上处于以太之中...
rthardin
秩
徽章
你好 @Roberteo，，，，

感谢您发送日志。我看到其中一个测试线程正在悬挂，这导致螺纹连接到阻塞。悬挂线负责在测试期间更新Prometheus配置。该线程还查询Prometheus以更新线图。

这是足够的信息来确定需要添加超时以防止测试完全悬挂的地方。如果我们可以更接近似乎是性能问题的根本原因，那么我们也许可以进行进一步的修复。

如果您有机会，请使用失败的测试登录X射线VM并运行一些命令？您可以使用用户名`nutanix`和密码nutanix/4U'连接到X射线VM。如果您可以运行下面的命令并发送输出，那将非常有用：

磁盘空间上的X射线VM低点吗？
  • DF -A
X射线VM的内存低吗？
  • 猫 /proc /meminfo
  • vmstat -s
X射线VM CPU忙吗？
  • 顶部
您已经提供的信息帮助我们开始了修复。非常感谢！

瑞安
@Roberteo<\/user-mention>,
\r\n
\r\nThanks for sending the logs. I see that one of the test threads is hanging, which is causing the thread join to block. The hung thread is responsible for updating the Prometheus configuration during the test. That thread also queries Prometheus to update the line plots.
\r\n
\r\nThis is enough information to identify a few places where timeouts need to be added to prevent the test from hanging altogether. If we can get closer to the root cause of what appears to be a performance issue, we may be able to make further fixes.
\r\n
\r\nIf you have a chance, would you please log in to the X-Ray VM with the failing tests and run a few commands? You can connect to the X-Ray VM via SSH with the username `nutanix` and password `nutanix\/4u`. If you could please run the commands below and send the output, that would be very useful:
\r\n
\r\nIs the X-Ray VM low on disk space?
\r\n
    \r\n
  • df -a \r\n<\/ul>\r\nIs the X-Ray VM low on memory?
    \r\n
      \r\n
    • cat \/proc\/meminfo \r\n
    • vmstat -s \r\n<\/ul>\r\nIs the X-Ray VM CPU busy?
      \r\n
        \r\n
      • top \r\n<\/ul>\r\nThe information you've already provided has helped us get started on a fix. Thank you very much!
        \r\n
        \r\nRyan","quoteUsername":"rthardin","translations":{"Common":{"like":"Like","unlike":"Unlike"},"Forum":{"Quote":"Quote","Share":"Share"}}}">
r
UserLevel 2
徽章 +3
它正在进入您的收件箱...
条款和条件
Learn more about our cookies.<\/a>","cookiepolicy.button":"Accept cookies","cookiepolicy.button.deny":"Deny all","cookiepolicy.link":"Cookie settings","cookiepolicy.modal.title":"Cookie settings","cookiepolicy.modal.content":"We use 3 different kinds of cookies. You can choose which cookies you want to accept. We need basic cookies to make this site work, therefore these are the minimum you can select. Learn more about our cookies.<\/a>","cookiepolicy.modal.level1":"Basic
Functional","cookiepolicy.modal.level2":"Normal
Functional + analytics","cookiepolicy.modal.level3":"Complete
Functional + analytics + social media + embedded videos"}}}">
Baidu