解决了

远程站点复制调整

  • 2018年5月29日
  • 6回复
  • 1966年观点
Penguindows.
UserLevel 2.
徽章 +5
Nutanix中有哪些选项可加快对远程站点的复制？我能够获得单一流的最佳速度约为20Mbps（160Mbps），总速度约为60Mbps（480mbps）展开了几个流。但是，我们在网站之间有一个10Gbps的链接，所以我期望获得更好的吞吐量。
图标

最好的答案Penguindows.2018年6月8日，16:35

@dlink7<\/user-mention> for the good info on expected performance under ideal situations. That does give me a good baseline.
\r\n
\r\nOur issue is definitely latency. Our remote site is across the continental US, and our per VM size is 30TB (10TB used) on a single disk. as you can imagine, this creates a bottleneck for us. Replications with default settings are not meeting RPO.
\r\n
\r\nOur AOS version is 5.5.2.
\r\n
\r\nI have found (with help from support) these tunable settings within the cluster:
\r\n
\r\n
\r\n
code:<\/b>
nutanix@NTNX:~$ python \/home\/nutanix\/serviceability\/bin\/edit-aos-gflags | grep stargate_
2018-06-08 10:00:09 INFO zookeeper_session.py:110 edit-aos-gflags is attempting to connect to Zookeeper
   stargate_cerebro_replication_max_rpc_vblocks = 16  #default 4
   stargate_cerebro_replication_max_rpc_data = 4194304 #default 1048576
   stargate_cerebro_max_outstanding_vdisk_replication_rpcs = 16 #default 4
   stargate_cerebro_replication_param_multiplier = 32 #default 16
   stargate_vdisk_read_extents_max_outstanding_egroup_reads = 6 #default 3
<\/pre><\/div>
\r\n
\r\nI am putting together a helpful howto on how to tune these parameters, but the gist of it is editing with...
\r\n
\r\n
code:<\/b>
nutanix@NTNX:~$ python \/home\/nutanix\/serviceability\/bin\/edit-aos-gflags --service=stargate
<\/pre><\/div>
\r\n
\r\n...then restarting stargate on each cvm.
\r\n
\r\nThe restarts went perfectly with a sleep 60 in there, we had no failure on running replications, and our throughput went up ~4x as expected.  our 12 day running replications began to finish.
\r\n
\r\nmy ultimate solution is two fold.  first, the above settings to accelerate each stream.  second, we are working to break out our 30TB disk in to 10x 3TB disks.
\r\n
\r\nRE: going off stock; I think that as nutanix grows in to more environments (and they will, the technology is amazing and i foresee continued adoption) there are going to be many more flavors of environments that sticking to stock options just wont solve.  I expect in the next few years that nutanix will pivot in to a more open position.  I'd love to see some of these settings within reach of the prism GUI along with some better man, info pages on what each individual setting and switch does.
\r\n
\r\nThat being said, your warning about going off stock is noted and appreciated.  i understand that nutanix attempts to tune AOS as well as possible out of the box, and that tuning these settings can have an impact (sometimes bad, sometimes good).  Therefore, I will only seek a behavior change in the technology to achieve some desired result.","className":"post__content__best_answer"}">

            查看原版
    此主题已关闭征询意见

    6回复

    Dlink7.
    UserLevel 4.
    徽章 +19
    嗨，你是谁的摇摆？

    在acropolis中，每个节点都可以在一次复制四个文件，最多可在100 MB / s的聚合。因此，在四节点配置中，群集可以复制400 MB / s或3.2 gb / s。

    什么可能导致这个数字下降？群集上的其他任务，如策展人，如果snapchain太高，延迟不好。

    如果你仍然遇到你的RPO，我就不会改变任何东西。off股票会回到以后困扰你。
    \r\n
    \r\nIn Acropolis, every node can replicate four files, up to an aggregate of 100 MB\/s at one time. Thus, in a four-node configuration, the cluster can replicate 400 MB\/s or 3.2 Gb\/s.
    \r\n
    \r\nWhat can cause this number to go down? Other tasks on the cluster, like curator running if they snapchain is too high and bad latency.
    \r\n
    \r\nIf your still meeting your RPO I wouldn't change anything. Going off stock will come back to haunt you later.","quoteUsername":"dlink7","translations":{"Common":{"like":"Like","unlike":"Unlike"},"Forum":{"Quote":"Quote","Share":"Share"}}}">
    aluciani
    UserLevel 7.
    徽章 +34
    你好 @penguindows.

    你有没有看到回复 @ dlink7.
    在推特上关注我：https://twitter.com/angeloluciani
    @penguindows<\/user-mention>
    \r\n
    \r\nDid you see the reply from @dlink7<\/user-mention>","quoteUsername":"aluciani","translations":{"Common":{"like":"Like","unlike":"Unlike"},"Forum":{"Quote":"Quote","Share":"Share"}}}">
    Penguindows.
    UserLevel 2.
    徽章 +5
    是的，我看到Dlink7，S的回复。谢谢 @ dlink7.在理想情况下对预期绩效的良好信息。这给了我一个很好的基线。

    我们的问题绝对是延迟。我们的远程网站位于美国大陆，我们的每个VM大小是单个磁盘上的30TB（10TB）。正如您可以想象的那样，这为我们创造了一个瓶颈。默认设置的复制不符合RPO。

    我们的AOS版本为5.5.2。

    我已经找到（借助于帮助）这些集群中的这些可调整设置：


    代码： 
    Nutanix @ NTNX：〜$ Python / Home / Nutanix / ServiceAility / Bin / Edit-AOS-GFLAGS |Grep Stargate_
    2018-06-08 10:00:09信息zookeeper_session.py：110 Edit-AOS-GFLAGS正在尝试连接到ZooKeeper
    Stargate_cerebro_replication_max_rpc_vblocks = 16 #default 4
    Stargate_cerebro_replication_max_rpc_data = 4194304 #default 1048576
    stargate_cerebro_max_out优效_vdisk_replication_rpcs = 16 #default 4
    Stargate_cerebro_replication_param_multiplier = 32 #default 16
    Stargate_vdisk_read_readtents_max_outivent_egroup_reads = 6 #default 3


    我在一起有用的如何调整这些参数的有用HOWTO，但它的主旨是编辑......

    代码： 
    Nutanix @ NTNX：〜$ Python / Home / Nutanix / ServiceAility / Bin / Edit-AOS-GFLAGS  -  Service = STARGATE


    ...然后在每个CVM上重新启动星形格。

    重启与那里的睡眠完全完美，我们在运行复制上没有失败，我们的吞吐量按预期上涨〜4倍。我们的12天运行重复开始完成。

    我的终极解决方案是两倍。首先，以上设置加速每个流。其次，我们正致力于将30TB磁盘分为10x 3TB磁盘。

    回复：脱掉股票;我认为随着Nutanix在更多的环境中增长（而且，这项技术是惊人的，我预见到持续采用）将有更多的旨在粘在股票期权的环境中的味道。我期待在未来几年内，Nutanix将枢转到更开放的位置。我很乐意在棱镜GUI的范围内看到一些这些设置以及每个单独的设置和交换机的信息页面。

    据说，关于脱离股票的警告被指出并赞赏。我明白Nutanix尝试在盒子中尽可能地调整AOS，并且调整这些设置可能会产生影响（有时差，有时好）。因此，我只会寻求技术的行为变化以实现一些所需的结果。
    @dlink7<\/user-mention> for the good info on expected performance under ideal situations. That does give me a good baseline.
    \r\n
    \r\nOur issue is definitely latency. Our remote site is across the continental US, and our per VM size is 30TB (10TB used) on a single disk. as you can imagine, this creates a bottleneck for us. Replications with default settings are not meeting RPO.
    \r\n
    \r\nOur AOS version is 5.5.2.
    \r\n
    \r\nI have found (with help from support) these tunable settings within the cluster:
    \r\n
    \r\n
    \r\n
    code:<\/b>
    nutanix@NTNX:~$ python \/home\/nutanix\/serviceability\/bin\/edit-aos-gflags | grep stargate_
    2018-06-08 10:00:09 INFO zookeeper_session.py:110 edit-aos-gflags is attempting to connect to Zookeeper
       stargate_cerebro_replication_max_rpc_vblocks = 16  #default 4
       stargate_cerebro_replication_max_rpc_data = 4194304 #default 1048576
       stargate_cerebro_max_outstanding_vdisk_replication_rpcs = 16 #default 4
       stargate_cerebro_replication_param_multiplier = 32 #default 16
       stargate_vdisk_read_extents_max_outstanding_egroup_reads = 6 #default 3
    <\/pre><\/div>
    \r\n
    \r\nI am putting together a helpful howto on how to tune these parameters, but the gist of it is editing with...
    \r\n
    \r\n
    code:<\/b>
    nutanix@NTNX:~$ python \/home\/nutanix\/serviceability\/bin\/edit-aos-gflags --service=stargate
    <\/pre><\/div>
    \r\n
    \r\n...then restarting stargate on each cvm.
    \r\n
    \r\nThe restarts went perfectly with a sleep 60 in there, we had no failure on running replications, and our throughput went up ~4x as expected.  our 12 day running replications began to finish.
    \r\n
    \r\nmy ultimate solution is two fold.  first, the above settings to accelerate each stream.  second, we are working to break out our 30TB disk in to 10x 3TB disks.
    \r\n
    \r\nRE: going off stock; I think that as nutanix grows in to more environments (and they will, the technology is amazing and i foresee continued adoption) there are going to be many more flavors of environments that sticking to stock options just wont solve.  I expect in the next few years that nutanix will pivot in to a more open position.  I'd love to see some of these settings within reach of the prism GUI along with some better man, info pages on what each individual setting and switch does.
    \r\n
    \r\nThat being said, your warning about going off stock is noted and appreciated.  i understand that nutanix attempts to tune AOS as well as possible out of the box, and that tuning these settings can have an impact (sometimes bad, sometimes good).  Therefore, I will only seek a behavior change in the technology to achieve some desired result.","quoteUsername":"penguindows","translations":{"Common":{"like":"Like","unlike":"Unlike"},"Forum":{"Quote":"Quote","Share":"Share"}}}">
    Dlink7.
    UserLevel 4.
    徽章 +19
    你很高兴知道Eng有项目让这些设置更加动态。很高兴你被排序了。
    布拉德利4681.
    是的，我看到Dlink7，S的回复。谢谢 @ dlink7.在理想情况下对预期绩效的良好信息。这给了我一个很好的基线。

    我们的问题绝对是延迟。我们的远程网站位于美国大陆，我们的每个VM大小是单个磁盘上的30TB（10TB）。正如您可以想象的那样，这为我们创造了一个瓶颈。默认设置的复制不符合RPO。

    我们的AOS版本为5.5.2。

    我已经找到（借助于帮助）这些集群中的这些可调整设置：


    代码： 
    Nutanix @ NTNX：〜$ Python / Home / Nutanix / ServiceAility / Bin / Edit-AOS-GFLAGS |Grep Stargate_
    2018-06-08 10.
    ：00：09信息zookeeper_session.py :110编辑-AOS-GFLAGS正在尝试连接到ZooKeeper
    Stargate_cerebro_replication_max_rpc_vblocks = 16 #default 4
    Stargate_cerebro_replication_max_rpc_data = 4194304 #default 1048576
    stargate_cerebro_max_out优效_vdisk_replication_rpcs = 16 #default 4
    Stargate_cerebro_replication_param_multiplier = 32 #default 16
    Stargate_vdisk_read_readtents_max_outivent_egroup_reads = 6 #default 3



    我在一起有用的如何调整这些参数的有用HOWTO，但它的主旨是编辑......

    代码： 
    Nutanix @ NTNX：〜$ Python / Home / Nutanix / ServiceAility / Bin / Edit-AOS-GFLAGS  -  Service = STARGATE



    ...然后在每个CVM上重新启动星形格。
    你是如何做出这些变化的或如何创造的？谢谢！
    Yes, I see dlink7,s reply. Thanks @dlink7<\/user-mention> for the good info on expected performance under ideal situations. That does give me a good baseline.
    \r\n
    \r\nOur issue is definitely latency. Our remote site is across the continental US, and our per VM size is 30TB (10TB used) on a single disk. as you can imagine, this creates a bottleneck for us. Replications with default settings are not meeting RPO.
    \r\n
    \r\nOur AOS version is 5.5.2.
    \r\n
    \r\nI have found (with help from support) these tunable settings within the cluster:
    \r\n
    \r\n
    \r\n
    code:<\/b>
    nutanix@NTNX:~$ python \/home\/nutanix\/serviceability\/bin\/edit-aos-gflags | grep stargate_
    2018-06-08 10
    :00:09 INFO zookeeper_session.py:110 edit-aos-gflags is attempting to connect to Zookeeper
       stargate_cerebro_replication_max_rpc_vblocks = 16  #default 4
       stargate_cerebro_replication_max_rpc_data = 4194304 #default 1048576
       stargate_cerebro_max_outstanding_vdisk_replication_rpcs = 16 #default 4
       stargate_cerebro_replication_param_multiplier = 32 #default 16
       stargate_vdisk_read_extents_max_outstanding_egroup_reads = 6 #default 3

    <\/pre><\/div>
    \r\n
    \r\nI am putting together a helpful howto on how to tune these parameters, but the gist of it is editing with...
    \r\n
    \r\n
    code:<\/b>
    nutanix@NTNX:~$ python \/home\/nutanix\/serviceability\/bin\/edit-aos-gflags --service=stargate

    <\/pre><\/div>
    \r\n
    \r\n...then restarting stargate on each cvm.
    \r\n
    \r\n<\/content-quote>
    \r\nHow did you make these changes or where is the how to you were creating? Thanks!","quoteUsername":"Bradley4681","translations":{"Common":{"like":"Like","unlike":"Unlike"},"Forum":{"Quote":"Quote","Share":"Share"}}}">
    Penguindows.
    UserLevel 2.
    徽章 +5
    @ Bradley4681.当你运行时：
    代码： 
    Nutanix @ NTNX：〜$ Python / Home / Nutanix / ServiceAility / Bin / Edit-AOS-GFLAGS  -  Service = STARGATE

    您将使用所有星际服务Glfags进入文本编辑器。在这里，您可以编辑设置以实现不同的结果。

    我调整的伤口为ME工作的设置基本上增加了每VDISK的流数量的增加，行上允许的数据饱和度和眩晕的优先级给予复制处理。这让我克服了我的十字架管道上的高层。

    您可以在引用的块中找到我的特定设置调整。此外，您可能需要最少获得创世纪重启。

    值得注意的是关于我的环境的一些事情，使得这种设置的组合工作，没有任何负面影响：

    1. 同质图像：整个群集中的每位客人都在执行相同类型的工作，是相同的大小，并具有相同的时间表。这意味着我可以在不对奇数球系统产生不利影响的情况下应用集群范围变化。
    2. 低计算需求：我有6位客人参加了8个节点群集。基本上，我可以为Nutanix服务提供负荷，而不是维修来宾计算，因为我的客人没有高计算需求。
    3. 预定的工作负载：这里的客人是备份媒体服务器。他们有一个“忙碌”的时间范围，为客人提供IO很重要，而“慢速”时间帧可以为Nutanix服务保留。基本上，我们在白天晚上和更少的时间备份。
    4. 高层岩，但高带宽。我的越野数据中心之间的管道为2 x 10Gbps。我们有一个QoS策略将我们的工作量限制为此群集的5Gbps，但这仍然是一个相当宽的管道。延迟是74ms。
    RE：指南：对于我的尴尬，我的指南仅限于一些内部注释和一些我的团队的Wiki页面。它不是以时尚的方式组织或清理，我很舒服地与社区分享。我有很高的愿望，以便将这些笔记在一起，并以对社区有用的形式。我确实需要乞求你的耐心，因为现在的新需求似乎是一个不断的溪流。
    @Bradley4681<\/user-mention> When you run:
    \r\n
    code:<\/b>
    nutanix@NTNX:~$ python \/home\/nutanix\/serviceability\/bin\/edit-aos-gflags --service=stargate
    <\/pre><\/div>
    \r\nYou enter in to a text editor with all the stargate service glfags.  Here, you can edit the settings to achieve different results.
    \r\n
    \r\nThe settings i adjusted that wound up working for me basically amounted to an increase in the number of streams per vdisk, the amount of data saturation allowed on the line, and the priority that stargate gave to replication processing.  This allowed me to overcome the high latancy on my cross US pipe.
    \r\n
    \r\nYou can find my specific setting adjustments in the block that you quoted.  Also, you'll probably need to get a genesis restart at a minimum.
    \r\n
    \r\nIts worth noting a few things about my environment that made this combination of settings work without any negative impacts:
    \r\n
    \r\n
       \r\n
    1. Homogeneous images:<\/b> every guest in the entire cluster is doing the same type of work, is the same size and has the same schedule.  This meant that i can apply cluster wide changes without adversely affecting odd ball systems. \r\n
    2. Low compute demand<\/b>: I have 6 guests in an 8 node cluster.  basically, i can afford to give loads of power to nutanix services rather than servicing guest compute because my guests do not have a high compute demand. \r\n
    3. Scheduled workloads<\/b>: The guests here are backup media servers.  They have a \"busy\" timeframe where IO for the guests is important, and a \"slow\" timeframe when IO can be reserved for nutanix services.  basically, we backup more at night and less during the day. \r\n
    4. High latancy, but high bandwidth<\/b>.  The pipe between my cross country datacenters is 2 x 10gbps.  we have a qos policy that limits our workload down to 5gbps for this cluster, but that is still a rather wide pipe. The latency is 74ms. \r\n<\/ol>\r\nRE: the guide<\/b>:  To my embarrassment, my guide is limited to some internal notes and a few wiki pages for my team.  It isn't organized or cleaned up in a fashion that i'd be comfortable sharing with the community.  I have high aspirations to get these notes together and in a form that would be useful to the community.  I do need to beg for your patience however, as new demand seems to be a constant stream right now.","quoteUsername":"penguindows","translations":{"Common":{"like":"Like","unlike":"Unlike"},"Forum":{"Quote":"Quote","Share":"Share"}}}">
    Learn more about our cookies.<\/a>","cookiepolicy.button":"Accept cookies","cookiepolicy.button.deny":"Deny all","cookiepolicy.link":"Cookie settings","cookiepolicy.modal.title":"Cookie settings","cookiepolicy.modal.content":"We use 3 different kinds of cookies. You can choose which cookies you want to accept. We need basic cookies to make this site work, therefore these are the minimum you can select. Learn more about our cookies.<\/a>","cookiepolicy.modal.level1":"Basic
    Functional","cookiepolicy.modal.level2":"Normal
    Functional + analytics","cookiepolicy.modal.level3":"Complete
    Functional + analytics + social media + embedded videos"}}}">
    Baidu