博客

在Nutanix上运行弹性研究：技术深潜力

2年前
2019年3月22日
0回复
4281的浏览量

UserLevel 7.

+ 34

aluciani
骑士
334回复

本文作者是Gabe Contreras, Nutanix服务咨询架构师

当涉及到在任何平台上运行Elasticsearch时，根据你的用例，获得正确的设置可能需要时间。吞吐量小的部署可能已经部署在云中，但是部署在一分钟处理数百万个文档的情况又如何呢?许多公司在裸金属上运行这些更大的部署也就不足为奇了。许多Nutanix客户已经在我们的企业云上运行了Elasticsearch。我们的客户发现了以下好处:

快提供
简化基础设施
改进可用性
更大的灵活性
提高效率

在大多数现实世界的Elasticsearch和一般大数据环境中，这些优势超过了在裸金属上运行可能获得的任何潜在好处。

对于Nutanix客户，能够更好地管理和运行大数据应用和一般服务器虚拟化是一个主要因素。
消除支持多个孤岛的需要并为这些工作负载部署虚拟化允许它们获得更好的效率。
迁移到Nutanix可以帮助应用程序团队提高可用性，包括如何从可能的中断中恢复Elasticsearch。我们已经多次演示了对Nutanix和Elasticsearch设置进行优化的能力。
配置的时间大大减少了，因为在获取裸金属和完成配置步骤之间不再需要等待时间。

在一个具体的例子中，Elasticsearch工作负载平均每分钟摄取七百万个文件，尖峰高达11百万。这些将是每日索引，每天具有高十亿数字文件的总文件计数，大约为16-18TB的主要数据，弹性科学副本设置为= 1总计32-36TB。文档大小在2KB左右。

该解决方案已设置为保持至少1周的索引，这意味着存储至少200TB的数据总数。这也必须考虑每天对此数据的数百万次搜索。在这种情况下，数据将流入来自Kafka的Elasticsearch。

Elasticsearch是一个IO密集型应用程序，我们设计的目的是合并，这样做可以大大提高IO和吞吐量。

在Nutanix上虚拟化Elasticsearch

下面是对Elasticsearch (ES)环境的总结

单独的机架中的2x 24节点集群（每个机架 - 努力和ES故障域）。每个群集都可以使用其他工作负载运行

每个集群使用RF2提供230TB的可用存储
每个Nutanix计算节点32核和512GB RAM
RF2在Nutanix上

Elasticsearch VM配置

3管理员节点
6摄取节点
90年的数据节点

每个虚拟机有16个VCPU和128GB内存

虚拟机磁盘布局:每个LVM虚拟机新增8个vdisk，条带1MB，总容量3.4TB

我们对Linux操作系统应用了一些设置来优化它，以保持IO流向我们的存储子系统。

Max_sectors_kb = 1024
nr_requests = 128

下面的设置应用于sysctl，因为最好让内核不断地将数据刷新到存储中，而不是将其保存在缓冲区中

vm.dirty_background_ratio = 1
vm.dirty_ratio = 40.

Elasticsearch测试

测试完成时，Kafka数据被重定向到Nutanix设置中。如果您已经部署了Elasticsearch (ES)，您会意识到您的设置是根据您的工作负载进行优化的，但是还有其他设置可以用来优化搜索。下面是我们选择改变的其他设置，以利用Nutanix的操作方式。我们调整的主要设置如下。

索引

index.refresh_interval = 600s.

此设置默认为1S，这意味着您将创建大量段，并且将不断同步。同步通常是效率低下，创造了很多不必要的IO，因为您将创建许多较小的细分市场，并且不断合并，这可能导致摄取较慢。此工作负载不需要实时搜索，每10分钟执行一次同步是最佳设置。

该图显示了何时开始合并。在正常的摄取过程中，写的IOPS可能是10,000 IOPS，但是当合并发生时，您可以看到峰值超过160,000 IOPS，并且吞吐量可以在操作过程中翻倍。

Index.translog。flush_threshold_size = 1024 mb

默认值通常为512MB。此设置已看到索引吞吐量得到改善
大约20％。更改此设置增加了重播单个转变的时间，但性能增益是值得的。

index.translog.durability =异步

默认是request，改变这个设置将把flush推到内核，这将阻止应用程序可能进入阻塞状态。

综上所述，总体索引将平均每分钟处理700万个文档，并在峰值时处理高达1100万个文档。对于平均文档级别和2Kb文档大小，主数据的直接写吞吐量约为12GBpm，所有索引数据的直接写吞吐量约为24GBpm。Nutanix集群有足够的吞吐量来处理搜索和上面显示的合并流量。

这张来自Kibana的图片显示了白天的交通流量。您可以看到集群每秒处理多少个文档。

搜索

除了正常的搜索优化外，没有真正的建议为您的工作负载执行。Nutanix是一个分布式可伸缩存储平台，为您提供低延迟，如数据位置，如裸机读取。Elasticsearch由于延迟要求推荐本地存储。使用Nutanix的数据点，您可以获得每天执行数百万搜索时所需的低延迟本地读取。

失败场景

Index.translog.retention.size = 10 gb

这个设置的默认值是512mb，对于高吞吐量工作负载来说，这个值很小。在恢复副本时将此设置为基于操作的同步，而不是执行完整的复制。这在裸金属环境中发挥的作用并不大，因为当硬件节点出现故障时，它通常会出现硬故障，无法恢复。而使用Nutanix，当节点出现故障时，另一个节点上的数据节点会在1分钟内重新启动，并且可以快速恢复副本，使集群更快地恢复绿色。

Index.unassigned.node_left.delayed_timeout = 15米

默认设置为1m。这种设置是对Nutanix比裸金属更多的要求。对于快速恢复时间，您不希望Elasticsearch在节点恢复时开始恢复碎片。如果你使用默认值运行，并且虚拟机在1分30秒内重启，Elasticsearch将不允许你再次使用这些索引。发生的情况是，该节点现在有0个分片，必须等待重新平衡，因为它已经恢复了该节点上的分片。这导致了恢复和再平衡的双重责任。

这些设置有助于充分利用Nutanix HA来完成关键任务的工作负载。不同之处在于，当一个裸金属节点出现严重故障时，Elasticsearch集群需要几个小时才能恢复为绿色，并使两个分片副本再次可用。而使用Nutanix HA，在节点下降的几分钟内，不活动的索引是绿色的，而活动的索引在30分钟内是绿色的。

这意味着对于裸金属，只有一个副本可以搜索碎片，从而降低了搜索时间。这也增加了IO和网络活动，因为Elasticsearch只做一对一的复制来恢复分片，这意味着一个节点有主副本到新节点，这可能需要几个小时。与此同时，随着Nutanix在后台恢复它的第二个副本，它花费的时间大大减少，因为整个集群都在为重建做出贡献。

对于这个客户来说，一个常见的场景是一个坏驱动器导致了集群中的问题。为了获得最好的性能，裸金属节点选择了RAID 0，这样可以获得最好的吞吐量，但也因为在RAID重建期间，会出现影响整个集群的IO降级。

在这个单个坏驱动器导致节点故障的场景中，Nutanix处理这个问题的方式要优雅得多。我们在Nutanix系统中有多个检查，所以当我们看到一个驱动器的性能下降，或者我们得到SMART错误消息预测驱动器可能会失败，我们将主动停止使用该驱动器或将其从系统中全部弹出。这与裸金属服务器不同，可以防止数据节点故障或单个节点成为性能瓶颈。

数据重建

正如我之前提到的那样，设置是弹星精灵层的单拷贝（RF2），并在Nutanix上的RF2，弹性型搜索副本在两个单独的簇之间分开。

Elasticsearch在出现故障时必须创建一个新的副本，当一个副本离线时，它的工作方式是直接的一对一副本。如果您有10个分片来重建，那么这些分片就是从主数据节点到新的辅助节点的一对一复制。如果整个集群很忙，或者它选择的单个节点很忙，那么恢复就会很慢，因此数据的重建就会遇到瓶颈。在此期间，搜索将会变慢，索引可能会遇到瓶颈或面临数据丢失的风险。

如果硬件节点在Nutanix集群中发生故障，它会实现需要复制哪些数据块以恢复绿色。通过这种情况，整个Nutanix群集参与数据的重建，因此在重建时没有超速单个节点，并且您具有群集的分布式计算和IO电源。即使在较重负载期间，这也意味着重建更快。在这种情况下，Elasticsearch还会看到两个副本，以帮助保持搜索劣化。

结论

在查看整个解决方案时，Nutanix将您的大数据应用组合到单个平台上，以实现更好的管理和可扩展性。您可以从此示例中看到，即使是最苛刻的工作负载也可以获得所需的性能。

有关运行弹性电池的更多信息，请阅读Nutanix AHV上的弹性堆栈的解决方案注意 -//www.jhbzcj.com/go/virtualizing-elastic-stack-on-ahv.php．您也可以通过发送电子邮件到我们的解决方案专家和服务来设置简报info@nutanix.com.．

资源：

关于Elasticsearch的Nutanix NEXT社区线程:https://next.nutanix.com/server-virtualization-27/elasticsearch-on-nutanix-31873
Elasticsearch入门 -https://www.elastic.co/webinars/getting-started-elasticsearch
Nutanix emoch elasticsearch文档 - https://docs.ech.nutanix.com/integrations/elasic/

免责声明:本博客可能包含链接到外部网站，而不是Nutanix.com的一部分。Nutanix不控制这些网站，并对任何外部网站的内容或准确性不负任何责任。我们链接到外部网站的决定不应被认为是对该网站上任何内容的认可。

©2019 Nutanix，Inc。保留所有权利。Untanix，Nutanix徽标和本文提到的其他Nutanix产品和特征是Nutanix，Inc。的注册商标或商标。本文提到的所有其他品牌名称仅供识别目的，可能是其各自持有人的商标。

在推特上关注我：https://twitter.com/angeloluciani

elasticsearch.

\r\n
\r\nWhen it comes to running Elasticsearch on any platform getting the right settings can take time to get right based on your use case. Small throughput deployments may have been deployed in the cloud, but what about deployments processing millions of documents a minute? It\u2019s no surprise that many companies run these larger deployments on bare-metal. Numerous Nutanix customers have made the leap to running Elasticsearch on our Enterprise Cloud. Our customers have found the following benefits:
\r\n

Faster provisioning \r\n
Simplified Infrastructure \r\n
Improved availability \r\n
Greater flexibility \r\n
Increased efficiency \r\n<\/ul>\r\nIn most real-world Elasticsearch and general big data environments, these advantages outweigh any potential benefits you might get from running on bare metal.
\r\n
- For Nutanix customers the ability to better manage and run big data applications and general server virtualization is a major factor. \r\n
- Eliminating the need to support multiple silos and deploy virtualization for these workloads allows them to get much better efficiency. \r\n
- The move to Nutanix helps application teams with availability including how Elasticsearch would recover from possible outages. We have been able to repeatably demonstrate the ability to optimize both Nutanix and Elasticsearch settings to make this optimal. \r\n
- The time for provisioning has been greatly decreased as there is no longer a wait time between procuring bare-metal and going through the provisioning steps. \r\n<\/ul>\r\nIn one specific example, the Elasticsearch workload ingests on average seven million documents a minute with spikes up to eleven million. These would be daily indexes that would have a total document count per day of high single billion digital documents with around 16-18TB a day of primary data with an Elasticsearch replica set to =1 totaling 32-36TB a day. The document sizes were around 2Kb each.
  \r\n
  \r\nThe solution has been set up to keep at least 1 week of indexes, meaning at least 200TB of data total is stored. This also had to account for the millions of searches on this data a day. In this scenario, the data is streaming in to Elasticsearch from Kafka.
  \r\n
  \r\nElasticsearch is an IO intensive application and we designed for merging that is done which can create a substantial increase in IO and throughput.
  \r\n
  \r\n
  Virtualizing Elasticsearch on Nutanix<\/h3>
  \r\nHere is a summary of the environment that this Elasticsearch (ES)
  \r\n
  \r\n2x 24 nodes clusters in separate racks (each rack--Nutanix and ES failure domain). Each cluster could run ES with other workloads
  \r\n
  - Each cluster provides 230TB of useable storage with RF2 \r\n
  - 32 cores and 512GB RAM per Nutanix compute node \r\n
  - RF2 on Nutanix \r\n<\/ul>\r\n
    <\/a><\/p>
    \r\n
    Elasticsearch VM Configurations<\/h3>
    \r\n
    3 Manager Nodes \r\n
    6 Ingest Nodes \r\n
    90 Data Nodes \r\n<\/ul>\r\nEach VM had 16 VCPU and 128GB RAM
    \r\n
    \r\nVM disk layout: 8 vdisks added into an LVM with a 1MB stripe totaling 3.4TB
    \r\n
    \r\nWe applied some settings to the Linux OS to optimize it to keep IO flowing to our storage subsystem.
    \r\n
    \r\n
    Max_sectors_kb = 1024 \r\n
    nr_requests = 128 \r\n<\/ul>\r\nThe following settings were applied to the sysctl as it is better to have the kernel constantly flushing data to the storage rather than keeping it in the buffer
    \r\n
    \r\n
    vm.dirty_background_ratio = 1 \r\n
    vm.dirty_ratio = 40 \r\n<\/ul>\r\n
    Elasticsearch Testing<\/h3>Testing was completed with the same production Kafka data just redirected to the Nutanix setup. If you have deployed Elasticsearch (ES), you realize that your settings are tuned based on your workload but there are other settings that can be used to optimize search. Here are other settings we chose to change from defaults to take advantage of how Nutanix operates. The main settings we tuned are below.
    \r\n
    \r\n
    Indexing<\/h3>
    \r\n
    Index.refresh_interval = 600s \r\n<\/ul>\r\nThis setting is defaulted to 1s which means you will be creating a lot of segments and will be constantly syncing. Syncing this often is inefficient and creates a lot of unnecessary IO as you will be creating many smaller segments and constantly merging which can cause slower ingestion. This workload was not required to be searchable in real time, doing a sync every 10 minutes turned out to be the optimal setting.
    \r\n
    \r\n
    <\/a><\/p>
    \r\nThis graph shows when the merging begins\ufffc\ufffc. IOPS during normal ingest could be 10,000 IOPS for writes but when merging happens you can see the peaks of over 160,000 IOPS and throughput can double during the operation.
    \r\n
    \r\n
    Index.translog.flush_threshold_size =1024mb \r\n<\/ul>\r\nDefault is normally 512mb. This setting has seen indexing throughput improved
    \r\nby around 20%. Changing this setting has increased the time it might take to replay a single translog but the performance gain is worth it.
    \r\n
    \r\n
    Index.translog.durability = async \r\n<\/ul>\r\nDefault is request, changing this setting will push the flush to the kernel which will stop the app from possibly going into a blocking state.
    \r\n
    \r\nOverall Indexing as mentioned would average seven million documents per minute and could handle peaks up to 11 million documents at peak. With the average document level and 2Kb document size straight write throughput is around 12GBpm for primary data and 24GBpm for all indexing data. The Nutanix clusters had plenty of throughput to be able to handle the searches as well as the merge traffic shown above.
    \r\n
    \r\nThis image from Kibana shows the flow of traffic during the day. You can see how many documents the cluster is handling per second.
    \r\n
    \r\n
    <\/a><\/p>
    \r\n
    Search<\/b><\/h3>There are no real recommendations aside from the normal search optimizations to do for your workload. Nutanix is a distributed scalable storage platform that gives you low latency reads like bare-metal due to data locality. Elasticsearch recommends local storage due to the latency requirements. With Nutanix\u2019s data locality you get the low latency local reads that are needed when doing millions of searches a day.
    \r\n
    \r\n
    Failure Scenarios<\/b><\/h3> \r\n
    Index.translog.retention.size = 10gb \r\n<\/ul>\r\nDefault for this setting is 512mb which for a high throughput workload is small. Setting this to a operation based sync when recovering replicas vs doing a full copy. This does not play much of a role in the bare-metal environment as when a hardware node went down it normally went down hard and did not recover. With Nutanix though when the node would go down the data nodes restarted within 1 minute on another node and could quickly recover copies and get the cluster to green much faster.
    \r\n
    \r\n
    \r\n
    Index.unassigned.node_left.delayed_timeout = 15m \r\n<\/ul>\r\nThe default setting is 1m. This setting is a requirement on Nutanix more than on bare metal. With quick recovery times you don\u2019t want Elasticsearch to start recovering shards when the node will recover. If you are running with the default and the VM restarts within 1 minute 30 seconds Elasticsearch won\u2019t allow you to use those indexes again. What happens is that node now has 0 shards and will have to wait to rebalance as it is already recovering shards that were on that node. This leads to double duty of recovery then rebalance.
    \r\n
    \r\nThese settings help take full advantage of Nutanix HA for this mission critical workload. The difference was when one of the bare-metal nodes would go down hard it would take hours for the Elasticsearch cluster to go green and have both copies of shards available again. While with Nutanix HA the inactive indexes are green within a couple minutes of the node going down and active index under 30 minutes.
    \r\n
    \r\nThis means for the bare-metal there was only one copy to search of the shards slowing down search times. This also adds IO and network activity as Elasticsearch only does a one to one copy to recover shards, meaning a node that has the primary copies to the new node which can take hours. Meanwhile with Nutanix recovering its second copy in the background, it takes significantly less time as the whole cluster contributes to the rebuild.
    \r\n
    \r\n\ufffcA common scenario for this customer was a bad drive causing issues in the cluster. To get the best performance RAID 0 was chosen for bare-metal nodes, one to get the best throughput but also because during RAID rebuild there would be IO degradation affecting the entire cluster.
    \r\n
    \r\nIn this scenario of a single bad drive bringing down nodes, Nutanix handles this much more gracefully. We have multiple checks in the Nutanix system so when we see degraded performance of a drive or we get SMART error messages predicting the drive might fail we will proactively stop using that drive or eject it from the system all together. This, unlike the bare-metal server, keeps the data nodes from failing or from a single node becoming a performance bottleneck.
    \r\n
    \r\n
    Data Rebuild<\/b><\/h3>As I mentioned before, the setup is single copy (RF2) at the Elasticsearch layer and RF2 on Nutanix with the Elasticsearch copies split between two separate clusters.
    \r\n
    \r\nThe way Elasticsearch works when there is a failure and it has to create a new copy when one goes offline it is a straight one to one copy. If you have 10s of shards to rebuild these are one to one copies from the primary data node to the new secondary. Rebuilds of data are bottlenecked in this way as if the cluster as a whole is busy or the individual node it has chosen is busy then the recovery can be slow. During this time searches will be slower, and indexing could be hitting a bottleneck or at risk of data loss.
    \r\n
    \r\nIf a hardware node fails within the Nutanix cluster it realizes what data blocks need to be replicated to get back to green. With this scenario the entire Nutanix cluster participates in the rebuilding of data so no single node is being overtaxed on the rebuild and you have the distributed computing and IO power of the cluster. This means rebuilds are much faster even during heavier load. In this instance Elasticsearch also sees both copies of data to help keep search from being degraded.
    \r\n
    \r\n
    Conclusion<\/b><\/h3>When looking at the overall solution Nutanix makes great sense for combining your big data applications onto a single platform for better management and scalability. You can see from this example that you can also get the performance you need for even the most demanding workloads.
    \r\n
    \r\nFor further information on running Elasticsearch, read the Solution Note on the Elastic Stack on Nutanix AHV -- \/\/www.jhbzcj.com\/go\/virtualizing-elastic-stack-on-ahv.php<\/a>. You can also setup a briefing with our solution experts and services by sending us an email at info@nutanix.com<\/a>.
    \r\n
    \r\n
    Resources:<\/h3>
    \r\n
    Nutanix NEXT community thread on Elasticsearch: https:\/\/next.nutanix.com\/server-virtualization-27\/elasticsearch-on-nutanix-31873 \r\n
    Getting Started with Elasticsearch -- https:\/\/www.elastic.co\/webinars\/getting-started-elasticsearch<\/a> \r\n
    Nutanix Epoch documentation for Elasticsearch -- https:\/\/docs.epoch.nutanix.com\/integrations\/elastic\/ \r\n<\/ul>\r\nDisclaimer: This blog may contain links to external websites that are not part of Nutanix.com. Nutanix does not control these sites and disclaims all responsibility for the content or accuracy of any external site. Our decision to link to an external site should not be considered an endorsement of any content on such site.
    \r\n
    \r\n \u00a9 2019 Nutanix, Inc. All rights reserved. Nutanix, the Nutanix logo and the other Nutanix products and features mentioned herein are registered trademarks or trademarks of Nutanix, Inc. in the United States and other countries. All other brand names mentioned herein are for identification purposes only and may be the trademarks of their respective holder(s).","quoteUsername":"aluciani","translations":{"Common":{"like":"Like","unlike":"Unlike"},"Forum":{"Quote":"Quote","Share":"Share"}}}">
    
    就像
    
    报价
    
    分享

此主题已关闭征询意见

由内

条款和条件

注册

已经有一个帐户？登录

用你的帐户登入

登录社区

用你的帐户登入

输入您的用户名或电子邮件地址。我们会给你发一封电子邮件，告诉你如何重置密码。

用户名或电子邮件

回到概述

用于病毒的扫描文件。

抱歉，我们仍在检查这个文件的内容，以确保下载是安全的。请几分钟后再试。
好吧

无法下载此文件

对不起，我们的病毒扫描程序检测到这个文件下载不安全。
好吧

Learn more about our cookies.<\/a>","cookiepolicy.button":"Accept cookies","cookiepolicy.button.deny":"Deny all","cookiepolicy.link":"Cookie settings","cookiepolicy.modal.title":"Cookie settings","cookiepolicy.modal.content":"We use 3 different kinds of cookies. You can choose which cookies you want to accept. We need basic cookies to make this site work, therefore these are the minimum you can select. Learn more about our cookies.<\/a>","cookiepolicy.modal.level1":"Basic
Functional","cookiepolicy.modal.level2":"Normal
Functional + analytics","cookiepolicy.modal.level3":"Complete
Functional + analytics + social media + embedded videos"}}}">

在Nutanix上虚拟化Elasticsearch

Elasticsearch VM配置

Elasticsearch测试

索引

搜索

失败场景

数据重建

结论

资源：

Elasticsearch VM Configurations<\/h3> \r\n3 Manager Nodes \r\n6 Ingest Nodes \r\n

Resources:<\/h3> \r\nNutanix NEXT community thread on Elasticsearch: https:\/\/next.nutanix.com\/server-virtualization-27\/elasticsearch-on-nutanix-31873 \r\n

注册

登录社区

用于病毒的扫描文件。

无法下载此文件