加米谷大数据吧 关注:18贴子:272
  • 0回复贴,共1

kafka设计动机

只看楼主收藏回复

We designed Kafka to be able to act as a unified platform for handling all the real-time data feeds a
large company might have. To do this we had to think through a fairly broad set of use cases.
我们设计kafka是必须能够作为一个统一的平台,来处理一家大公司可能有的所有实时数据。要做到这一点,我们不得不
思考一个相当广泛的用例。
It would have to have high-throughput to support high volume event streams such as real-time log
aggregation.
它必须具有高吞吐量,以支持大容量的事件流,如实时日志聚集。
It would need to deal gracefully with large data backlogs to be able to support periodic data loads from
offline systems.
它将需要优雅地处理大型数据积压,要能够支持从脱机系统的定期数据装载。
It also meant the system would have to handle low-latency delivery to handle more traditional messaging
use-cases.
这也意味着该系统将不得不处理低延迟交付,以处理更传统的消息传递的用例。
We wanted to support partitioned, distributed, real-time processing of these feeds to create new, derived
feeds. This motivated our partitioning and consumer model.
我们想要支持以分区,分布式的,实时处理来这些创建新派生的feeds,促使我们的分区和消费模式。
Finally in cases where the stream is fed into other data systems for serving we new the system would have
to be able to guarantee fault-tolerance in the presence of machine failures.
最后,在其中该流被送入其他数据系统用于服务的情况下,我们的新的系统必须能够保证在机器故障存在的容错。
Supporting these uses led use to a design with a number of unique elements, more akin to a database log
then a traditional messaging system. We will outline some elements of the design in the following
sections


1楼2018-01-16 16:09回复