Quotas and Limitations of Various Managed Kafaka Services

This post contains a comparison of different managed Kafka-enabled message/event processing services available for AWS, Azure and Google Cloud Platform (GCP).

  • I know I need to format this a bit better.
Description Azure Event Hub Azure Event Hub Dedicated Confluent Cloud Confluent Cloud Enterprise AWS MSK
Brokers Scales automatically Scales automatically Scales automatically Scales automatically Number of brokers have to be pre-defined. Up to 90 per account 15 Brokers per cluster
Max clusters/namespaces 100 50 per CU* 5 No Limit 90 single broker clusters per account
Max topics/event hubs 10 per namespace 1000 per namespace No Limit No Limit No Limit
Max partitions 32 2000 per CU* 2048 100000 Only limited by the open file handle limit in the underlying operating system
Max consumer groups 20 No limit per CU*, 1000 per event hub No Limit No Limit No Limit
Max message size 1 MB 1 MB 2 MB Default is 2 MB, but can be increased to 6 MB No limit. Batch size can even be specified at topic level.
Multi-zone high availability Yes Yes Coming soon Yes Yes
ACLs Yes Yes Yes Yes Yes
Cluster throughput Write: Up to 20 TUs **(20 MBps) Read: Up to 20 TUs** (40 MBps) Write: Up to 20 CUs* Read: Up to 20 Cus* Write: Up to 100 MBps Read: Up to 100 MBps Write: Unlimited Read: Unlimited No listed limit as you can procure as many EC2 instances as you like
Data retention 84 GB/TU** 10 TB/CU* 5 TB Unlimited A maximum of 16384 GB per broker
Encryption at rest Yes  (Azure SSE Yes  (Azure SSE & CM Keys) Yes (encrypted volumes) Yes (encrypted volumes) Yes (EBS SS & KMS)
Encryption in motion Yes Yes Yes (SSL/TLS) Yes (SSL/TLS) Yes
Exactly Once Semantics Yes Yes Yes Yes Yes
Terraform Support Yes Yes Yes Yes Yes
SLA 99.95% 99.99% 99.95% 99.95% 99.9%

You May Also Like

About the Author: Irtaza

Leave a Reply

Your email address will not be published. Required fields are marked *