Amazon MSK – Managed Streaming for Apache Kafka

Fully managed, highly available, and secure Apache Kafka service.

Efficiently scale your data streaming

Amazon Managed Streaming for Apache Kafka (Amazon MSK) is a fully managed service that makes it easy for you to build and run applications that use Apache Kafka to process streaming data. Apache Kafka is an open-source platform for building real-time streaming data pipelines and applications.

With Amazon MSK, you can use Apache Kafka APIs to populate data lakes, stream changes to and from databases, and power machine learning and analytics applications. Apache Kafka clusters are challenging to setup, scale, and manage in production. When you run Apache Kafka on your own, you need to provision servers, configure Apache Kafka manually, replace servers when they fail, orchestrate server patches and upgrades, architect the cluster for high availability, ensure data is durably stored and secured, setup monitoring and alarms, and carefully plan scaling events to support load changes. Amazon MSK makes it easy for you to build and run production applications on Apache Kafka without needing Apache Kafka infrastructure management expertise. That means you spend less time managing infrastructure and more time building applications.

With a few clicks in the Amazon MSK console you can create highly available Apache Kafka clusters with settings and configuration based on Apache Kafka’s deployment best practices. Amazon MSK automatically provisions and runs your Apache Kafka clusters. Amazon MSK continuously monitors cluster health and automatically replaces unhealthy nodes with no downtime to your application. In addition, Amazon MSK secures your Apache Kafka cluster by encrypting data at rest.

Picture

Benefits

Amazon MSK handles provisioning, configuration, and maintenance of Apache Kafka clusters and Apache ZooKeeper nodes. Key performance metrics are accessible via the AWS web console, allowing users to focus on creating streaming applications without the operational overhead.

Amazon MSK ensures high availability by monitoring cluster health and replacing unhealthy brokers without downtime. It manages Apache ZooKeeper nodes, deploys necessary software patches, and uses multi-Availability Zone replication.

Amazon MSK offers multiple security layers for Apache Kafka clusters, including VPC network isolation, AWS IAM for API authorization, encryption at rest, and TLS encryption in-transit.

Amazon MSK runs and manages Apache Kafka for you. Enabling seamless migration and operation of existing applications on AWS without code changes. It maintains open-source compatibility, supporting custom and community-built tools like MirrorMaker.

There are two kind of scaling: broker scaling, it is possible to increase or decrease the number of the brokers of the cluster, and storage scaling, by modifying the amount of storage provisioned per broker to match changes in storage requirements.

Data Reply best practices

Data Reply, an AWS Premier Consulting Partner, has honed its expertise in implementing AWS Big Data platforms. Over time, they have mastered the use of AWS MSK, ensuring reliability and cost savings.Their best practices are built on four main features of Amazon MSK.

Integration

Deep integration with several other AWS services (IAM. KMS, Lambda, etc.).

Innovation

Fast innovation and constantly improved (42+ new features in 2 years).

Costs

Up to 40% cheaper compare to a self-managed Kafka cluster.

Compatibility

Fully compatible with Kafka APIs but not necessary to manage Zookeeper.

Data Reply Migration Approach

As Data Reply we provide our expertise in AWS Migration, built in different industrial sectors, among several years of projects. We distilled our expertise into our Migration Approach, that consists in 4 different modules which can be adapted to the customer’s needs.

Requirements & business use case

Understand key business challenges and goals, in order to identify gaps and opportunity, and plan current and future state

Technical workshop

During workshop phase, we can perform a Technical & Opportunity assessment, planning technical deep dive session, in order to identify migration success criteria, business & IT Data Lake outcomes

Pilot

Scope of the Pilot phase is to create a simple Pilot of the target use case, in order to allow customers to have a concrete way to test the solution. We define target architecture & component level mapping, according to requirements collected in previous phases, and execute incremental data migration and automation. After the UAT step, the Pilot is ready to Go Live!

Implementation

This is typically the last step, where we implement the final solution, split into Waves to guarantee a progressive release of the solution. We define full migration strategy and schedule, and the application code migration with Dual Target approach. Later, we can start with waves of implementation, including bulk import/export and the validation and audit of the solution. After the Test & UAT phases, we are ready for a successful GO LIVE!