clusters

Creates, updates, deletes or gets a cluster resource or lists clusters in a region

Overview

Name	clusters
Type	Resource
Description	Resource Type definition for AWS::MSK::Cluster
Id	aws.msk.clusters

Fields

Name	Datatype	Description
broker_node_group_info	object
enhanced_monitoring	string
kafka_version	string
number_of_broker_nodes	integer
encryption_info	object
open_monitoring	object
cluster_name	string
arn	string
current_version	string	The current version of the MSK cluster
client_authentication	object
logging_info	object
tags	object	A key-value pair to associate with a resource.
configuration_info	object
storage_mode	string
region	string	AWS region.

Methods

Name	Accessible by	Required Params
create_resource	INSERT	BrokerNodeGroupInfo, KafkaVersion, NumberOfBrokerNodes, ClusterName, region
delete_resource	DELETE	data__Identifier, region
update_resource	UPDATE	data__Identifier, data__PatchDocument, region
list_resources	SELECT	region
get_resource	SELECT	data__Identifier, region

SELECT examples

Gets all clusters in a region.

SELECT
region,
broker_node_group_info,
enhanced_monitoring,
kafka_version,
number_of_broker_nodes,
encryption_info,
open_monitoring,
cluster_name,
arn,
current_version,
client_authentication,
logging_info,
tags,
configuration_info,
storage_mode
FROM aws.msk.clusters
WHERE region = 'us-east-1';

Gets all properties from an individual cluster.

SELECT
region,
broker_node_group_info,
enhanced_monitoring,
kafka_version,
number_of_broker_nodes,
encryption_info,
open_monitoring,
cluster_name,
arn,
current_version,
client_authentication,
logging_info,
tags,
configuration_info,
storage_mode
FROM aws.msk.clusters
WHERE region = 'us-east-1' AND data__Identifier = '<Arn>';

INSERT example

Use the following StackQL query and manifest file to create a new cluster resource, using stack-deploy.

Required Properties

/*+ create */
INSERT INTO aws.msk.clusters (
 BrokerNodeGroupInfo,
 KafkaVersion,
 NumberOfBrokerNodes,
 ClusterName,
 region
)
SELECT 
'{{ BrokerNodeGroupInfo }}',
 '{{ KafkaVersion }}',
 '{{ NumberOfBrokerNodes }}',
 '{{ ClusterName }}',
'{{ region }}';

All Properties

/*+ create */
INSERT INTO aws.msk.clusters (
 BrokerNodeGroupInfo,
 EnhancedMonitoring,
 KafkaVersion,
 NumberOfBrokerNodes,
 EncryptionInfo,
 OpenMonitoring,
 ClusterName,
 CurrentVersion,
 ClientAuthentication,
 LoggingInfo,
 Tags,
 ConfigurationInfo,
 StorageMode,
 region
)
SELECT 
 '{{ BrokerNodeGroupInfo }}',
 '{{ EnhancedMonitoring }}',
 '{{ KafkaVersion }}',
 '{{ NumberOfBrokerNodes }}',
 '{{ EncryptionInfo }}',
 '{{ OpenMonitoring }}',
 '{{ ClusterName }}',
 '{{ CurrentVersion }}',
 '{{ ClientAuthentication }}',
 '{{ LoggingInfo }}',
 '{{ Tags }}',
 '{{ ConfigurationInfo }}',
 '{{ StorageMode }}',
 '{{ region }}';

Manifest

version: 1
name: stack name
description: stack description
providers:
  - aws
globals:
  - name: region
    value: '{{ vars.AWS_REGION }}'
resources:
  - name: cluster
    props:
      - name: BrokerNodeGroupInfo
        value:
          StorageInfo:
            EBSStorageInfo:
              VolumeSize: '{{ VolumeSize }}'
              ProvisionedThroughput:
                Enabled: '{{ Enabled }}'
                VolumeThroughput: '{{ VolumeThroughput }}'
          ConnectivityInfo:
            PublicAccess:
              Type: '{{ Type }}'
            VpcConnectivity:
              ClientAuthentication:
                Tls:
                  Enabled: '{{ Enabled }}'
                Sasl:
                  Scram:
                    Enabled: '{{ Enabled }}'
                  Iam:
                    Enabled: '{{ Enabled }}'
          SecurityGroups:
            - '{{ SecurityGroups[0] }}'
          BrokerAZDistribution: '{{ BrokerAZDistribution }}'
          ClientSubnets:
            - '{{ ClientSubnets[0] }}'
          InstanceType: '{{ InstanceType }}'
      - name: EnhancedMonitoring
        value: '{{ EnhancedMonitoring }}'
      - name: KafkaVersion
        value: '{{ KafkaVersion }}'
      - name: NumberOfBrokerNodes
        value: '{{ NumberOfBrokerNodes }}'
      - name: EncryptionInfo
        value:
          EncryptionAtRest:
            DataVolumeKMSKeyId: '{{ DataVolumeKMSKeyId }}'
          EncryptionInTransit:
            InCluster: '{{ InCluster }}'
            ClientBroker: '{{ ClientBroker }}'
      - name: OpenMonitoring
        value:
          Prometheus:
            JmxExporter:
              EnabledInBroker: '{{ EnabledInBroker }}'
            NodeExporter:
              EnabledInBroker: '{{ EnabledInBroker }}'
      - name: ClusterName
        value: '{{ ClusterName }}'
      - name: CurrentVersion
        value: '{{ CurrentVersion }}'
      - name: ClientAuthentication
        value:
          Sasl:
            Iam:
              Enabled: '{{ Enabled }}'
      - name: LoggingInfo
        value:
          BrokerLogs:
            S3:
              Enabled: '{{ Enabled }}'
              Prefix: '{{ Prefix }}'
              Bucket: '{{ Bucket }}'
            CloudWatchLogs:
              LogGroup: '{{ LogGroup }}'
              Enabled: '{{ Enabled }}'
            Firehose:
              Enabled: '{{ Enabled }}'
              DeliveryStream: '{{ DeliveryStream }}'
      - name: Tags
        value: {}
      - name: ConfigurationInfo
        value:
          Revision: '{{ Revision }}'
          Arn: '{{ Arn }}'
      - name: StorageMode
        value: '{{ StorageMode }}'

DELETE example

/*+ delete */
DELETE FROM aws.msk.clusters
WHERE data__Identifier = '<Arn>'
AND region = 'us-east-1';

Permissions

To operate on the clusters resource, the following permissions are required:

Create

ec2:DescribeSecurityGroups,
ec2:DescribeSubnets,
ec2:DescribeVpcs,
iam:AttachRolePolicy,
iam:CreateServiceLinkedRole,
iam:PutRolePolicy,
kms:CreateGrant,
kms:DescribeKey,
kafka:CreateCluster,
kafka:DescribeCluster,
kafka:TagResource,
logs:CreateLogDelivery,
logs:GetLogDelivery,
logs:UpdateLogDelivery,
logs:DeleteLogDelivery,
logs:ListLogDeliveries,
s3:GetBucketPolicy,
s3:PutBucketPolicy,
logs:PutResourcePolicy,
logs:DescribeResourcePolicies,
logs:DescribeLogGroups,
firehose:TagDeliveryStream,
acm-pca:GetCertificateAuthorityCertificate

Update

kafka:UpdateMonitoring,
kafka:UpdateClusterKafkaVersion,
kafka:UpdateClusterConfiguration,
kafka:UpdateBrokerType,
kafka:UpdateBrokerCount,
kafka:UpdateBrokerStorage,
kafka:UpdateStorage,
kafka:UpdateSecurity,
kafka:UpdateConnectivity,
kafka:DescribeCluster,
kafka:DescribeClusterOperation,
kafka:TagResource,
kafka:UntagResource,
ec2:DescribeSubnets,
ec2:DescribeVpcs,
ec2:DescribeSecurityGroups,
iam:AttachRolePolicy,
iam:CreateServiceLinkedRole,
iam:PutRolePolicy,
kms:DescribeKey,
kms:CreateGrant,
logs:CreateLogDelivery,
logs:GetLogDelivery,
logs:UpdateLogDelivery,
logs:DeleteLogDelivery,
logs:ListLogDeliveries,
s3:GetBucketPolicy,
logs:PutResourcePolicy,
logs:DescribeResourcePolicies,
logs:DescribeLogGroups,
firehose:TagDeliveryStream,
acm-pca:GetCertificateAuthorityCertificate

Delete

kafka:DeleteCluster,
kafka:DescribeCluster

List

kafka:ListClusters

Read

kafka:DescribeCluster