connectors

Creates, updates, deletes or gets a connector resource or lists connectors in a region

Overview

Name	connectors
Type	Resource
Description	Resource Type definition for AWS::KafkaConnect::Connector
Id	aws.kafkaconnect.connectors

Fields

Name	Datatype	Description
capacity	object	Information about the capacity allocated to the connector.
connector_arn	string	Amazon Resource Name for the created Connector.
connector_configuration	object	The configuration for the connector.
connector_description	string	A summary description of the connector.
connector_name	string	The name of the connector.
kafka_cluster	object	Details of how to connect to the Kafka cluster.
kafka_cluster_client_authentication	object	Details of the client authentication used by the Kafka cluster.
kafka_cluster_encryption_in_transit	object	Details of encryption in transit to the Kafka cluster.
kafka_connect_version	string	The version of Kafka Connect. It has to be compatible with both the Kafka cluster's version and the plugins.
log_delivery	object	Details of what logs are delivered and where they are delivered.
plugins	array	List of plugins to use with the connector.
service_execution_role_arn	string	The Amazon Resource Name (ARN) of the IAM role used by the connector to access Amazon S3 objects and other external resources.
tags	array	A collection of tags associated with a resource
worker_configuration	object	The configuration of the workers, which are the processes that run the connector logic.
region	string	AWS region.

Methods

Name	Accessible by	Required Params
create_resource	INSERT	Capacity, ConnectorConfiguration, ConnectorName, KafkaConnectVersion, KafkaCluster, KafkaClusterClientAuthentication, KafkaClusterEncryptionInTransit, Plugins, ServiceExecutionRoleArn, region
delete_resource	DELETE	data__Identifier, region
update_resource	UPDATE	data__Identifier, data__PatchDocument, region
list_resources	SELECT	region
get_resource	SELECT	data__Identifier, region

SELECT examples

Gets all connectors in a region.

SELECT
region,
capacity,
connector_arn,
connector_configuration,
connector_description,
connector_name,
kafka_cluster,
kafka_cluster_client_authentication,
kafka_cluster_encryption_in_transit,
kafka_connect_version,
log_delivery,
plugins,
service_execution_role_arn,
tags,
worker_configuration
FROM aws.kafkaconnect.connectors
WHERE region = 'us-east-1';

Gets all properties from an individual connector.

SELECT
region,
capacity,
connector_arn,
connector_configuration,
connector_description,
connector_name,
kafka_cluster,
kafka_cluster_client_authentication,
kafka_cluster_encryption_in_transit,
kafka_connect_version,
log_delivery,
plugins,
service_execution_role_arn,
tags,
worker_configuration
FROM aws.kafkaconnect.connectors
WHERE region = 'us-east-1' AND data__Identifier = '<ConnectorArn>';

INSERT example

Use the following StackQL query and manifest file to create a new connector resource, using stack-deploy.

Required Properties

/*+ create */
INSERT INTO aws.kafkaconnect.connectors (
 Capacity,
 ConnectorConfiguration,
 ConnectorName,
 KafkaCluster,
 KafkaClusterClientAuthentication,
 KafkaClusterEncryptionInTransit,
 KafkaConnectVersion,
 Plugins,
 ServiceExecutionRoleArn,
 region
)
SELECT 
'{{ Capacity }}',
 '{{ ConnectorConfiguration }}',
 '{{ ConnectorName }}',
 '{{ KafkaCluster }}',
 '{{ KafkaClusterClientAuthentication }}',
 '{{ KafkaClusterEncryptionInTransit }}',
 '{{ KafkaConnectVersion }}',
 '{{ Plugins }}',
 '{{ ServiceExecutionRoleArn }}',
'{{ region }}';

All Properties

/*+ create */
INSERT INTO aws.kafkaconnect.connectors (
 Capacity,
 ConnectorConfiguration,
 ConnectorDescription,
 ConnectorName,
 KafkaCluster,
 KafkaClusterClientAuthentication,
 KafkaClusterEncryptionInTransit,
 KafkaConnectVersion,
 LogDelivery,
 Plugins,
 ServiceExecutionRoleArn,
 Tags,
 WorkerConfiguration,
 region
)
SELECT 
 '{{ Capacity }}',
 '{{ ConnectorConfiguration }}',
 '{{ ConnectorDescription }}',
 '{{ ConnectorName }}',
 '{{ KafkaCluster }}',
 '{{ KafkaClusterClientAuthentication }}',
 '{{ KafkaClusterEncryptionInTransit }}',
 '{{ KafkaConnectVersion }}',
 '{{ LogDelivery }}',
 '{{ Plugins }}',
 '{{ ServiceExecutionRoleArn }}',
 '{{ Tags }}',
 '{{ WorkerConfiguration }}',
 '{{ region }}';

Manifest

version: 1
name: stack name
description: stack description
providers:
  - aws
globals:
  - name: region
    value: '{{ vars.AWS_REGION }}'
resources:
  - name: connector
    props:
      - name: Capacity
        value:
          AutoScaling:
            MaxWorkerCount: '{{ MaxWorkerCount }}'
            MinWorkerCount: '{{ MinWorkerCount }}'
            ScaleInPolicy:
              CpuUtilizationPercentage: '{{ CpuUtilizationPercentage }}'
            ScaleOutPolicy:
              CpuUtilizationPercentage: '{{ CpuUtilizationPercentage }}'
            McuCount: '{{ McuCount }}'
          ProvisionedCapacity:
            McuCount: '{{ McuCount }}'
            WorkerCount: '{{ WorkerCount }}'
      - name: ConnectorConfiguration
        value: {}
      - name: ConnectorDescription
        value: '{{ ConnectorDescription }}'
      - name: ConnectorName
        value: '{{ ConnectorName }}'
      - name: KafkaCluster
        value:
          ApacheKafkaCluster:
            BootstrapServers: '{{ BootstrapServers }}'
            Vpc:
              SecurityGroups:
                - '{{ SecurityGroups[0] }}'
              Subnets:
                - '{{ Subnets[0] }}'
      - name: KafkaClusterClientAuthentication
        value:
          AuthenticationType: '{{ AuthenticationType }}'
      - name: KafkaClusterEncryptionInTransit
        value:
          EncryptionType: '{{ EncryptionType }}'
      - name: KafkaConnectVersion
        value: '{{ KafkaConnectVersion }}'
      - name: LogDelivery
        value:
          WorkerLogDelivery:
            CloudWatchLogs:
              Enabled: '{{ Enabled }}'
              LogGroup: '{{ LogGroup }}'
            Firehose:
              DeliveryStream: '{{ DeliveryStream }}'
              Enabled: '{{ Enabled }}'
            S3:
              Bucket: '{{ Bucket }}'
              Enabled: '{{ Enabled }}'
              Prefix: '{{ Prefix }}'
      - name: Plugins
        value:
          - CustomPlugin:
              Name: '{{ Name }}'
              Description: '{{ Description }}'
              ContentType: '{{ ContentType }}'
              Location:
                S3Location:
                  BucketArn: '{{ BucketArn }}'
                  FileKey: '{{ FileKey }}'
                  ObjectVersion: '{{ ObjectVersion }}'
              Tags:
                - Key: '{{ Key }}'
                  Value: '{{ Value }}'
      - name: ServiceExecutionRoleArn
        value: '{{ ServiceExecutionRoleArn }}'
      - name: Tags
        value:
          - null
      - name: WorkerConfiguration
        value:
          Name: '{{ Name }}'
          Description: '{{ Description }}'
          PropertiesFileContent: '{{ PropertiesFileContent }}'
          Tags:
            - null

DELETE example

/*+ delete */
DELETE FROM aws.kafkaconnect.connectors
WHERE data__Identifier = '<ConnectorArn>'
AND region = 'us-east-1';

Permissions

To operate on the connectors resource, the following permissions are required:

Create

kafkaconnect:CreateConnector,
kafkaconnect:DescribeConnector,
kafkaconnect:TagResource,
kafkaconnect:ListTagsForResource,
iam:CreateServiceLinkedRole,
iam:PassRole,
ec2:CreateNetworkInterface,
ec2:DescribeSecurityGroups,
ec2:DescribeSubnets,
ec2:DescribeVpcs,
logs:CreateLogDelivery,
logs:GetLogDelivery,
logs:ListLogDeliveries,
logs:PutResourcePolicy,
logs:DescribeResourcePolicies,
logs:DescribeLogGroups,
s3:GetBucketPolicy,
s3:PutBucketPolicy,
firehose:TagDeliveryStream

Read

kafkaconnect:DescribeConnector,
kafkaconnect:ListTagsForResource

Delete

kafkaconnect:DeleteConnector,
kafkaconnect:DescribeConnector,
logs:DeleteLogDelivery,
logs:GetLogDelivery,
logs:ListLogDeliveries

Update

kafkaconnect:UpdateConnector,
kafkaconnect:DescribeConnector,
kafkaconnect:TagResource,
kafkaconnect:ListTagsForResource,
kafkaconnect:UntagResource,
iam:CreateServiceLinkedRole,
logs:UpdateLogDelivery,
logs:GetLogDelivery,
logs:ListLogDeliveries,
logs:PutResourcePolicy,
logs:DescribeResourcePolicies,
logs:DescribeLogGroups,
s3:GetBucketPolicy,
s3:PutBucketPolicy,
firehose:TagDeliveryStream

List

kafkaconnect:ListConnectors