pipelines

Creates, updates, deletes or gets a pipeline resource or lists pipelines in a region

Overview

Name	pipelines
Type	Resource
Description	An OpenSearch Ingestion Service Data Prepper pipeline running Data Prepper.
Id	aws.osis.pipelines

Fields

Name	Datatype	Description
buffer_options	object	Key-value pairs to configure buffering.
encryption_at_rest_options	object	Key-value pairs to configure encryption at rest.
log_publishing_options	object	Key-value pairs to configure log publishing.
max_units	integer	The maximum pipeline capacity, in Ingestion OpenSearch Compute Units (OCUs).
min_units	integer	The minimum pipeline capacity, in Ingestion OpenSearch Compute Units (OCUs).
pipeline_configuration_body	string	The Data Prepper pipeline configuration.
pipeline_name	string	Name of the OpenSearch Ingestion Service pipeline to create. Pipeline names are unique across the pipelines owned by an account within an AWS Region.
tags	array	An array of key-value pairs to apply to this resource.
vpc_options	object	Container for the values required to configure VPC access for the pipeline. If you don't specify these values, OpenSearch Ingestion Service creates the pipeline with a public endpoint.
vpc_endpoints	array	The VPC interface endpoints that have access to the pipeline.
pipeline_arn	string	The Amazon Resource Name (ARN) of the pipeline.
ingest_endpoint_urls	array	A list of endpoints that can be used for ingesting data into a pipeline
region	string	AWS region.

Methods

Name	Accessible by	Required Params
create_resource	INSERT	MaxUnits, MinUnits, PipelineConfigurationBody, PipelineName, region
delete_resource	DELETE	data__Identifier, region
update_resource	UPDATE	data__Identifier, data__PatchDocument, region
list_resources	SELECT	region
get_resource	SELECT	data__Identifier, region

SELECT examples

Gets all pipelines in a region.

SELECT
region,
buffer_options,
encryption_at_rest_options,
log_publishing_options,
max_units,
min_units,
pipeline_configuration_body,
pipeline_name,
tags,
vpc_options,
vpc_endpoints,
pipeline_arn,
ingest_endpoint_urls
FROM aws.osis.pipelines
WHERE region = 'us-east-1';

Gets all properties from an individual pipeline.

SELECT
region,
buffer_options,
encryption_at_rest_options,
log_publishing_options,
max_units,
min_units,
pipeline_configuration_body,
pipeline_name,
tags,
vpc_options,
vpc_endpoints,
pipeline_arn,
ingest_endpoint_urls
FROM aws.osis.pipelines
WHERE region = 'us-east-1' AND data__Identifier = '<PipelineArn>';

INSERT example

Use the following StackQL query and manifest file to create a new pipeline resource, using stack-deploy.

Required Properties

/*+ create */
INSERT INTO aws.osis.pipelines (
 MaxUnits,
 MinUnits,
 PipelineConfigurationBody,
 PipelineName,
 region
)
SELECT 
'{{ MaxUnits }}',
 '{{ MinUnits }}',
 '{{ PipelineConfigurationBody }}',
 '{{ PipelineName }}',
'{{ region }}';

All Properties

/*+ create */
INSERT INTO aws.osis.pipelines (
 BufferOptions,
 EncryptionAtRestOptions,
 LogPublishingOptions,
 MaxUnits,
 MinUnits,
 PipelineConfigurationBody,
 PipelineName,
 Tags,
 VpcOptions,
 region
)
SELECT 
 '{{ BufferOptions }}',
 '{{ EncryptionAtRestOptions }}',
 '{{ LogPublishingOptions }}',
 '{{ MaxUnits }}',
 '{{ MinUnits }}',
 '{{ PipelineConfigurationBody }}',
 '{{ PipelineName }}',
 '{{ Tags }}',
 '{{ VpcOptions }}',
 '{{ region }}';

Manifest

version: 1
name: stack name
description: stack description
providers:
  - aws
globals:
  - name: region
    value: '{{ vars.AWS_REGION }}'
resources:
  - name: pipeline
    props:
      - name: BufferOptions
        value:
          PersistentBufferEnabled: '{{ PersistentBufferEnabled }}'
      - name: EncryptionAtRestOptions
        value:
          KmsKeyArn: '{{ KmsKeyArn }}'
      - name: LogPublishingOptions
        value:
          IsLoggingEnabled: '{{ IsLoggingEnabled }}'
          CloudWatchLogDestination:
            LogGroup: '{{ LogGroup }}'
      - name: MaxUnits
        value: '{{ MaxUnits }}'
      - name: MinUnits
        value: '{{ MinUnits }}'
      - name: PipelineConfigurationBody
        value: '{{ PipelineConfigurationBody }}'
      - name: PipelineName
        value: '{{ PipelineName }}'
      - name: Tags
        value:
          - Key: '{{ Key }}'
            Value: '{{ Value }}'
      - name: VpcOptions
        value:
          SecurityGroupIds:
            - '{{ SecurityGroupIds[0] }}'
          SubnetIds:
            - '{{ SubnetIds[0] }}'

DELETE example

/*+ delete */
DELETE FROM aws.osis.pipelines
WHERE data__Identifier = '<PipelineArn>'
AND region = 'us-east-1';

Permissions

To operate on the pipelines resource, the following permissions are required:

Create

osis:CreatePipeline,
osis:GetPipeline,
osis:TagResource,
osis:ListTagsForResource,
iam:PassRole,
iam:CreateServiceLinkedRole,
logs:CreateLogDelivery,
kms:DescribeKey

Read

osis:GetPipeline,
osis:ListTagsForResource

Update

osis:UpdatePipeline,
osis:GetPipeline,
osis:ListTagsForResource,
osis:TagResource,
osis:UntagResource,
iam:PassRole,
logs:GetLogDelivery,
logs:UpdateLogDelivery,
logs:ListLogDeliveries,
kms:DescribeKey

Delete

osis:DeletePipeline,
osis:GetPipeline,
logs:GetLogDelivery,
logs:DeleteLogDelivery,
logs:ListLogDeliveries

List

osis:ListPipelines