Building Scalable Data Pipelines on AWS: Kinesis, Glue, and Redshift

As a senior software engineer at Khadervali.com, I’ve seen firsthand how critical efficient data pipelines are for modern businesses. In today’s data-driven world, the ability to collect, process, and analyze vast amounts of data in near real-time is no longer a luxury but a necessity. From understanding customer behavior to monitoring application performance and detecting anomalies, timely insights are key to staying competitive.

However, building these pipelines can be challenging. You need to handle diverse data sources, varying data volumes and velocities, complex transformations, and ultimately, deliver data to an analytical store that can serve diverse business intelligence (BI) and reporting needs. Traditional on-premise solutions often struggle with scalability, maintenance, and cost-effectiveness.

Enter Amazon Web Services (AWS). With its comprehensive suite of managed services, AWS provides an incredibly powerful and flexible platform for constructing robust data pipelines. In this article, we’ll dive deep into building a scalable, near real-time data pipeline using three cornerstone AWS services: Amazon Kinesis for data ingestion, AWS Glue for serverless ETL (Extract, Transform, Load), and Amazon Redshift as our petabyte-scale data warehouse. This combination allows us to ingest streaming data, transform it efficiently, and make it available for high-performance analytical queries.

By the end of this guide, you’ll have a solid understanding of how these services work together, practical code examples, and best practices to implement your own robust data pipelines.

Understanding the Core AWS Services

Before we piece together our pipeline, let’s briefly introduce the key players and understand their individual strengths.

Amazon Kinesis: The Real-Time Data Stream

Amazon Kinesis is a fully managed service that makes it easy to collect, process, and analyze real-time, streaming data. It’s designed for high-throughput, low-latency data ingestion. Kinesis offers several capabilities, but for our pipeline, we’ll primarily focus on Kinesis Data Streams and potentially Kinesis Data Firehose.

• Kinesis Data Streams (KDS): This is the backbone for real-time data processing. It allows you to continuously capture and store gigabytes per second of data from hundreds of thousands of sources. Data records are stored in shards for a configurable retention period (1-365 days) and can be consumed by multiple applications concurrently. KDS is ideal when you need custom processing logic for your streaming data.

  • Pros: High throughput, low latency, flexible consumption by multiple applications, custom processing.
  • Cons: Requires consumers to be built and managed (e.g., Lambda, EC2).

• Kinesis Data Firehose (KDF): A fully managed service for delivering real-time streaming data to destinations like Amazon S3, Amazon Redshift, Amazon OpenSearch Service, and Splunk. Firehose automatically scales to match the throughput of your data and requires no administration. It can also transform, compress, and encrypt data before delivery.

  • Pros: Fully managed, auto-scaling, simpler setup for direct delivery to destinations, built-in transformation/compression.
  • Cons: Less flexible for complex, multi-stage real-time processing compared to KDS.

For our pipeline, Kinesis Data Streams will serve as the initial ingestion point for raw, real-time data, allowing us to capture events as they happen.

AWS Glue: The Serverless ETL Engine

AWS Glue is a fully managed, serverless ETL service that makes it easy to prepare and load your data for analytics. It’s particularly powerful for working with data lakes, as it integrates seamlessly with Amazon S3. Glue comprises several key components:

• Glue Data Catalog: A persistent metadata store for all your data assets, regardless of where they reside (S3, RDS, etc.). It stores table definitions, schemas, and partitions, making your data discoverable and queryable by other AWS services like Athena, Redshift Spectrum, and EMR.

• Glue Crawlers: Programs that connect to a data store, progress through a prioritized list of classifiers to extract the schema of your data, and then write the metadata to the Glue Data Catalog. They can infer schemas for various data formats (JSON, CSV, Parquet, ORC, etc.).

• Glue Jobs: The ETL engine itself. Glue Jobs run Apache Spark or Python shell scripts to perform data transformations. You can author these scripts using the Glue console, or develop them locally and upload. Glue takes care of provisioning, configuring, and scaling the underlying compute resources.

• Glue Triggers: Mechanisms to start Glue jobs automatically, either on a schedule or in response to events (e.g., new data arriving in an S3 bucket).

AWS Glue will be our workhorse for transforming raw data from our S3 landing zone into a clean, structured format suitable for Redshift.

Amazon Redshift: The Petabyte-Scale Data Warehouse

Amazon Redshift is a fully managed, petabyte-scale data warehousing service. It’s designed for analytical workloads and excels at complex queries over large datasets, making it ideal for business intelligence, reporting, and big data analytics.

• Columnar Storage: Redshift stores data in a columnar format, which significantly reduces the I/O operations required to process analytical queries. Instead of reading entire rows, it only reads the columns relevant to the query.

• Massively Parallel Processing (MPP): Redshift uses an MPP architecture, distributing data and query processing across multiple compute nodes. This parallel execution enables fast performance for complex queries.

• Integration with Data Lake: Redshift Spectrum allows you to query data directly from S3 without loading it into Redshift, enabling you to combine data in your data lake with data in your data warehouse.

• Scalability and Performance: Redshift can scale up or down as your data needs change, and offers various node types optimized for different workloads.

Redshift will be the final destination for our processed data, providing a high-performance environment for analysts and BI tools to extract valuable insights.

Designing the Architecture: Kinesis, Glue, and Redshift Data Pipeline

Let’s put these services together to form a robust, near real-time data pipeline. Our goal is to ingest streaming data, land it in a raw format on S3, transform it using a serverless ETL service, and finally load it into a data warehouse for analytical queries.

Overall Architecture Flow (in words)

1. Data Producers: Your applications, IoT devices, or log aggregators continuously generate data (e.g., clickstream events, sensor readings, transaction records).

2. Real-time Ingestion (Amazon Kinesis Data Streams): Data producers send their events to a Kinesis Data Stream. KDS acts as a highly scalable, durable buffer for streaming data.

3. Raw Data Landing (Kinesis Data Firehose to Amazon S3): A Kinesis Data Firehose subscribes to the Kinesis Data Stream. Firehose then buffers the incoming data, optionally transforms it (e.g., format conversion), and delivers it in batches to an Amazon S3 bucket. This S3 bucket serves as our raw data landing zone or a “data lake” layer for unprocessed data.

   Alternative: Instead of KDS + KDF, producers could directly send to KDF if no custom real-time processing is needed before S3. However, KDS offers more flexibility for multiple consumers and custom real-time processing (e.g., anomaly detection with Kinesis Data Analytics) before landing in S3.

4. Metadata Discovery (AWS Glue Crawler): An AWS Glue Crawler periodically scans the newly arrived raw data in the S3 landing zone. It infers the schema and partitions, then updates the AWS Glue Data Catalog, making the raw data structure discoverable.

5. Serverless ETL (AWS Glue Job): An AWS Glue Spark ETL job is triggered (e.g., on a schedule or via an S3 event when new files land). This job reads the raw data from S3 (using the schema from the Glue Data Catalog), performs necessary transformations (filtering, cleaning, joining, aggregation, data type conversions), and then loads the cleaned, transformed data into Amazon Redshift.

6. Data Warehousing (Amazon Redshift): The processed data resides in Amazon Redshift, optimized for fast analytical queries. BI tools and analysts can now query this data for insights.

7. Monitoring & Logging (Amazon CloudWatch): All services generate logs and metrics, which are collected by CloudWatch for monitoring the pipeline’s health and performance.

Component-wise Breakdown and Implementation Details

1. Data Ingestion Layer: Amazon Kinesis Data Streams and Firehose

Our pipeline begins with capturing data. Let’s assume we’re collecting real-time application logs or clickstream events.

a. Setting up Kinesis Data Stream (KDS)

You can create a Kinesis Data Stream via the AWS Management Console, CLI, or CloudFormation.

aws kinesis create-stream \
    --stream-name my-app-event-stream \
    --shard-count 1

Choose your shard-count based on your expected throughput. Each shard can typically handle up to 1 MB/sec or 1000 records/sec for writes, and 2 MB/sec for reads.

b. Writing a Kinesis Producer

Data producers push records to KDS. Here’s a Python example using the Boto3 SDK:

import boto3
import json
import random
import time
from datetime import datetime

# Initialize Kinesis client
kinesis_client = boto3.client('kinesis', region_name='your-aws-region')
stream_name = 'my-app-event-stream'

def generate_event():
    """Generates a sample event record."""
    event_types = ['page_view', 'add_to_cart', 'purchase', 'login', 'logout']
    products = ['Laptop', 'Mouse', 'Keyboard', 'Monitor', 'Webcam']
    users = [f'user_{i}' for i in range(1, 101)]

    event = {
        'event_id': str(random.randint(10000, 99999)),
        'timestamp': datetime.utcnow().isoformat() + 'Z',
        'user_id': random.choice(users),
        'event_type': random.choice(event_types),
        'product_id': random.choice(products) if random.random() > 0.3 else None,
        'price': round(random.uniform(10.0, 1000.0), 2) if random.random() > 0.5 else None,
        'session_id': f'sess_{random.randint(1000, 9999)}',
        'ip_address': f'192.168.1.{random.randint(1, 254)}'
    }
    return event

def put_record_to_kinesis(record):
    """Puts a single record into the Kinesis stream."""
    try:
        response = kinesis_client.put_record(
            StreamName=stream_name,
            Data=json.dumps(record),
            PartitionKey=record['user_id'] # Use user_id to ensure related events go to the same shard
        )
        print(f"Successfully put record: {record['event_id']} to shard: {response['ShardId']}")
    except Exception as e:
        print(f"Error putting record: {e}")

if __name__ == '__main__':
    print(f"Producing events to Kinesis stream: {stream_name}...")
    while True:
        event_data = generate_event()
        put_record_to_kinesis(event_data)
        time.sleep(random.uniform(0.1, 1.0)) # Simulate varying event frequency

Note on PartitionKey: The PartitionKey is crucial for Kinesis. It determines which shard a record is sent to. Using a value like user_id ensures that all events for a particular user are processed in order within the same shard, which can be important for certain analytical workloads.

c. Setting up Kinesis Data Firehose (KDF)

Kinesis Data Firehose will consume from our Kinesis Data Stream and deliver data to S3.

• Source: Choose “Direct PUT or other sources” and then select “Kinesis Data Stream” as the source, pointing to my-app-event-stream.

• Process records: You can enable record transformation with AWS Lambda if needed (e.g., to flatten JSON, add metadata). For our basic pipeline, we might skip this here and do transformations in Glue.

• Destination: Select “Amazon S3”. Specify an S3 bucket (e.g., my-app-raw-data-lake) and an S3 prefix (e.g., events/year=!{timestamp:yyyy}/month=!{timestamp:MM}/day=!{timestamp:dd}/). Firehose automatically partitions data by date, which is excellent for Glue Crawlers and query performance.

• Buffer hints: Configure the buffer size (e.g., 128 MB) and buffer interval (e.g., 300 seconds). Firehose delivers data to S3 when either condition is met.

• Error logging: Enable CloudWatch logging for Firehose to monitor delivery status.

Ensure Firehose has an IAM role with permissions to read from KDS and write to S3.

2. Data Landing Zone: Amazon S3

Amazon S3 is the perfect landing zone for raw data. It’s highly durable, scalable, and cost-effective. Firehose will automatically create folders based on the timestamp prefix we configured.

s3://my-app-raw-data-lake/events/year=2023/month=10/day=26/my-app-event-stream-1-2023-10-26-10-30-00-abc.gz
s3://my-app-raw-data-lake/events/year=2023/month=10/day=26/my-app-event-stream-1-2023-10-26-10-35-00-def.gz
...

This date-based partitioning is crucial for efficient ETL processing and querying later on.

3. Data Transformation Layer: AWS Glue

Once data lands in S3, AWS Glue takes over for schema discovery and ETL.

a. AWS Glue Data Catalog and Crawler

We need to tell Glue about the structure of our raw data.

# Create an S3 bucket for your Glue scripts and temporary files
aws s3 mb s3://my-glue-scripts-and-temp

# Create an IAM role for Glue Crawler
aws iam create-role --role-name AWSGlueServiceRole-Crawler --assume-role-policy-document file://glue-crawler-trust-policy.json

# Attach policies (S3 read-only for data, Glue service policy)
aws iam attach-role-policy --role-name AWSGlueServiceRole-Crawler --policy-arn arn:aws:iam::aws:policy/service-role/AWSGlueServiceRole
aws iam attach-role-policy --role-name AWSGlueServiceRole-Crawler --policy-arn arn:aws:iam::aws:policy/AmazonS3ReadOnlyAccess # For raw data bucket

# Create Glue Crawler
aws glue create-crawler \
    --name my-app-event-crawler \
    --role arn:aws:iam::YOUR_ACCOUNT_ID:role/AWSGlueServiceRole-Crawler \
    --database-name my_app_db \
    --targets '{"S3Targets": [{"Path": "s3://my-app-raw-data-lake/events/"}]}' \
    --schedule "cron(0 0 * * ? *)" # Run daily at midnight UTC

The crawler will scan s3://my-app-raw-data-lake/events/, infer the schema (e.g., event_id, timestamp, user_id, etc.), and create a table (e.g., events) in the my_app_db Glue Data Catalog database. It will also recognize the date partitions (year, month, day).

b. AWS Glue ETL Job

This is where we transform the raw data into a clean, analytical-ready format. We’ll use a PySpark script for this.

First, create an IAM role for the Glue Job with permissions to:

• Read from S3 (my-app-raw-data-lake).
• Write to S3 (for temporary files and processed data if you have another S3 layer).
• Write to Redshift (covered by Redshift connection setup).
• Access the Glue Data Catalog.

Let’s consider a simple transformation: select relevant columns, cast data types, and load into Redshift. We’ll create a Glue connection to Redshift.

Glue ETL PySpark Script (glue_events_to_redshift.py):

import sys
from awsglue.transforms import *
from awsglue.utils import getResolvedOptions
from pyspark.context import SparkContext
from awsglue.context import GlueContext
from awsglue.job import Job
from datetime import datetime, timedelta

# @params: [JOB_NAME, S3_OUTPUT_PATH, REDSHIFT_TABLE_NAME, GLUE_CONNECTION_NAME, S3_TEMP_DIR, GLUE_DATABASE_NAME]
args = getResolvedOptions(sys.argv,
                          ['JOB_NAME',
                           'S3_OUTPUT_PATH',
                           'REDSHIFT_TABLE_NAME',
                           'GLUE_CONNECTION_NAME',
                           'S3_TEMP_DIR',
                           'GLUE_DATABASE_NAME',
                           'DATE_TO_PROCESS']) # New argument for dynamic partitioning

sc = SparkContext()
glueContext = GlueContext(sc)
spark = glueContext.spark_session
job = Job(glueContext)
job.init(args['JOB_NAME'], args)

# --- Configuration ---
s3_output_path = args['S3_OUTPUT_PATH'] # e.g., s3://my-app-processed-data/
redshift_table_name = args['REDSHIFT_TABLE_NAME'] # e.g., public.app_events
glue_connection_name = args['GLUE_CONNECTION_NAME'] # e.g., redshift-connection
s3_temp_dir = args['S3_TEMP_DIR'] # e.g., s3://my-glue-scripts-and-temp/temp/
glue_database_name = args['GLUE_DATABASE_NAME'] # e.g., my_app_db
date_to_process_str = args.get('DATE_TO_PROCESS', (datetime.now() - timedelta(days=1)).strftime('%Y-%m-%d')) # Default to yesterday

print(f"Processing data for date: {date_to_process_str}")

# Source data path for specific date partition
source_path = f"s3://my-app-raw-data-lake/events/year={date_to_process_str.split('-')[0]}/month={date_to_process_str.split('-')[1]}/day={date_to_process_str.split('-')[2]}/"

# Read data from the Glue Data Catalog (which points to S3)
# Using a catalog table for partitioning awareness
datasource = glueContext.create_dynamic_frame.from_catalog(
    database=glue_database_name,
    table_name='events', # Table created by the crawler
    transformation_ctx="datasource",
    push_down_predicate=f"year='{date_to_process_str.split('-')[0]}' and month='{date_to_process_str.split('-')[1]}' and day='{date_to_process_str.split('-')[2]}'"
)

print(f"Number of records read: {datasource.count()}")
if datasource.count() == 0:
    print("No data to process for the specified date. Exiting.")
    job.commit()
    sys.exit(0)

# Convert to Spark DataFrame for easier transformations
df = datasource.toDF()

# --- Transformations ---
# Select, rename, and cast columns
transformed_df = df.select(
    df.event_id.cast("string").alias("event_id"),
    df.timestamp.cast("timestamp").alias("event_timestamp"),
    df.user_id.cast("string").alias("user_id"),
    df.event_type.cast("string").alias("event_type"),
    df.product_id.cast("string").alias("product_id"),
    df.price.cast("decimal(10,2)").alias("price"),
    df.session_id.cast("string").alias("session_id"),
    df.ip_address.cast("string").alias("ip_address"),
    df.year.cast("int").alias("event_year"),
    df.month.cast("int").alias("event_month"),
    df.day.cast("int").alias("event_day")
).filter(df.event_id.isNotNull()) # Example: filter out records with no event_id

# You can add more complex transformations here:
# - Joins with other datasets (e.g., product dimensions from another S3 location)
# - Aggregations (e.g., daily unique users)
# - Data cleaning (e.g., standardizing text fields)

print(f"Number of records after transformation: {transformed_df.count()}")

# Convert back to DynamicFrame for writing to Redshift
output_dynamic_frame = DynamicFrame.fromDF(transformed_df, glueContext, "output_dynamic_frame")

# --- Write to Redshift ---
# Note: Redshift connection needs to be configured in Glue with JDBC URL, username, password.
# The IAM role for the Glue job must have permissions to access the Redshift cluster
# (e.g., by being specified in the Redshift cluster's IAM role for S3 access, or by VPC security groups).
glueContext.write_dynamic_frame.from_jdbc_conf(
    frame=output_dynamic_frame,
    catalog_connection=glue_connection_name,
    connection_options={
        "dbtable": redshift_table_name,
        "database": "dev", # Default Redshift database
        "aws_iam_role": "arn:aws:iam::YOUR_ACCOUNT_ID:role/RedshiftS3AccessRole", # Role for Redshift to access S3 for COPY command
        "preactions": f"DELETE FROM {redshift_table_name} WHERE event_year={date_to_process_str.split('-')[0]} AND event_month={date_to_process_str.split('-')[1]} AND event_day={date_to_process_str.split('-')[2]};" # Example: UPSERT logic, delete existing data for the day
    },
    redshift_tmp_dir=s3_temp_dir,
    transformation_ctx="redshift_output"
)

job.commit()
print("Glue job finished successfully!")

Explanation of the Glue Job:

• The script reads raw data from the events table in the Glue Data Catalog. Crucially, it uses a push_down_predicate to only process data for a specific date partition, making it efficient for incremental loads.
• It converts the DynamicFrame to a Spark DataFrame for easier manipulation.
• It performs basic transformations: selecting columns, casting data types (e.g., timestamp, decimal), and a simple filter.
• It writes the transformed data to Redshift using a Glue JDBC connection. The aws_iam_role in connection_options is critical; it’