AWS Glue

Overview

AWS Glue is a serverless data integration service with two distinct and independent functions — a Data Catalog (metadata repository) and an ETL engine (Apache Spark-based data transformation). Both functions are independent and can be used separately or together.

The simplest mental model:

"Glue Catalog" = the map  →  describes WHERE data is and WHAT shape it's in
"Glue ETL"     = the truck →  moves and transforms the data itself

How Glue Works Under the Hood

Glue never copies or stores actual data. It only reads and stores metadata — descriptions of the data.

The Glue Data Catalog is a library index, not a library

Glue Data Catalog (metadata only — tiny, cheap)
├── Database: "analytics"
│   ├── Table: "orders"
│   │   ├── column: id        → INT
│   │   ├── column: amount    → DECIMAL
│   │   ├── column: date      → STRING
│   │   ├── location: s3://my-bucket/orders/   ← a pointer, not a copy
│   │   └── format: parquet
│   └── Table: "customers"
│       └── location: s3://my-bucket/customers/

The Catalog holds the schema and the S3 location. The actual data stays exactly where it was.

When Athena runs SELECT * FROM analytics.orders:

1. Athena looks up orders in the Glue Catalog → gets the S3 path and column types
2. Athena goes directly to S3 to read the actual Parquet files
3. Results returned — Glue Catalog was only consulted for the schema, never touched the data

What the Crawler actually does

A Crawler does not import or replicate data. It:

1. Connects to the data source (S3, RDS via JDBC, DynamoDB, etc.)
2. Samples a small portion of the files — just enough rows to infer column names and types
3. Writes the discovered schema as a table definition into the Glue Catalog
4. Disconnects — the source data is untouched

S3: s3://bucket/orders/year=2024/month=01/data.parquet
         │
         │  Crawler reads ~100 rows to infer schema
         │  (does NOT copy the file)
         ▼
Glue Catalog: table "orders" { id INT, amount DECIMAL, ... }
              location: s3://bucket/orders/
              partitions: year=2024/month=01

What Glue ETL Jobs actually do

ETL Jobs are where data does move — but only because the script explicitly reads from a source and writes to a destination. Glue:

1. Spins up a managed Apache Spark cluster
2. Runs the script (reads data, transforms it, writes output)
3. Tears the Spark cluster down when done

The output goes wherever the script specifies — S3, Redshift, RDS. There is no "Glue storage." Glue is the execution environment, not the storage.

Glue ETL Job execution:
  [Spark cluster spins up]
      │
      ├── reads from: S3 CSV files / RDS via JDBC
      ├── transforms: rename columns, convert to Parquet, deduplicate
      └── writes to:  S3 Parquet / Redshift table
  [Spark cluster tears down]
  (billed only for cluster runtime)

The Two Jobs of AWS Glue

This split is critical — many services use the Catalog without ever using Glue ETL, and Glue ETL jobs can run without relying on the Catalog.

Part 1: Glue Data Catalog

The AWS Glue Data Catalog is a managed metadata repository — essentially a database of databases. It stores:

• Databases — logical groupings of tables
• Tables — schema definitions (column names, data types, partition keys, S3 location of the actual data)
• Partitions — sub-paths within a table (e.g. s3://bucket/year=2024/month=01/)
• Connections — credentials and network config for JDBC sources (RDS, Redshift, etc.)

The Catalog itself stores only metadata — it never holds actual data. Data stays in S3 (or wherever it lives).

Who Uses the Glue Data Catalog

Glue Crawlers

Glue Crawlers automate schema discovery, eliminating the need to write CREATE TABLE DDL manually:

1. Configure a Crawler with a target S3 path (or JDBC database, DynamoDB table, etc.)
2. The Crawler samples the data and infers the schema (column names, types, partition structure)
3. The discovered table definition is written into the Glue Data Catalog automatically
4. Crawlers can be scheduled to run periodically to detect new partitions or schema changes

S3 bucket: s3://my-data-lake/orders/year=2024/month=01/orders.parquet
                                   ─────────────────────────────────────
                                   Crawler discovers this partition structure
                                   and writes it to the Catalog automatically

Part 2: Glue ETL Jobs

A Glue ETL Job is a managed Apache Spark (or Python shell) script that extracts data from a source, transforms it, and loads it to a destination — without provisioning a Spark cluster.

How a Glue Job Works

Source                  Glue ETL Job                 Destination
S3 (CSV/JSON)  ──▶  [ Extract → Transform → Load ]  ──▶  S3 (Parquet)
RDS / Aurora   ──▶  [ Spark transforms (Python/Scala)]──▶  Redshift
DynamoDB       ──▶  [ Generated or custom code      ] ──▶  S3 data lake

Common Transformations

Job Types

Glue Studio

A visual drag-and-drop interface for building Glue ETL jobs without writing Spark code. Source, transform, and destination nodes connect visually; Studio generates the underlying PySpark script.

DynamicFrame vs. DataFrame

Glue introduces its own data abstraction called a DynamicFrame (on top of Spark's DataFrame):

• Handles schema inconsistencies gracefully — different records can have different columns without crashing
• Useful for messy real-world data where CSV/JSON schemas vary between files
• Can be converted to a standard Spark DataFrame when needed for Spark SQL operations

Part 3: AWS Glue DataBrew

Glue DataBrew is a visual, no-code data preparation tool aimed at data analysts. Key capabilities:

• Preview a dataset sample in a spreadsheet-like UI
• Apply 250+ pre-built transformations (trim whitespace, normalize dates, mask Personally Identifiable Information (PII) data, etc.)
• Profile data quality (detect nulls, outliers, distributions)
• Execute transformations as a job on the full S3 dataset

DataBrew vs. Glue Studio:

How Glue Fits Into the Analytics Ecosystem

                     ┌─────────────────────────────────────┐
                     │         Data Sources                │
                     │  S3  │  RDS  │  DynamoDB  │  JDBC  │
                     └──────────────┬──────────────────────┘
                                    │
                          ┌─────────▼──────────┐
                          │  Glue Crawler       │  ← auto-discovers schema
                          │  (metadata only)    │
                          └─────────┬──────────┘
                                    │ writes to
                          ┌─────────▼──────────┐
                          │  Glue Data Catalog  │  ← central schema registry
                          └──┬──────────────┬───┘
                             │              │
               ┌─────────────▼──┐     ┌─────▼──────────────┐
               │  Amazon Athena │     │  Redshift Spectrum  │
               │  (SQL on S3)   │     │  (SQL via Redshift) │
               └────────────────┘     └────────────────────┘

                          ┌─────────────────────┐
                          │  Glue ETL Jobs       │  ← transforms & moves data
                          │  (Spark / PySpark)   │
                          └──────────┬──────────┘
                                     │
                    ┌────────────────▼────────────────┐
                    │  Destinations                    │
                    │  S3 (Parquet)  │  Redshift       │
                    │  RDS           │  OpenSearch      │
                    └─────────────────────────────────┘

SAA/SAP Exam Tips

Cross-Cloud Equivalents

Pricing Model

• Glue ETL Jobs: per Data Processing Unit (DPU) per hour (1 DPU = 4 vCPUs + 16 GB RAM); 10-minute minimum per job run
• Glue Crawlers: per DPU-hour consumed during crawl
• Glue Data Catalog: first 1 million objects free; then per 100,000 objects/month stored; per million requests
• Glue DataBrew: per node-hour for job runs; per session-hour for interactive sessions

Related Services / See Also

• Amazon Athena — primary consumer of the Glue Data Catalog for S3 SQL queries
• Amazon Redshift — Glue ETL loads transformed data into Redshift; Spectrum uses the Catalog
• Amazon S3 — Glue's primary data source and destination
• Amazon Kinesis and Managed Flink — streaming ETL alternative to Glue for real-time pipelines
• AWS Lake Formation — builds on Glue Catalog; adds fine-grained access control for data lakes