# AWS Glue

> AWS's service for finding, describing and cleaning up data, with no servers to run.

- Level: Intermediate
- Topics: AWS, Data
- URL: https://tidbit.legios.com.ar/aws-glue/
- In Spanish: [AWS Glue](https://tidbit.legios.com.ar/es/aws-glue/)

**AWS Glue** is the AWS service for ETL: taking data from where it lands, cleaning and reshaping it, and putting it where people can analyze it. A typical case: every night an app drops thousands of JSON files into Amazon S3, and a **Glue job** reads them, removes duplicates, fixes dates, joins them with a customer table and writes the result as Parquet files ready for queries.

Its second half is the **Glue Data Catalog**, a register of every dataset: where the files live, what columns they have and of what type. A **crawler** can fill it for you by opening the files and guessing their structure. The catalog is what lets Amazon Athena or Amazon Redshift query a folder of files in S3 as if it were a table.

Glue is serverless: you write the transformation (in Python or SQL on top of Apache Spark, or by dragging boxes in a visual editor), say when it should run, and AWS spins up the machines for the duration of the job. You pay for the processing time used, not for idle clusters. That makes it the glue, as the name says, of many data lakes built on AWS.

## An analogy

A librarian and a bookbinder in one. The librarian walks the shelves writing down what each book contains; the bookbinder takes loose, messy pages and turns them into clean volumes that readers can actually use.

## Keep in mind

- Runs ETL jobs on Apache Spark without you managing a cluster.
- The Data Catalog describes datasets so Athena and Redshift can query them as tables.
- Crawlers inspect files and infer their columns and types automatically.
- Billed by processing time while jobs run; nothing runs between jobs.

## Go deeper

- [What is AWS Glue? (official docs)](https://docs.aws.amazon.com/glue/latest/dg/what-is-glue.html)
- [AWS Glue](https://aws.amazon.com/glue/)
- [What is ETL? (AWS)](https://aws.amazon.com/what-is/etl/)

## Related terms

- [ETL](https://tidbit.legios.com.ar/etl/): Copy data out of its sources, clean it up, load it where it can be analyzed.
- [JSON](https://tidbit.legios.com.ar/json/): The plain-text format programs use to hand each other structured data.
- [Amazon S3](https://tidbit.legios.com.ar/amazon-s3/): A giant online drawer for files, rented by the gigabyte.
- [Parquet](https://tidbit.legios.com.ar/parquet/): A file format that stores tables by column, compressed, so analytics reads far less.
- [Amazon Athena](https://tidbit.legios.com.ar/amazon-athena/): Run SQL directly on files sitting in S3, with no database to set up.
- [Amazon Redshift](https://tidbit.legios.com.ar/amazon-redshift/): AWS's data warehouse: SQL over huge tables, built for analysis rather than apps.
- [Serverless](https://tidbit.legios.com.ar/serverless/): Code that runs only when called; you never see or manage the server.
- [Python](https://tidbit.legios.com.ar/python/): A popular programming language known for being easy to read and quick to write.