AWS Cloud TutorialLesson 58Free preview

AWS Data Pipelines

This lesson is part of AWS · All levels

Summary

Data pipelines are a series of data processing steps from source to destination, including extract-transform-load (ETL) and analytics in real-time. AWS Glue provides serverless ETL along with automated data catalogs, while Kinesis is used for continuous data flow. The batch pipeline is suitable for periodic reporting, while the streaming is for instantaneous needs.

Key points

ETL: extract, transform, load
AWS Glue: Serverless ETL with automatic Glue Data Catalog
Kinetic data streams to ingest in real-time; Firehose to load to S3/Redshift
AWS Step Functions orchestrates the pipeline steps
Batch or streaming options tailored to the data speed needs

Full lesson

Code examples and quizzes

Full lessons, code examples, and quizzes are available to subscribers. 5 questions.