Introduction

Every organization creates data long before it knows what to do with it.

A customer places an order. A mobile app records a click. A warehouse scanner updates inventory. A payment system emits a transaction. A support agent closes a ticket. Each of these events leaves behind a record: a small description of something that happened. Alone, one record may not mean much. Together, records become the raw material for decisions, products, automation, reporting, machine learning, fraud detection, forecasting, and operational insight.

But raw data does not automatically become useful information. It may be scattered across databases, APIs, files, queues, logs, and third-party systems. It may arrive late, contain duplicates, use inconsistent names, or change shape without warning. It may be too large for one machine, too sensitive to expose broadly, or too expensive to query carelessly. The work of data engineering begins in this gap between data that exists and data that can be trusted and used.

This book is about learning to close that gap on AWS.

Data engineering is the practice of designing, building, operating, and improving systems that collect, store, transform, serve, and monitor data. A data engineer does not only write scripts. A data engineer thinks about reliability, cost, security, ownership, scale, recovery, latency, and meaning. A good pipeline is not merely code that works once; it is a system that can keep working when input data changes, upstream systems fail, usage grows, and people depend on its output.

A useful first mental model is this:

Data engineering turns events and records from the real world into dependable data products for people and systems.

A data product is any data asset that has users and expectations. It might be a table used by analysts, a dashboard dataset used by executives, a feature table used by a machine learning model, or an aggregated feed consumed by another application. If people make decisions from it, if software depends on it, or if it must be correct at a known time, then it deserves engineering discipline.

The path from source to value

Imagine an online store. When a customer buys a product, the operational application writes a row into a database table called orders. That table is designed to support the application: create an order quickly, update its payment status, show the customer their receipt, and allow support staff to look it up. This is an operational system, meaning a system built to run day-to-day business processes.

Now imagine the finance team asks a different question:

“What was our net revenue by country, product category, and week for the last two years?”

The orders table alone may not answer that well. You may need product data, customer geography, discounts, refunds, currency conversions, tax rules, shipment status, and historical changes. You may also need to preserve the meaning of the data as it looked at the time of purchase. This is where analytical systems appear. Analytical systems are designed to answer questions across large amounts of historical data.

Operational systems and analytical systems have different priorities. Operational systems usually care about fast, correct transactions for current business activity. Analytical systems usually care about scanning, joining, aggregating, and comparing large amounts of data over time. This distinction is one reason data warehouses and dimensional modeling became important in analytics work; Kimball and Ross describe dimensional models as structures designed to make business measurements understandable and queryable for analytic users (Kimball and Ross, 2013).

A simple analytical journey might look like this:

1. Extract order records from the application database.
2. Store a raw copy in Amazon S3.
3. Clean and standardize the records with PySpark.
4. Save the transformed data as partitioned Parquet files.
5. Register the table in the AWS Glue Data Catalog.
6. Query it with Amazon Athena or Amazon Redshift.
7. Monitor the pipeline with Datadog and alert when freshness, volume, or error rate becomes abnormal.

Each step has a purpose. The raw copy protects you from losing the original input. The transformation makes the data easier to understand. The table catalog tells query engines what the data means. The monitoring system helps you notice when reality no longer matches expectation.

This book will teach those ideas gradually, from first principles to production habits.

Data, pipelines, and systems

Before touching AWS services, we need a few foundational words.

Data is recorded information. In data engineering, data is usually represented as files, rows, columns, events, messages, objects, or documents. A CSV file of sales records is data. A JSON response from an API is data. A stream of click events is data. A Parquet dataset in S3 is data.

A source is where data comes from. A source can be a relational database such as PostgreSQL or MySQL, a SaaS platform such as Salesforce, an application API, a file upload, an event stream, or a logging system.

A sink is where data is written. A sink might be S3, Redshift, OpenSearch, a relational database, a message queue, or another API.

A pipeline is a repeatable process that moves data from sources to sinks, usually transforming or validating it along the way. For example, a daily pipeline may read yesterday’s transactions from a database, write them into S3, remove duplicates, calculate daily revenue, and publish a curated table for analysts.

A transformation changes data from one form into another. It may rename columns, parse timestamps, join records, remove invalid rows, aggregate values, mask sensitive fields, or convert file formats. For example, transforming "2026-01-05T12:03:00Z" into a standardized timestamp column is a small transformation. Joining orders to products and calculating revenue by category is a larger transformation.

A workflow coordinates multiple pipeline steps. If ingestion must finish before transformation, and transformation must finish before data quality checks, and checks must pass before publishing, then you need workflow logic. Tools such as AWS Step Functions, AWS Glue Workflows, Managed Workflows for Apache Airflow, and Apache Airflow help express these dependencies.

A data platform is the broader environment that supports many pipelines and users. It includes storage, compute, catalogs, permissions, orchestration, testing, observability, cost controls, documentation, and operating practices.

These definitions matter because production data engineering is not a single skill. It is the combination of programming, SQL, distributed processing, cloud infrastructure, security, architecture, and operations.

Why AWS is central in this book

AWS is a cloud computing platform. In cloud computing, you rent computing resources—such as storage, virtual machines, managed databases, serverless functions, and networking capabilities—instead of buying and operating all hardware yourself. AWS organizes these capabilities into services. For data engineering, the most important AWS services include Amazon S3 for object storage, AWS Glue for cataloging and ETL, Amazon Athena for serverless SQL queries on S3 data, Amazon Redshift for data warehousing, Amazon EMR for managed big data clusters, Amazon Kinesis for streaming data, AWS Lambda for event-driven compute, and AWS Step Functions for workflow orchestration.

A central idea in AWS data engineering is that storage and compute are often separated. Storage means where bytes live. Compute means the CPU and memory used to process those bytes. Amazon S3 is commonly used as durable object storage for data lakes; AWS describes S3 as object storage designed to store and retrieve any amount of data from anywhere (Amazon Web Services, 2024a). A query engine such as Athena, a Spark job on Glue or EMR, or a Redshift Spectrum query can then read data from S3 when needed.

This separation creates flexibility. For example, you can store raw clickstream files in S3 once, then process them with Spark for large transformations, query them with Athena for ad hoc analysis, and load selected aggregates into Redshift for high-performance dashboards. The same stored data can support multiple compute patterns.

But flexibility also creates responsibility. If files are too small, queries may become inefficient. If partitions are poorly chosen, users may scan too much data. If permissions are too broad, sensitive information may be exposed. If lifecycle policies are missing, storage costs may grow unnoticed. If nobody monitors freshness, a dashboard may quietly show stale numbers. The cloud gives you building blocks; engineering turns them into a reliable system.

AWS’s own architectural guidance emphasizes operational excellence, security, reliability, performance efficiency, cost optimization, and sustainability as pillars of well-architected workloads (Amazon Web Services, 2024b). In this book, those ideas will appear repeatedly in practical form. We will not treat a pipeline as “done” merely because it produced a file. We will ask: Is it secure? Is it repeatable? Is it observable? Is it cost-aware? Can it recover? Can another engineer understand it?

Why SQL remains essential

SQL, pronounced either as “S-Q-L” or “sequel,” is a language for working with relational data: data organized in tables with rows and columns. Even in modern cloud data platforms, SQL remains a central language because many analytical questions are naturally expressed as filtering, joining, grouping, and aggregating tables.

Suppose you have an orders table:

A business question such as “How much revenue did we make per day?” maps cleanly to SQL:

SELECT
    order_date,
    SUM(amount) AS daily_revenue
FROM orders
GROUP BY order_date
ORDER BY order_date;

This query groups rows by order_date and sums the amount values in each group. The output is no longer raw transaction data; it is an analytical result.

SQL is also important because many data tools expose a SQL interface. Amazon Athena uses SQL to query data in S3. Amazon Redshift uses SQL for warehouse analytics. Spark includes Spark SQL for querying structured data using SQL syntax and DataFrame operations; the Apache Spark project describes Spark SQL as a module for structured data processing (Apache Software Foundation, 2024).

As you progress through this book, SQL will become more than a query language. You will use it as a design tool. You will learn to think about grain, joins, window functions, deduplication, incremental logic, and slowly changing dimensions. In real data engineering work, a well-written SQL transformation can be as important as a well-written Python program.

Why PySpark matters

Some datasets are small enough to process on one machine. Others are too large, too slow, or too expensive to process that way. Distributed processing means splitting work across multiple machines so they can process data in parallel. Apache Spark is a widely used open-source engine for large-scale data processing. It provides APIs in languages including Python, Scala, Java, and R, and it supports SQL, batch processing, streaming, machine learning libraries, and graph processing components (Apache Software Foundation, 2024).

PySpark is the Python interface to Apache Spark. It lets you write Spark jobs using Python syntax while Spark handles distributed execution. For example, a PySpark job can read many Parquet files from S3, filter invalid records, join them with reference data, aggregate by date and region, and write the result back to S3.

A tiny PySpark transformation might look like this:

from pyspark.sql import functions as F

orders = spark.read.parquet("s3://example-raw/orders/")

daily_revenue = (
    orders
    .groupBy("order_date")
    .agg(F.sum("amount").alias("daily_revenue"))
)

daily_revenue.write.mode("overwrite").parquet("s3://example-curated/daily_revenue/")

This example resembles SQL because the idea is the same: group orders by date and sum the amount. The difference is execution. Spark can distribute the work across a cluster when the dataset is large.

PySpark is powerful, but it is not magic. To use it well, you need to understand partitions, shuffles, joins, file formats, schemas, memory, and lazy evaluation. This book will build those concepts carefully. You will learn not only how to write PySpark code, but how to reason about what Spark is likely to do when that code runs.

Why observability belongs in data engineering

A beginner often asks, “How do I build the pipeline?” A production engineer also asks, “How will I know when the pipeline is wrong?”

Observability is the ability to understand a system’s internal state from the signals it emits. In software and data systems, common signals include metrics, logs, and traces. A metric is a numerical measurement over time, such as job duration, number of rows processed, error count, or data freshness in minutes. A log is a timestamped record of something that happened, such as “started processing file X” or “failed to connect to API.” A trace follows a request or operation across multiple services, helping you see where time was spent and where failure occurred. Datadog describes metrics, traces, and logs as major telemetry types used in observability practices (Datadog, 2024).

For data engineering, observability must include both system health and data health.

System health asks:

• Did the job run?
• How long did it take?
• Did it fail?
• Did memory, CPU, or network usage become abnormal?
• Did queue backlog grow?

Data health asks:

• Did the expected number of records arrive?
• Is the data fresh?
• Did the schema change?
• Are important columns unexpectedly null?
• Did revenue suddenly drop to zero?
• Are duplicate records increasing?

Consider a daily orders pipeline. If the Spark job finishes successfully but processes only 10 rows instead of the usual 2 million, the system may look healthy while the data is wrong. A dashboard built from that output may show a dramatic revenue collapse. Without data observability, the first alert may come from a confused stakeholder.

In this book, Datadog will appear as a practical observability platform. You will learn how metrics, logs, dashboards, monitors, alert routing, service-level objectives, and AWS integrations help operate data systems. The goal is not to collect telemetry for decoration. The goal is to make failures visible early, explain them quickly, and reduce the time between “something is wrong” and “we know what to do.”

The production mindset

A script can be useful. A production pipeline must be dependable.

A production system is one that real users, teams, or business processes rely on. In production, correctness and maintainability matter as much as cleverness. A production-grade data pipeline should usually have clear ownership, documented behavior, controlled permissions, repeatable deployment, tests, monitoring, and recovery procedures.

For example, imagine a pipeline that ingests customer data from an API every hour. A quick script may work like this:

1. Call the API.
2. Save the response to a file.
3. Transform the file.
4. Append rows to a table.

A production version asks deeper questions:

• What happens if the API returns duplicate records?
• What happens if the API is unavailable for three hours?
• How do we resume without losing or double-counting data?
• Where are credentials stored?
• Are personally identifiable fields encrypted or masked?
• How do we know the latest data arrived?
• How do we backfill last month if a bug is discovered?
• Can we deploy a fix without manually editing servers?
• Who receives the alert when the pipeline fails?
• How do we estimate the cost of running it?

These are not advanced concerns reserved for senior engineers. They are the normal concerns of responsible data engineering. You will meet them throughout the book in small pieces, then bring them together in the end-to-end platform chapter.

The learning journey ahead

This book starts with concepts before tools because tools are easier to learn when you know what problem they solve.

You will first develop the data engineering mindset: how data moves from source systems into analytical systems, and how lakes, warehouses, and lakehouses differ. Then you will study the raw materials of data work: files, tables, events, schemas, partitions, compression, and metadata. From there, you will build practical skill in SQL and Python, because they are the daily working languages of many data engineers.

After that foundation, the book moves into cloud infrastructure. You will learn enough Linux, networking, IAM, encryption, and AWS service design to understand the environment in which pipelines run. You will then study core AWS services for data engineering and go deeper into S3 as the foundation of a data lake.

The middle of the book focuses on data modeling, ingestion, streaming, Spark, PySpark, AWS Glue, EMR, Athena, and Redshift. These chapters are where data begins to move and transform at realistic scale. You will learn batch patterns such as full loads, incremental loads, watermarks, idempotency, and backfills. You will also learn streaming concepts such as events, ordering, windows, late data, and delivery guarantees.

The later chapters turn working pipelines into professional systems. You will study orchestration, data quality, Datadog observability, security, governance, CI/CD, infrastructure as code, cost optimization, and performance tuning. Finally, you will assemble the pieces into an end-to-end AWS data platform and prepare for career-ready practice.

The sequence is intentional. You do not need to master everything before building anything, but you do need stable mental models. When you understand the lifecycle of data, each service and technique has a place.

A small example to carry through the book

To make the ideas concrete, keep this example in mind:

A company wants a daily analytics table called daily_product_revenue. It should show revenue by date and product category. Analysts will query it in SQL. Dashboards will refresh every morning. If the data is late, incomplete, or suspicious, the data team should know before business users complain.

A beginner may see this as one SQL query. A data engineer sees a system:

• Where do orders come from?
• Where do product categories come from?
• How often do they change?
• How do we store raw data?
• What schema do we expect?
• How do we handle refunds?
• How do we avoid double-counting?
• What file format should we use?
• How should the data be partitioned?
• Should we process with SQL, PySpark, or both?
• Should the serving layer be Athena, Redshift, or another engine?
• What tests prove the output is credible?
• What Datadog monitor tells us the table is stale?
• What IAM role can read or write each location?
• How do we redeploy safely when the transformation changes?

By the end of the book, these questions should feel natural rather than intimidating. You will not memorize every AWS option. Instead, you will learn how to reason: from data shape to storage design, from processing requirement to compute choice, from user need to serving layer, from failure mode to monitor, and from risk to control.

What success looks like

Success in this book is not merely recognizing service names. It is being able to explain and build a pipeline responsibly.

You are progressing well when you can say things like:

• “This dataset belongs in a raw S3 zone first, because we need an immutable copy before transformation.”
• “This Athena query is expensive because it scans too much data; partitioning and columnar formats may reduce scanned bytes.”
• “This PySpark join may cause a shuffle, so I should check data size, skew, and whether a broadcast join is appropriate.”
• “This pipeline needs idempotency, because a retry should not duplicate records.”
• “This dashboard needs a freshness monitor, because a successful job yesterday does not prove today’s data arrived.”
• “This IAM policy is too broad; the job role should have only the permissions it needs.”

That is the voice of a data engineer: practical, precise, skeptical, and calm.

The cloud will change. AWS services will gain features. Datadog interfaces will evolve. File formats, orchestration tools, and processing engines will continue to improve. But the core discipline will remain: understand the data, design the flow, control the risk, observe the system, and serve trustworthy results.

Let us begin there.

References

Amazon Web Services. (2024a). Amazon Simple Storage Service User Guide: What is Amazon S3? AWS Documentation. https://docs.aws.amazon.com/AmazonS3/latest/userguide/Welcome.html

Amazon Web Services. (2024b). AWS Well-Architected Framework. AWS Documentation. https://docs.aws.amazon.com/wellarchitected/latest/framework/welcome.html

Apache Software Foundation. (2024). Apache Spark Documentation. https://spark.apache.org/docs/latest/

Datadog. (2024). The Three Pillars of Observability. https://www.datadoghq.com/knowledge-center/three-pillars-of-observability/

Kimball, R., & Ross, M. (2013). The Data Warehouse Toolkit: The Definitive Guide to Dimensional Modeling (3rd ed.). Wiley.