Q23Data Ingestion and Transformation
A company maintains multiple extract, transform, and load (ETL) workflows that ingest data from the company's operational databases into an Amazon S3 based data lake. The ETL workflows use AWS Glue and Amazon EMR to process data. The company wants to improve the existing architecture to provide automated orchestration and to require minimal manual effort. Which solution will meet these requirements with the LEAST operational overhead?
← → navigate · a answer
Community votes
Discussion · 17
B 20
Selected Answer: B
Glue Workflow only orchestrates crawlers and glue jobs
B 9
Selected Answer: B
For me it's B because I couldn't find a way for Glue to trigger/orchestrate EMR processes OOTB.
But with StepFunction there is a way: https://aws.amazon.com/blogs/big-data/orchestrate-amazon-emr-serverless-jobs-with-aws-step-functions/
A 6
Selected Answer: A
Since it seems to me that this pipeline is complex, with multiple workflows, I would choose Glue workflows.
B 4
Selected Answer: B
There is no way for Glue Workflow to trigger EMR
3
Yo me voy por la D) Amazon MWAA porque Glue Workflows solo admite Jobs de Glue y Step Function puede fucionar pero no son workflows de datos. Amazon MWAA son workflows de datos y esta integrado tanto con Glue como EMR: https://aws.amazon.com/blogs/big-data/simplify-aws-glue-job-orchestration-and-monitoring-with-amazon-mwaa/
B 3
Selected Answer: B
EMR in workflows, I don't think so
B 3
Selected Answer: B
We have both Glue job and EMR job, so we need Step Functions to connect them.
Airflow can do it, but it needs more dev work.
2
Here's an example of how you can use AWS Glue to start an EMR (Elastic MapReduce) job:
Let's assume you have an AWS Glue job that does ETL tasks on data stored in Amazon S3. You want to use EMR for a specific task inside this job, such as running a complex Spark job.
1. Define a Glue Job: Create an AWS Glue job using the AWS Glue console, SDK, or CLI. Define the input and output data sources, along with the transformations you want to apply.
2. Add EMR Step: Inside the Glue job script, include a section where you define an EMR step. An EMR step is a unit of work that carries out a specific task on an EMR cluster.
Code follows in the next entry...
A 2
Selected Answer: A
https://aws.amazon.com/blogs/big-data/orchestrate-an-etl-pipeline-using-aws-glue-workflows-triggers-and-crawlers-with-custom-classifiers/
B 2
Selected Answer: B
The company wants to improve the existing architecture, so A cannot be the right choice
B 1
Selected Answer: B
B - because AWS Glue can't trigger EMR
A 1
Selected Answer: A
AWS Glue Workflows are built specifically for orchestrating ETL jobs in AWS Glue. They let you define and manage complex workflows that include multiple jobs and triggers, all inside the AWS Glue environment.Integration: AWS Glue workflows integrate smoothly with other AWS Glue components, which makes it easier to manage ETL processes without needing external orchestration tools.Minimal Operational Overhead: Because AWS Glue is a fully managed service, using Glue workflows will lower the operational overhead compared to managing separate orchestrators or building custom solutions.While D. Amazon Managed Workflows for Apache Airflow (Amazon MWAA) is also a good option for more complex orchestration, it may bring more management overhead than the more straightforward AWS Glue workflows. So, AWS Glue workflows offer the least operational overhead for this scenario.
A 1
Selected Answer: A
Glue workflows are managed services and are best when considering the least operational overhead.
1
Answer A: Glue workflows
A 1
Selected Answer: A
glue workflows is part of the glue ecosystem, so it provides seamless integration with minimal changes
B 1
Selected Answer: B
it's interesting to pick A for minimum effort, but only step functions can trigger the work both on EMR and on GLUE jobs
A 1
Selected Answer: A
Why Not the Other Options?
B. AWS Step Functions More flexible, but it needs manual setup of states and transitions for Glue & EMR. Higher operational overhead than Glue Workflows.
C. AWS Lambda Lambda is not a good fit for long-running ETL workflows. It is better for lightweight data transformations or event-driven tasks.
D. Amazon MWAA (Apache Airflow) More control, but it needs cluster management and custom DAGs. More maintenance than Glue Workflows.