Apache Airflow & Aws (s3/emr/bedrock) Data Platform Administrator
- Zensar
- 3 - 8 years
- Pune
- 15 hours ago
- Email to a friend
- Report this job
Job Description
Short DescriptionDescription
- Strong experience in Airflow DAG monitoring, including tracking task states, resolving DAG execution delays, and ensuring reliability across distributed environments.
- Expertise in failure recovery, including retry strategies, SLA miss handling, backfilling, re-running failed task instances, and ensuring consistent pipeline execution across environments.
- Handson experience providing SLAbased job execution support, ensuring timecritical pipelines meet business deadlines and production SLAs are continuously maintained.
- Skilled in performing root cause analysis (RCA) for pipeline failures, including dependency failures, tasklevel exceptions, scheduler issues, and platformlevel bottlenecks.
- Experience in managing S3 storage optimization, including lifecycle policies, intelligent tiering, storage class transitions, versioning, and costeffective data retention strategies.
- Expertise in securing S3 environments using IAM policies, bucket policies, encryption (KMS), access logging, and objectlevel permissions.
- Skilled in conducting cost usage analysis for S3 storage and recommending optimization strategies to reduce operational spend.
- Strong background in administering Amazon EMR clusters, including cluster provisioning, configuration, autoscaling, and lifecycle management.
- Experience supporting Amazon Bedrock environments, including model endpoint configuration, invocation monitoring, access control, and cost governance.
Responsibilities
- Monitor and manage Apache Airflow DAGs, ensuring timely execution, resolving delays, and maintaining reliability across distributed environments.
- Perform failure recovery activities, including retries, SLAmiss handling, backfilling, and rerunning failed task instances for consistent pipeline execution.
- Provide SLAdriven operational support to ensure critical data pipelines meet business timelines and production availability targets.
- Conduct indepth RCA for pipeline issues such as dependency failures, task exceptions, scheduler disruptions, and platform bottlenecks.
- Optimize AWS S3 storage through lifecycle policies, intelligent tiering, storage class transitions, and costeffective data retention strategies.
- Implement strong S3 security using IAM roles, bucket policies, KMS encryption, access logging, and objectlevel access controls.
- Analyze S3 usage patterns and recommend costoptimization measures to minimize storage and operational spend.
- Administer Amazon EMR clusters, including provisioning, configuration management, autoscaling, and endtoend lifecycle operations.
- Support Amazon Bedrock environments with model endpoint configuration, monitoring invocations, managing access controls, and ensuring cost governance.
Qualifications
- Strong handson experience administering Apache Airflow in distributed, production-grade environments.
- Deep understanding of DAG orchestration, task execution states, scheduler behavior, and pipeline reliability practices.
- Proven expertise in workflow recovery techniquesretries, backfilling, SLA handling, and task reruns.
- Solid background in troubleshooting and performing RCA for pipeline, platform, and dependency failures.
- Practical experience managing and optimizing AWS S3 storage, lifecycle rules, tiering, and cost management.
- Strong knowledge of S3 security controls including IAM roles, bucket policies, KMS encryption, and access logging.
- Experience conducting S3 usage/cost analysis and recommending optimization strategies.
- Handson expertise in provisioning, configuring, and managing Amazon EMR clusters and autoscaling policies.
- Working experience with Amazon Bedrockmodel endpoint setup, usage monitoring, access governance, and cost oversight.
- Ability to support 247 production environments with a strong focus on operational excellence and SLAs.
- Strong analytical, problemsolving, and crossteam coordination skills for cloud and data platform operations.
Job Classification
Industry: IT Services & ConsultingFunctional Area / Department: IT & Information SecurityRole Category: IT Infrastructure ServicesRole: IT Operations ManagementEmployement Type: Full timeContact Details:
Company: ZensarLocation(s): Pune+ View Contact
Keyskills: Airflow Administration EMR AWS Bedrock Bedrock Administrator
Fraud Alert to job seekers!
₹ Not Disclosed
Similar positions
Zensar
Zensar Technologies Limited Zensar Technologies is among the top 25 software and BPO services providers in India. It is an RPG Group company. Headquartered in India, Zensar Technologies has marketing presence in US, Europe and Asia Pacific regions. The company has operations and a customer bas...
Job Listings
We are in open beta version of website, please let us know if any issues, at: info(at)indigojobs.in