QA Professional - Azure and Databricks
- Infobeans
- 7 - 10 years
- Indore
- 14 days ago
- Email to a friend
- Report this job
Job Description
1) Test Strategy & Planning
- Define and maintain test strategies for data pipelines, notebooks, jobs, and APIs across Databricks & Azure.
- Translate business rules and data quality SLAs into testable acceptance criteria.
- Establish risk-based test plans across functional, integration, system, and non-functional (performance, scalability, security) testing.
- Develop automated tests for Spark transformations (e.g., PySpark, Spark SQL) and Delta Lake features (MERGE, time travel, schema evolution).
- Implement data quality checks (completeness, accuracy, consistency, referential integrity, drift detection).
- Validate batch and streaming pipelines, job dependencies, and SLAs.
- Create reproducible test datasets and use Delta Live Tables (if applicable) to validate expectations.
- Test data flows across Azure Data Factory / Synapse pipelines, Databricks Jobs, Azure Storage/ADLS, Azure SQL/SQL MI, Event Hub/Kafka, and Key Vault access patterns.
- Verify IAM/ACLs/Unity Catalog permissions, secrets handling, and network boundary controls.
- Validate infrastructure configurations across environments (Dev/Test/Prod) and support IaC validation (e.g., Bicep/Terraform plan diffs).
- Integrate tests into CI/CD (Azure DevOps/GitHub Actions): unit, integration, data-validation, and post-deployment smoke tests.
- Implement quality gates (coverage thresholds, schema checks, contract tests, static checks like Pylint/flake8).
- Build test utilities and shared libraries for data assertions, fixtures, and synthetic data generation.
- Use feature branches, PR checks, and merge policies to enforce quality.
- Instrument tests and pipelines with logging & metrics (e.g., Azure Monitor, Log Analytics, Databricks metrics).
- Define SLIs/SLOs for data quality and pipeline reliability; participate in incident reviews and RCA.
- Establish quality dashboards (test pass rate, DQ incidents, recovery time, flaky test rate).
- Partner with data engineers, platform engineers, product owners, and data stewards.
- Contribute to test data management practices and a data contract approach with upstream/downstream teams.
- Document test plans, runbooks, and traceability from requirements to tests.
Required Qualifications
- 3-8+ years in QA/Test Engineering with 2+ years in data platform or Spark-based environments.
- Hands-on with Databricks: Notebooks, Jobs, Delta Lake, cluster policies.
- Strong Python for tests/utilities; solid SQL for validation and profiling.
- Experience testing ETL/ELT patterns, data partitioning, late-arriving data, slowly changing dimensions, idempotency.
- Azure experience: Data Factory, ADLS Gen2, Key Vault, Azure DevOps/GitHub.
- CI/CD for data: pipelines, artifacts, environments, approvals, test stages and post-deploy validations.
- Familiar with schema validation and API testing (REST/Databricks Jobs API).
- Clear understanding of security (RBAC/ABAC, managed identities), cost & performance considerations.
- Unity Catalog, data lineage, and governance testing.
- Streaming (Structured Streaming, Delta Live Tables), event-driven validation.
- Performance testing of Spark jobs (cluster sizing, shuffle hotspots, skew).
- Exposure to privacy & regulatory controls (PII masking, GDPR/CCPA, SOX).
Job Classification
Industry: IT Services & ConsultingFunctional Area / Department: Engineering - Software & QARole Category: DBA / Data warehousingRole: Database Architect / DesignerEmployement Type: Full timeContact Details:
Company: InfobeansLocation(s): Indore
Keyskills: QA pr data management data pipelines performance testing Azure Databricks iac sql analytics automation iam spark devops etl rest python github azure sql data lineage elt gdpr data quality kafka terraform api testing ccpa
Similar positions
Infobeans
InfoBeans Systems is a fast growing software development company with offices in Pune and Indore and business development centre in US. We develop customized business applications for large and mid-size organizations in the US. We strongly believe that customer success defines our success. Hence, we...
Job Listings
We are in open beta version of website, please let us know if any issues, at: info(at)indigojobs.in