QA with Azure and Databricks Professional @ Infobeans

Key Responsibilities

1) Test Strategy Planning

• Define and maintain test strategies for data pipelines, notebooks, jobs, and APIs across Databricks Azure.
• Translate business rules and data quality SLAs into testable acceptance criteria.
• Establish risk-based test plans across functional, integration, system, and non-functional (performance, scalability, security) testing.

• Develop automated tests for Spark transformations (e.g., PySpark, Spark SQL) and Delta Lake features (MERGE, time travel, schema evolution).
• Implement data quality checks (completeness, accuracy, consistency, referential integrity, drift detection).
• Validate batch and streaming pipelines, job dependencies, and SLAs.
• Create reproducible test datasets and use Delta Live Tables (if applicable) to validate expectations.

• Test data flows across Azure Data Factory / Synapse pipelines, Databricks Jobs, Azure Storage/ADLS, Azure SQL/SQL MI, Event Hub/Kafka, and Key Vault access patterns.
• Verify IAM/ACLs/Unity Catalog permissions, secrets handling, and network boundary controls.
• Validate infrastructure configurations across environments (Dev/Test/Prod) and support IaC validation (e.g., Bicep/Terraform plan diffs).

• Integrate tests into CI/CD (Azure DevOps/GitHub Actions): unit, integration, data-validation, and post-deployment smoke tests.
• Implement quality gates (coverage thresholds, schema checks, contract tests, static checks like Pylint/flake8).
• Build test utilities and shared libraries for data assertions, fixtures, and synthetic data generation.
• Use feature branches, PR checks, and merge policies to enforce quality.

• Instrument tests and pipelines with logging metrics (e.g., Azure Monitor, Log Analytics, Databricks metrics).
• Define SLIs/SLOs for data quality and pipeline reliability; participate in incident reviews and RCA.
• Establish quality dashboards (test pass rate, DQ incidents, recovery time, flaky test rate).

• Partner with data engineers, platform engineers, product owners, and data stewards.
• Contribute to test data management practices and a data contract approach with upstream/downstream teams.
• Document test plans, runbooks, and traceability from requirements to tests.

• 3-8+ years in QA/Test Engineering with 2+ years in data platform or Spark-based environments.
• Hands-on with Databricks: Notebooks, Jobs, Delta Lake, cluster policies.
• Strong Python for tests/utilities; solid SQL for validation and profiling.
• Experience testing ETL/ELT patterns, data partitioning, late-arriving data, slowly changing dimensions, idempotency.
• Azure experience: Data Factory, ADLS Gen2, Key Vault, Azure DevOps/GitHub.
• CI/CD for data: pipelines, artifacts, environments, approvals, test stages and post-deploy validations.
• Familiar with schema validation and API testing (REST/Databricks Jobs API).
• Clear understanding of security (RBAC/ABAC, managed identities), cost performance considerations.

• Unity Catalog, data lineage, and governance testing.
• Streaming (Structured Streaming, Delta Live Tables), event-driven validation.
• Performance testing of Spark jobs (cluster sizing, shuffle hotspots, skew).
• Exposure to privacyregulatory controls (PII masking, GDPR/CCPA, SOX).

Location - Pune,Nagar,Indore,Bengaluru