Senior Consultant | Databricks + Pyspark | Mumbai | Engineering | Data Modernization & Migration

Job Title: Databricks - Sr. Consultant

Role Overview: As a Senior Consultant, you will lead the design and optimization of high-performance data processing frameworks on Azure Databricks. You will be responsible for advanced Spark tuning, implementing robust data governance through Unity Catalog, and driving the transition toward a modern Medallion architecture.

Experience:

• 5–7 years of experience in data engineering, with deep expertise in AzureDatabricks and Apache Spark.

• Advanced experience with Delta Lake and Medallion Architecture (Bronze/Silver/Gold).

• Proven track record in Spark Performance Tuning (handling data skew, shuffle tuning, etc.).

• Strong experience with Unity Catalog for data governance and security.

Key Responsibilities: Technical Skills:

• Design and implement scalable Medallion Architectures using Delta LiveTables (DLT) and Databricks.

• Lead the optimization of complex Spark workloads to improve execution speed and reduce cloud compute costs.

• Implement enterprise-grade data governance and access control using Unity Catalog.

• Architect automated CI/CD pipelines for Databricks using Databricks Asset Bundles (DABs) or Terraform.

• Design sophisticated data integration patterns between Databricks, Azure Synapse, and various RDBMS.

• Act as a technical mentor to junior engineers, conducting code reviews and enforcing best practices.

Soft Skills:

• Strong leadership and ability to drive technical delivery.

• Advanced problem-solving skills for complex architectural challenges.

• Excellent communication skills for interacting with stakeholders and technical leads.

Preferred Qualifications:

• Bachelor’s or Master’s degree in a technical field.

• Databricks Certified Data Engineer Professional is highly desirable.

• Experience with DataOps and Infrastructure as Code (Terraform).

Job Details:

• Location: Mumbai (Hybrid)