Responsibilities

End-to-End Technical Delivery:

• Take ownership of technical initiatives from design through production deployment.
• Operate autonomously within client and internal environments to resolve technical blockers and accelerate delivery timelines.

Databricks Platform Engineering:

• Design, build, and maintain scalable data platforms using Databricks and Apache Spark.
• Optimize distributed processing workloads for performance, scalability, and cost efficiency.
• Implement Delta Lake architectures and medallion data models.

Data Pipeline Development:

• Build and maintain reliable batch and streaming ETL/ELT pipelines using Python and Databricks.
• Develop reusable frameworks and orchestration workflows for enterprise-scale data processing.
• Integrate structured and unstructured data from multiple internal and external sources.

Cloud & Data Architecture:

• Design modern cloud-native data solutions across AWS environments.
• Collaborate with architecture and infrastructure teams to ensure scalability, resiliency, and maintainability of data platforms.

AI/ML Enablement:

• Support AI/ML initiatives by preparing and optimizing datasets for advanced analytics and GenAI workloads.
• Collaborate with Data Science and AI teams to productionize ML pipelines and feature engineering workflows.

Data Governance & Quality:

• Ensure high standards of data quality, lineage, governance, and observability.
• Implement monitoring, validation, and alerting mechanisms across data workflows.
• Maintain security and compliance best practices for enterprise data environments.

Stakeholder Collaboration:

• Partner with engineering, analytics, product, and business stakeholders to translate requirements into scalable technical solutions.
• Provide technical guidance and consulting support across multiple concurrent initiatives.
‍

Skills & Experience

• Bachelor’s degree in Computer Science, Engineering, Information Systems, or related technical field.
• 5+ years of experience in Data Engineering or Data Platform Engineering roles.
• Strong expertise in Python for data engineering and backend development.
• Deep hands-on experience with Databricks, Apache Spark, Delta Lake, and distributed data processing.
• Strong understanding of modern ETL/ELT architectures and data modeling principles.
• Experience building and optimizing large-scale data pipelines in cloud environments.
• Proficiency with SQL and performance optimization techniques for analytical workloads.
• Experience with orchestration and workflow management tools (e.g., Airflow, Databricks Workflows).
• Hands-on experience with at least one major cloud platform: AWS, Azure, or GCP.
• Familiarity with CI/CD practices, Git-based development workflows, and Infrastructure as Code concepts.
• Experience supporting AI/ML data workflows and analytics platforms is highly preferred.
• Strong troubleshooting, analytical, and performance optimization skills.
• Excellent communication and stakeholder management capabilities.
• Ability to work independently in fast-paced and ambiguous environments.