Lead Data Engineer

Role & responsibilities Lead the design, development, and implementation of scalable, reliable, and high-performance data engineering solutions. Architect, build, and maintain robust ETL/ELT pipelines for batch and real-time data processing. Lead the development and optimization of data warehouses, data lakes, and lakehouse architectures.

Design and manage data ingestion, transformation, integration, and orchestration workflows. Work with large-scale datasets using Apache Spark, PySpark, Kafka, Hadoop, and other big data technologies. Implement and manage cloud-based data platforms using AWS, Microsoft Azure, or Google Cloud Platform.

Develop and optimize data models, database schemas, and data processing frameworks. Ensure data quality, consistency, security, governance, and regulatory compliance across data platforms. Implement workflow orchestration and automation using Apache Airflow, dbt, and related tools.

Monitor data pipelines, troubleshoot failures, and optimize system performance and infrastructure costs. Establish data engineering standards, best practices, documentation, and CI/CD processes. Collaborate with data scientists, data analysts, software engineers, and business stakeholders to deliver data-driven solutions.

Lead technical discussions, architecture reviews, code reviews, and technology evaluations. Mentor junior and mid-level data engineers and provide technical guidance to the engineering team. Drive continuous improvements in data infrastructure, scalability, reliability, and operational efficiency.

Preferred candidate profile Bachelor's or Master's degree in Computer Science, Information Technology, Data Science, Software Engineering, or a related technical field. 5 to 10 years of professional experience in data engineering, big data development, or data platform engineering. Strong proficiency in Python, SQL, PySpark, and experience with large-scale data processing. Hands-on experience designing and developing complex ETL/ELT pipelines and data integration workflows.

Strong knowledge of Apache Spark, Kafka, Airflow, Hadoop, and distributed data processing frameworks. Experience with modern data platforms such as Snowflake, Databricks, Amazon Redshift, Google BigQuery, or Azure Synapse Analytics. Proven expertise in cloud technologies, including AWS, Azure, or Google Cloud Platform.

Strong understanding of data warehousing, data lakes, lakehouse architecture, data modeling, and database optimization. Experience with real-time streaming, batch processing, and workflow orchestration. Knowledge of data governance, data quality frameworks, metadata management, and data security practices.

Experience with Docker, Kubernetes, Git, CI/CD pipelines, Terraform, and infrastructure automation. Ability to lead technical projects, define data architecture, and establish engineering best practices. Proven experience mentoring engineers, conducting code reviews, and collaborating with cross-functional teams.

Strong analytical, problem-solving, communication, and stakeholder-management skills. Ability to translate business requirements into scalable and efficient data engineering solutions.