Overview

Infogain

Onboarding n laptop collection from Noida or Gurgaon Infogain office

Job Title: Data Engineer

Experience: 8 Years

Job Type: Full-Time

Key Responsibilities

• Design, develop, and maintain scalable data pipelines using Azure Data Factory (ADF).
• Develop and optimize data processing solutions using Azure Databricks and PySpark.
• Write efficient Python scripts for data extraction, transformation, and loading (ETL).
• Build and maintain robust data ingestion and transformation workflows.
• Work with large datasets and perform data cleansing, validation, and transformation.
• Optimize data pipelines and Spark jobs for performance, scalability, and reliability.
• Collaborate with data architects, analysts, and other engineering teams to deliver data solutions.
• Troubleshoot and resolve data pipeline failures and performance-related issues.
• Ensure data quality, consistency, and security across data platforms.
• Follow best practices for coding, deployment, monitoring, and documentation.

Required Skills

• Experience: 8 years of relevant experience in Data Engineering.
• Azure Data Factory (ADF): Strong hands-on experience in building and managing ETL/ELT pipelines.
• Azure Databricks: Experience developing and managing data processing workflows using Databricks.
• Python: Strong programming skills for data processing, automation, and transformation.
• PySpark: Hands-on experience with Spark DataFrames, transformations, and large-scale distributed data processing.
• Strong understanding of data warehousing concepts, ETL processes, and data integration.
• Experience in troubleshooting and optimizing data pipelines and processing jobs.
• Strong analytical, problem-solving, and communication skills.

Preferred Skills

• Experience working with Azure cloud services and cloud-based data platforms.
• Knowledge of SQL and database concepts.
• Experience with CI/CD and version control tools such as Git and Azure DevOps.
• Understanding of data lake architecture and data pipeline monitoring.

Qualifications

The ideal candidate should have strong hands-on expertise in ADF, Databricks, Python, and PySpark, with the ability to independently design, develop, optimize, and maintain scalable data engineering solutions.

Other Details

Interested candidates with relevant experience in the above-mentioned skills are encouraged to apply.