Software Developer / Application Developer

JD: Data Engineer/Lead with Azure DataBricks , Python and Azure Data Factory

Min Experience should have 8+ years of IT Experience, with at least 3 years of Python and DataBricks experience

We need a Data Engineer/Lead to develop reusable ETL framework on Azure DataBricks using Spark 3.0.

This custom framework will be used to create a template driven data load and transformation system on Data Lake and Big Databases. The Framework shall use components from Azure and DataBricks (Spark).

The developer is expected to write Python Test cases and Data Quality checks for all code produced as a precondition for CI/CD and higher environments.

Must Have Skills:

·         Must have Azure Cloud experience and understand Azure components like ADF, Azure SQL and Azure DataBricks.

·         Very Strong DataBricks, Spark, Pyspark, DataBricks SQL with Azure

·         Must have Strong ETL and ELT experience

·         Must have strong Python, DataBricks SQL skills beyond just calling Spark Api, must be fluent in Python programming language

·         Must have Relational Database knowledge for optimum loading of Data from on premise system and data lake

·         Must have experience with Data Lake and Data Bases

·         Must have knowledge of OOP and functional programming to create reusable framework for ETL

·         Must understand encryption and security required for PII, Financial and other sensitive data.

·         Must understand Delta Lake and other big data file formats

·         Good to have exposure to DevOps and CICD skills in Big Data Space

·         Good to have Airflow or AppWorx.

·         Good to have exposure to Manufacturing Domain

·         Good to have SQL as well as No SQL databases

·         Good to have AD security experience

Job Description:

·         Work closely with Architect and Business units in understanding technical requirement and implement independently reusable code.   

·         Develop ETL Framework for template driven ETL

·         Develop Databricks code that can call Python and other required libraries

·         Work with offshore and on shore teams and mentor team members on ETL and do KT on framework and design

·         Implement transformations and aggregations as requirement

·         Work in Agile manner and resolve ambiguous requirements and communicate effectively with his peers

• It is a Full Time IT / Software / Data Analyst job for candidates with 2 - 6 years of experience.