Job Summary: IT Software - Engineering Development Role !
Position: Data Engineer
Shift: 11.00AM to 8.00 PM
Mode of work: Remote
Exp: 2 to 5 Yrs
Education Mandate : BE / BTech Comp Science 2020• (Batch or later) 2021- 2022•
Mandatory Skills : Strong proficiency in SQL for querying and managing data from relational databases.
Excellent problem-solving, analytical skills.
Familiarity with DevOps for CI/CD in data integration pipelines
Experience with additional cloud platforms (e.g., Azure, AWS, Google Cloud).
Built and optimized any ETL pipelines using [e.g., Python, ADF, etc.]
Notice Period: Immediate to 15 days
Budget : Standard Market –Hike for Services firms • Background (Consulting Background)
JOB Description
Requirements
Key Responsibilities:
· Develop, and maintain ETL pipelines to move, transform, and load data from multiple sources into data warehouses or data lakes.
· Work with large datasets and ensure data quality and integrity throughout the ETL process.
· Integrate various data sources (on-premises and cloud) using Cloud databases, ensuring scalability and performance.
· Develop complex SQL queries to extract, transform, and load data efficiently.
· Optimize SQL queries for performance and ensure data consistency across systems.
· Manage and maintain relational databases to ensure reliable data storage and retrieval.
· Integrate data from different sources (including external APIs, third-party applications, etc.) into centralized data storage systems.
· Continuously monitor, optimize, and troubleshoot data pipelines to ensure high performance and reliability.
· Implement error handling, logging, and monitoring mechanisms to ensure smooth operations.
· Documentation and Best Practices: Create and maintain clear documentation for ETL processes, data pipelines, and data models.
Skills (Mandatory) (Note: list of skills needed to perform the role):
· Strong proficiency in SQL for querying and managing data from relational databases.
· Excellent problem-solving, analytical, and leadership skills.
· Familiarity with DevOps for CI/CD in data integration pipelines.
· Experience with Data bricks is a plus.
· Built and optimized ETL pipelines using [e.g., Python, ADF, etc.]
· Worked with large datasets and structured/unstructured data
· Designed and maintained SQL databases & NoSQL databases
· Implemented data warehouse solutions using [e.g., AWS Redshift, Azure Synapse]
· Developed real-time & batch data processing pipelines
· Improved query performance and database optimization
· Automated data pipeline monitoring and troubleshooting issues
Projects (Optional if work experience is strong)
· Built an end-to-end data pipeline using [tools like Apache Airflow, Python, and SQL]
· Created a real-time data streaming solution using Kafka and Spark
· Developed an ETL pipeline for structured & semi-structured data
Skills (Good to Have) (Note: list of skills that will add value to the role):
· Experience with additional cloud platforms (e.g., Azure, AWS, Google Cloud).
· Advanced skills in data visualization tools (e.g., Power BI, SSRS, Tableau).
· Proficiency in Python or other scripting languages.