Data Engineer-
Diverse Data Engineering activities on Azure Data platform.
· Build & Deliver Data pipeline connecting various enterprise data sources both RDBMS, NoSQL & APIs. Develop data mappings to existing data sources.
· Design and develop big data processing notebooks using Azure Databricks
· Scripting and programming using programming languages such as Python, PySpark etc.
· Lead the data identification, data analysis efforts for data sources and work directly with data owner teams
· Lead a team of junior Data Engineers in an Agile/Scrum setting
· Develop data requirements for new data sources.
· Design and development of data extraction, data ingestion, data quality rules implementation
· Understand the data model, transform data to target schema from relational and semi structured source data.
· Clean and process the data for Machine Learning consumption.
· Provide Business Intelligence (PowerBI) and Data Warehousing (DW) solutions and support by leveraging project standards and leading analytics platforms
· Evaluate and define functional requirements for BI and DW solutions
· Design and development of data extraction, data ingestion, data quality rules implementation
· Involvement in architecture and design of data
· Work with Data Governance team to implement Data Quality and Security guidelines
· Work with DevOps team to implement CI/CD pipelines
Required Skills :-
5+ years overall experience preferably in data domain (data analysis, database developer)
Minimum 5 years' experience as a cloud-based Data Engineer
Minimum 3 years' experience and strong knowledge of Azure Data Services such as Azure Databricks, Delta Lake, Azure Data Lake Storage, Blob Storage, Azure Data Factory
Minimum 2 years' experience on Analytics Visualization tools like Power BI, Qlik Sense etc.
Strong knowledge of Big Data and Analytics, including Apache Spark, Delta Lake
Good understanding of data integration and warehousing (ETL, ELT) processes
Strong knowledge of Database concepts, SQL and experience in working with Database clients like, TOAD, SQL developer, SQL Server Management Studio
Good knowledge of NoSQL/ Document DB.
Good understanding of Enterprise Information Model and Enterprise Data Warehouse
Proficient in analyzing business requirements and mapping them to technical requirements
Data Analysis and Interpretation, and Data Issue Debugging Skills
Good knowledge of DevOps tools and processes as it applies to Data Engineering