Digital Engineering Lead Engineer

• Experience in implementing data storage solutions, managing, and developing data processing, and optimizing data solutions (Azure).• Expertise in integrating, transforming, and consolidating data from structured and semi-structured data systems suitable for analytics solutions.• Hands-on experience in integrating event based or streaming data into an existing data model. • Consuming stream payloads using Azure Service Bus.• Engaging with business users to gather requirements, design visualizations, and provide training to use self-service BI tools.• Developing conceptual solutions & create proof-of-concepts to demonstrate viability of solutions.• Architecting & implementing medium to large scale BI solutions on Azure using Azure Data Platform services (Azure Data Lake, Data Factory, Data Lake Analytics, Stream Analytics, Azure SQL DW, HDInsight/Databricks, NoSQL DB).• Estimating the cluster size, monitoring, and troubleshooting of the Spark data bricks cluster.• Experience in solving business problems using SQL.• Experience in the financial services industry and experience managing teams in technology consulting.• Experience in the following next gen tools and technologies: Hadoop, HIVE, Databricks, Python, Spark, Kafka, Dremio, Azure Platform, NoSQL, HBase, Snowflake.• Experience in Data Management and Data Governance.• Experience in data sourcing, data warehousing implementation, data management and Data-Sourcing ETL.• Experience in data quality and designing and implementing end-to-end data controls.• Experience in Events integration into a data modelPrimary Skills: Azure Databricks, Azure Service Bus, PySpark, SQL #LI-INCDTS