Azure Data Engineer (Data Lake Expert)
Area of expertise:
As an Azure Data Engineer (Data Lake) at Avanade, you will use comprehensive modern data engineer techniques and methods with Advanced Analytics to support business decisions for Avanade clients. You’re a data designer who knows how to find, store, and present a range of information from different sources so that everyone can access what they need quickly and simply, and use it effectively.
Key Role Responsibilities:
- You design and set up projects that bring together information from a variety of sources, to enable analysis and decision-making.
- You make sure that data is accessible and easy to use so that it can be used for routine and ad-hoc analysis.
- Be able to analyze current business practices, processes, and procedures as well as identifying future business opportunities for leveraging Microsoft Azure Data & Analytics PaaS Services.
- Support the planning and implementation of data design services, providing sizing and configuration assistance, and performing needs assessments.
- Delivery of architectures for transformations and modernizations of enterprise data solutions using Azure cloud data technologies.
- Design and Build Modern Data Pipelines and Data Streams.
- Design and Build Data Service APIs.
- Develop and maintain data warehouse schematics, layouts, architectures, and relational/non-relational databases for data access and Advanced Analytics.
- Expose data to end-users using PowerBI, Azure API Apps, or any other modern visualization platform or experience.
- Implement effective metrics and monitoring processes.
- Travel as needed
Key Role Skill & Capability Requirements:
- Strong experience as an Azure Data Engineer and must have Azure Data Lakes experience.
- Demonstrated experience of turning business use cases and requirements into technical solutions.
- Experience in business processing mapping of data and analytics solutions.
- Ability to conduct data profiling, cataloging, and mapping for technical design and construction of technical data flows.
- The ability to apply such methods to solve business problems using one or more Azure Data and Analytics services in combination with building data pipelines, data streams, and system integration.
- Azure IoT, Azure HDInsight + Spark, Azure Cosmos DB, Azure Stream Analytics is a plus.
- Knowledge of Python is a plus.
- Experience preparing data for Data Science and Machine Learning.
- Experience preparing data for use in Azure Machine Learning and/or Azure Databricks is a plus.
- Demonstrated experience preparing data and building data pipelines for AI Use Cases (text, voice, image, etc).
- Big Data components such as Kafka, Spark SQL, Data frames, HIVE DB, etc. and their implementation using Azure Data Bricks
- Experience in migration from on-premises Spark environment to Azure using Azure Data bricks
- Azure Data bricks integration with other services
- Read and write data in Azure Data bricks
- Best practices in Azure Data bricks
- Coding complex U-SQL, Spark (Scala or Python). T-SQL
- Good Understanding of Modern Data Warehouse/Lambda Architecture, Data warehousing concepts
- PowerShell scripting
- Effort estimation in support of project delivery or pre-sales effort
- Experience in structured problem-solving. You don’t hesitate to roll up your sleeves to solve code-related issues.
Preferred Education Background:
- You likely possess MS/ Ph.D. in a quantitative field such as computer science, applied mathematics, statistics, or machine learning. An equivalent combination of education and experience will also suffice.
Benefits of being an Avanade Employee
- Additional 5 days personal Leave + Opportunity to purchase extra annual leave
- Wellbeing & Gadget Allowance
- Flexible working arrangements
- Generous parental leave
- Corporate private health insurance discounts
- Up to 80 hours of training per year
Share this job:
Avanade leads in providing innovative digital services, business solutions and design-led experiences for its clients, delivered through the power of people and the Microsoft ecosystem. Our professionals combine technology, business and industry expertise to build and deploy solutions to realize results for clients and their customers. Avanade has 29,000 digitally connected people across 23 countries, bringing clients the best thinking through a collaborative culture that honors diversity and reflects the communities in which we operate. We welcome all, and seek talented individuals who can bring their whole self to work, build inclusive teams and encourage diversity inside and outside the organization. Majority owned by Accenture, Avanade was founded in 2000 by Accenture LLP and Microsoft Corporation. Learn more at www.avanade.com.