Country:

United States

Cities:

Dallas

Area of expertise:

Analytics

Job Description

Big Data & PySpark Engineer 

 

Do you view data as an art and a science? So do we. 

 

Avanade leads in providing creative digital services, business solutions and design-led experiences for its clients, delivered through the power of people and the Microsoft ecosystem. Our professionals combine technology, business, and industry expertise to build and deploy solutions to realize results for clients and their customers. Avanade has 34,000 digitally connected people across 24 countries, bringing clients the best thinking through a collaborative culture that honors diversity and reflects the communities in which we operate. We welcome all and seek talented individuals who can bring their whole self to work, build inclusive teams and encourage diversity inside and outside the organization. Majority owned by Accenture, Avanade was founded in 2000 by Accenture LLP and Microsoft Corporation. 

 

How we support you: 

 

As an Avanade Short-term Resource, we offer compelling project work, competitive hour rates, paid company holidays, 401k eligibility with company match, and paid sick leave.

 

About you: 

 

You draw on your considerable experience in bringing data and statistics to life to solve sometimes complex problems, and you’re comfortable looking after several projects at once. You’re able to make your own decisions while at the same time supporting more junior team members. As a Big Data/PySpark Engineer at Avanade, you will have a deep understanding of the architecture, performance characteristics and limitations of modern storage and computational frameworks, with experience implementing solutions that leverage: HDFS/Hive; Spark/MLlib; Kafka, etc. You will have knowledge in Apache Spark and/or Python programming, deep experience in developing data processing using PySpark such as reading data from external sources, merge data, perform data enrichment and load into target data destinations. Deep experience in developing data processing tasks using PySpark such as reading data from external sources, merge data, perform data enrichment and load into target data destinations. Knowledge of python packaging, azure/data lake, and data bricks. Experience with ML. Manage large volumes of structured and unstructured data and extract & clean data to make it amenable for analysis. 80% travel is required. 

 

Key Technologies: 

 

Must-haves: 

Apache Spark 

Python 

PySpark  

Hadoop 

SQL 

Azure or AWS 

 

 

Pluses:  

Cosmos DB 

TensorFlow 

Apache airflow  

Snowflake  

Linux OS 

. 

 

Key Role Responsibilities: 

 

Demonstrated expertise working with and maintaining open-source data analysis platforms, including but not limited to:  

Pandas, Scikit-Learn, Matplotlib, TensorFlow, Jupyter and other Python data tools  

Spark (PySpark), HDFS, Kafka and other high-volume data tools  

SQL and NoSQL storage tools, such as MySQL, Postgres, Cassandra, MongoDB and ElasticSearch. 

Deep understanding of the architecture, performance characteristics and limitations of modern storage and computational frameworks, with experience implementing solutions that leverage: HDFS/Hive; Spark/MLlib; Kafka, etc. 

Technical background in computer science, data science, machine learning, artificial intelligence, statistics or other quantitative and computational science  

Hands-on experience using Big Data and statistical analysis tools such as Hadoop/Spark, SQL 

Experience transforming Data at a scale using Spark/PySpark 

Working experience with Linux OS (Redhat/Ubuntu) 

Excellent communication skills (speaking, presenting) 

 

Preferred Years of Work Experience: 

You likely have about 3-6 + years of relevant professional experience. 

 


Apply now

Share this job:

Share Facebook Twitter Email

About Avanade

Avanade leads in providing innovative digital services, business solutions and design-led experiences for its clients, delivered through the power of people and the Microsoft ecosystem. Our professionals combine technology, business and industry expertise to build and deploy solutions to realize results for clients and their customers. Avanade has 29,000 digitally connected people across 23 countries, bringing clients the best thinking through a collaborative culture that honors diversity and reflects the communities in which we operate. We welcome all, and seek talented individuals who can bring their whole self to work, build inclusive teams and encourage diversity inside and outside the organization. Majority owned by Accenture, Avanade was founded in 2000 by Accenture LLP and Microsoft Corporation. Learn more at www.avanade.com.