Purpose

• To build and manage AWS data pipelines to ensure systems are scalable, reliable, and optimized for analytics, machine learning, and various business needs.

Key Responsibilities

• Design and implement large scale enterprise data solutions by using a combination of the of the following technologies – AWS Glue, AWS Step-functions, AWS RedShift, AWS Lambda, AWS Athena, AWS Lakeformation, Spark, Python.
• Analyze, re-architect and re-platform on-premise data warehouses to data platforms on AWS cloud using AWS and 3rd party services.
• Design and build production data pipelines from ingestion to integration within a big data architecture, using PySpark, Python and SQL.
• Design, implement and support an analytical data infrastructure providing ad-hoc access to large datasets and computing power.
• Interface with other technology teams to extract, transform, and load data from a wide variety of data sources using SQL and AWS big data technologies.
• Continual research of the latest big data and visualization technologies to provide new capabilities and increase efficiency.
• Collaborate with other tech teams to implement advanced analytics algorithms that exploit our rich datasets for statistical analysis, prediction, clustering and machine learning.
• Help continually improve ongoing reporting and analysis processes, automating or simplifying self-service support for customers.

Minimum Experience

• Advanced Data Engineering knowledge and experience working with modern data practices.
• Experience building/operating highly available, distributed systems of data extraction, ingestion, and processing of large data sets.
• Experience working with distributed systems as it pertains to data storage and computing.
• Experience performing root cause analysis on internal and external data and processes to answer specific business questions and identify opportunities for improvement.
• Strong analytic skills related to working with unstructured datasets.
• Build processes supporting data transformation, data structures, meta data, dependency, and workload management.

Additional Minimum Experience

• Proficiency in SQL with expertise in writing and optimizing complex queries.            
• A successful history of manipulating, processing and extracting value from large, disconnected data sets.
• Experience supporting and working with cross-functional teams in a dynamic environment.
• Experience in a Data Engineering roles.
• Experience with the following is a must: AWS Glue, PySpark, SQL
• Experience with data pipeline and workflow management tools like AWS StepFunctions and Control-M would be beneficial but not required.
• Experience with AWS cloud services: EC2, EMR, RDS, DynamoDB would be beneficial but not required.

Minimum Qualifications

• Bachelor’s Degree in Computer Science, Information Technology, or other relevant fields.
• Has experience in any of the following AWS Athena, AWS Glue, Pyspark, AWS DynamoDB, AWS Redshift, AWS Lambda and AWS Step Functions.
• Proficient in SQL, Python and PySpark
• Proficient in utilizing a cloud platform and services
• Knowledge of software engineering best practices across the development lifecycle, including agile methodologies, coding standards, code reviews, source management, build processes, testing, devops and operations.