Key Purpose

• The primary role of a Data Engineer it to provide thought leadership and technical expertise related to the full lifecycle of the data management discipline. This role requires extensive knowledge and experience related to data analysis, design, manipulation, development and operational insights. The role requires a candidate to be able to use a wide variety of data tooling as well as being able to craft tooling where required.

Areas of responsibility may include but not limited to

Strategy & Planning

• Create data lifecycle management architectures, standards, statistical reporting, and data analysis methodologies.
• Create and deploy scalable data lifecycle management solution, including functional areas such as ETL, ELT, Data streaming, data quality, test data management, analytics and business intelligence.
• Assist in the development of data management policies and procedures.
• Provide thought leadership and innovation related to the data lifecycle management discipline.

Support and collaboration

• The Data Engineer plays a collaborative role where, in collaboration with architecture teams, engineering and operations management, develops and implements optimisation and innovative solutions for database maintenance, monitoring, and performance tuning to be applied across the business. The Data Engineer plays a supporting role to various departments across the business where supports is provided to gain useful insights on the implementation of data related workflows.
• Key for this role, the Data Engineer also plays a driving and supporting role to data warehousing and analytics initiative in the implementation of the data warehouse for the new big data platforms. He works collaboratively with the entire Data and Analytics team, providing support to the entire department for its data centric needs.

Acquisition & Deployment

• Conduct research and make recommendations on data mining products, services, protocols, and standards in support of procurement and development efforts.

Operations and Operations Management

• Work with application development staff to coordinate the creation and management of data queries.
• Collaborate with unit managers, end users, development staff, and other stakeholders to integrate data mining applications with existing systems.
• Provide and apply quality assurance best practices for data management services across the organization.
• Develop, implement, and maintain change control and testing processes for modifications to data models.
• Create data definitions for new database file/table development and/or changes to existing ones.
• Determine required network components to ensure data access, as well as data consistency and integrity.
• Develop routines for end users to facilitate best practices use of data management tools.
• Collaborate with various administrators, including database and disaster recovery, to ensure effective protection and integrity of data assets.
• Monitor data system details within the database, including queries, resource utilization and execution times, and provide guidance to implement efficiency improvements.
• Respond to and resolve data related performance issues.
• Manage and/or provide guidance to other members of the team.

Personal Attributes and Skills

• Drives Results
• Values Driven
• Optimistic
• Learns on the Fly
• Resilient
• Instils Trust
• People Savvy
• Problem Solver
• Manages complexity
• Balances Stakeholders
• Transparency
• Multitasking
• Organizational skills
• Analytics
• Good written and oral communication skills.
• Strong technical documentation skills.
• Good interpersonal skills.
• Ability to conduct research into data management practices, and products as required.
• Ability to present ideas in a user-friendly language.
• Highly self-motivated and directed.
• Keen attention to detail.
• Proven analytical and problem-solving abilities.
• Ability to effectively prioritize and execute tasks in a high-pressure environment.
• Strong customer service orientation.
• Experience working in a team-oriented, collaborative environment.

Education and Experience

Education:

• Diploma or university degree in the field of computer science or statistics, and/or 8 years equivalent work experience.
• Certifications in one or more industry standard relational database management systems

Experience:

• Experience in ETL tooling such as SQL Server Integration Services etc. 
• Experience in Java development (any amount)
• Direct experience with data management and data lifecycle management techniques.
• Extensive experience with database change management.

Knowledge:

• Strong understanding of data mining models, structures, theories, principles, and practices.
• Excellent knowledge of data modeling tools.
• Strong familiarity with data preparation, processing, classification, and forecasting.
• Working technical experience with at least 2 different relational database servers, including PostgreSQL, Microsoft SQL Server, MYSQL, and Oracle.
• Knowledge and experience with NoSQL databases an advantage.
• Hands-on database tuning and troubleshooting experience.
• Good understanding of the organization’s goals and objectives.
• Knowledge of applicable data privacy practices and laws.
• Knowledge and experience with cloud-based deployments of database systems (Azure/AWS) advantageous
• Processes: SDLC. Unit Testing..
• Technical Change Control.

Data privacy

• Base understanding of POPIA and/or GDPR
• Technical Skills: Software development within SDLC.

Relational Database Modelling and design

• UML modelling in Enterprise Architect
• Database structure design
• Normalisation techniques
• Business Writing

Database

• Experience with 2 or more Relational Database technologies
• Working knowledge of NoSQL databases

Data management  experience

• Replication
• Large database query optimisation
• Data Lifecycle Management
• Horizontal and vertical data partitioning
• Exposure or involvement with operations management, although not required, would be an advantage

Data Warehouse and BI:

• Understanding of DW 2.0
• Exposure to Machine Learning
• Working knowledge of at least one visualisation tool (eg. PowerBI)
• Experience in modelling facts and dimensions
• Monitoring / Logging / Tracing techniques

Platform Service Technologies

Exposure/experience in (advantageous):

• Freemarker or similar templatization tools
• Kafka 

Infrastructure

•  Containerisation knowledge

Dev/Ops

• Git / Gerrit
• Jenkins

Testing

• Exposure to at least 1 automated test framework/Toolset