Data Engineer needed at CDW

Key responsibilities

• Ingest data from various sources, including on premise SQL databases, REST API’s, Apache Kafka streams, etc.
• Define and apply cleansing rules to the data to ensure meets quality expectations.
• Define and model data into single source of truth Kimball datasets, e.g. dimensions and facts.
• DevOps approach to software; create high quality code using traditional software practices, such as building, linting, unit and integration testing, repositories, CI/CD, peer reviews, etc.
• DevOps approach to support; monitor pipelines to ensure businesscritical data pipelines are processed meeting time constraint and data quality expectations (including implementing reactive changes where applicable).
• Drives and Integrates into the data team’s WOW, including backlog refinement, sprint planning, demonstrations, and retrospectives, including translation of user requirements into technical requirements including complexity estimation, implement into sprint cycles, collaboration into process improvement, etc.
• Based on requirements, perform ad-hoc analysis of structured and unstructured data across multiple data sources to inform solution design.
• Document datasets in data catalogue, including ownership, stewardship, dictionaries, glossaries, lineage, sensitivity, etc.
• Ownership of Work Items and works with data owners / stewards to ensure high quality, aligned deliveries, and compliance with legislation, such as GDPR, e.g. PII vs non-PII, data retention, etc.
• Document solution design in wiki.
• The above is not an exhaustive list of duties and you will be expected to perform different tasks as necessitated by your changing role within the organisation and the overall business objectives of the organisation.

Qualifications, Skills & Experience

Must Have:

• Architecture, modelling and leadership skills
• Strong Azure data skills, including:
• Azure Data Factory V2
• Azure Data Lake Storage V2
• Azure Databricks
• Azure Function Apps& Logic Apps
• Azure Stream Analytics o Azure Resource Manager skills (Terraform, Azure Portal, Az CLI and Az PowerShell)
• Strong PySpark, Delta Lake, Unity CatLog and Python skills. o Includes ability to write unit and integration tests in Python with unittest, pytest, etc.
• Strong understanding of software development practices, such as SOLID principals, structuring code, testing, IOC, etc.
• Strong repositories, CI/CD skills.
• Strong knowledge of Kimball data modelling, such as star schema, snowflake, etc.
• Strong SQL skills.
• Strong data analysis skills.
• Excellent written and verbal communication skills
• A minimum of 2 years of experience as an Architect.
• A demonstrable track record of getting stuff done whilst managing competing pressures and deadlines and retaining an eye for detail and quality.
• A passion for technology and its ability to have a positive impact on business.

Nice To Have:

• Azure DevOps (git, multistage YAML, etc.)
• Other languages, such as C#, PowerShell
• IaC, e.g. Terraform, ARM, Bicep, etc.
• Test driven development (TDD)
• Streaming technologies, such as Azure Stream Analytics, Spark Structured Streaming, etc.
• Power BI Engineering experience
• Certified SCRUM Developer (CSD)
• Machine learning and Artificial intelligence