Guidance

Data engineer: skills they need

Updated 2 January 2019

This content is part of the Digital, Data and Technology (DDaT) Capability Framework which describes the DDaT roles in government and the skills needed to do them.

1. What a data engineer does

A data engineer delivers the designs set by more senior members of the data engineering community.

Data engineers:

• implement data flows to connect operational systems, data for analytics and BI systems
• document source-to-target mappings
• re-engineer manual data flows to enable scaling and repeatable use
• support the build of data streaming systems
• write ETL scripts and code to make sure the ETL process performs optimally
• develop business intelligence reports that can be re-used
• build accessible data for analysis

2. What skills they need

A data engineer needs specific technical skills.

All roles have essential skills, and some have desirable skills.

Each skill has one of 4 skill levels associated with it:

• Expert
• Practitioner
• Working
• Awareness

2.1 Essential skills

Skill	Description of the skill	Skill level	What the skill level means
Communicating between the technical and the non-technical	Is able to communicate effectively across organisational, technical and political boundaries, understanding the context. Makes complex and technical information and language simple and accessible for non-technical audiences. Is able to advocate and communicate what a team does to create trust and authenticity, and can respond to challenge.	Awareness	Is aware of the need to translate technical concepts into non-technical language and understands what communication is required to internal and external stakeholders.
Data analysis and synthesis	Translates data into valuable insights that inform decisions. Involves teams in analytics and synthesis to increase consensus and challenge assumptions. Identifies and utilises the most appropriate analytical techniques. Has an understanding of analytical tools and is numerate. Is aware of and keeps up to date with advances in digital analytics tools and data manipulation products. Collects, collates, cleanses, synthesises and interprets data to derive meaningful and actionable insights.	Working	Undertakes data profiling and source system analysis and can present clear insights to colleagues to support the end use of the data.
Data development process	Integrates and separates data feeds in order to map, produce, transform and test new data products.	Working	Designs, builds and tests data products based on feeds from multiple systems using a range of different storage technologies and/or access methods. Creates repeatable and reusable products.
Data integration design	Develops fit for purpose, resilient, scalable and future-proof data services to meet user needs. Has a demonstrable understanding of how to expose data from systems (for example, through APIs), link data from multiple systems and deliver streaming services.	Working	Delivers data solutions in accordance with agreed organisational standards that ensure services are resilient, scalable and future-proof.
Data modelling	Produces data models and understands where to use different types of data models. Understands different tools and is able to compare between different data models. Able to reverse engineer a data model from a live system. Understands industry recognised data modelling patterns and standards.	Working	Understands the concepts and principles of data modelling and is able to produce, maintain and update relevant data models for specific business needs. Reverse engineers data models from a live system.
Programming and build (data engineering)	Designs, writes and iterates code from prototype to production-ready. Understands security, accessibility and version control. Can use a range of coding tools and languages.	Working	Designs, codes, tests, corrects and documents simple programs or scripts under the direction of others.
Technical understanding (data engineering)	Has knowledge of specific technologies which underpin an individual’s ability to deliver the responsibilities and tasks of their role. Applies the required breadth and depth of technical knowledge.	Working	Understands core technical concepts related to their role and is able to apply them with guidance.
Testing	Plans, designs, manages, executes and reports tests, using appropriate tools and techniques, and works within regulations. Ensures risks associated with deployment are adequately understood and documented.	Awareness	Correctly executes test scripts under supervision. Understands the role of testing and how it works.

2.2 Desirable skills

Skill	Description of the skill	Skill level	What the skill level means
Data Innovation	Recognises and exploits business opportunities to ensure more efficient and effective performance of organisations. Explores new ways of conducting business and organisational processes.	Awareness	Aware of opportunities for innovation with new tools and uses of data.
Metadata management	Understands a variety of metadata management tools. Designs and maintains the appropriate metadata repositories to enable the organisation to understand their data assets.	Working	Works with metadata repositories to complete complex tasks such as data and systems integration impact analysis. Maintains a repository to ensure information remains accurate and up to date.
Problem resolution (data)	Logs, analyses and manages problems in order to identify and implement the appropriate solution. Ensures that the problem is fixed.	Awareness	Aware of the types of problems in databases, data processes, data products and services.

3. Civil Service Success Profiles Framework

The Civil Service uses The Success Profiles Framework to assess candidates during recruitment.

It is a flexible framework, used to assesses a range of experiences, abilities, strengths, behaviours and technical/professional skills required for different roles.

Find out more about Success Profiles.

4. Other roles in data engineering

There are 3 other role levels in data engineering:

• head of data engineering
• lead data engineer
• senior data engineer