Job Description
Position Overview
The Assoc Data Engineer supports the design, implementation and maintenance of data flow channels and data processing systems that support the collection, storage, batch and real-time processing, and analysis of information in a scalable, repeatable and secure manner.
This role focuses
- on defining optimal solutions to data collection, processing and warehousing
- on designs, codes and tests data systems and works on implementing those into the internal infrastructure
- on collecting, parsing, managing, analysing and visualising large sets of data to turn information into insights accessible through multiple platforms
- on development tools to generate reports, dashboards and analytical solutions according to business rules and specifications.
Role & Responsibilities
Identify business needs
• Work with stakeholders to understand needs for data structure, availability and accessibility
• Support the translation of data needs into technical system requirements
• Advise on latest technologies, strategies, and products in database and data processing software
• Recommend types of data and data sources needed
Build data processing systems
• Support the building of data flow channels and processing systems to extract, transform, load and integrate data from various sources
• Develop complex code, scripts and data pipelines to process structured and unstructured data near real-time
• Develop prototypes and proof of concepts for the selected solutions
• Support the identification of data sources (both internal and external)
• Perform operational ad-hoc data provision to extract/load data based on stakeholders requirements using ETL tools such as Informatica
• Research hardware and software needs to support selected solutions
Analyse Data
• Gather data from internal systems and external sources
• Enter information into data collection systems
• Clean databases from duplicated, outdated or irrelevant information
• Perform data validation and quality control checks
• Mine data to identify trends, patterns and correlations
• Develop solutions and recommendations to address information need
Report Insights
• Develop automated and logical data models and data output methods
• Design data reports and visualisation tools to facilitate data understanding
Optimise solution performance
• Assist with integration of data systems into existing infrastructure
• Develop tools to improve data flows between internal/external systems and the data warehouse
• Automate data collection and analysis processes, data releasing and reporting tools
• Test data system configurations and perform regular tuning to increase efficiency
• Facilitate data cleansing, enrichment and data quality improvements
• Enable searching, data visualisation, and advanced analytics functionality
Maintain data processing solutions
• Monitor data system performance
• Support the handling and logging of errors
Requirements
• Bachelor’s or master’s degree in computer science, data engineering, or a related field.
• Minimum 10 years of experience in data engineering, with minimum AWS Cloud Practitioner certification or higher expertise in AWS services, Databricks, and/or Informatica IDMC.
• Proficiency in programming languages such as Python, Java, or Scala for building data pipelines.
• Evaluate potential technical solutions and make recommendations to resolve data issues especially on performance assessment for complex data transformations and long running data processes.
• Strong knowledge of SQL and NoSQL databases.
• Familiarity with data modeling and schema design.
• Excellent problem-solving and analytical skills.
• Strong communication and collaboration skills.
Apply Now
#LI-SYNX32