Back to jobsDevelop new and improve existing data pipeline on Big Data platform (Hadoop, mapR or equivalent).
Build new and enhance existing data application for streaming and batch datasets.
Work on Apache Airflow on data pipeline, Python, Spark, PySpark, Scala, Java, SQL, etc for data application open sources technologies.
Responsible in developing frontend dashboards and integration to back end. Using frontend and backend API Frameworks such as React, Angular Django, FastAPI, Springboot.
Perform R&D and conduct PoC (proof-of-concept) for new data solution.
Perform metadata and data management, data security
Drive optimization, testing and tooling to improve data quality & efficiency in data lake and streaming platform
Lead and monitor the performance of Junior Data Engineer and providing them with practical guidance, solution validation and implementation.
Lead and manage DevOps, DataOps and Streaming Operations (or equivalent) in Data Engineering
Design high level & detailed design to ensure that the solution delivers to the business needs and align to the data & analytics architecture principles and roadmap.
Collaborate with different stakeholders from business, technical, project management and operation to design and implement the solution.
Ensure best practices, frameworks and re-useable components are employed in the development project.
Ability to lead troubleshooting efforts for complex design and eliminate application issue faced by the project and operation team.
Understand various data security standards, information security standard and to apply and adhere to the required data controls for user access.
