CloudFactory is seeking a highly skilled and experienced Senior Data Engineer to join our dynamic team. As a global leader in combining people and technology to provide a cloud workforce solution, CloudFactory works on numerous machine learning and core business data processing projects. We are dedicated to creating economic and leadership opportunities for talented individuals in developing nations.

As a Senior Data Engineer at CloudFactory, you will play a key role in developing and maintaining our data infrastructure to support our growing operations. You will have the opportunity to work with cutting-edge technologies and collaborate with a diverse and passionate team to scale our data processing capabilities. If you are passionate about making a difference in the world through meaningful work, enjoy building relationships, and thrive as part of a unified team, CloudFactory is the place for you.

Responsibilities

The successful candidate will translate user requirements into actionable insights by leveraging existing tools and technologies. Your responsibilities will include:

• Regular interaction with PMs/stakeholders to understand their requirements.
• Investigation to ascertain if the requirements can be met using existing technology and data resources.
• Data extraction using Python and other tools like Fivetran.
• Data modelling using SQL and the concept of Star and Snowflake schema.
• Assisting in cost and performance optimization choices within the data warehouse.
• Communicating reasoning and logic behind opinions to help make optimal choices.
• Write data quality and other metadata collection systems within data pipelines to promote the observability and recoverability of pipelines.
• Write unit and integration tests to bolster system reliability
• Mentoring and guidance

Tech Stack

Our tech stack:

• Data extraction and orchestration tools - Fivetran, Prefect
• Data modelling tool - DBT
• Data Warehouse - Snowflake
• Data quality cheeks - DBT, Elementary
• Data visualisation - Tableau and Quicksight
• Data catalogue and lineage - Atlan
• Source code management - Github
• Infrastructure as Code - Terraform
• Programming Language - Python
• AWS services (ECS, S3)

Interview Process

Candidates will be expected to complete an initial coding and analytical thinking test, followed by an interview with a Senior Product Manager where they will be expected to talk through previous projects, including approaches taken and results achieved.

Successful candidates will then meet with the Engineering Director before a final interview with the CTO.

The process is expected to take 1-2 weeks.

• The ideal candidate will possess:
  
  • Minimum 4 years of experience as a Data Engineer.
  • Mastery of SQL in the context of building and running an enterprise data warehouse.
  • Mastery of data orchestration and related tools in the context of transporting data.
  • Proficiency with OLAP systems and related use cases.
  • Proficiency in extracting data from source systems using Python.
  • Deep understanding of and proficiency in using MPP architectures.
  • Proficiency in modelling data using Kimball, Inmon or Data Vault.
  • Proficiency in writing unit tests and integration tests.
  • Understanding of microservices and event driven systems.
  • Excellent communication skills, able to translate complex data into understandable insights and narratives.