Data Processing Quality Engineer
TLDR
Ensure quality of the data processing engine, in terms of result accuracy, performance fidelity, and robust execution at scale.
Ensure quality of the data processing engine, in terms of result accuracy, performance fidelity, and robust execution at scale.
Requirements
• BS EE/CS or equivalent
• 5+ years of experience in data processing quality or performance testing for database, data warehouse, or query engine applications. Experience testing for platforms such as Apache Spark, Gluten, Velox, DataFusion preferred.
• Solid knowledge of SQL, Python, and similar data processing languages
• Automation-first mindset, experienced with programming/scripting languages and automation tools.
• Strong in problem-solving and coming up with the test strategy for the complex system.
• Strong in debugging, root cause, and narrowing down the failures.
• Experience in Functional, Performance, Integration, System Level testing
• Experience with the use of public cloud platforms such as AWS, GCP, and MS Azure
• Good Knowledge of tools like Jira, Confluence, Git, Jenkins.
• Good understanding of SDLC and agile methodologies.
• Good understanding of CI/CD implementations.
DataPelago develops a groundbreaking data processing engine that optimizes the performance of Apache Spark, Apache Flink, and Ray for large-scale data analytics and GenAI preprocessing. Our technology leverage advancements in hardware-accelerated computing and parallel processing to eliminate data bottlenecks, making it easier for engineers to handle diverse datasets efficiently.