Pivotal Software is hiring a

Software Engineer - Data Engineering

New York, United States


You have a passion for developing large distributed systems to manage data. You may have worked on Massively Parallel Processing [MPP] systems solutions, SQL on Hadoop, in memory grid systems or perhaps even NoSQL systems. Whichever kinds of ‘Big Data’  solutions you’ve worked on, you understand that software that manages data at scale is a little different from other kinds of systems. Solutions that work fine on single nodes with gigabytes of data can fall over when scaled to petabytes of data and thousands of nodes. You relish discovering solutions that perform at scale while preserving the sanctity of customer data. You are curious, love to learn, and love to share your latest discoveries. Above all, you love shipping software as a member of a collaborative team.


At Pivotal, our mission is to enable customers to build a new class of applications, leveraging big and fast data, and do all of this with the power of cloud-independence. We are at the epicenter of Cloud, Big Data and Mobile and we’re actively investing in all three areas.. Pivotal’s broad portfolio of products includes the Big Data Suite, the most complete approach to enterprise data lakes and advanced analytics; Pivotal Cloud Foundry, the industry leading Platform as a Service product; and world leading ultra-agile application development through Pivotal Labs where we’re transforming how the world builds software.

The Big Data Suite includes HAWQ, our SQL on Hadoop solution; Greenplum Database (GPDB), our massively parallel data warehouse; GemFire, our distributed in-memory key-value store; and MADlib, our machine learning solution. Open source is an important part of our strategy and all of our products are open source.

The Pivotal Data engineering team tackles the technical challenges that come with massively parallel distributed systems operating on petabytes of data across thousands of nodes. We delve into areas like query optimization, high-performance in-memory transaction and query processing, parallel and distributed execution of advanced data processing algorithms, resource management, and storage. Here at Pivotal you'll be working on hard, worthwhile problems with a collaborative team, accelerating your growth as an engineer.


Pivotal Data engineering in the US is located in our light-filled spacious offices in Palo Alto, CA right off  280, close to the VMware campus. We are also open to candidates working at our downtown San Francisco location off Howard. 

To make sure you start your day energized, we provide a catered breakfast every weekday morning, and unlimited snacks/drinks are of course available all day. Our collaborative, open-plan office space is filled with talented, like-minded engineers who enjoy taking advantage of our weekly Tech Talks, yoga classes, playing ping pong, board games and hanging out with their co-workers.

Desired Skills/ Experience:

Deep, meaningful, recent development experience with one or more of the following is required:

  • Database kernel or related internals
  • In-memory object stores
  • SQL on hadoop
  • Machine learning
  • Query execution or cost based query optimization
  • Highly scalable, distributed, data driven applications
  • Linux systems code development in c/c++ and/or java
  • Products deployed in highly available, mission critical, enterprise environments

Optional, but highly desirable skill sets:

  • A deep understanding of Postgres internals from versions 8.2 to 9.x
  • Domain knowledge of high speed transaction processing
  • Development experience with in-memory grid systems
  • Extremely large scale application development – data systems in the multi-petabyte range
  • Exposure to pair-programming, test-driven development, continuous integration and other agile or XP engineering practices
  • Experience with the challenges of testing distributed systems.


***Please answer application questions if they are not included in your cover letter****

Other jobs at Pivotal Software