Data lies at the heart of every product we build at SocialCops. It breathes life into our visualizations, serves as the starting point for our indices, powers our data intelligence platform, and plays the central character in every story that we write. Data is our rockstar and, as a Data Engineer, you will be its manager. You will handle all aspects of its life inside SocialCops, making sure that it is cleaned, transformed, and appropriately dressed before every performance.
You will design and build both search and storage infrastructure for the vast amounts of data that flow into SocialCops. You will help the Data Scientists on your team build models and apply complex transformations at scale. You will be tasked with firefighting open-ended data acquisition issues and help create generic solutions for scraping and parsing data over time.
An ideal Data Engineer is someone who understands how to scale operations on data. They have worked with large datasets before and have a functional understanding of various statistical methods and machine-learning algorithms. They love working on problems related to natural language processing, information retrieval, and data mining. At SocialCops, these individuals understand the importance of quick execution, and you will find that they come prepared with or quickly build a tool belt full of data utilities that allow them to execute almost instantly.
Technology Stack: Elastic Search, NLP, Machine learning, NoSQL, Python, PostgreSQL
THE CHALLENGES WE FACE EVERY DAY
- Sheer volumes! As a Data Engineer, you will deal with over 100 million data points (at the least!) and build systems that can scale.
- The data we work with comes from the developing world, bringing with it local languages, bad structures, and terrible formats.
- Our data comes from multiple sources at different granularities. This means that you need to devise novel ways of searching through the data and correlating data.
- Your role is to keep refining our platform to be able to transform data at scale. You will work on novel challenges faced by data analysts around the world, ranging from sourcing data and cleaning data to creating indices and running regressions.
- Our users begin at home, with our internal data team being the first users of what we build. Gear up for loads of feedback. After all, the best products solve your own problem!
IN MONTH 1...
- As soon as introductions are over on day 1, you are asked to put on your tool belt, get into your fire retardant suit and start firefighting on some of our open-ended data problems. To make matters worse, you only have until the end of the week to make significant progress. You showcase your hack that Friday during Demo and take a bow as you officially join the ranks as an Engineer at SocialCops.
- You start by working with data scientists on your team and help them scale some of their data wrangling tasks. By the end of the month you have a significant understanding of the wrangling needs and have considerably increased or improved the wrangling tools in your tool belt.
- By the end of the month you have participated in many discussions and wrangled many a data set. Discussions ranged from technology stacks to product features to the food menu at the office. You now know to always back your arguments with sound data at SocialCops.
IN MONTH 3...
- You know everyone at the office, even the quiet introverted types. You are up to speed with everything that is being worked on at the office, either through direct participation or via the demos on Friday.
- You have shipped many new data pipelines and made marked improvements to our data infrastructure
- Over the last 3 months, you've come into your own and have now become the go-to person to solve certain classes of problems at the office.
IN MONTH 6 (AND BEYOND)...
- You hire a couple of engineers to help manage the recent deluge of data. The data scrapers and parsers that were built over the last few months are bringing in large amounts of data and you need help managing this increased output. You have also implemented a good interim solution for one of the open-ended data problems that have been plaguing us for the last 3 months.
- You mentor some of the interns who have joined SocialCops. While showing them the ropes, you learn a whole lot from the experience as well as from the interns themselves. Some of them make it to the Intern Hall of Fame when they leave. But almost all of them leave with their fails on the infamous Fail Wall, right next to some fails by the founders.
- Over the last 6 months, along with your team of Data engineers, you've helped power many decisions that make our world a better place.
- SocialCops is a data intelligence company. Our platform brings the entire decision-making process to one place — from collecting primary data and accessing secondary data to merging internal data and visualizing data via easy-to-use dashboards.
- We were on the Forbes Asia 30 Under 30 2016 and Fortune India 40 Under 40 2016 lists. Nascomm recognized us as one of India's 10 emerging startups to meet PM Modi in 2015.
- We're a diverse team. Americans, Brits, Indians — all in a cosy office in New Delhi. The youngest member in our team is 18 and oldest is 42. Read more on our blog here: 24 or 42 - It doesn't really matter.
- Our Friday Demos allow everyone on the team to contribute to building a better product and company. Come learn from our diverse team of economists, data scientists, engineers, entrepreneurs, and field managers! Read more on our blog here: TGIF: What SocialCops does on Fridays.
- Take complete ownership of your work. Build stuff from the ground up. Hear it from the horse's mouth. Read more: Giving Thanks for my Job: A non-traditional Thanksgiving and Engineer Differently: My Experience as an Engineer at SocialCops.
We're not hiring employees. Solving the world's biggest problems doesn't need employees. Come join us in changing the way the world makes decisions :)