Data Engineer

Data Engineer – Cloud Big Data Role Summary: Responsible for designing, developing, and maintaining scalable cloud-based data pipelines and data platforms across AWS, GCP, Kafka, and Databricks. The role involves building reliable ETL/ELT pipelines, real-time data ingestion, data transformation, monitoring, and performance/cost optimization. In this role you will... Design and develop ETL/ELT data pipelines using AWS Glue, Databricks, and Spark. Build and maintain batch and real-time data pipelines using Apache Kafka / Confluent Kafka. Develop CDC-based data ingestion and event-driven data processing. Work with AWS services including Glue, S3, Lambda, IAM, KMS, Secrets Manager, and related services. Develop and optimize data solutions on GCP, including BigQuery, Cloud Storage, Cloud Functions, and related services. Implement data processing using Databricks, PySpark, Delta Lake, Bronze/Silver/Gold architecture, and DLT. Manage and optimize Elasticsearch/OpenSearch data ingestion, indexing, queries, and cluster performance. Implement data quality, validation, monitoring, logging, and alerting across data pipelines. Troubleshoot production issues involving Kafka, schema compatibility, CDC, Spark, Glue, Databricks, and cloud infrastructure. Optimize pipeline performance and cloud infrastructure costs. Implement secure data solutions using IAM, encryption, network controls, secrets management, and least-privilege access. Collaborate with application, DevOps, QA, analytics, and product teams to deliver reliable data solutions. You've got what it takes if you have… 5+ years of experience in Data Engineering / Big Data Engineering. Strong hands-on experience with AWS + Databricks + Kafka. Experience working in multi-cloud environments (AWS/GCP). Strong understanding of distributed systems, streaming, CDC, schema evolution, and data governance. Experience troubleshooting high-volume production data pipelines. Experience with cloud performance and cost optimization. #LI-Onsite