Senior SRE Engineer (Observability Focus)

We are a leading trading platform that is ambitiously expanding to the four corners of the globe. Our top-rated products have won prestigious industry awards for their cutting-edge technology and seamless client experience. We deliver only the best, so we are always in search of the best people to join our ever-growing talented team. We're building out our observability practice and need a senior engineer who can own it end to end. This is a hands-on role. You'll design and operate the telemetry stack that gives our engineering teams real visibility into production — across a hybrid AWS and on-premise environment, at scale. Responsibilities:

Own the full observability stack: metrics (VictoriaMetrics), logs (OpenSearch), and traces (OpenTelemetry) — from pipeline design to day-2 operations.

Architect and run VictoriaMetrics cluster topology (vmstorage/vminsert/vmselect), including vmagent scraping, remote write configuration, vmalert rules, and cardinality control.

Operate OpenSearch clusters: index lifecycle management (ISM), hot-warm-cold architecture, shard tuning, and ingest pipelines via Data Prepper.

Build and maintain OTEL Collector pipelines — receivers, processors, exporters — and instrument services across Java, Python, and JS/TS stacks (auto and manual).

Run Kafka as the telemetry transport layer (OTEL Collector → Kafka → backends), including topic design, partition strategy, consumer group lag monitoring, and throughput tuning for high-volume telemetry.

Manage log shipping infrastructure using Fluent Bit, Vector, or Fluentd; define structured logging standards and field normalization across services.

Build Grafana dashboards and alerting that engineers actually use — clear, actionable, with well-structured variables and thresholds.

Work with platform and application teams to improve sampling strategies (head/tail), batching, and context propagation across distributed services.

Contribute to incident response, post-mortems, and reliability improvements driven by observability signals.

Mentor engineers on observability practices, tooling, and structured logging standards.

Requirements:

6+ years in a DevOps, SRE, or platform engineering role, with at least 2 years focused on observability tooling at production scale.

Deep hands-on experience with VictoriaMetrics (or Prometheus) — MetricsQL/PromQL, exporters, service discovery, remote write, downsampling, and retention management.

Solid OpenSearch or Elasticsearch skills: cluster operations, Query DSL, ISM policies, and ingest pipeline design.

Production experience with OpenTelemetry: Collector configuration, OTLP, context propagation, and instrumentation across multiple languages.

Strong Kafka skills — producer/consumer patterns, consumer group management, Kafka Connect, Schema Registry, and JMX-based monitoring. Strimzi experience a plus if you've run Kafka on Kubernetes.

Proficiency with log shippers (Fluent Bit, Vector, Fluentd) and structured log parsing/normalization.

Working knowledge of Kubernetes (operators, Helm), Argo CD/GitOps, and Terraform/Ansible.

Comfortable in a hybrid AWS + on-prem environment; solid understanding of networking as it applies to scraping and shipping pipelines.

Scripting ability in Bash or Python for automation and tooling.

Strong communication skills — you can explain observability tradeoffs clearly to engineers and non-engineers alike.

English proficiency.