AI/ML Engineer

Job Description

Role Overview

You will design and implement server-side systems that power all 14 platform modules, from real-time incident ingestion and AI model inference endpoints to FinOps aggregation and SLA management APIs. Paired with the Engineering Lead and AI/ML Engineer, you will build the data and API backbone that supports every frontend view. Two positions are available; engineers will be aligned to different capability layers during parallel build sprints.

Key Responsibilities

• API design and implementation: Design and build RESTful APIs for every platform module, authoring OpenAPI 3.0 specifications in collaboration with the Engineering Lead.
• Multi-tenant data isolation: Implement architecture ensuring strict client data separation and safe multi-tenant operations.
• Real-time ingestion pipelines: Build high-throughput event ingestion capable of processing 2.4M+ events/day.
• Observability integrations: Integrate with Prometheus and Grafana for Digital Twin and telemetry data feeds.
• Incident Control backend: Build services for AI root-cause enrichment, resolution workflows, and escalation routing.
• Predictive Models API: Implement model registry, accuracy/metrics feeds, and inference request handling.
• FinOps aggregation: Develop cloud spend normalisation, baseline comparison, and trend calculation services.
• SLA tracking services: Implement attainment calculations, breach detection, and at-risk client alerting.
• Authentication & access control: Implement authentication, RBAC, and SSO/SAML integrations for Tenants and Integrations modules.
• Testing and quality: Write comprehensive unit, integration, and contract tests; participate in performance and load testing.
• Collaboration and reviews: Participate in architecture reviews, API contract design sessions, and code reviews with cross-functional teams.

Performance Expectations

• Latency targets: Primary dashboard data endpoints should deliver sub-100 ms response times.
• Availability: Maintain 99.9% API availability across platform services.
• Real-time support: Ensure data endpoints support WebSocket or Server-Sent Events for live dashboard updates.
• Documentation: Produce clear OpenAPI documentation for every endpoint before frontend integration begins.
• Security-first: Follow strict security practices—no sensitive data exposure, validate all inputs, and apply least-privilege access.
• Observability-by-default: Ensure structured logging, distributed tracing, health checks, and metric emission are implemented for all services.

Technical Skills (Required)

• Backend languages: Strong proficiency in Node.js (TypeScript) and/or Python for microservice development.
• API design: RESTful API design, OpenAPI 3.0 authorship, versioning, and consumer-focused contracts.
• Databases: Deep knowledge of PostgreSQL (schema design, query optimisation, indexing, migrations).
• Time-series: Experience with InfluxDB, TimescaleDB, or Prometheus remote-write patterns.
• Event streaming: Experience with Kafka, RabbitMQ, or AWS SQS/SNS for high-throughput pipelines.
• Container orchestration: Kubernetes (manifests, Helm charts, pod lifecycle understanding).
• Authentication: JWT, OAuth 2.0, OIDC, SAML, RBAC patterns and implementation.
• Caching: Redis for hot paths and session management.
• Testing & load: Jest or Vitest for unit tests, Pact for contract testing, k6 or Artillery for load testing.
• Observability: OpenTelemetry SDK integration, structured logs, and metrics.

Functional & Soft Skills

• API-first mindset: Prioritizes contract design, versioning strategy, and consumer experience.
• Technical communication: Clear documentation and ability to align with frontend and ML teams.
• Debugging: Systematic root-cause analysis and strong troubleshooting skills.
• Collaboration: Works closely with cross-functional teams during design and delivery.
• Reliability: Delivers predictably and escalates scope or complexity risks proactively.

Hiring Signals (Nice-to-have)

• Experience building multi-tenant SaaS platforms.
• Prior work with ML model serving infrastructure (e.g., Triton, TorchServe, KFServing).
• Familiarity with cloud cost modelling or FinOps tooling.
• Experience with policy-driven RBAC and tenant-level data encryption strategies.

Operational & Development Practices

• Contract-first API development: OpenAPI definitions written before implementation and used for contract testing.
• CI/CD: Automated pipelines for build, test, and deploy; staging and canary releases for major changes.
• Observability standards: Traces propagated across services, structured JSON logs, and actionable SLO/SLA dashboards.
• Security practices: Automated secret management, input validation, encryption in transit and at rest, and periodic security audits.

Interview Topics

• Design a multi-tenant PostgreSQL schema that supports strict separation and efficient querying.
• Architect a pipeline to ingest 2.4M events/day with low-latency enrichment and downstream consumers.
• Define an OpenAPI contract for a Predictive Models inference endpoint with versioning and canary rollout strategy.
• Debugging and performance: How you would diagnose a sub-100 ms endpoint suddenly increasing to 300 ms under load.
• Implementing RBAC and SSO: Walk through SAML integration and tenant-specific role mapping.

Next Steps

• If this aligns with your experience and interests, please prepare examples of:
• A public OpenAPI spec or API documentation you authored.
• A high-throughput ingestion or event-streaming architecture you contributed to.
• Test or observability configurations showing how you achieved high availability and low latency.