Location
London

Hours
Full Time

Salary
Competitive, commensurate with experience

About the Role
We have an opportunity to impact your career and provide an adventure where you can push the limits of what's possible. As a Software Engineer III at JPMorganChase within the Firmwide LLM Serving Platform team, you will be an integral part of an agile team that designs, builds, and operates services that make large language models usable at scale. This is an infrastructure-meets-ML role: you don't need to be an ML researcher, but you should be excited to learn how model architectures and inference constraints translate into real production systems.

You will contribute to a living platform where performance is optimized by pushing down latency, increasing throughput, maximizing GPU utilization, and eliminating waste across the request lifecycle.

Key responsibilities include:
- Building core backend services for LLM inference, including request routing, batching, scheduling, streaming responses, and quota/limits
- Implementing and maintaining APIs and SDKs used by product and application teams across the firm
- Profiling and optimizing performance end-to-end across CPU, memory, network, serialization, concurrency, GPU utilization, and caching
- Improving reliability and operability through health checks, graceful degradation, autoscaling behaviors, incident follow-ups, and runbooks
- Contributing to system design by breaking down ambiguous problems, proposing approaches, and making pragmatic tradeoffs
- Adding observability with metrics, tracing, logging, dashboards, and actionable alerts tied to SLOs
- Supporting safe deployments through CI/CD improvements, canarying, feature flags, backward compatibility, and rollback plans
- Learning LLM serving fundamentals such as tokenization costs, KV cache, quantization, context length tradeoffs, and throughput vs. latency
- Leveraging enterprise-authorized AI coding assist tools to improve code quality, delivery speed, and productivity while validating outputs through peer review, automated testing, and secure coding standards
- Applying knowledge of tools within the Software Development Life Cycle toolchain, including AI-assisted development and automation capabilities
- Contributing to a team culture of diversity, opportunity, inclusion, and respect

Experience
- Formal training or certification in software engineering concepts with applied experience
- Hands-on experience using enterprise-authorized AI-assisted software development tools within the work environment, with demonstrated ability to critically evaluate, validate, and refine AI-generated outputs for correctness, performance, and security
- Interest in distributed systems and system design, even if you haven't built large systems yet
- Curiosity about LLMs and AI model architecture with a willingness to learn quickly
- Experience with performance profiling tools such as pprof, flamegraphs, or distributed tracing systems is preferred
- Familiarity with containers and orchestration (Docker, Kubernetes) and service-to-service networking is preferred
- Understanding of inference concepts including batching, streaming tokens, GPU memory constraints, and KV cache is preferred
- Experience with high-throughput APIs (gRPC/HTTP), eventing/queues, or caching layers such as Redis is preferred
- Exposure to reliability practices such as SLOs/SLIs, on-call rotations, and incident reviews is preferred

About you
- A measurement-driven mindset: you enjoy profiling, benchmarking, and proving improvements with data
- Understanding of responsible AI use in engineering workflows, including data sensitivity considerations, secure handling of inputs/outputs, and adherence to resiliency and security expectations
- Ability to guide peers on safe and effective usage of AI-assisted tools within team practices
- Strong collaboration skills and commitment to fostering a diverse and inclusive team culture

Qualifications
- Bachelor's Degree in Computer Science or equivalent
- Solid programming fundamentals including data structures, concurrency basics, debugging, and testing
- Comfort working in one or more of Go, Python, or TypeScript, with the ability to ramp up quickly on the others