• LLM Inference Sizing and Performance Guidance
  When planning to deploy a chatbot or simple Retrieval-Augmentation-Generation (RAG) pipeline, you may have questions about sizing (capacity) and performance based on your existing GPU resources or potential future GPU acquisitions. For instance: What is the maximum number of concurrent requests that can be supported for a specific Large Language Model (LLM) on a specific GPU? What is the maximum sequence length (or prompt size) that a user can send to the chat app without experiencing a noticeably slow response time? What is the estimated response time (latency) for generating output tokens, and how does it vary with different input sizes and LLM sizes? Conversely, if you have specific capacity or latency requ
  in Computers > Hårdvara > AI with concurrency gpu llm memory requirements scaling sizing

requirements from all users