{"meta":{"title":"Scaling and multi-tenancy","intro":"Design your Copilot SDK deployment to serve multiple users, handle concurrent sessions, and scale horizontally across infrastructure. This guide covers session isolation patterns, scaling topologies, and production best practices.","product":"GitHub Copilot","breadcrumbs":[{"href":"/en/copilot","title":"GitHub Copilot"},{"href":"/en/copilot/how-tos","title":"How-tos"},{"href":"/en/copilot/how-tos/copilot-sdk","title":"Copilot SDK"},{"href":"/en/copilot/how-tos/copilot-sdk/setup","title":"Set up Copilot SDK"},{"href":"/en/copilot/how-tos/copilot-sdk/setup/scaling","title":"Scaling"}],"documentType":"article"},"body":"# Scaling and multi-tenancy\n\nDesign your Copilot SDK deployment to serve multiple users, handle concurrent sessions, and scale horizontally across infrastructure. This guide covers session isolation patterns, scaling topologies, and production best practices.\n\n<!-- markdownlint-disable GHD046 GHD005 -->\n\n<!-- Suppressed: GHD046 (outdated release terminology), GHD005 (hardcoded data variable) -->\n\nFor SDK-level options and patterns, see [Multi-tenancy and server deployments](/en/copilot/how-tos/copilot-sdk/setup/multi-tenancy).\n\n**Best for:** Platform developers, SaaS builders, any deployment serving more than a handful of concurrent users.\n\n## Core concepts\n\nBefore choosing a pattern, understand three dimensions of scaling:\n\n![Diagram: Flowchart showing the described process.](/assets/images/help/copilot/copilot-sdk/setup-scaling-diagram-0.png)\n\n## Session isolation patterns\n\n### Pattern 1: isolated CLI per user\n\nEach user gets their own CLI server instance. Strongest isolation—a user's sessions, memory, and processes are completely separated.\n\n![Diagram: Flowchart showing the described process.](/assets/images/help/copilot/copilot-sdk/setup-scaling-diagram-1.png)\n\n**When to use:**\n\n* Multi-tenant SaaS where data isolation is critical\n* Users with different auth credentials\n* Compliance requirements (SOC 2, HIPAA)\n\n```typescript\n// CLI pool manager — one CLI per user\nclass CLIPool {\n    private instances = new Map<string, { client: CopilotClient; port: number }>();\n    private nextPort = 5000;\n\n    async getClientForUser(userId: string, token?: string): Promise<CopilotClient> {\n        if (this.instances.has(userId)) {\n            return this.instances.get(userId)!.client;\n        }\n\n        const port = this.nextPort++;\n\n        // Spawn a dedicated CLI for this user\n        await spawnCLI(port, token);\n\n        const client = new CopilotClient({\n            cliUrl: `localhost:${port}`,\n        });\n\n        this.instances.set(userId, { client, port });\n        return client;\n    }\n\n    async releaseUser(userId: string): Promise<void> {\n        const instance = this.instances.get(userId);\n        if (instance) {\n            await instance.client.stop();\n            this.instances.delete(userId);\n        }\n    }\n}\n```\n\n### Pattern 2: shared CLI with session isolation\n\nMultiple users share one CLI server but have isolated sessions via unique session IDs. Lighter on resources, but weaker isolation.\n\n![Diagram: Flowchart showing the described process.](/assets/images/help/copilot/copilot-sdk/setup-scaling-diagram-2.png)\n\n**When to use:**\n\n* Internal tools with trusted users\n* Resource-constrained environments\n* Lower isolation requirements\n\n```typescript\nconst sharedClient = new CopilotClient({\n    cliUrl: \"localhost:4321\",\n});\n\n// Enforce session isolation through naming conventions\nfunction getSessionId(userId: string, purpose: string): string {\n    return `${userId}-${purpose}-${Date.now()}`;\n}\n\n// Access control: ensure users can only access their own sessions\nasync function resumeSessionWithAuth(\n    sessionId: string,\n    currentUserId: string\n): Promise<Session> {\n    const [sessionUserId] = sessionId.split(\"-\");\n    if (sessionUserId !== currentUserId) {\n        throw new Error(\"Access denied: session belongs to another user\");\n    }\n    return sharedClient.resumeSession(sessionId);\n}\n```\n\n### Pattern 3: shared sessions (collaborative)\n\nMultiple users interact with the same session—like a shared chat room with Copilot.\n\n![Diagram: Flowchart showing the described process.](/assets/images/help/copilot/copilot-sdk/setup-scaling-diagram-3.png)\n\n**When to use:**\n\n* Team collaboration tools\n* Shared code review sessions\n* Pair programming assistants\n\n> ⚠️ **Important:** The SDK doesn't provide built-in session locking. You **must** serialize access to prevent concurrent writes to the same session.\n\n```typescript\nimport Redis from \"ioredis\";\n\nconst redis = new Redis();\n\nasync function withSessionLock<T>(\n    sessionId: string,\n    fn: () => Promise<T>,\n    timeoutSec = 300\n): Promise<T> {\n    const lockKey = `session-lock:${sessionId}`;\n    const lockId = crypto.randomUUID();\n\n    // Acquire lock\n    const acquired = await redis.set(lockKey, lockId, \"NX\", \"EX\", timeoutSec);\n    if (!acquired) {\n        throw new Error(\"Session is in use by another user\");\n    }\n\n    try {\n        return await fn();\n    } finally {\n        // Release lock (only if we still own it)\n        const currentLock = await redis.get(lockKey);\n        if (currentLock === lockId) {\n            await redis.del(lockKey);\n        }\n    }\n}\n\n// Usage: serialize access to shared session\napp.post(\"/team-chat\", authMiddleware, async (req, res) => {\n    const result = await withSessionLock(\"team-project-review\", async () => {\n        const session = await client.resumeSession(\"team-project-review\");\n        return session.sendAndWait({ prompt: req.body.message });\n    });\n\n    res.json({ content: result?.data.content });\n});\n```\n\n## Comparison of isolation patterns\n\n|                      | Isolated CLI Per User | Shared CLI + Session Isolation | Shared Sessions         |\n| -------------------- | --------------------- | ------------------------------ | ----------------------- |\n| **Isolation**        | ✅ Complete            | ⚠️ Logical                     | ❌ Shared                |\n| **Resource usage**   | High (CLI per user)   | Low (one CLI)                  | Low (one CLI + session) |\n| **Complexity**       | Medium                | Low                            | High (locking)          |\n| **Auth flexibility** | ✅ Per-user tokens     | ⚠️ Service token               | ⚠️ Service token        |\n| **Best for**         | Multi-tenant SaaS     | Internal tools                 | Collaboration           |\n\n## Horizontal scaling\n\n### Multiple CLI servers behind a load balancer\n\n![Diagram: Flowchart showing the described process.](/assets/images/help/copilot/copilot-sdk/setup-scaling-diagram-4.png)\n\n**Key requirement:** Session state must be on **shared storage** so any CLI server can resume any session.\n\n```typescript\n// Route sessions to CLI servers\nclass CLILoadBalancer {\n    private servers: string[];\n    private currentIndex = 0;\n\n    constructor(servers: string[]) {\n        this.servers = servers;\n    }\n\n    // Round-robin selection\n    getNextServer(): string {\n        const server = this.servers[this.currentIndex];\n        this.currentIndex = (this.currentIndex + 1) % this.servers.length;\n        return server;\n    }\n\n    // Sticky sessions: same user always hits same server\n    getServerForUser(userId: string): string {\n        const hash = this.hashCode(userId);\n        return this.servers[hash % this.servers.length];\n    }\n\n    private hashCode(str: string): number {\n        let hash = 0;\n        for (let i = 0; i < str.length; i++) {\n            hash = (hash << 5) - hash + str.charCodeAt(i);\n            hash |= 0;\n        }\n        return Math.abs(hash);\n    }\n}\n\nconst lb = new CLILoadBalancer([\n    \"cli-1:4321\",\n    \"cli-2:4321\",\n    \"cli-3:4321\",\n]);\n\napp.post(\"/chat\", async (req, res) => {\n    const server = lb.getServerForUser(req.user.id);\n    const client = new CopilotClient({ cliUrl: server });\n\n    const session = await client.createSession({\n        sessionId: `user-${req.user.id}-chat`,\n        model: \"gpt-5.4\",\n    });\n\n    const response = await session.sendAndWait({ prompt: req.body.message });\n    res.json({ content: response?.data.content });\n});\n```\n\n### Sticky sessions vs. shared storage\n\n![Diagram: Flowchart showing the described process.](/assets/images/help/copilot/copilot-sdk/setup-scaling-diagram-5.png)\n\n**Sticky sessions** are simpler—pin users to specific CLI servers. No shared storage needed, but load distribution is uneven.\n\n**Shared storage** enables any CLI to handle any session. Better load distribution, but requires networked storage for `~/.copilot/session-state/`.\n\n## Vertical scaling\n\n### Tuning a single CLI server\n\nA single CLI server can handle many concurrent sessions. Key considerations:\n\n![Diagram: Flowchart showing the described process.](/assets/images/help/copilot/copilot-sdk/setup-scaling-diagram-6.png)\n\n**Session lifecycle management** is key to vertical scaling:\n\n```typescript\n// Limit concurrent active sessions\nclass SessionManager {\n    private activeSessions = new Map<string, Session>();\n    private maxConcurrent: number;\n\n    constructor(maxConcurrent = 50) {\n        this.maxConcurrent = maxConcurrent;\n    }\n\n    async getSession(sessionId: string): Promise<Session> {\n        // Return existing active session\n        if (this.activeSessions.has(sessionId)) {\n            return this.activeSessions.get(sessionId)!;\n        }\n\n        // Enforce concurrency limit\n        if (this.activeSessions.size >= this.maxConcurrent) {\n            await this.evictOldestSession();\n        }\n\n        // Create or resume\n        const session = await client.createSession({\n            sessionId,\n            model: \"gpt-5.4\",\n        });\n\n        this.activeSessions.set(sessionId, session);\n        return session;\n    }\n\n    private async evictOldestSession(): Promise<void> {\n        const [oldestId] = this.activeSessions.keys();\n        const session = this.activeSessions.get(oldestId)!;\n        // Session state is persisted automatically — safe to disconnect\n        await session.disconnect();\n        this.activeSessions.delete(oldestId);\n    }\n}\n```\n\n## Ephemeral vs. persistent sessions\n\n![Diagram: Flowchart showing the described process.](/assets/images/help/copilot/copilot-sdk/setup-scaling-diagram-7.png)\n\n### Ephemeral sessions\n\nFor stateless API endpoints where each request is independent:\n\n```typescript\napp.post(\"/api/analyze\", async (req, res) => {\n    const session = await client.createSession({\n        model: \"gpt-5.4\",\n    });\n\n    try {\n        const response = await session.sendAndWait({\n            prompt: req.body.prompt,\n        });\n        res.json({ result: response?.data.content });\n    } finally {\n        await session.disconnect();  // Clean up immediately\n    }\n});\n```\n\n### Persistent sessions\n\nFor conversational interfaces or long-running workflows:\n\n```typescript\n// Create a resumable session\napp.post(\"/api/chat/start\", async (req, res) => {\n    const sessionId = `user-${req.user.id}-${Date.now()}`;\n\n    const session = await client.createSession({\n        sessionId,\n        model: \"gpt-5.4\",\n        infiniteSessions: {\n            enabled: true,\n            backgroundCompactionThreshold: 0.80,\n        },\n    });\n\n    res.json({ sessionId });\n});\n\n// Continue the conversation\napp.post(\"/api/chat/message\", async (req, res) => {\n    const session = await client.resumeSession(req.body.sessionId);\n    const response = await session.sendAndWait({ prompt: req.body.message });\n\n    res.json({ content: response?.data.content });\n});\n\n// Clean up when done\napp.post(\"/api/chat/end\", async (req, res) => {\n    await client.deleteSession(req.body.sessionId);\n    res.json({ success: true });\n});\n```\n\n## Container deployments\n\n### Kubernetes with persistent storage\n\n```yaml\napiVersion: apps/v1\nkind: Deployment\nmetadata:\n  name: copilot-cli\nspec:\n  replicas: 3\n  selector:\n    matchLabels:\n      app: copilot-cli\n  template:\n    metadata:\n      labels:\n        app: copilot-cli\n    spec:\n      containers:\n        - name: copilot-cli\n          image: your-registry/copilot-cli:latest  # See backend-services.md for how to build and push this image\n          args: [\"--headless\", \"--host\", \"0.0.0.0\", \"--port\", \"4321\"]\n          env:\n            - name: COPILOT_GITHUB_TOKEN\n              valueFrom:\n                secretKeyRef:\n                  name: copilot-secrets\n                  key: github-token\n          ports:\n            - containerPort: 4321\n          volumeMounts:\n            - name: session-state\n              mountPath: /root/.copilot/session-state\n      volumes:\n        - name: session-state\n          persistentVolumeClaim:\n            claimName: copilot-sessions-pvc\n---\napiVersion: v1\nkind: Service\nmetadata:\n  name: copilot-cli\nspec:\n  selector:\n    app: copilot-cli\n  ports:\n    - port: 4321\n      targetPort: 4321\n```\n\n![Diagram: Flowchart showing the described process.](/assets/images/help/copilot/copilot-sdk/setup-scaling-diagram-8.png)\n\n### Azure Container Instances\n\n```yaml\ncontainers:\n  - name: copilot-cli\n    image: your-registry/copilot-cli:latest  # See backend-services.md for how to build and push this image\n    command: [\"copilot\", \"--headless\", \"--host\", \"0.0.0.0\", \"--port\", \"4321\"]\n    volumeMounts:\n      - name: session-storage\n        mountPath: /root/.copilot/session-state\n\nvolumes:\n  - name: session-storage\n    azureFile:\n      shareName: copilot-sessions\n      storageAccountName: myaccount\n```\n\n## Production checklist\n\n![Diagram: Flowchart showing the described process.](/assets/images/help/copilot/copilot-sdk/setup-scaling-diagram-9.png)\n\n| Concern             | Recommendation                                                |\n| ------------------- | ------------------------------------------------------------- |\n| **Session cleanup** | Run periodic cleanup to delete sessions older than your TTL   |\n| **Health checks**   | Ping the CLI server periodically; restart if unresponsive     |\n| **Storage**         | Mount persistent volumes for `~/.copilot/session-state/`      |\n| **Secrets**         | Use your platform's secret manager (Vault, K8s Secrets, etc.) |\n| **Monitoring**      | Track active session count, response latency, error rates     |\n| **Locking**         | Use Redis or similar for shared session access                |\n| **Shutdown**        | Drain active sessions before stopping CLI servers             |\n\n## Limitations\n\n| Limitation                      | Details                                                   |\n| ------------------------------- | --------------------------------------------------------- |\n| **No built-in session locking** | Implement application-level locking for concurrent access |\n| **No built-in load balancing**  | Use external LB or service mesh                           |\n| **Session state is file-based** | Requires shared filesystem for multi-server setups        |\n| **30-minute idle timeout**      | Sessions without activity are auto-cleaned by the CLI     |\n| **CLI is single-process**       | Scale by adding more CLI server instances, not threads    |\n\n## Next steps\n\n* **[Session resume and persistence](/en/copilot/how-tos/copilot-sdk/features/session-persistence)**: Deep dive on resumable sessions\n* **[Backend services setup](/en/copilot/how-tos/copilot-sdk/setup/backend-services)**: Core server-side setup\n* **[GitHub OAuth setup](/en/copilot/how-tos/copilot-sdk/setup/github-oauth)**: Multi-user authentication\n* **[BYOK (bring your own key)](/en/copilot/how-tos/copilot-sdk/auth/byok)**: Use your own model provider"}