Skip to content

fix(vllm): add tool support detection so agent mode sends tools parameter#12591

Open
rodboev wants to merge 1 commit into
continuedev:mainfrom
rodboev:pr/vllm-tool-support
Open

fix(vllm): add tool support detection so agent mode sends tools parameter#12591
rodboev wants to merge 1 commit into
continuedev:mainfrom
rodboev:pr/vllm-tool-support

Conversation

@rodboev

@rodboev rodboev commented Jun 10, 2026

Copy link
Copy Markdown

Fixes #5508

Summary

Agent mode with the vLLM provider never executes tools: the model replies with plain text or raw tool-call syntax leaks into chat. The request sent to vLLM never includes the tools parameter, so the model has nothing to call, regardless of how well the server supports OpenAI-compatible tool calling.

Root cause

PROVIDER_TOOL_SUPPORT in core/llm/toolSupport.ts has no vllm entry. modelSupportsNativeTools() (core/llm/toolSupport.ts:513) returns false for any provider absent from the map, so gui/src/redux/thunks/streamNormalInput.ts:116 sets useNativeTools = false, routes the request through the system-message code-block framework, and never populates completionOptions.tools. The downstream streaming path (fromChatCompletionChunk in core/llm/openaiTypeConverters.ts, addToolCallDeltaToState in gui/src/util/toolCallState.ts) already reassembles streamed tool_calls deltas correctly; it is simply never reached.

Changes

  • core/llm/toolSupport.ts: add a vllm entry to PROVIDER_TOOL_SUPPORT, delegating to the existing ollama model heuristic (the same delegation pattern lmstudio uses), since vLLM commonly serves the same model families (Qwen3, Llama 3.x, DeepSeek, Hermes)
  • core/llm/toolSupport.test.ts: new describe("vllm") block covering Qwen3, Llama 3.1, DeepSeek (true) and a non-tool model (false)

What this doesn't change

  • The SSE chunk reassembly path: fromChatCompletionChunk and addToolCallDeltaToState are untouched
  • Vllm.ts: it inherits the OpenAI adapter path, which correctly sends tools once populated
  • Other providers' PROVIDER_TOOL_SUPPORT entries
  • The capabilities: [tools] config override remains the escape hatch for unusual vLLM deployments

Checklist

  • I've read the contributing guide
  • The relevant docs, if any, have been updated or created
  • The relevant tests, if any, have been updated or created

Screen recording or screenshot

N/A — no UI change.

Tests

  • cd core && npm test -- llm/toolSupport.test.ts — 78/78 passing (73 existing + 5 new). Covers: vLLM provider tool-support detection for Qwen3 (the model cited in the issue), Llama 3.1, and DeepSeek model names, plus the negative case for llama2.

Summary by cubic

Fixes agent mode with vllm so tool calls work. We now detect tool support and send the tools parameter, enabling proper tool execution.

  • Bug Fixes
    • Added vllm to PROVIDER_TOOL_SUPPORT, reusing the ollama heuristic so compatible models (Qwen3, Llama 3.1, DeepSeek) include tools in requests.
    • Added tests for Qwen3, Llama 3.1, DeepSeek, and a negative case for Llama 2.

Written for commit 7ad76ae. Summary will update on new commits.

Review in cubic

@rodboev rodboev requested a review from a team as a code owner June 10, 2026 19:50
@rodboev rodboev requested review from sestinj and removed request for a team June 10, 2026 19:50
@dosubot dosubot Bot added the size:XS This PR changes 0-9 lines, ignoring generated files. label Jun 10, 2026

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No issues found across 2 files

Re-trigger cubic

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:XS This PR changes 0-9 lines, ignoring generated files.

Projects

Status: Todo

Development

Successfully merging this pull request may close these issues.

Agent Mode: Streamed tool_calls from vLLM ignored - no action performed

1 participant