fix(vllm): add tool support detection so agent mode sends tools parameter#12591
Open
rodboev wants to merge 1 commit into
Open
fix(vllm): add tool support detection so agent mode sends tools parameter#12591rodboev wants to merge 1 commit into
rodboev wants to merge 1 commit into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #5508
Summary
Agent mode with the vLLM provider never executes tools: the model replies with plain text or raw tool-call syntax leaks into chat. The request sent to vLLM never includes the
toolsparameter, so the model has nothing to call, regardless of how well the server supports OpenAI-compatible tool calling.Root cause
PROVIDER_TOOL_SUPPORTincore/llm/toolSupport.tshas novllmentry.modelSupportsNativeTools()(core/llm/toolSupport.ts:513) returnsfalsefor any provider absent from the map, sogui/src/redux/thunks/streamNormalInput.ts:116setsuseNativeTools = false, routes the request through the system-message code-block framework, and never populatescompletionOptions.tools. The downstream streaming path (fromChatCompletionChunkin core/llm/openaiTypeConverters.ts,addToolCallDeltaToStatein gui/src/util/toolCallState.ts) already reassembles streamedtool_callsdeltas correctly; it is simply never reached.Changes
core/llm/toolSupport.ts: add avllmentry toPROVIDER_TOOL_SUPPORT, delegating to the existingollamamodel heuristic (the same delegation patternlmstudiouses), since vLLM commonly serves the same model families (Qwen3, Llama 3.x, DeepSeek, Hermes)core/llm/toolSupport.test.ts: newdescribe("vllm")block covering Qwen3, Llama 3.1, DeepSeek (true) and a non-tool model (false)What this doesn't change
fromChatCompletionChunkandaddToolCallDeltaToStateare untouchedVllm.ts: it inherits the OpenAI adapter path, which correctly sendstoolsonce populatedPROVIDER_TOOL_SUPPORTentriescapabilities: [tools]config override remains the escape hatch for unusual vLLM deploymentsChecklist
Screen recording or screenshot
N/A — no UI change.
Tests
cd core && npm test -- llm/toolSupport.test.ts— 78/78 passing (73 existing + 5 new). Covers: vLLM provider tool-support detection for Qwen3 (the model cited in the issue), Llama 3.1, and DeepSeek model names, plus the negative case for llama2.Summary by cubic
Fixes agent mode with
vllmso tool calls work. We now detect tool support and send thetoolsparameter, enabling proper tool execution.vllmtoPROVIDER_TOOL_SUPPORT, reusing theollamaheuristic so compatible models (Qwen3, Llama 3.1, DeepSeek) includetoolsin requests.Written for commit 7ad76ae. Summary will update on new commits.