LLM Provider Compatibility

Documentation Index

Fetch the complete documentation index at: https://docs.fim.ai/llms.txt

Use this file to discover all available pages before exploring further.

​

Provider detection

1. Explicit provider (from DB ModelConfig.provider field) — highest priority. If the provider matches a known domain in the URL, no api_base is returned (LiteLLM routes natively). Otherwise, api_base is set to the relay URL.
2. Domain match against KNOWN_DOMAINS — official API endpoints are recognized by hostname.
3. URL path hint against PATH_PROVIDER_HINTS — common on relay platforms like UniAPI where /claude or /anthropic in the path indicates the upstream protocol.
4. Fallback — openai/ prefix (generic OpenAI-compatible).

​

tool_choice — the four modes

​

Where tool_choice is used

​

ReAct engine — tool_choice=“auto”

​

structured_llm_call — tool_choice=forced

​

The Bedrock prefill trap

{"role": "assistant", "content": "{"}

ValidationException: This model does not support assistant message prefill.
The conversation must end with a user message.

1. The model is resolved with the anthropic/ prefix (via domain match or URL path hint).
2. response_format={"type":"json_object"} is passed (the json_mode code path in structured_llm_call).
3. The actual backend is AWS Bedrock (which rejects prefill).

​

The fix: json_mode_enabled

• DB-configured models: toggle in Admin → Models → Advanced settings. The flag is stored on ModelProviderModel.json_mode_enabled (default TRUE).
• ENV-configured models: set LLM_JSON_MODE_ENABLED=false in your environment.
• Effect: when disabled, abilities["json_mode"] returns False → response_format is never passed → no prefill → Bedrock works. The degradation chain becomes native_fc → plain_text, skipping the doomed json_mode call entirely.
• No quality loss: the model still returns valid JSON because the system prompt instructs it to. The plain_text level uses extract_json() to parse JSON from free-form content, which works reliably with modern models.

​

Thinking models + forced tool_choice

tool_choice 'specified' is incompatible with thinking enabled

​

The fix: tool_choice_enabled

• DB-configured models: toggle in Admin → Models → Advanced → “Native Function Calling”. The flag is stored on ModelProviderModel.tool_choice_enabled (default TRUE).
• ENV-configured models: set LLM_TOOL_CHOICE_ENABLED=false in your environment.
• Effect: when disabled, abilities["tool_choice"] returns False → the degradation chain starts from Level 2 (json_mode) or Level 3 (plain_text), skipping native_fc entirely. This eliminates the ~10s penalty per structured call for incompatible models.
• ReAct agent unaffected: tool_choice_enabled only controls forced tool selection in structured_llm_call. The ReAct engine uses tool_choice="auto" (model freely decides), which works with all models regardless of this setting.

​

Quick reference: what works where

​

Recommended per-model configuration

# Disable native_fc globally (for thinking-model-only deployments)
LLM_TOOL_CHOICE_ENABLED=false

# Disable json_mode globally (for Bedrock relay deployments)
LLM_JSON_MODE_ENABLED=false

​

Reasoning effort and thinking configuration

1. Temperature is forced to 1.0. Bedrock rejects temperature != 1.0 when thinking is enabled. FIM One handles this automatically — no user action needed.
2. GPT-5.x with tools: reasoning_effort is silently dropped when tools are present, because the GPT-5 /v1/chat/completions endpoint rejects the combination. This only affects the ReAct tool loop; structured_llm_call calls that lack a tools parameter are unaffected.

​

Defensive parsing for structured output

1. Single object instead of array. Some models return {"steps": {"id": "1", "task": "..."}} (a single step object) instead of {"steps": [{"id": "1", "task": "..."}]} (an array). The parser detects this by checking for id or task keys and wraps the object in a list.
2. Double-encoded JSON string. When structured output falls through to json_mode (which lacks schema enforcement), some providers return the steps value as a JSON string rather than a native array — e.g., {"steps": "[{\"id\": \"1\", ...}]"}. This string may also contain literal newlines (from the model’s formatting) that break standard json.loads. The parser uses extract_json_value() (which includes _repair_json_strings) to handle:
   • Literal newlines inside JSON string values
   • Invalid escape sequences (common with LaTeX or code content)
   • Other serialization quirks from compatibility layers
3. Missing steps wrapper. The model may return a single step as the top-level object without the steps wrapper key. The parser detects id and task at the root level and wraps accordingly.

​

Prompt caching (cross-provider)

​

Architecture

• PromptSection — a named fragment with either a static content: str or a dynamic content: Callable
• PromptRegistry — a memoized store (static sections render once, dynamic sections re-render per call)
• DYNAMIC_BOUNDARY — a sentinel marker the registry inserts between the last static section and the first dynamic one, so callers can split the rendered prompt at the cache breakpoint

• Static prefix (~95% of the prompt) — identity, core guidelines, tool descriptions
• Dynamic suffix — current datetime, per-request language directive, handoff context

​

Capability detection

​

Cross-provider coverage

​

Observability

"cache": {
  "read_tokens": 1067,
  "creation_tokens": 0
}

​

Multi-turn cache ROI

​

Reasoning replay policy (modelless correctness)

​

Three policies

• anthropic_thinking — Claude family (including anthropic/, bedrock/anthropic, vertex_ai/claude). Thinking blocks MUST be replayed with signature attached; Anthropic rejects subsequent turns if the signature is missing or altered.
• informational_only — models that emit CoT but do NOT expect replay: DeepSeek R1 / R1-Distill, Qwen QwQ, Gemini 2.x thinking, OpenAI o1 / o3 / o4. Their documentation explicitly says “do not send reasoning_content back in message history”. Sending it anyway:
  • Violates the provider contract (may start rejecting in future versions)
  • Silently invalidates their automatic prefix cache — message bytes mutate on every turn, breaking the hash
• unsupported — models without reasoning capability (GPT-4o, GPT-4 Turbo, Gemini 1.5, Mistral, Llama). No CoT to replay; field should never appear. This policy is also the safe default for unknown model ids.

​

Enforcement

​

For users

• If you switch agents between Claude and DeepSeek in the same conversation, history is stored with thinking blocks intact; on the next turn, the outgoing message shape adapts per the current model.
• If you use a proxy / custom BaseLLM subclass, make sure its model id is recognizable (contains one of the fragments) or the default unsupported policy will apply — which is safe but means Claude behind an unusual proxy might lose thinking replay. Add the model-id fragment to _CACHE_CAPABLE_MODEL_FRAGMENTS (in core/prompt/caching.py) and/or the reasoning policy lookup.

​

Troubleshooting