Summary: Most AnythingLLM problems fall into four buckets: the model ignores your documents, Ollama shows no models, embedding fails with "Fetch failed", or an agent never calls its tools. Each has a specific cause that AnythingLLM's own documentation explains, and most are fixed with one setting or one URL change rather than a reinstall. This guide walks through each symptom in the order you should check it, plus the October 2026 engine change that confuses people upgrading the Desktop app.
Before you start: find your version, logs and storage folder
AnythingLLM comes in three forms that fail in different ways: the free Desktop app (macOS, Windows, Linux), the free self-hosted Docker image, and the paid hosted Cloud. Know which one you are on before changing anything, because the fix for "Ollama not connecting" is completely different on Desktop and in Docker.
- Open the app info panel. In the Desktop app, click the version number in the top-right corner. The panel shows the installed version and the latest available one, the local ports for the API server and the document collector, and two buttons: Open logs and Open storage.
- Update if you are behind. Several behaviours in this guide depend on the version. Document attaching in chat changed in 1.8.5, Agent mode became a workspace option after 1.11.1, and Pro needs Desktop 1.15.0 or later.
- Know where the storage folder lives. On macOS it is
/Users/<user>/Library/Application Support/anythingllm-desktop/storage, on Linux~/.config/anythingllm-desktop/storage/, and on WindowsC:\Users\<user>\AppData\Roaming\anythingllm-desktop\storage. Inside you will findlogs,models,documents,lancedb(the default vector database) andanythingllm.db. - Get verbose errors if the app crashes. Run the executable from a terminal so its logs print live. On Windows the usual path is
C:\Users\<user>\AppData\Local\Programs\anythingllm-desktop\AnythingLLMDesktop.exe. For Docker, open Container Logs in Docker Desktop or print them with the docker logs command.
One easy-to-miss conflict: Desktop and the default Docker setup both use port 3001. If both run at once, the Desktop app can stop accepting text or opening its menu. Stop the container, or remap Docker to another host port (for example -p 3002:3001) and open it at http://localhost:3002.
Problem 1: "The AI ignores my documents"
This is the most common complaint, and AnythingLLM's docs answer it bluntly: the model does not read your whole library. When a document is embedded, it is split into chunks and stored as vectors. Your question is embedded the same way, and only the nearest chunks (usually four to six) are passed to the model. If the right chunk is not retrieved, the model never sees it. That is why "list the three highest-calorie recipes in my collection" fails even though each recipe is technically in the workspace.
Step 1: Loosen the similarity threshold
Hover over the workspace, click the gear icon, and open Vector Database. The Document similarity threshold filters out chunks below a score; the docs say the default minimum is 20% and that this setting is "likely the cause" of most missing-answer reports. Set it to No restriction, save with Update workspace, and ask again. The other options are Low (≥ .25), Medium (≥ .50) and High (≥ .75).
Step 2: Try Accuracy Optimized search
In the same tab, Search Preference can be switched from Default to Accuracy Optimized. It searches more chunks and re-ranks them. AnythingLLM's own testing puts the extra cost at roughly 100 to 500 ms per reply, and the reranking model downloads once on first use. This option is only available with the default LanceDB vector database.
Step 3: Adjust Max Context Snippets carefully
This controls how many chunks reach the model. The docs recommend 4 to 6 for most models. Going higher can help with large-context models, but more snippets also means more noise, and AnythingLLM trims context that would overflow the model.
Step 4: Use the document's own words, or pin it
Retrieval is mathematical (cosine distance), not a reading of intent, so phrase questions the way the document is written. If a file is critical and small enough to fit the context window, open the workspace's document list and click the pushpin icon. Pinning inserts the full text into every chat in that workspace. It is slower and costs more tokens with paid APIs, but the model sees everything. You can only pin a document that has already been embedded.
Attached vs embedded: a newer source of confusion
Since version 1.8.5, dragging a file into the chat box attaches it: the full text goes into that one thread only, not the rest of the workspace. If the attachments exceed the model's context window, AnythingLLM asks whether to Cancel, Continue Anyway or Embed. The docs advise against Continue Anyway because the app silently prunes text to fit, which produces inaccurate answers. Choose Embed if you want the file in every thread, and remember that in multi-user mode an embedded file becomes visible to everyone with access to that workspace.
Finally, check the chat mode under Chat Settings. Query mode answers only from your documents and says so when it finds nothing; Chat mode mixes documents with the model's general knowledge. If Query mode keeps replying that no relevant information was found, that is the similarity threshold again.
Problem 2: Ollama connected, but no models appear
AnythingLLM tries to detect Ollama automatically at http://127.0.0.1:11434, http://host.docker.internal:11434 and http://172.17.0.1:11434. If your Ollama is elsewhere, you have to type the URL into the LLM settings yourself.
- Confirm the Ollama server is running. Open
http://127.0.0.1:11434in a browser; you should see the plain text "Ollama is running". The docs point out thatollama run model-namealone does not start the API server, so useollama serve(or the Ollama desktop app) if the page does not load. - If AnythingLLM runs in Docker, stop using localhost. Inside a container,
localhost,127.0.0.1and0.0.0.0point at the container itself, not your computer. Usehttp://host.docker.internal:11434on Windows and macOS, orhttp://172.17.0.1:11434on Linux. The same rule applies to LM Studio (port 1234), Chroma, PostgreSQL or anything else on the host. - If Ollama is on another machine, use that machine's IP address and make sure its firewall allows the port. AnythingLLM cannot discover it for you.
- AnythingLLM Cloud cannot reach your home Ollama. The docs say it would require exposing your machine to the internet with a tunnel, which they explicitly do not recommend or support.
If Ollama itself is the problem (GPU not used, models failing to load), our Ollama troubleshooting guide covers that side.
The October 2026 engine change
Separate from "your own Ollama", the Desktop app used to ship a bundled copy of Ollama as its built-in engine. According to AnythingLLM's October 2026 blog post, new installs now use llama.cpp managed by AMD's open-source Lemonade server instead. Existing setups keep working until you click Engine upgrade in the top bar, and a future update will move everyone automatically. The catch: models must be downloaded again for the new engine, so do the upgrade when you have time and bandwidth. Connecting to an Ollama you installed yourself is unaffected. The same post says the new engine installs only the GPU libraries your machine can use, around 137 MB on a typical Windows PC without an NVIDIA card versus about 1.8 GB for the old bundled engine.
Problem 3: "Fetch failed" when embedding a file
The built-in embedder model is not bundled with the app; it downloads on first use. The docs list three causes, most likely first:
- Downloads are blocked. Look in your storage folder for
models/Xenova. If it does not exist, a firewall is probably blocking the model download. Allowhuggingface.co,api.huggingface.coandhttps://cdn.anythingllm.com/support/models/, then try again. Company laptops are the usual culprits. - Windows is missing the Visual C++ Redistributable. Install Microsoft's Visual C++ Redistributable v14.x and embed again.
- Your CPU lacks AVX2. The default LanceDB vector database needs it. Virtual machines sometimes hide AVX2 from the guest. The fix is a different machine or a different vector database provider.
On the hosted Cloud, the equivalent failure is a 502 error after uploading something large. The built-in embedder runs on the instance's limited CPU, and a big document (measured in words, not file size) can kill the process. The help page suggests about 10,000 words per file on the smaller tier and 50,000 on the larger one, using the older tier names Starter and Professional. Re-uploading the same file will crash it again. Reboot the instance from the my.mintplexlabs.com dashboard, then split the file or switch to a cloud embedder.
Problem 4: The agent never uses its tools
Agent tool calls are just the model writing JSON that matches a tool's schema. Small, heavily quantized local models are bad at that, so the docs' advice centres on the model rather than AnythingLLM:
- Check whether a tool actually ran. A real tool call shows a "thought" step in the chat. If you get an answer without one, the model probably made the result up.
- Use a larger or less quantized model, or a cloud model just for agent calls while keeping a local model for normal chat.
- Reset and trim. Type
/resetto clear chat history, then ask again, and turn off tools you are not using so the prompt is smaller. - Know which mode you are in. If you see the @ symbol in the prompt box, you are not in Agent mode and need to start with
@agent. If a provider's native tool calling misbehaves, the docs mention thePROVIDER_DISABLE_NATIVE_TOOL_CALLINGenvironment variable to fall back to prompt-based calling.
On Cloud, custom agent skills and MCP servers are not supported at all for security reasons. If you need them, use Desktop or self-host with Docker.
Which AnythingLLM option avoids which problem
| Option | Price (official) | Built-in local LLM | Custom agents / MCP | Typical trouble spot |
|---|---|---|---|---|
| Desktop app | Free, no account needed; optional Pro license removes daily limits on Magic features | Yes (llama.cpp via Lemonade on new installs) | Yes | Firewall blocking the embedder download; port 3001 clash with Docker |
| Self-hosted Docker | Free | Off by default; enable with ENABLE_BUILT_IN_LLM=true (releases after v1.18.0), or connect Ollama, LM Studio, etc. |
Yes | Using localhost instead of host.docker.internal or 172.17.0.1 |
| Cloud Basic | $50/month; private instance, custom subdomain, bring an LLM API key | No | No | 502 errors from large uploads with the built-in embedder |
| Cloud Pro | $99/month; priority resources, 72-hour support SLA | No | No | Same limits, more compute headroom |
Enterprise (on-premise, SSO, RBAC, custom SLA) is quote-only. If you are weighing alternatives, see our Open WebUI playbook for small teams and the LM Studio tutorial, which covers chatting with files in another desktop app.
Editor's note
Good fit: people who want private document chat on their own computer without an account, and small teams willing to self-host. Poor fit: anyone expecting the model to "know" every file in a big library. Retrieval only ever returns a handful of chunks, so questions that need a full scan (totals, rankings across hundreds of files) will disappoint unless you pin small documents.
Gotchas we confirmed in the official docs:
- Embedding a file shares it with every thread in the workspace, and with every user of that workspace in multi-user mode.
- "Continue Anyway" on the context warning silently drops text. Choose Embed instead.
- Upgrading to the new built-in engine means downloading your models again.
- Cloud has no built-in LLM and cannot use your home Ollama.
Quick fixes: no relevant information found → similarity threshold to No restriction. No Ollama models in Docker → host.docker.internal:11434 (Linux: 172.17.0.1:11434). Fetch failed → check for models/Xenova and unblock Hugging Face. Desktop frozen → stop the Docker container using port 3001.
FAQ
Q: Is AnythingLLM free?
A: The Desktop app and the self-hosted Docker version are free. The optional Pro license only removes daily limits on the Magic features (Magic Echo, Magic Beacon, Magic Tab) and the watermark on generated documents; if it lapses, those features fall back to their free daily allowance. The hosted Cloud starts at $50 a month.
Q: Why does Query mode say it found nothing when the answer is in my PDF?
A: The chunk holding the answer was probably filtered out by the similarity threshold, or your wording doesn't match the document. Set the threshold to No restriction, reuse the document's own terms, and try Accuracy Optimized search. For background on how vector search behaves, see our Pinecone vs Qdrant vs Weaviate comparison.
Q: Do I still need Ollama to run local models?
A: No. New Desktop installs run models with the built-in llama.cpp engine and can download GGUF files from Hugging Face directly. Ollama remains a supported provider if you prefer to run it yourself. The Docker image also has a built-in engine on releases after v1.18.0, but it is off until you start the container with -e ENABLE_BUILT_IN_LLM=true; it cannot be switched on from the UI.
Check the version, the URL and the similarity threshold before anything else; that order solves most AnythingLLM problems. The official screenshots from AnythingLLM's documentation are at the end of this post.
Images: official screenshots from the AnythingLLM documentation (docs.anythingllm.com).
Last verified: 2026-11-05
Sources: Attaching vs RAG, RAG in AnythingLLM, Chat modes, Ollama connection debugging, Connecting to localhost (Docker), Built-in LLM for Docker, Fetch failed error on embed, AI agent not using tools, Desktop debugging and logs, Where is my data stored, Cloud limitations, 502 error on hosted, AnythingLLM Pro, AnythingLLM Cloud pricing, Moving from Ollama to Lemonade and llama.cpp.
댓글
댓글 쓰기