Short answer: Pick an LLM development company that tells you in writing which model it will use and why, where your data goes, how it will test answers, what the app costs to run each month, and how you can switch later. Ask for a small paid pilot on your own documents first. Walk away from vendors who promise “no hallucinations” or will not hand over the code and prompts.
We build custom AI apps and run our own product on LLM APIs, so we see this choice from both sides. This guide is for founders and operations or IT heads comparing vendors for an assistant, document Q&A or a support bot. On 2 November 2026 we read the data and pricing pages of OpenAI and Anthropic and the NIST AI Risk Management Framework page; the facts below come from them.
What does an LLM development company actually build?
The phrase covers four different jobs. Ask each vendor which one they propose.
| Approach | What it is | Typical use |
|---|---|---|
| Prompts over an API | An existing model (from OpenAI, Anthropic or others) with instructions you control | Drafting emails, summaries, sorting tickets |
| RAG (retrieval) | The app searches your documents and passes the best matches to the model with each question | Internal assistant, policy Q&A, support bot |
| Fine-tuning | Training an existing model further on your examples | A fixed output format or a narrow task |
| Agents | The model can call tools: look up an order, create a ticket, update a record | Workflows that need actions as well as answers |
We compare these in depth, including self-hosting, in custom LLM vs ChatGPT API. One change matters for proposals: OpenAI’s model optimization guide says “OpenAI is winding down the fine-tuning platform” and that it “is no longer accessible to new users”. If a plan depends on fine-tuning an OpenAI model for a new account, ask how. For assistants that answer from company documents, we think RAG is the right start. For actions, see our AI agent development page.
Which questions should you ask an LLM development company?
The checklist we would use if hiring a vendor ourselves. Send it before the first call and compare written answers.
| Ask this | A good answer sounds like |
|---|---|
| Which model will you use, and why? | A named model, a reason tied to your task, and a plan to test a cheaper one too |
| Where does our data go? | Every service that sees it: model provider, document store, logs, hosting, region |
| Is our data used to train anyone’s model? | A link to the provider’s own page, plus what the vendor does with its own logs |
| Do we need zero data retention? | A clear yes or no for your data, and who applies to the provider |
| How will you measure quality? | A test set of real questions with agreed answers, scored before every release |
| What if the answer is not in our documents? | It says it does not know, points to a person, and shows sources for every answer |
| Who can see what? | The same access rights as your files, so staff never see HR or finance documents they could not open before |
| Who owns the code, prompts and test sets? | You do, in the contract, in a repository you control |
| What will it cost to run each month? | A sheet with your expected questions, tokens per question and the provider’s list prices, like ours below |
| How do we switch models or leave you? | Model calls in one place in the code, a documented setup, and a handover plan |
| Can we see your security reports? | If they claim SOC 2 or similar, the report itself and its date, not a badge on a website |
Where does your data go, and is it used for training?
It depends on the provider and product, not the vendor’s promise. Read the provider’s own page and keep a copy.
OpenAI. Its data controls page says: “As of March 1, 2023, data sent to the OpenAI API is not used to train or improve OpenAI models (unless you explicitly opt in to share data with us).” Abuse monitoring logs are kept “for up to 30 days” by default, and Zero Data Retention and Modified Abuse Monitoring “are subject to prior approval by OpenAI and acceptance of additional requirements”.
Anthropic. Its privacy center says the following about commercial products, which include the Anthropic API:
The exception: if you “explicitly report feedback or bugs” (such as thumbs up or down) or otherwise allow it, Anthropic may train on those chats. So ask your vendor: which product and plan will the app use, will feedback buttons send anything to the provider, and what do the vendor’s own logs keep, and for how long?
How much will an LLM app cost to run each month?
Providers charge per token (roughly a piece of a word), with separate prices for text in and text out. A RAG assistant sends a lot in (instructions, question, document sections) and gets a short answer back. Here is a worked example with list prices from the OpenAI and Anthropic pricing pages, read on 2 November 2026.
Our assumptions: 4,000 tokens in and 400 out per question. A is an internal assistant with 3,000 questions a month (12 million tokens in, 1.2 million out). B is a support bot with 20,000 (80 million in, 8 million out). No caching or batch discounts.
| Model (price per million tokens in / out) | A: 3,000 questions | B: 20,000 questions |
|---|---|---|
| OpenAI gpt-6-luna ($0.10 / $0.50) | $1.80 | $12 |
| Anthropic Claude Haiku 5.5 ($0.10 / $0.50, prompts up to 100,000 tokens) | $1.80 | $12 |
| OpenAI gpt-6.1-terra ($1 / $5) | $18 | $120 |
| Anthropic Claude Sonnet 5.5 ($2 / $10) | $36 | $240 |
| OpenAI gpt-6-astra ($10 / $50) | $180 | $1,200 |
| Anthropic Claude Fable 5.1 ($10 / $50) | $180 | $1,200 |
Our arithmetic, not a provider quote. Example for gpt-6.1-terra in scenario A: 12 million tokens in at $1 per million is $12, plus 1.2 million out at $5 per million is $6, so $18.
Three things stand out. The cheapest and dearest models differ about 100 times for the same traffic, so “which model and why” is a money question too. Token counts differ by provider: Anthropic’s pricing page says the tokenizer of Claude 4.7 and later models “produces approximately 30% more tokens for the same text”, so a fair estimate counts tokens on your own documents. And the model bill is only part of it: hosting, the document store, monitoring and fixing wrong answers come on top, so ask for those lines separately.
How should a vendor prove the answers are right?
OpenAI’s own guide says why this matters: “LLM output is non-deterministic, and model behavior changes between model snapshots and families.” A serious vendor tests on a schedule. What we ask for in a pilot:
- A test set from your real work. Questions people really asked, each with a correct answer agreed by your side.
- Clear scores. Correct, partly correct, wrong, or “I don’t know”. Track each number, not one average.
- Sources on every answer, so your team can check it in seconds.
- Questions it must refuse: answers not in the documents, and documents a user is not allowed to see.
- A re-run before every change of model, prompt or documents.
No vendor can promise zero wrong answers. They can show how often it happens on your questions, and what the assistant does when unsure.
What are the red flags when hiring an LLM developer?
Good signs
- They ask for your documents and real questions before quoting.
- A small pilot with a written pass mark.
- They name the model provider and link its data terms.
- A monthly running cost with workings.
- Code, prompts and test set live in your account.
- They say when a plain search box would do the job better.
Red flags
✗ “Our AI never hallucinates.”
✗ Training a model from scratch for a document assistant.
✗ Fine-tuning plans without checking the provider still offers it.
✗ A demo only on questions they chose.
✗ Your data runs through their account with no written terms.
✗ A build price but no running cost.
✗ Security badges with no report behind them.
Can the NIST AI Risk Management Framework help you choose?
Yes, as a source of questions. NIST says the AI RMF “is intended for voluntary use”; it was released on 26 January 2023 and is being revised as part of the White House AI Action Plan. On 26 July 2024 NIST released the Generative AI Profile (NIST-AI-600-1), which “can help organizations identify unique risks posed by generative AI”.
The framework core has four functions: govern, map, measure and manage. As vendor questions: who on our side owns this system (govern), what could go wrong for our users and data (map), how will you measure it (measure), and what happens when a test fails or a user reports a bad answer (manage).
How we work on LLM projects
We build custom AI apps on models from providers such as OpenAI and Anthropic, starting with a scoping call and a pilot on the client’s own documents. We also run our own product on LLM APIs: wptaskify, an AI agent and MCP platform for websites and stores, is ours. In October 2026 we moved its text features from one model provider to another, which is why switching is on our checklist. See our custom AI and LLM development page, and for a support bot, read AI chatbot for a website.
Send us your use case and sample questions and we will answer the checklist for your project. From the US or India, book a free 30-minute call, or call our Fatehabad office on +91 70151 78387. In Canada, see our Toronto page or call +1 437-563-8080. In Australia, see our Australia page or message us on WhatsApp on 0494 866 992.
Common questions
What does an LLM development company do?
It builds software around a large language model: an internal assistant, a tool that answers from your documents, a support bot or an agent that takes actions. Most of the work is around the model: your data, search, access control, testing and running costs.
Will OpenAI or Anthropic train on our company data?
Not by default on their business products. OpenAI says data sent to its API since 1 March 2023 is not used for training unless you opt in. Anthropic says it will not use inputs or outputs from commercial products, such as its API, to train its models by default; feedback you send is an exception.
How much does it cost to run an LLM app each month?
In our worked example, 3,000 questions a month cost about $1.80 on the cheapest models and about $180 on the most expensive, at list prices read on 2 November 2026. Hosting, monitoring and upkeep come on top.
Should we fine-tune a model or use RAG?
For most business assistants, RAG, because documents are easier to update than a trained model. OpenAI also says its fine-tuning platform is winding down and is no longer accessible to new users.
What is zero data retention?
A setting where the provider does not keep your prompts and answers in its abuse monitoring logs. OpenAI says it needs prior approval by OpenAI, so ask your vendor whether you need it and who applies.
Sources read 2 November 2026: OpenAI, Data controls in the OpenAI platform; OpenAI, Model optimization guide; OpenAI API pricing; Anthropic Privacy Center, Is my data used for model training?; Anthropic pricing; NIST AI Risk Management Framework; NIST AIRC, AI RMF Core. Prices are US dollar list prices on that date and change often. The cost table is our own arithmetic. General guidance, not legal advice.