Buyer guides
How to Evaluate AI Software for Manufacturing: A Buyer’s Checklist
Twenty questions to ask vendors, a scoring sheet, and the red flags we would watch for if we were buying.
Trikaan Research & Engineering4 min read
KEY TAKEAWAYS
- Judge AI on your own data and your own questions, not on a polished demo.
- Ask where answers come from, who can see what, and who approves actions.
- Integration and adoption usually decide success more than model quality.
- Insist on a measured pilot with agreed success criteria before scaling.
Almost every manufacturing software vendor now claims AI. Some have built genuinely useful capabilities; others have added a chat window to an existing product. For a buyer, the challenge is telling the difference before signing a multi-year contract.
Here is the checklist we would use if we were on the buying side.
Start with the problem, not the AI
Write down the two or three business problems you want to solve and how you will measure success. If a vendor cannot connect their AI features to those problems in concrete terms, the features are unlikely to matter.
Questions about data and accuracy
- Where does each answer come from? Can users see the source records?
- How are numeric answers produced — from queries or from generated text?
- How do you test accuracy on our data before go-live?
- What happens when the AI is unsure?
- How does the system improve from corrections?
Questions about control and governance
- Which actions can the AI take on its own, and which need approval?
- Does the AI respect user roles and plant-level permissions?
- Are prompts, outputs and approvals logged?
- Can AI be switched off per module or role?
- Is our data used to train models for anyone else?
Questions about integration and fit
- How do you integrate with our ERP and which system owns which data?
- Can you connect to our machines safely, and is that read-only?
- Does it work offline where our teams need it?
- Which parts are configuration and which are custom development?
- What does a similar customer’s rollout look like in practice?
Questions about security, hosting and exit
- Where is data hosted, and can we choose the region or our own cloud?
- What security certifications or independent tests can you show?
- Which AI model providers do you use, and on what terms?
- How do we export all our data if we leave?
- What does pricing look like at three times our current scale?
A simple scoring sheet
Adjust weights to your situation, but agree them before demos, not after.
| Area | Weight | What good looks like |
|---|---|---|
| Business fit | 25% | Clear link to your top problems, with references |
| Accuracy & transparency | 20% | Tested on your data, source-linked answers |
| Integration | 15% | Clear ownership map, proven connectors |
| Governance & security | 15% | Approvals, permissions, logs, documented controls |
| Adoption | 15% | Designed for floor and field users, training plan |
| Commercials & exit | 10% | Transparent pricing, data portability |
Red flags
- Demos only on the vendor’s sample data.
- Vague answers about where numbers come from.
- AI that can change records without approval.
- No clear answer on data use for model training.
- Pricing that depends on unpredictable usage with no cap.
Make vendors demo on your data
A scripted demo on the vendor’s own sample data tells you very little. Prepare three or four scenarios from your business — a real drawing, a month of anonymised production data, a typical management question — and ask every shortlisted vendor to run them. Give them the same material and the same time. Watch where they need to edit things manually, how they handle ambiguity, and whether answers can be traced to sources.
This is also the fastest way to see how much of a promised capability is product and how much is a services project in disguise.
Total cost of ownership, not licence price
| Cost element | Questions to ask |
|---|---|
| Subscription | How does it scale with plants, users and usage? |
| Implementation | Fixed or time-based? What is excluded? |
| Integration | Which connectors exist; what is custom? |
| Hardware | Gateways, sensors, tablets — who supplies them? |
| AI usage | Is model usage included or metered? Is there a cap? |
| Support | What response times, at what tier? |
| Internal time | How much of your team’s time will it need? |
References that actually tell you something
When you speak to reference customers, ask specific questions: what took longer than planned, what they would do differently, how responsive the vendor was when something broke, and whether the promised results were measured against a baseline. A vendor confident in their delivery will connect you with customers who can answer honestly.
Running the pilot
Pick a contained use case — reporting, drawing-to-BOM, or maintenance alerts on a bottleneck line. Measure the baseline first. Run for long enough to cover normal variation, involve real users, and review results against the criteria you agreed. Then decide on scale with evidence.
Frequently asked questions
What should be in a manufacturing AI RFP?
Business problems and success measures, data and integration landscape, security and hosting requirements, governance expectations for AI, pilot terms and exit provisions.
How long should an AI pilot run?
Long enough to cover normal variation — often eight to fourteen weeks for operational use cases.
Should AI be allowed to act automatically?
Only within narrow, pre-approved limits. Anything that changes recipes, plans, orders or payments should need human approval.
How do we compare vendors fairly?
Use the same scripted scenarios on your own data for every vendor and score against weights agreed in advance.
- #AI software for manufacturing
- #evaluate AI vendors
- #manufacturing AI RFP
- #AI buyer checklist
- #industrial AI
Trikaan Research & Engineering
Trikaan
Written by the researchers and engineers who build Anvaya, ASTR and Seed Track, with input from Trikaan's delivery and engineering teams.
