How We Vet and Evidence-Check Every AI Tool at useToolCraft
Most directories list hundreds of tools and call it curation. We do something narrower and more useful for a solo founder: for every tool used in our curated stacks, we read the vendor’s own pricing page and compare it with the band we publish, we collect what real users reported in public over the last 90 days with a link for each point, and a person reviews the result before it goes live. The check date and the sources sit on the tool profile, so you can see when it was last checked and judge the evidence yourself. We do not claim hands-on testing of every tool; where a claim comes from our own use, the profile says so. This page is the full method: what an evidence check covers, how we score fit, how we handle affiliate bias, and the bar a tool has to clear.
Every profile and stack on the operator guides & content hub follows this methodology. Selecting tools? Start with our 2026 selection framework — this page documents the vetting and evidence layer behind every recommendation.
Data as of October 2026
useToolCraft Workflow Lab
Implementation & Automation Specialists
·Data as of October 2026
Our Vetting Method
Three layers back every recommendation. First, the catalog: each tool has a hand-written entry with a pricing band, a skill level, the jobs it fits and the people it does not fit; entries are reviewed when vendors change plans or readers report an error. Second, the weekly evidence check for every tool used in a curated stack: the vendor pricing page is read and compared with the catalog band, public user reports from the last 90 days are collected with a source link for each point, and a person reviews the result before it is published; the check date and the sources appear on the tool profile. Third, editorial judgement: stack pages pair tools around one weekly job and say who should skip each tool. Where a note comes from our own use of a tool, the profile says so; where it does not, we do not imply it. “Data as of” dates on profiles and stack pages tell you how current the information is, and a pricing mismatch found by the check puts the tool under review on the site until it is corrected.
Sources consulted
- useToolCraft tool catalog — data freshness policy
- useToolCraft (accessed 2026-10-05)
- How to Find the Right AI Tools (2026 framework)
- useToolCraft (accessed 2026-10-05)
What we look for that directories skip
A tool that demos well and fails in production does not belong in your stack. These are the failure modes the evidence check and the editorial review look for, with the signal each one relies on.
- Price and plan changes
- The lowest paid price and the plan tiers are read from the vendor pricing page on every check. A change in the starting price or a removed free tier appears in the tool’s evidence section and in the monthly “what changed” digest.
- Limits that block week one
- Task, seat and usage caps are taken from the vendor page and from user reports. Where users report hitting a cap on a free or entry plan, the complaint is listed with its source so you can read the thread yourself.
- Tools that disappear or stall
- A product page that stops responding marks the tool unreachable, and we review every stack that uses it. Tools that are superseded or shut down are marked legacy in the catalog and ranked last in the finder.
- What users actually complain about
- Public reports from the last 90 days are summarised into praise, complaints and dealbreakers. We keep the URL for each point and drop anything without one, so a summary never says more than its sources do.
What an Evidence Check Covers
The badge on a tool card means these four steps ran within the last 45 days and a person merged the result. Each step leaves something you can inspect on the profile: a date, a price as printed, a list of sources.
Availability
The product page and the pricing page are requested. A page that no longer responds marks the tool as unreachable until the next check, and the stacks that use it are reviewed.
Pricing as printed by the vendor
Tiers and the lowest paid price are read from the vendor’s own pricing page and compared with the catalog band we publish (Free, $1–50, $51–200, $200+). A mismatch puts the tool “under review” on the site until the band is corrected.
Public user reports, last 90 days
Hacker News threads, Reddit discussions, Product Hunt reviews and YouTube reviews from the last 90 days are collected and summarised into praise, complaints and dealbreakers. Every point keeps the URL it came from; anything without a source is dropped.
Human review before publishing
The run opens a change request. A person reads the diff, rejects anything that looks wrong, and merges. Nothing reaches a tool card or stack page before that step, and the check date on the page is the date of that run.
Only tools used in our curated stacks carry the evidence badge. The rest of the catalog is listed from the catalog entry and labelled “not yet evidence-checked”. A check is current for 45 days; after that the badge reads “check overdue” until the next run.
The Criteria We Use to Evaluate Tools
Four gates every tool passes before it earns a profile, stack slot, or wizard recommendation. Skip a gate — or hide an API limit, a loop risk, or a tier cap that blocks week one — and the tool stays out. No exceptions for launch-week hype.
1. Scope the workflow
Before a tool enters our database, we define the operator job it should solve — lead capture, support triage, SEO briefs, invoice follow-up. No job, no listing.
- Check
- Clear input → output for a non-technical founder (not “AI-powered platform”)
- Check
- Repeatable weekly use case — the kind that hurts when it fails
- Check
- Who should skip it, written as plainly as who should use it
2. Evidence check
Every tool used in a curated stack goes through a weekly check that leaves something you can inspect: a check date, the price as printed on the vendor page, and the public user reports behind each summary.
- Check
- Product and pricing pages respond; a tool that disappears is marked unreachable
- Check
- Pricing tiers read from the vendor page and compared with the catalog band
- Check
- Public user reports from the last 90 days collected with a link for each point; a person reviews the result before it is published
3. Fit scoring
Tools are scored for skill level, implementation difficulty, pricing tier, and integration friction — not marketing claims or launch-week hype.
- Check
- Beginner / Intermediate / Advanced skill alignment
- Check
- Risks users report — plan caps, loops, silent failures — passed on with the source
- Check
- Best-for vs not-recommended guidance in plain language
4. Ongoing verification
A check is current for 45 days. Pricing that no longer matches the band puts the tool under review on the site until the catalog is corrected, and every material change goes into the monthly “what changed” digest.
- Check
- Vendor pricing page re-read on every check
- Check
- Profiles marked “under review” on a pricing mismatch and “unreachable” on an outage
- Check
- “Data as of” and check dates on every profile, stack and recommendation
Operator Reliability Score (ORS)
The editorial score (1–5) measures operator fit for solopreneurs. The Operator Reliability Score (ORS, 0–100) is a separate, deterministic formula — same inputs always produce the same number. No star ratings, no vibes.
Tool-level formula
Score = (S×0.30 + P×0.25 + A×0.25 + F×0.20) × 10, where S=production stability, P=pricing transparency, A=API rate limits, F=community failure inverse (all 0–10).
Stack-level formula
Stack score = 0.60 × mean(core tool scores) + 0.40 × min(core tool score). Supplementary tools are listed but do not dilute the composite — your stack fails on the weakest core glue.
Four dimensions (0–10 inputs)
- Production stability & reliability (30%)
- Webhook delivery, scheduled job survival, incident frequency in live workflows.
- Pricing transparency (25%)
- Penalizes hidden credit systems, task multipliers, and surprise overage invoices.
- API rate limits & throttling (25%)
- Documented ceilings, predictable 429 behavior, batch-export viability on paid tier.
- Documented failure reports (20%)
- Inverse of known community/operator failure patterns — fewer documented issues = higher score.
Score bands
| Band | Range | Meaning |
|---|---|---|
| A | 85–100 | Production-grade reliability |
| B | 72–84 | Reliable with documented limits |
| C | 58–71 | Usable — watch pricing & rate limits |
| D | 45–57 | High friction — narrow scope only |
| F | 0–44 | Operator caution — verify before paying |
A handful of high-traffic tools carry editorial inputs set by hand in June 2026. Every other tool starts from catalog inference, and the profile breakdown always says which of the two drove the score. The ORS is a fit heuristic, not a measurement.
How We Handle Bias and Conflicts of Interest
Affiliate economics are real. So is the trust cost of pretending they do not exist. Here is exactly how we keep commissions from corrupting rankings.
- Affiliate revenue does not buy placement
- We may earn commission when you subscribe through our links. That does not change rank order, “best for” copy, or whether a tool appears in a stack. Vendors cannot pay to skip our vetting steps or override a not-recommended verdict.
- No pay-to-play listings
- There is no sponsored “featured tool” slot in the wizard shortlist. If a product is missing, it is because we have not verified it yet or it failed the operator-first bar — not because a competitor paid more.
- We disclose what we have not checked
- Tools outside the curated stacks are labelled “not yet evidence-checked”. We do not invent hands-on notes from press releases, and we do not describe the weekly check as testing. When a profile rests on the catalog entry alone, it says so.
- Corrections are public
- Readers can email support@usetoolcraft.com with pricing errors or broken onboarding steps. Material corrections update the profile and the “data as of” date — we do not silently edit affiliate copy without noting the change in our review log.
The Operator-First Standard We Follow
Operator-first means we optimize for the solo founder answering client email at 9pm — not the enterprise buyer with a 12-person procurement team. When users report that a scenario loops, a sync rate-limits, or a model output needs three edit passes before it can ship, we pass that on with the source. These four standards are non-negotiable.
- Workflow-first, not feature-first
- We scope the job before the SKU: “capture budget-qualified leads from site chat” beats “AI chatbot with 47 integrations.” Feature dumps without a defined input → output do not ship in recommendations.
- How you see it: Every stack page and wizard result ties tools to a named weekly task — not a category label.
- Budget-honest tiers
- Pricing bands are checked against vendor billing pages, not affiliate landing pages with promotional discounts. Free-tier caps (seats, tasks, conversations) that block solopreneurs in week one are recorded when the vendor page or user reports state them.
- How you see it: “Under $50” stacks sum real paid tiers; a tool whose vendor page no longer matches its band shows “under review” until it is corrected.
- Skill-level match
- Beginner, Intermediate, Advanced labels reflect who completes onboarding without API keys or Zapier surgery. A tool that wins for agencies loses for a non-technical founder — we say both.
- How you see it: ToolEvaluationSection best-for / not-recommended blocks on every profile and curated stack.
- Brutal not-recommended lines
- If a tool is wrong for a persona, we say why in plain language — “referral-only businesses with no site traffic” not “may not fit all use cases.” Fluff hides subscription mistakes.
- How you see it: Curated stacks add supplementary tools only around a core job that is already covered — not as bundle upsells.
Try the tool finder — get an evidence-checked shortlist in 30 seconds
This is the process behind every recommendation. To see it applied to your own work, describe what you do each week, your budget and your experience level. The finder returns a short list ranked for fit, with first setup steps for each tool.
Find AI tools matched to your workflow
Describe your project in plain English and get a curated shortlist plus step-by-step implementation plan — built for solopreneurs and small business operators.
Try the free AI tool finder wizardCommon questions about our vetting process
- What does “evidence-checked” mean?
- Within the last 45 days, the product and pricing pages responded, the pricing tiers were read from the vendor’s own page and matched the band we publish, public user reports from the last 90 days were collected with a source for each point, and a person reviewed the result before it was published. The check date and the sources are on the tool profile. It does not mean we used the tool ourselves.
- Do affiliate links change which tools you recommend?
- No. Commission does not change rank order, stack placement, or “best for” copy. Partner links are marked, the disclosure appears on every page that has one, and we publish this methodology so you can judge fit independently.
- What happens when a tool changes price or stops working after you checked it?
- The next weekly check records the change: a new starting price, a removed free tier or an unreachable product page. The profile shows the new check date, the change appears in the monthly “what changed” digest, and a pricing mismatch marks the tool under review until the catalog band is corrected.
- How is this different from reading G2, Product Hunt or asking ChatGPT?
- Review sites optimise for breadth and social proof, and an assistant answers from memory with no date and no source. We cover a small set of tools that fit one operator job, one budget band and one skill level, show the date each one was checked, link the public reports behind every summary, and say who should skip a tool.
- What is the Operator Reliability Score (ORS)?
- ORS is a 0–100 composite from four weighted 0–10 dimensions: production stability (30%), pricing transparency (25%), API rate limits (25%), and inverse failure-report density (20%). It is separate from our 1–5 editorial fit score. Every tool profile shows the full breakdown.