Why the old vendor checklist no longer works
Every software company now calls itself AI native. Some have rebuilt how they work around coding agents, automated testing and senior review. Others have added a copilot license and a paragraph to the website. From the outside, the pitch decks look identical. The difference shows up six months later in your delivery speed, your defect rate and your invoice.
The vendor evaluation questions most CTOs still use were written for a different era: team size, years in business, hourly rates, a reference call. Those still matter, but they do not tell you whether a partner can deliver the two to three times throughput that AI native engineering makes possible, or whether they have the discipline to keep that speed from turning into a quality problem. The twelve questions below do. Ask them in the first two conversations, and pay close attention to how specific the answers are.
The 12 questions
1. Show me how a feature moves from ticket to production on your team
You want a walkthrough, not a diagram. Who writes the architecture note, which coding agents are used and for what, how tests are generated, who reviews, what gates a merge, how it is deployed. An AI native team can describe this in concrete terms because they do it every day. A team that is retrofitting AI will talk about "leveraging AI tools" without specifics.
2. What percentage of your code is written by agents, and what is never delegated to them?
Good answers are honest and bounded. Scaffolding, CRUD, integrations, test suites and documentation are heavily agent written. Data models, security boundaries, authentication, payment flows and anything touching compliance are designed by a senior engineer and reviewed line by line. A partner who claims either "almost none" or "almost all" has not thought about the risk profile.
3. Who reviews agent written code, and how do I verify that?
The correct answer is a named senior engineer reviewing every pull request, with the review visible in your repository history. Ask to see a real pull request from a recent project (with the client's permission) including the review comments. If review is optional, sampled or done by another model only, you will inherit the defects.
4. What automated test coverage do you commit to in writing?
AI native teams can write tests cheaply, so there is no excuse for low coverage. Look for a contractual target (we commit to 80% or higher from day one) plus a description of what kinds of tests: unit, integration, end to end, load, and security scanning in CI. Ask how coverage is reported to you and how often. Our approach is described in why we insist on 80%+ coverage from day one.
5. How has your pricing changed since 2023?
This is the question vendors dislike most, and it is the most revealing. If agents have doubled or tripled their output, their fixed price quotes for equivalent scope should be substantially lower than two years ago, or their monthly rate should buy visibly more delivery. A partner whose prices have not moved is either not using the tools seriously or is keeping the entire gain. Ask for a concrete example: the same kind of module, then and now.
6. What does your written estimate contain, and how fast do I get it?
Speed of scoping is a proxy for engineering maturity. A team with strong architecture habits and agents that can prototype in hours can turn a discovery call into a written scope, timeline and cost within a couple of days. We commit to 48 hours. Ask what the document includes: architecture outline, assumptions, risks, milestones, what is excluded, and how change is handled.
7. How do you handle security for AI assisted development?
Three things to listen for. First, how client code and data are kept out of model training and out of shared contexts (enterprise agreements with model vendors, no personal accounts). Second, static analysis and dependency scanning on every pull request, because agents can introduce vulnerable patterns and outdated packages at speed. Third, secrets management and access control that does not depend on a developer's laptop. Our security page is the standard we hold ourselves to; ask any partner for the equivalent.
8. What do I own at the end, and can my team maintain it?
You should own the code, the infrastructure definitions, the tests, the documentation and any prompts and evaluation sets used in AI features. Ask how documentation is produced (AI native teams generate it as the code is written, not as a handover exercise) and ask to see a sample. Then ask the harder question: could a mid level engineer on your own team pick this up in a month? If the partner cannot answer confidently, the speed you gain now becomes lock in later.
9. How do you keep speed from becoming sprawl?
Agents make it cheap to write more code. That is not always good. Ask how the team keeps architecture coherent when output triples: shared conventions enforced in CI, architecture reviews, refactoring budgets, and a senior engineer who says no. Ask how many services a recent project ended up with and why.
10. Show me a modernization or rescue project, not just a greenfield one
Greenfield builds flatter every team. Legacy modernization, where the old system must keep running while the new one replaces it module by module, tests judgment. Ask for a case with a real cutover: how traffic was shifted, how data was migrated and reconciled, what went wrong and how it was reversed. Our case studies include the numbers; insist on the same from anyone you evaluate.
11. What happens when a defect reaches production?
Look for a written warranty period (we provide 30 days of post launch stabilization), defined response times for severity levels (four hours for a P1 in our case), and a description of how the fix is tested and deployed. Ask for the team's post release defect rate and how it is measured. "Very low" is not a number.
12. If this does not work, how do I leave?
Notice periods, knowledge transfer obligations, repository and infrastructure ownership, and what happens to in flight work. A confident partner makes exit easy because they expect you to stay for the results, not the friction. Month to month terms and free replacement of an engineer who is not a fit are reasonable expectations in 2026.
Any partner can say they are AI native. Ask to see the pull request, the coverage report, the estimate and the price history. The evidence is either there or it is not.
Scoring the answers
Use a simple scorecard. For each question, score zero for a vague or evasive answer, one for a credible verbal answer, and two for an answer backed by evidence you can inspect (a document, a repository, a report, a reference who confirms it). Weight questions 3, 4, 5 and 8 double, because they are the ones that predict your outcome.
| Signal | Weak answer | Strong answer |
|---|---|---|
| Code review | "Our senior team reviews regularly" | Every PR reviewed by a named senior engineer, visible in repo history |
| Test coverage | "We follow best practices" | 80%+ committed in the contract, reported weekly, enforced in CI |
| Pricing | Rates unchanged since 2023, "quality costs" | Same scope quoted at materially lower cost or shorter timeline than 2023, with an example |
| Estimate | Two to three weeks for a proposal | Written scope, timeline and cost within 48 hours of discovery |
| Ownership | "You will have access" | Client owns repo, infra as code, tests, docs and prompts from day one |
| Warranty | "We stand behind our work" | 30 day stabilization, P1 response in hours, defect rate measured and shared |
A partner scoring above 70% of the maximum is worth a paid discovery sprint. Below 50%, keep looking regardless of how good the demo was. Our free vendor evaluation scorecard puts this into a spreadsheet you can use across several candidates.
Red flags that end the conversation
- Developers using personal AI accounts with client code.
- No test coverage number, or a number that is not enforced anywhere.
- Pricing that has not changed since before coding agents existed.
- Inability to show a real pull request with human review.
- Documentation described as a deliverable at the end of the project rather than a by product of the work.
- A case study portfolio with no numbers, or numbers with no method behind them.
How RG INSYS answers these questions
We wrote this list because it is the list we want to be evaluated against. Senior engineers design every system and review every pull request. Coding agents write the scaffolding, integrations, tests and documentation under that direction. Test coverage of 80% or higher is in the contract, static analysis runs on every change, and you own the code, infrastructure, tests and documentation from the first commit. Delivery is typically three times faster than a traditional team at around 60% below onshore rates, with a written scope within 48 hours, a 30 day stabilization warranty and four hour P1 response.
Read our engineering methodology and delivery process, compare options on our in house versus freelance versus partner page, or book a consultation and put the twelve questions to us directly.
Frequently asked questions
What does AI native actually mean for a software development company?
It means the delivery process has been rebuilt around AI tools rather than having them added on. Senior engineers design the system, coding agents produce most of the routine code and tests under their direction, every change is reviewed by a human, and the cost and speed gains are passed to the client. The tell is measurable: faster delivery, higher test coverage and lower prices for equivalent scope.
Should an AI native partner be cheaper than a traditional one?
For equivalent scope, yes, or it should deliver materially more for the same monthly cost. Coding agents have changed the cost of producing tested software. A partner whose pricing has not moved since 2023 is either not using the tools seriously or is keeping the entire gain. Ask for a like for like example of the same kind of module quoted then and now.
Is AI written code safe to run in production?
It is when it is reviewed by a senior engineer, covered by automated tests, scanned for vulnerabilities and outdated dependencies in CI, and kept away from security critical paths unless a human designed them. Unreviewed agent output is not production code. The safety comes from the process around the tools, which is exactly what your evaluation questions should probe.
How long should it take a partner to produce an estimate?
A mature team should turn a discovery conversation into a written scope, timeline and cost within two to three days; we commit to 48 hours. The document should include an architecture outline, assumptions, risks, milestones and exclusions. Longer turnaround is usually a sign of thin senior capacity or a sales process that is disconnected from engineering.
Put the twelve questions to us
Book a consultation and ask us anything on this list. You will get direct answers, a real pull request to inspect, and a written scope, timeline and cost estimate within 48 hours.
Book a Free Consultation