Almost every GxP audit programme operates at Level 1 or Level 2, and does it competently. Neither level examines the software itself. Across 52 vendor audits between 2019 and 2025, 88% of vendors' quality systems did not manage their software technical process — no technical process instruction, no process management, and no baseline level of quality.
All IT vendor audits differ in depth, and that difference decides what you can conclude at the end of one. The EmpowermentQE Audit Depth Model names three levels of audit, preceded by a questionnaire stage that is not an audit at all. Almost every programme operates at Level 1 or 2. The software product itself is assessed only at Level 3.
In order, this page covers:
If you chose any of the first four, your programme is operating at Level 1 or Level 2. The last two are Level 3, and they are the reason this page exists.
| Level | Areas of interest | What it examines | Outcome | Skill level |
|---|---|---|---|---|
| 0Questionnaire | What does the vendor say about itself? | Company background and history. Whether a QMS structure exists. | Nothing is verified. | Access to a questionnaire template. |
| 1Audit | QMS structure, documentation and records. |
|
|
Trained auditor. Knowledge of quality management practices and documentation management. |
| 2Audit | Adequacy of documented processes, and whether QMS processes are reflected in working practices. |
|
|
As Level 1, plus experience in technical operational controls such as disaster recovery and role access. |
| 3Audit | Scope of technical processes defined and managed by the QMS. QMS maturity. Software product quality level. Total cost of software ownership. |
|
|
As Level 2, plus experience in software lifecycle activities, in product and service quality measurement, and in the economics of software quality. |
Where this model comes from. The Audit Depth Model is EmpowermentQE's, built from more than 300 computerised-system audits. It is not derived from a standard and no standard endorses it. It aligns with ISO/IEC/IEEE 15026-4:2021 on assurance in the life cycle — set out in the questions below.
The model builds on the capabilities and experience of the previous level. The difference at Level 3 is that a Level 2 outcome may give the impression of a mature vendor when, in reality, that vendor's quality levels are inconsistent.
“The absence of defects, incidents or problems that cause operational failure or incorrect data.” (EmpowermentQE). Defined this way, quality stops being an opinion and becomes something that can be counted, trended and compared between vendors.
Not sure which level your programme reaches? Send us one recent vendor audit report and we will tell you — free, one working day, nothing to prepare.
See how it works →A production line makes a product in phases. Each phase has a plan-do-check-act, each check is measured against a baseline level of quality, and work falling below that baseline is reworked before it moves on. That is how a manufacturer knows what is leaving the line.
Software is built in logical phases — requirements, design, code, unit test, integration test, system test. Each is a production process, each needs an instruction, and each needs a check in order to decide an action. The process instruction is what makes the output uniform, and uniformity is what makes a baseline level of quality possible.
A baseline quality level is a stated, measurable threshold — for example, a maximum defect density per module before code is accepted, a minimum proportion of defects removed before a phase closes, or a defect trend that must be falling release on release. A strong QMS states the quality level each process output is checked against. Where it does not, defects lie latent for later discovery — in the vendor's own process, in your validation cycle, or in operational use. Regulators are well acquainted with what happens when they surface in the last two.
ICH E6(R3), Glossary — Standard Operating Procedure: “Detailed, documented instructions to achieve uniformity of the performance of a specific activity.”
That is the production line argument, written into a GxP glossary. A document that names an activity without instructing anyone how to carry it out is not a thin SOP — it does not meet the definition of one.
Auditor hint. You do not need to be technical to establish whether an SOP describes an activity or instructs on one. It is the difference between what and how.
These levels are performed competently across the industry every day, and each establishes something real. The model has defined boundaries; in practice they overlap, depending on the expertise available and the audit scenario.
“What does the vendor say about itself?”
Completed questionnaires, desk checks of documents the vendor elected to provide, and ISO 9001, ISO 13485, ISO 27001 or SOC 2 certification provided at face value. Nothing can be independently verified. The activity is genuinely useful for quantifying vendor selection, for audit scoping, and for proportionate treatment of lower-risk systems.
“Is a QMS defined, and are its outputs produced?”
The vendor is conditioned to build artefacts that meet auditor expectations, rather than to meet the needs of software product quality.
Whether any of it worked. A complete, correctly sequenced document set is compatible with software of any quality.
“Are the processes adequate, and how completely are they applied?”
What those activities found. Code reviews occurred — but was their yield examined? Tests ran — but was their capability to reveal defects assessed? The SOP describes the production line; the QMS describes the delivery of documents. What is the quality of the software product?
A Level 2 audit is scoped to establish whether the QMS is applied. It cannot tell you the level of software product quality. You may have documentation quality, and documentation quality does not equate to software product quality.
The vendor's validation or test report records that only a few defects were raised. Documentation was correct, the process was followed, testing occurred, and the decision to release was made. The vendor claims fitness for release; you claim fitness for intended use; the vendor's claim is inherited by yours.
A Level 2 audit confirms that the test report stated testing was performed and that defects were managed — some fixed, some deemed acceptable to defer to a later release. It does not assess how capable those technical processes were at finding and removing problems. There is no indication of the number of test iterations, or of how issues were distributed across phases and features. The QMS described an approach; the report records only what happened in the final test phase.
So a low defect count arrives without an explanation. Read as a good sign it is reassuring, but it has three possible causes and they point in different directions.
Too little testing to know whether the product is any good. A low defect count here proves nothing.
Extensive testing, few defects found. Testing appears to be effective and the product looks to be holding up.
Defects are present and testing is limited. Whatever has been found, more likely remains.
Extensive testing is uncovering many defects. The production line is finding its own faults before you do.
Many defects does not mean bad testing. Where the test strategy is strong and thorough, finding many defects is a positive sign, because the testing is successfully exposing problems. Equally, few defects and few tests does not mean a high-quality product — you may simply not have looked hard enough, or in enough different ways.
Many varied tests, many defects → testing is effective; product quality needs work.
Many varied tests, few defects → testing is effective; the product appears stable.
Few tests, few defects → insufficient evidence.
Few tests, many defects → an obvious quality and testing problem.
There are many test techniques available, and there is a reason for that: individually, each has relatively low defect detection efficiency. Applying varied techniques accumulates that efficiency and improves the likelihood of finding defects before operational use. A vendor relying on a single technique, or on a small set, will report few defects and will have earned none of the confidence that figure appears to offer. Where the same approach is replicated across phases — often so the test documentation will survive a customer audit — the vendor's capability is diminished rather than demonstrated.
FDA General Principles of Software Validation (2002), section 5.2.5: “Software testing is one of many verification activities intended to confirm that software development output meets its input requirements. Other verification activities include various static and dynamic analyses, code and document inspections, walkthroughs, and other techniques.”
Four classes of verification activity are named. So where is the analysis of the others?
When software product quality is never determined, expensive consequences arrive later, as work your own teams absorb — recorded against the business process rather than against the vendor who built it.
Together these are the real total cost of ownership of the software — the licence and implementation cost plus everything above, carried for the life of the system. It is rarely calculated, and it is almost never attributed to the vendor who caused it. A Level 3 audit makes that figure visible, and it makes it comparable between vendors before you sign.
Levels 1 and 2 may not identify subtle system faults that compromise data which is otherwise accepted as correct. Those are the expensive ones, because nothing flags them.
Health warning. Poor software behaviour in production is frequently dismissed as the nature of software. This is incorrect. It is a reflection of the vendor's QMS maturity level — and the audit activity can be both a cause of that and a solution to it.
Most vendor quality systems are built to govern validation. Very few govern software engineering.
“Is the QMS any good? Prove it.” Those are the two questions often asked at the start of a Level 3 audit. What is being sought is a discussion of how the technical processes on the production line are measured, and of how those measures then influence the subsequent process activities and the management of the processes themselves.
Auditor hint. Check the development SOP history table. Do the changes arise from audits, or from quality measurement trends taken off the production line? The latter is an indicator of a medium-to-high maturity QMS.
Across 52 vendor audits conducted between 2019 and 2025, 46 vendors' quality systems did not manage their software technical process — no defined technical process steps for the software production line. Documentation compliance was present throughout. Work instructions frequently contained no instructions. Where the technical “how” existed at all, it was often held in engineering wikis — outside the QMS, and outside the remit of internal audit.
EmpowermentQE vendor audit observations, 52 vendor audits, 2019–2025. This is an observation drawn from our own audit practice, not an industry-wide prevalence study.
A Level 3 audit conducts a deep dive of the software production line to understand the processes actually in use to build the software. It then cross-checks whether the QMS has defined the technical process instructions needed for a baseline level of quality to be achieved. What comes back reflects the vendor's quality maturity.
Technical experts are often invited to participate in vendor audits, and this is where they earn their place. An auditor with technical expertise can interrogate the production line through deep dives and identify the inherent risks quickly. That understanding comes with experience, built across many years of auditing or of building software. A Level 2 auditor can also gather considerable intelligence during a Level 3 audit.
The last three require no technical expertise. Measurement, improvement and quality oversight can all be assessed by a non-technical auditor, which means a Level 3 scope is within reach of your existing team.
Level 3 is not a deeper reading of the same documents. It is a different class of evidence, and it may not be visible in a structured document set at all if the technical process instructions sit outside the formal QMS.
“How good is your QMS at producing a quality software product?” — “Prove it.”
EmpowermentQE has been measuring software product quality in regulated environments since 1997. The approach is older than our practice: we did not discover it, we were taught it. Measurement of product characteristics has sat inside the quality management standard your vendors scope to for more than thirty years.
| Standard | Where the requirement sits |
|---|---|
| ISO 9001:1994 | 4.9(d) monitoring and control of process parameters and product characteristics · 4.20 Statistical Techniques — “required for establishing, controlling and verifying process capability and product characteristics” |
| ISO 9001:2000 and 2008 | 8.2.3 monitoring and measurement of processes · 8.2.5 monitoring and measurement of product · 8.4 analysis of data, including conformity to product requirements and the characteristics and trends of processes and products |
| ISO 9001:2015 | 9.1.3 analysis and evaluation — analyse appropriate data to evaluate conformity of products and services, the effectiveness of the QMS, and the performance of external providers |
| ISO/IEC 90003:2018 | 9.1 monitoring, measurement, analysis and evaluation — the application of ISO 9001:2015 to computer software |
| FDA GPSV 2002 | 3.1.2 Verification and Validation (measures) · 5.2.5 Testing by the Software Developer (metrics) |
Where a vendor claims conformance with ISO 9001, clause 9.1.3 asks them to analyse data to evaluate the conformity of their product. Requesting that analysis holds a vendor to a standard they invoked themselves.
“You state conformance with ISO 9001. Show me the clause 9.1.3 analysis for this product.”
Regulatory harmonisation with wider software quality practice has strengthened over the years — from the FDA's General Principles of Software Validation in 2002, to the Quality Management System Regulation aligning 21 CFR 820 with ISO 13485:2016 on 2 February 2026. The emerging European conformity standards for artificial intelligence centre on vendor management and post-market monitoring in addition to technical documentation. An AI or machine learning system has to be monitored in operational use, and monitoring is measurement under another name. It is a Level 3 activity.
ISO/IEC 9126-1 set out a chain in which each stage determines the next: process quality determines internal quality measurements, which determine validation quality measurements, which determine quality in operational use. Measuring early in that chain predicts what arrives at the end of it. We have been predicting the quality of software releases as part of mature quality systems since 2003. This is not theory — it is an activity that relieves the time pressure on validation teams when a release reaches the environment.
Quality metrics provide the pulse of software product quality, regardless of the state of the documentation.
Because what is compared is the product quality level rather than the process used to reach it, vendors become directly comparable regardless of their development methodology. This is akin to comparing car models from different manufacturers — agile, linear and hybrid stop being an obstacle to comparison.
Reaching Level 3 does not begin with a vendor audit. It begins with establishing what your current programme asks, what it is contractually able to ask, and how far your existing reports actually reach.
You learn which level your programme is reaching, from a single document you already hold.
If the review shows a gap, we assess the whole programme and give you a plan to close it.
Your auditors learn to measure software product quality and to run a Level 3 scope themselves.
Send one recent vendor or system audit report. You receive a written note assessing the level of investigation achieved, whether a conclusion on QMS maturity and software product quality could be drawn from it, and what a deeper examination would have covered.
What happens next. The note is yours to use, with no obligation. If it shows a gap you want to close, the natural next step is the Readiness Assessment below — and the report you sent becomes part of its sample at no extra cost.
Request a free review →An assessment of your audit programme, not of a vendor. If you wanted to ask Level 3 questions, could you? Findings in a sample of your reports are classified by level; your SOP, checklists and questionnaire are reviewed; and your contracts are examined for whether they grant any right of access to defect data, test evidence or code review records. They rarely do, and that gap can only be closed at contract renewal.
A two-day course. The production line and what each phase produces, what each test phase can detect, the shapes defect distributions take and what each shape reveals, and worked audit scenarios covering where to obtain the information and how to interpret what comes back.
Drawn from Software Testing: Measuring Vendor Software Quality, Testing and Software Quality Analytics, and IT Vendor Audits: A Paradigm Shift from Documentation Quality to Product Quality, RQA Quasar. Read the full series.
Send it to us and we will tell you — that is the free Audit Report Review above, and it is the fastest way to find out. As a rough guide in the meantime: read the findings rather than the scope statement. Findings about missing, late or out-of-sequence documents indicate Level 1. Findings showing that a defined control was not being applied indicate Level 2. It reached Level 3 only if the report says something about the quality of the software itself.
It is the arrangement we find most often, and it is why Level 2 audits pass. The first tier is formal and covers the documents you see — specifications, reviews, test records, release documentation, all under document control. The second tier is informal and internal to the technical team, held in wikis, team conventions and engineering habit, and outside the QMS entirely.
The consequence is that the production line runs on the second tier while the audit examines the first. Both tiers are real. Only one of them is governed.
A single request will tell you. Ask to see the specific instruction for performing one technical task. You do not need to understand the task — you need to see whether the instruction contains logical steps. If it names the activity without telling anyone how to carry it out, the QMS is not managing the production line.
No. A questionnaire is self-declaration: the vendor answers questions about itself, in its own words, choosing what evidence to attach. Nothing in it is independently verified. The same applies to a desk check of documents the vendor selected, and to an ISO or SOC 2 certificate provided without its scope statement being read. These are genuinely useful for triage and for scoping an audit, and they provide no assurance about the system.
Yes, and it is the right starting point for an auditor new to the technical arena. Level 1 establishes that a quality system exists and produces its outputs, and it builds the familiarity an auditor needs before assessing whether those processes are applied. The levels are a progression rather than a judgement. The argument on this page is not that Levels 1 and 2 are wrong; it is that most programmes stop at 2, so the product itself is never assessed by anyone.
Through process metrics, which act as indicators of the production line's condition. Defect analysis is the most accessible — where defects were introduced, where they were caught and where they escaped tells you how effective each technical process actually is. Note that organisations name these differently: problems, faults, incidents, tickets and observations are frequently the same object under another label, and rarely called defects.
Service measures are also good indicators and you may already hold them: fault turnaround time, query turnaround time, number of patches per year, and the volume of corrections to new features per patch. A high correction rate against new features says something specific about the production line that no document set will tell you.
Defect density, defect criticality and defect removal efficiency are ordinary engineering measures. Their absence is what allows a system to carry a clean audit history and an unknown quality level at the same time.
As noted above, the model is EmpowermentQE's own. It does align with ISO/IEC/IEEE 15026-4:2021, Systems and software assurance — Part 4: Assurance in the life cycle, which describes how confidence in a stated claim is built and demonstrated. A claim, in that standard, is a stated assertion about the system — a vendor's statement that a build is fit for release is one.
| Level | What is assessed |
|---|---|
| 0 | The claim is accepted as stated. No evidence is sought. |
| 1 | Evidence exists, and the documentation is correct and in order. |
| 2 | The documentation is complete, and the processes producing it are applied. |
| 3 | The evidence supports the claim, by meeting a baseline quality measure. |
These are the responses recorded across the audits behind this page. None of them is offered in bad faith, and each has an answer.
CAPA addresses nonconformity in the quality system. When the software production line sits outside the QMS, CAPA takes no input from design, code review or test and never reaches it. It is also post-hoc, and its volumes are too low to trend. One request settles it: show me a CAPA that originated from a code review finding, a unit test failure or a static analysis result. A device vendor working under ISO 13485 and 21 CFR 820 may well be able to, because design controls sit inside their QMS by regulation. It is a question, not an assumption.
Agile development produces defect data in greater volume, including technical debt, and often at higher frequency than a linear approach. What is compared is the software product quality level, not the method used to reach it.
Actually no — some do not have issues. The expectation is that the vendor strives to catch them all. The question is what proportion of issues was caught before a release, which is a measurable quantity and the basis of defect removal efficiency.
The regulatory accountability for fitness for intended use remains with the regulated user. It cannot be transferred by a vendor's statement about its own testing.
Read the scope statement. SOC 2 addresses service organisation controls. It does not examine the software production line, and it makes no claim to. Vendors often fail to assess their own SOC 2 content against the risks their software offering actually poses.
Ask for totals across phases and releases, rather than individual defects that might identify a customer — although it is worth noting that the vendor was proud to share customer identities in their introduction slides. Where they still decline, your own validation and operational defect records support high-level trending.
Agreed, and it should not be applied to every vendor. Level 3 is for systems where the consequences of poor software product quality are real — business-critical, data-critical, patient-impacting, or a vendor that has already given you trouble. For everything else, Levels 1 and 2 remain proportionate and correct. The question is not whether to audit every vendor at Level 3. It is whether any of your vendors currently receive it.
No. The Audit Depth Model is a practical framework, not a regulation, and no authority mandates it by name. What regulation does expect — risk-based oversight of suppliers, evidence to support a released state, measurement, and continual improvement — has been in place for decades. Level 3 turns those existing expectations into questions an auditor can actually ask, and evidence an inspector can actually read.
We will tell you the level of investigation it achieved, and whether a conclusion on QMS maturity and software product quality could be drawn from it. No system access, no vendor involvement, nothing to prepare.
Request an Audit Report Review