Industrial Visual Inspection: The Allure of Multimodal Large Models

Shortly after the launch of GPT-4V in early 2023, we received a call from a long-term client.

He served as the technical director of a home appliance manufacturer. Two years prior, we had deployed a surface inspection system based on YOLOv5 for their factory, which had been operating stably ever since.

He raised a thought-provoking question over the phone:

“I’ve seen that GPT-4V can interpret all kinds of images and recognize nearly everything. Can we adopt it directly for quality inspection? Would that eliminate the need for data labeling entirely?"

I held back a straightforward answer back then.

Truth be told, we were equally captivated by the idea ourselves.

Demos of multimodal large models are undeniably impressive. Feed the model any random image, and it can outline contents, pinpoint defects and classify fault types. No training or labeling is required; it delivers zero-shot performance out of the box.

If this capability translated seamlessly to factories, the entire rulebook for industrial visual inspection would be rewritten.

We spent nearly two years testing diverse multimodal large model solutions across multiple projects.

Our conclusion is clear: tempting as the technology may seem, real-world industrial application comes with harsh limitations.

This article documents all the pitfalls we encountered over these two years.

Before diving into multimodal large models, it is critical to lay out the industry baseline:

The dominant solution for today’s industrial visual inspection relies on object detection and segmentation models represented by the YOLO series.

This is hardly a new trend. Starting from YOLOv3, through the widely deployed YOLOv8, YOLOv9 and YOLOv10, the YOLO family has been implemented in industrial production lines for years, boasting a fully mature technical stack.

First, ultra-fast inference speed.

Equipped on standard edge computing boxes paired with industrial cameras, YOLOv8 completes inference for one frame within 10 to 30 milliseconds, matching the takt time of most production lines.

Second, sufficient detection accuracy.

With adequate labeled datasets, the YOLO series achieves outstanding precision for common defect categories, easily hitting an mAP of over 90%.

Third, mature deployment ecosystem.

Ready-made toolchains support multiple deployment frameworks including ONNX, TensorRT and OpenVINO. The full workflow from model training to on-site deployment has been validated by countless industrial projects.

Fourth, comprehensive open-source ecosystem.

The active open-source community provides accessible fixes for most technical hurdles, with abundant pre-trained weights, data augmentation kits and labeling tools readily available.

Therefore, the YOLO series is practically the default choice for industrial visual inspection projects launched in 2024.

There is no need to debate whether deep learning should be adopted — that question was settled a decade ago.

The new core question now arises: With the emergence of multimodal large models, does YOLO still remain the optimal solution?

2023 witnessed an explosive wave of multimodal large model releases.

Models including GPT-4V, Gemini and Claude 3 deliver powerful general image comprehension capabilities.

Traditional workflow: To inspect a specific type of defect, you first need to collect, label and train on images of that defect. No data means no usable model.

Multimodal large models: Simply describe your demand in natural language, such as “Check whether there are scratches in this image", and the model will return results instantly. No training or labeling required.

What does this mean? The cold-start cost drops close to zero.

When launching new products, there is no need to spend two weeks on data collection, labeling and model training. You can put the model into use merely with a few lines of prompts.

Traditional models only output bounding boxes and confidence scores, e.g. “A defect exists within this box with a confidence of 0.87".

Multimodal large models generate descriptive natural language: “A scratch of around 2cm appears at the top-left corner of the picture, likely formed during transportation. It is recommended to optimize the packaging process."

What does this mean? Inspection results can be directly converted into formal quality inspection reports.

Traditional models can only recognize defect types seen during training; they fail to identify brand-new unseen defects.

In theory, multimodal large models have processed massive images sourced from the internet, enabling them to potentially recognize all kinds of rare and irregular defects.

What does this mean? Coverage for long-tail defects and abnormal edge cases is drastically improved.

Traditional solutions embed fixed inspection rules into the model. Revising inspection criteria requires full retraining.

Multimodal large models support dynamic adjustment of standards via prompts. For instance, you can set the threshold as “scratches over 1cm count as NG" one day and switch it to “0.5cm" the next without modifying the underlying model.

What does this mean? Tuning inspection standards becomes extremely flexible.

Reading all these advantages, you may also be tempted — just as we were back then. That’s why we decided to deploy multimodal large models in several real projects, only to run into a string of costly pitfalls afterward.

Our pilot project focused on appearance inspection for mobile phone housings.

The production line processes one workpiece every 3 seconds, meaning total inspection latency must stay below 2 seconds to reserve 1 second for robotic sorting.

We tested the GPT-4V API workflow:

1. Upload the image and input the prompt
2. Wait for server response
3. Receive inspection results

Average latency hit 4–6 seconds, and could exceed 10 seconds amid network fluctuations — far too slow for the assembly line.

You might suggest self-hosted open-source multimodal models such as LLaVA and Qwen-VL instead. We tested these as well. Running LLaVA-13B on an A100 GPU yields single-image inference latency of roughly 800ms to 1.2 seconds.

While faster than cloud APIs, it remains dozens of times slower than YOLO.

Even if we tolerate the latency for argument’s sake, the cost calculation tells a harsh story.

How many images does one production line process daily?

Assuming one workpiece every 3 seconds and 20 hours of daily operation, a single line generates around 24,000 inspection images per day.

For GPT-4V API, unit pricing ranged from $0.01 to $0.03 per image, depending on resolution and token consumption:

• Daily cost per line: $240–$720
• Monthly cost per line: $7,200–$21,600
• Annual cost per line: $86,400–$259,200

This only accounts for one line, while our client operated 12 production lines — an unaffordable expense for manufacturers.

What about self-hosted open-source models?

A single A100 GPU delivers roughly 1–2 QPS (queries per second). A single line peaks at around 0.3 QPS, seemingly manageable with one card for multiple lines.

However, factoring in servers, IDC space and maintenance, the annual operating cost for an A100 deployment runs into hundreds of thousands of RMB.

In contrast, a YOLO deployment only requires an edge computing box costing a few thousand RMB to support one full production line.

The cost gap spans two orders of magnitude.

This proved our most frustrating roadblock.

Industrial inspection demands absolute determinism: identical images must yield identical inspection results every single time, otherwise standardized quality control and traceability become impossible.

Multimodal large models, however, produce probabilistic outputs.

We ran a controlled test: feeding the same defective image with an identical prompt to GPT-4V ten separate times. The outcomes varied drastically:

• 7 runs labeled the product defective
• 2 runs marked it suspected defective requiring manual review
• 1 run claimed no obvious defects existed

All from the exact same input and prompt.

Such randomness is fatal for factory quality control. Inspectors cannot act on a “70% chance of defect" output — every workpiece needs a definitive OK or NG verdict.

Some propose setting temperature to 0 for consistency. We tried this method, which improved stability yet failed to guarantee 100% identical outputs. Large models generate results via sampling mechanisms, and minor deviations persist for edge cases even with temperature = 0.

Multimodal model performance hinges entirely on prompt design, which we spent extensive manpower optimizing to boost accuracy and stability.

We soon discovered prompts are extremely sensitive to wording changes.

Three prompts with nearly identical core requests delivered vastly different inspection outcomes:

Prompt A: “Check whether surface defects exist in this image."

Prompt B: “Carefully examine the product surface and identify scratches, pits, foreign matter and other defects."

Prompt C: “Act as a professional quality inspector. Locate and classify any appearance defects on the product in this image."

Worse still, prompts fine-tuned for Product A lose efficacy when applied to Product B, requiring full rework of prompt logic for every new product variant.

How does this differ from retraining YOLO models for new products?

YOLO training relies on quantifiable evaluation metrics to clearly signal when the model meets standards; prompt tuning depends entirely on subjective trial and error, with no clear benchmark for optimal performance.

Hallucination is a well-documented flaw of large language and multimodal models: the system confidently invents details that do not exist.

In industrial inspection, this manifests as three typical failures:

1. Flagging defect-free products as defective
2. Misstating defect positions (e.g. locating scratches on the left when they appear on the right)
3. Misclassifying defect types (e.g. labeling pits as scratches)

One test case exemplifies the severity: an entirely flawless product image triggered a highly detailed fabricated analysis: “A shallow scratch approximately 3mm long is detected at the bottom-right corner, functional impact assessment recommended."

Upon close visual review, no mark or scratch was present in that region at all.

If such hallucinations infiltrate mass production lines, severe consequences follow: either defective goods slip through undetected (missed inspection) or qualified products get wrongly rejected (false rejection).

As cloud APIs suffer high latency and excessive cost, self-hosted deployment seems like an alternative. We evaluated hardware and software requirements for mainstream open-source multimodal models:

YOLOv8-m runs smoothly even on a GTX 1080 with 8GB VRAM.

It can even be deployed on edge computing hardware such as NVIDIA Jetson modules with power consumption of merely tens of watts.

The computational resource threshold differs by an entire order of magnitude.

For most factories, installing an A100 server on the production floor is impractical in terms of both capital expenditure and daily operation & maintenance.

After stumbling through all the above pitfalls, we stepped back to reflect on a fundamental question:

What core capabilities are essentially demanded by industrial visual inspection?

1. Deterministic Output

   Identical images must yield 100% consistent results. This forms the foundation of standardized quality control and full traceability; probabilistic outputs are unacceptable.

2. Ultra-Low Latency

   Millisecond-level response. Production line takt time is rigid, and inspection cannot become a bottleneck.

   A 10ms inference time and a 1,000ms inference time represent entirely different operational realities.

3. High Throughput

   How many frames can be processed per second? How many workpieces can be inspected daily?

   Computational costs must remain controllable, avoiding annual expenses of hundreds of thousands of US dollars for a single production line.

4. Edge Deployment Compatibility

   Factory network environments are complex; many workshops lack stable or accessible internet connections.

   Models must operate locally on edge devices rather than relying on cloud APIs.

5. Interpretable Inspection Results

   When a defect is detected, the system needs to clearly inform inspectors of its exact location and category.

   Ideally, it should output defect coordinates, area and confidence scores for downstream system integration.

6. Controllable Maintenance Costs

   Products get upgraded and inspection standards are revised on a regular basis.

   The adaptation cost for every iteration must be manageable, without full reconstruction each time.

Matching these six core requirements against the two technical routes reveals a clear contrast:

• Determinism: 100% consistent outputs given identical input
• Low latency: 10–30 millisecond inference
• High throughput: Dozens to over a hundred QPS per single GPU
• Edge-deployable: Fully compatible with Jetson hardware and industrial PCs
• Interpretable outputs: Bounding boxes, defect categories and confidence values
• Low maintenance overhead: Mature toolchains for incremental training and transfer learning

• Determinism: Inherently probabilistic output
• Latency constraint: Second-scale inference
• Throughput limit: Single GPU only supports single-digit QPS
• Edge deployment barrier: Demands A100-class high-end GPUs
• Interpretability gap: Raw natural language descriptions require secondary parsing
• Unpredictable maintenance: Prompt engineering lacks quantifiable optimization standards

So can multimodal large models replace YOLO? The conclusion is unambiguous:

At the current stage of technical maturity, multimodal large models are unsuitable as the primary solution for industrial visual inspection.

Its strengths including zero-shot reasoning, deep semantic comprehension and strong generalization deliver little practical value on production lines; meanwhile its critical flaws — high latency, prohibitive costs and unstable outputs — are catastrophic for industrial quality control.

This does not mean multimodal large models are completely useless for industrial visual inspection.

The key lies in identifying their proper niche.

After two years of field trials, we have summarized four scenarios where multimodal large models create tangible value:

Annotation constitutes the biggest cost driver of traditional inspection projects.

An industrial vision task usually requires thousands to tens of thousands of annotated images. Outsourcing annotation services costs several tenths to several US dollars per frame, with labeling expenses accounting for 30%–50% of total project investment.

Multimodal large models deliver pre-labeling capability:

The model generates preliminary annotation masks and boxes from raw images first. Human staff only need to review and revise results instead of labeling from scratch.

Our field tests prove this workflow boosts annotation efficiency by 3–5 times, cutting average labeling time per image from 30 seconds to under 10 seconds.

The performance ceiling of YOLO models is straightforward: they can only recognize defect types featured in training datasets.

Unprecedented rare defects will trigger missed detection by YOLO.

Although such long-tail anomalies occur infrequently, they often signal severe abnormal manufacturing conditions, carrying higher operational risks.

Multimodal large models act as a fallback verification layer:

When YOLO outputs a borderline confidence score (roughly 0.3–0.7, the gray zone of uncertainty), the corresponding image is sent to the multimodal model for secondary judgment.

The zero-shot generalization strength of large models covers these unseen rare anomalies.

Under this mechanism, only 5%–10% of all images are forwarded to the multimodal model, keeping total costs manageable while drastically improving coverage of long-tail defects.

YOLO only outputs structured data: bounding boxes, defect categories and confidence scores.

While sufficient for backend industrial systems, these raw metrics are unintuitive for human inspectors, who need answers to practical questions: How severe is the defect? What caused it? What corrective action should be taken?

Multimodal large models perform semantic report generation:

Input: Defect coordinates, classification labels, product model and manufacturing process parameters

Output: Natural language inspection report, e.g. “A 5mm scratch is detected on the left edge of the product, likely caused by mold abrasion; mold maintenance is recommended."

This task is latency-insensitive (reports can be generated asynchronously) and cost-efficient (only executed on NG non-conforming products with limited volume).

Clients occasionally face tight deadlines: new products scheduled for mass production the following week with merely dozens of defective sample images, insufficient for full YOLO training.

Traditional workflow cannot launch inspection under such limited data.

Multimodal large models serve as a transitional temporary solution:

Zero-shot capability enables immediate deployment with acceptable yet imperfect accuracy, far outperforming full manual inspection. Data can be continuously collected during pilot operation to train a formal YOLO model for long-term use once sufficient samples are accumulated.

Based on the above analysis, we have adopted a hybrid dual-channel architecture for recent industrial projects:

• Handles over 95% of all inspection workloads
• Deployed locally on edge hardware with 10–20ms inference latency
• Outputs structured bounding boxes, defect types and confidence scores

• Only processes borderline low-confidence images within the gray zone
• Invoked asynchronously without disrupting main line throughput
• Functions for long-tail defect fallback verification, semantic report generation and auxiliary labeling

Core design principles of this hybrid framework:

1. YOLO acts as the core primary system; multimodal models serve as auxiliary tools — avoid reversing their roles
2. Data shunting instead of serial processing: multimodal models stay off the critical production path and impose no impact on main-line latency or throughput
3. Confidence-based traffic splitting: high-confidence results pass through directly, while ambiguous samples are forwarded for secondary multimodal validation
4. Predictable cost control: only a small fraction of images consumes multimodal model computing resources

Below is a summarized decision tree for teams selecting industrial visual inspection algorithms:

1. Latency Requirement
   • Required inference <100ms → Choose YOLO
   • Second-scale latency is acceptable → Multimodal large models are viable
2. Throughput Requirement
   • Over 1 frame per second → Choose YOLO
   • Only hundreds of images processed daily → Multimodal large models are viable
3. Deployment Environment
   • Edge offline deployment required → Choose YOLO
   • Stable dedicated cloud computing resources available → Multimodal large models are viable
4. Data Availability
   • Thousands of annotated samples on hand → Choose YOLO
   • Only dozens of samples with urgent launch timeline → Adopt multimodal models as temporary transition
5. Budget Constraints
   • Annual single-line operating budget below 100,000 RMB → Choose YOLO
   • Ample financial budget → Hybrid architecture is recommended

For the vast majority of industrial scenarios, YOLO will remain the optimal choice.

Multimodal large models are only suitable as primary solutions under specific conditions: latency tolerance, low throughput demand, stable cloud computing support and extreme data scarcity.

The most pragmatic industry solution is the hybrid architecture:

• YOLO undertakes core real-time inspection tasks
• Multimodal large models provide auxiliary value in annotation, fallback verification and automated report writing
• Leverage the respective strengths of both technologies while maintaining cost control

Revisiting the opening question: Can multimodal large models replace YOLO?

After two years of hands-on trial and error, our conclusion is clear:

This is the wrong framing of the question.

It is not a zero-sum “A replaces B" competition, but a matter of each technology occupying its own unique ecological niche.

Multimodal large models possess formidable capabilities, yet their core strengths — zero-shot reasoning, deep semantic understanding and broad generalization — deliver limited value for core production inspection workflows.

Meanwhile their inherent drawbacks: high latency, excessive operating costs and unstable outputs, are exactly the non-negotiable pain points industrial manufacturing cannot tolerate.

The essence of technical selection is not chasing the latest trending technology, but matching the solution to real-world scenario demands.

The YOLO series has been widely deployed across industrial visual inspection for years, and its status as the de facto standard is well justified.

Multimodal large models are powerful supplementary tools, yet they are not qualified full replacements under current technical maturity.

Perhaps in three or five years, the landscape will shift: inference speed will drastically improve, deployment costs will drop sharply, and the determinism issue will be fully resolved.

Only then can we revisit the discussion of full-scale replacement.