A product inspection model can score well on its test set and fail on the first night shift. The test photos were taken in daylight on a desk; the line camera at two in the morning sees glare off shrink-wrap and a conveyor darker than anything in training. The model is the last step. What decides whether inspection works is a set of photos from the real camera, in the real place, labelled by people who agreed on what each label means.
Photos from the camera the model will use
Take a packaged-food manufacturer near Sonipat that wants a camera on the line to say whether each pack is the right SKU, whether the seal is damaged and whether the carton label is readable. Or an FMCG distributor that wants field reps to photograph a kirana shelf and have the app say whether it matches the planogram. In both, the training photos have to come from the same source as production: the fixed line camera in one case, a rep's phone in the other.
Supplier renders, catalogue shots and a few photos taken in the office have little in common with a line camera on the night shift or a phone under a shop's tube light. Collect with the same camera, lens, distance and mounting as production. Cover the lighting across the day and across the year, including the monsoon weeks when the shop shutter stays half down. Include the old packaging still in circulation in smaller towns months after a redesign. Collect over weeks, so ordinary variation ends up in the set.
Include the boring majority. A model trained only on carefully chosen interesting photos learns that every photo is interesting.
Rare defects, collected on purpose
Defects are rare, which is why anyone inspects for them, and a set made almost entirely of acceptable units teaches a model to say acceptable. Gather defective examples from returns, quality holds, and the line during known bad periods, such as the week a sealing machine was misbehaving. If you stage defects by hand, mark those photos as staged and keep them out of the test set.
Record the real defect rate from production as well. Evaluation has to use that rate, and results have to be reported per class. A model that is right on every acceptable unit and wrong on half the defects can still produce a flattering overall number, and that number is the one that ends up in the review deck. Before anyone looks at a score, write down what pass means here: which classes, at which recall, on photos from which weeks.
Write the labelling guideline first
"Damaged" means nothing until it is defined. A dent of what size? A torn label on an intact pack? A scuff no customer would notice? The guideline is a short document with one definition per class, example photos for each, and a decision for every hard case. The hard cases are the inputs labellers will argue about, so list them as concrete photos:
- A pack half out of frame at the edge of the conveyor.
- Two packs overlapping in one frame.
- Motion blur on an otherwise clear label.
- Glare that hides the seal but not the pack.
- A shelf photo where a competitor's product sits in your facing.
- The old carton design with a correct SKU.
Pick the simplest label type that answers the business question. If the question is "is this unit acceptable", one class per photo is enough. Bounding boxes are needed only when the location of the defect matters, and outlines only when its area matters. Each step up multiplies the labelling effort, and labelling is usually the largest line in the budget.
Measuring agreement between labellers
Give the same sample of photos to two labellers. Where they disagree, the guideline is unclear, so fix the guideline. Read the disagreements, rewrite the definition, add the disputed photo to the guideline as an example, and re-label the affected photos. Agreement between two careful people is the ceiling for the model; it cannot be more consistent than its labels.
- 01Collectproduction camera
- 02Guidelineclasses, hard cases
- 03Labeltwo people on a sample
- 04Resolvefix the guideline
- 05Split and train
Add production misses
Photos the model gets wrong in production re-enter at the first step.
Pretrained vision models reduce the number of photos you need, because they already know what an edge or a texture looks like before they see your first pack. Fine-tuning on your labelled set teaches them your products and your camera. The set still decides the result. The same is true of reading forms and invoices with vision models, where accuracy breaks on the documents nobody collected.
A test set split by time, then left alone
Split the photos before training, by time or by batch, never by random photo. Consecutive frames from a line camera are near duplicates. A random split puts one frame in training and its twin in test, and the test score then measures memory. Have the most careful labellers do the test set, and do not use it to choose anything during training. Camera placement matters as much here as it does for counting and tracking in video: if the camera moves, the old test set stops describing production.
- Catalogue and desk photos
- One person, labels from memory
- Staged defects, unmarked
- Random split of near-duplicates
- Production camera, weeks of photos
- Written guideline with examples
- Real defects from returns
- Split by time, test set untouched
Where photo inspection goes wrong
Labelling stock photos, then finding on the line that the model has never seen a shadow.
One person labelled everything, has since left, and the definitions were in their head.
The class list changed halfway through and nobody re-labelled the first half. Expected: one meaning per label. Actual: two, split by date.
A random split and a test score that was never going to survive the first shift.
No route for the photos the model gets wrong in production, so the set stops growing on launch day. Every miss a supervisor overrides should be saved, labelled under the guideline and added to the set, and a model nobody watches drifts whether or not anyone is collecting them.
Checklist before training
- Photos from the production camera, across weeks and lighting conditions.
- A guideline with a definition and examples for every class and hard case.
- Real defect examples at a recorded rate, staged ones marked.
- Agreement measured on a shared sample, disagreements resolved in the guideline.
- A test set split by time or batch, labelled carefully, untouched during training.
- A route for production misses to be labelled and added.
Start with the camera. Mount it where it will stay, collect two weeks of photos from it, and write the guideline against those photos before anyone draws a single box.
