Start from observed errors
Define a difficult example through a concrete failure pattern rather than a vague request for challenging images. Examples include overlapping tools, reflective containers, or nearly identical packaging. Describe the mistaken prediction and the visual evidence a reviewer can verify. This turns model feedback into a sourcing instruction that does not depend on private intuition.
Separate difficulty from corruption
A small visible object may be useful, while an unreadable or broken image may simply be unsuitable. State minimum usability requirements alongside the desired challenge. Keep an uncertainty category when even a careful human cannot determine the target. Training and evaluation should not silently reward guesses that the provided image cannot support with visible evidence.
Keep the challenge set interpretable
Tag the condition being tested and allow more than one condition per image. Low light, occlusion, and clutter may coexist. Report the number of independent examples per condition and the degree of overlap. Avoid presenting a deliberately difficult collection as a prevalence estimate for ordinary use; it serves a different evaluation purpose. Keep the original failure examples attached to the acquisition rationale so later reviewers can tell whether the new material addresses the same observable issue.
Acquire in focused rounds
Review a small round before commissioning many similar images. Confirm that the examples produce the intended ambiguity without introducing an unrelated capture artifact. Compare the new material with existing source groups and update the coverage plan. Project-specific sourcing and sample review establish whether a proposed hard-case collection fits the intended model task and permitted uses.
Review checklist
- Describe the observed error in visible terms.
- Separate difficult examples from unusable files.
- Tag challenge conditions independently.
- Review each acquisition round before expansion.