dog/bicycle/car example error

Question

I am getting this error:

java.lang.ArrayIndexOutOfBoundsException: Index 1 out of bounds for length 1
	at ai.djl.modality.cv.translator.SingleShotDetectionTranslator.processOutput(SingleShotDetectionTranslator.java:63)
	at ai.djl.modality.cv.translator.SingleShotDetectionTranslator.processOutput(SingleShotDetectionTranslator.java:29)
	at ai.djl.inference.Predictor.processOutputs(Predictor.java:220)
	at ai.djl.inference.Predictor.batchPredict(Predictor.java:176)
	at ai.djl.inference.Predictor.predict(Predictor.java:128)
	at SimpleObjectDetector.detectObject(SimpleObjectDetector.java:32) ---> predictor.predict(BufferedImageFactory.getInstance().fromImage(image));

code:

public class SimpleObjectDetector implements ObjectDetector<BufferedImage> {

    private final ZooModel<Image, DetectedObjects> model;

    public SimpleObjectDetector(ZooModel<Image, DetectedObjects> model) {
        this.model = model;
    }

    @Override
    public Map<String, List<BoundingBox>> detectObject(BufferedImage image)
            throws ClassificationException {
        try (Predictor<Image, DetectedObjects> predictor = model.newPredictor()) {
            DetectedObjects detectedObjects =
                    predictor.predict(BufferedImageFactory.getInstance().fromImage(image));
            Map<String, List<BoundingBox>> ret = new ConcurrentHashMap<>();

            int imageWidth = image.getWidth();
            int imageHeight = image.getHeight();

            List<DetectedObjects.DetectedObject> detections = detectedObjects.items();
            for (DetectedObjects.DetectedObject detection : detections) {
                String className = detection.getClassName();
                float probability = (float) detection.getProbability();
                Rectangle rect = detection.getBoundingBox().getBounds();

                int x = (int) (rect.getX() * imageWidth);
                int y = (int) (rect.getY() * imageHeight);
                float w = (float) (rect.getWidth() * imageWidth);
                float h = (float) (rect.getHeight() * imageHeight);

                ret.compute(
                        className,
                        (k, list) -> {
                            if (list == null) {
                                list = new ArrayList<>();
                            }
                            list.add(new BoundingBox(className, probability, x, y, w, h));
                            return list;
                        });
            }

            return ret;
        } catch (TranslateException e) {
            throw new ClassificationException("Failed to process output", e);
        }
    }
}

public class ImageDetectPlugin implements FilePlugin {

    @Override
    public Iterator getIterator(ByteSource byteSource)
    {
      ArrayList al = new ArrayList();
        try (ThreadContextClassLoader ignored = new ThreadContextClassLoader(getClass().getClassLoader())) {
            Criteria<Image, DetectedObjects> criteria =
                    Criteria.builder()
                            .setTypes(Image.class, DetectedObjects.class)
                            .optApplication(Application.CV.OBJECT_DETECTION)
                            .optArgument("threshold", 0.3)
                            .build();
            //.optArtifactId("yolo")
            System.out.println("criteria:");
            System.out.println(criteria);
            try {
                ModelZoo.listModels();
                System.out.println("inb:");
                ModelZoo.listModels(criteria);
            } catch (IOException e) {
                e.printStackTrace();
            } catch (ModelNotFoundException e) {
                e.printStackTrace();
            }
            try (ZooModel<Image, DetectedObjects> model = ModelZoo.loadModel(criteria)) {
                    System.out.println("model:");
                    System.out.println(model);
                SimpleObjectDetector objectDetector = new SimpleObjectDetector(model);
                    System.out.println("objectDetector:");
                    System.out.println(objectDetector);
                URL imageUrl = null;
                imageUrl = new URL("https://djl-ai.s3.amazonaws.com/resources/images/dog_bike_car.jpg");

                BufferedImage inputImage = null;
                inputImage = ImageIO.read(imageUrl);
                Map<String, List<BoundingBox>> result = objectDetector.detectObject(inputImage);

                for (List<BoundingBox> boundingBoxes : result.values()) {
                    for (BoundingBox boundingBox : boundingBoxes) {
                        System.out.println(boundingBox.toString());
                        al.add(Arrays.asList(boundingBox.toString()));
                    }
                }
            } catch (IOException | ModelNotFoundException | MalformedModelException e) {
                throw new RuntimeException(e.getMessage());
            }
        }
        return al.iterator();
    }
}

logs printed before the stack trace:

	stdout	criteria:
	stdout	Criteria:
	Application: CV.OBJECT_DETECTION
	Input: interface ai.djl.modality.cv.Image
	Output: class ai.djl.modality.cv.output.DetectedObjects
	Arguments: {"threshold":0.3}
	No translator supplied

	stdout	inb:
	ai.djl.repository.zoo.ModelZoo	Loading model with Criteria:
	Application: CV.OBJECT_DETECTION
	Input: interface ai.djl.modality.cv.Image
	Output: class ai.djl.modality.cv.output.DetectedObjects
	Arguments: {"threshold":0.3}
	No translator supplied

	ai.djl.repository.zoo.ModelZoo	Searching model in zoo provider: ai.djl.repository.zoo.DefaultZooProvider
	ai.djl.repository.zoo.ModelZoo	No model zoo found in zoo provider: ai.djl.repository.zoo.DefaultZooProvider
	ai.djl.repository.zoo.ModelZoo	Searching model in zoo provider: ai.djl.basicmodelzoo.BasicZooProvider
	ai.djl.repository.zoo.ModelZoo	Checking ModelLoader: ai.djl.zoo:resnet CV.IMAGE_CLASSIFICATION [
	ai.djl.zoo:resnet:0.0.2 {"layers":"50","flavor":"v1","dataset":"cifar10"}
]
	ai.djl.repository.zoo.ModelZoo	application mismatch for ModelLoader: ai.djl.zoo:resnet
	ai.djl.repository.zoo.ModelZoo	Checking ModelLoader: ai.djl.zoo:mlp CV.IMAGE_CLASSIFICATION [
	ai.djl.zoo:mlp:0.0.3 {"dataset":"mnist"}
]
	ai.djl.repository.zoo.ModelZoo	application mismatch for ModelLoader: ai.djl.zoo:mlp
	ai.djl.repository.zoo.ModelZoo	Checking ModelLoader: ai.djl.zoo:ssd CV.OBJECT_DETECTION [
	ai.djl.zoo:ssd:0.0.2 {"flavor":"tiny","dataset":"pikachu"}
]
	ai.djl.repository.Resource	Preparing artifact: zoo, ai.djl.zoo:ssd:0.0.2 {"flavor":"tiny","dataset":"pikachu"}
	ai.djl.repository.AbstractRepository	Files have been downloaded already: /Users/me/.djl.ai/cache/repo/model/cv/object_detection/ai/djl/zoo/ssd/tiny/pikachu/0.0.2
	ai.djl.util.cuda.CudaUtils	cudart library not found.
	ai.djl.mxnet.jna.LibUtils	Using cache dir: /Users/me/.djl.ai/mxnet
	ai.djl.mxnet.jna.LibUtils	Downloading libmxnet.dylib ...
	ai.djl.mxnet.jna.LibUtils	Loading mxnet library from: /Users/me/.djl.ai/mxnet/1.7.0-backport-mkl-osx-x86_64/libmxnet.dylib
	ai.djl.engine.Engine	Engine loaded from provider: MXNet
	ai.djl.engine.Engine	Found default engine: MXNet
	ai.djl.BaseModel	Try to load model from /Users/me/.djl.ai/cache/repo/model/cv/object_detection/ai/djl/zoo/ssd/tiny/pikachu/0.0.2/ssd-0000.params
	ai.djl.BaseModel	Loading saved model: pikachu-ssd parameter
	ai.djl.BaseModel	DJL model loaded successfully
	stdout	model:
	stdout	ai.djl.repository.zoo.ZooModel@7e4d43ea
	stdout	objectDetector:

[frankfliu]

@tooptoop4 I think this is model issue. For this particular case, yolo model seems not able to distinguish between cat and dog. if you set threshold to 0.2, you can see yolo model detected the dog:

 [
	class: "car", probability: 0.98366, bounds: [x=0.620, y=0.138, width=0.266, height=0.162]
	class: "bicycle", probability: 0.97306, bounds: [x=0.146, y=0.223, width=0.595, height=0.575]
	class: "cat", probability: 0.87311, bounds: [x=0.173, y=0.409, width=0.240, height=0.509]
	class: "dog", probability: 0.27398, bounds: [x=0.173, y=0.409, width=0.240, height=0.509]
]

Looks like “resnet50” did a better job on this image:

.optFilter("backbone", backbone)

[lanking520]

@tooptoop4 close this issue for now. If you have futher question about SSD model, please feel free to reopen. 👍