topic Training Service for Customizable Image Object Detection - long runtime, no result in Artificial Intelligence Forum

Training Service for Customizable Image Object Detection - long runtime, no result

kmeersman — Tue, 18 Dec 2018 08:24:11 GMT

Dear Experts,

we are using this training service for fine tuning the default model. However, submitted jobs run for a very long time(days). Afterwards, the "jobs" folder and sub-folders remain mostly empty. There is 1 pipeline_xxx.log file but it has only 1 line: INFO:__main__:Using default checkpoint for training.

Training data:

Job submit body:

{ "jobName": "boxes_od2", "dataset": "boxes_dataset", "modelName": "boxes_od2", "iterations": "1000" }

Please advise.

Thanks & Regards,

Klaas.

Re: Training Service for Customizable Image Object Detection - long runtime, no result

kmeersman — Wed, 19 Dec 2018 09:09:52 GMT

Update:

Turns out the annotation filename was wrong. It should be "annotation" and not "annotations".

I'm surprised the job will continue to run for days even if no annotation file can be found. Also nothing in the log about a missing annotation file.

Now we are facing the next problem. For every annotation file entry we see this log entry: "row x removed as object size is too small"

We have double checked, height and width are correct and the objects are larger than 1/200th of the image size.

This is our annotation file:

IMG_0835;1008;756;Box;78;239;753;499
IMG_0836;1008;756;Box;219;242;528;358

And this is the resulting log:

INFO:__main__:Using default checkpoint for training.

INFO:check_annotations:#####################################

INFO:check_annotations:Checking Annotation File

INFO:check_annotations:#####################################

INFO:check_annotations:

INFO:check_annotations:Row 0 removed as object size is too small

INFO:check_annotations:Row 1 removed as object size is too small

INFO:check_annotations:Annotation file has passed the validity test

INFO:check_annotations:#####################################

INFO:check_annotations:Annotation Cleared

INFO:check_annotations:#####################################

INFO:check_annotations:

INFO:create_tfrecord:#####################################

INFO:create_tfrecord:Creating TFRecord

INFO:create_tfrecord:#####################################

INFO:create_tfrecord:

Re: Training Service for Customizable Image Object Detection - long runtime, no result

kmeersman — Thu, 20 Dec 2018 12:21:01 GMT

Update2:

Finally got the service working: the annotation file should be a ',' separated file and not a ';' separated file.

So if you are using Excel to create the file, make sure to use comma delimited!

Maybe this information will be helpful to someone.

Regards,

Klaas.

Re: Training Service for Customizable Image Object Detection - long runtime, no result

AndreasForster — Fri, 21 Dec 2018 08:12:34 GMT

Hello Klaas,

Thank you very much for the feedback on how you made it work.

I am sure this will be helpful for others and I will pass it on to our product group for consideration.

Many Greetings

Andreas