Thank you for providing the shapeworks software and I have been enjoying trying to use the deepSSM model. I am getting poor validation dataset performance using both the provided femur dataset as well as my own. Attached you can see the training and validation losses for the femur example which I run using the example deep_ssm python script (I had to reduce the number of augmented examples to 1000 due to RAM limitations).
I’m not entirely sure if it is an overfitting problem or if the validation dataset is distributionally different or if there is an issue with the validation particles. I have tried adding dropout which did not seem to improve things. When visualizing the generated meshes in test mode I can see that the train particles are able to generate meshes that overlay with the femur in the image very well, however, the validation meshes do not align well at all. Are there any tips to improve the validation performance?
The first thing I would check is that your validation images are cropped and aligned well with the training images. If this is the case, then this is likely an overfitting problem. You could try randomly splitting the training and validation sets again to see if the problem persists with a different validation set. If it does, then it is likely the case that there is simply not enough training data for the model to learn to generalize. This behavior is observed with the provided femur example, given the low training set size. Increasing the number of augmented samples should help to some degree. Additional augmentation, such as intensity-based methods, could also help (this is on our list to add to ShapeWorks). Let us know how we can help further.
Thanks Jadie for your reply. I can confirm that the images align well in the world coordinates. I noticed on this webpage Femur SSM Directly from Images - ShapeWorks (sciinstitute.github.io) that the femur model validation error is quite high even with 3000 samples. So I guess the output I see is not unexpected. I will definitely try different random splits and intensity augmentations for my own dataset and see if that improves validation error.
Great let us know how it goes! In the next phase of DeepSSM in Studio, we plan to expose more hyperparameters and architecture options. Those options may help improve accuracy as well.
