Hi all,

I've adapted my training script following the tutorials and it seems all the gpu's are communicating. However, when testing it out with loading an xception model (with imagenet weights) there's some odd things happening:

1. If I set the xception model's layers to non-trainable then the weights are loaded. I know this because my model accuracy starts at somewhere near 20% accuracy. However, I'd like to train these layers to improve accuracy so,
2. If I set the xception model's layers to trainable, it seems the weights aren't loaded. Becauese my model accuracy starts at 0% and struggles to get anywhere beyond 0% even after 20 epochs. Train and validation accuracy bounce around 0% during training.

Has anyone noticed this behaviour before and had found a solution? (Or am I missing something that's not in the basics tutorials). My problem is highly imbalanced image multi-class classification so I'd expect pretty low accuracy but I was expecting some accuracy a little higher than 0%! Especially if the Xception base model hits 20% when frozen.

A couple more details:

Tensorflow version 2.1 and 2.5,
Using tf.keras,
This is running on AWS SageMaker (4 GPU's, 1 instance).

When using frozen xception model + a dense and dropout layer:

When allowing the xception model layers to be trainable + a dense and dropout layer:

I'm gonna keep looking through the script to see I've not done anything silly but I'm out of ideas :/

Thanks,
Jonathan