Looks good! Thank you.
Have you tried running any experiments on Movielense?

I did, here the results so far using 1 NVIDIA P100:

*10 epochs,1gpu,bs=128,lr=0.005,hidden=512,512,1024,tr_time=1113s(18min),RMSE=0.8869814801625296
*50 epochs,1gpu,bs=256,lr=0.005,hidden=512,512,1024,tr_time=5026s(1.4h),RMSE=0.8500938244957209
*200 epochs,1gpu,bs=256,lr=0.005,hidden=512,512,1024,tr_time=19456s(5.4h),RMSE=0.8430963735001938

The rmse is lower than if you do ALS with a rank of 12, which I got 0.80. The next thing I will do is to try different layers. Have you done experiments with configurations in a diminishing order? like 512,512,256 or 512,256,128?

I noticed that when doing ALS if I use a rank of 200 I get RMSE of 0.97, but if I reduce the rank to 12 I get near 0.8. So maybe in this dataset we need a smaller latent space.

What is you experience on this?

If ALS of rank 12 gets 0.80, I would try DeepRecommender with an architecture of 12,12.
Making first layer large (e.g. 512,....) is dangerous because it can trigger severe over-fitting, because then first fully connected layer will be of size num_movies_x_512.
In fact, even with Netflix, the most time I spent fighting over-fitting.

ok, thank you for the guidance, I'll do experiments on my end and I will share them with you.