In the ResNet paper, the residual connection of each downsampling block is not just R(x) = F(x) + x, but R(x) = F(x) + G(x) with G being a Conv2D with 1 filter.
This can probably be easily implemented by giving 2 modules for one Residual connection (and then just adding their values).

Edit: Working on it, though it might take longer because I first have to understand all the gradients/tape stuff xD

Edit2: I think I almost finished