Upscale2D and ConvTrans2d #603

opfromthestart · 2023-03-23T03:05:57Z

Fixes #574

Adds the two image operations.

To use upscale, add an Upscale2D<OH, OW, UpscaleType>(not yet done, but functionality is there) module, which will upscale the input image to a (B, C, OW, OH) from a (B, C, W, H) using a given type of upscaling (planned nearest neighbor and bilinear).

It also will have an Upscale2DBy<S> module which will upscale the image by an integer factor.

It will include a ConvTrans2D<K,S,P> module which will give the transpose of the Conv2D<K,S,P> operation, so having Conv2D then ConvTrans2D will result in having the same shape.

I may need some help on other upscaling algorithms and the conv transpose implementation.

PReLU forward and backward now working properly

Actually runs now

Changed some defaults to match Pytorch

just a normal pull

coreylowman · 2023-03-23T14:44:06Z

src/tensor_ops/upscale2d/mod.rs

+        impl<
+                C: Dim,
+                const H: usize,
+                const W: usize,
+                const OH: usize,
+                const OW: usize,


I'm actually thinking we don't need const dimensions for upscale. I think we can take the approach we have with broadcast, where we do:

trait Upscale<E> { fn upscale2d<OH: ConstDim, OW: ConstDim>(self) -> ...; fn try_upscale2d<OH: ConstDim, OW: ConstDim>(self) -> ...; fn upscale2d_like<OH: Dim, OW: Dim>(self, size: (OH, OW)) -> ...; fn try_upscale2d_like<OH: Dim, OW: Dim>(self, size: (OH, OW)) -> ...; }

For the trait yeah, the module still will though.

coreylowman · 2023-03-23T14:45:05Z

src/tensor_ops/upscale2d/mod.rs

+    Meth = nearest_upscale2d,
+    TryMeth = try_nearest_upscale2d


Should we support other options for how to upscale? Could accept an enum, but that'd be much more complex of an implementation

I think having an Upscale operation could be a trait, which is then implemented on Nearest and Bilinear, I'd like at least those two.

coreylowman · 2023-03-23T14:47:29Z

I think we could probably separate these two into separate PRs if thats easier for you. And then adding nn layers for both can be separate PRs as well.

coreylowman · 2023-03-23T14:48:00Z

src/tensor_ops/convtrans2d/mod.rs

Is the conv transpose still WIP or is it all good for reviewing?

It is very WIP, I'm pretty sure that the implementation is still straight from Conv

Ok, it was actually much closer to the forward direction than I thought, all I had to do is switch the index calulating code between forward and backward. I have 2 tests written for it so far.

opfromthestart · 2023-03-23T15:14:37Z

~~It may be easier as multiple PRs but it isn't really usable until all of those are done, so I think it makes more sense to just have a large PR with all of it.~~

Uses trait bounds instead of individually names types. Formatting

This should be the last non-fix commit

opfromthestart · 2023-03-24T20:19:53Z

I think this is now officially in the tesing phase, all of the things that needed done are done.

coreylowman · 2023-03-24T21:04:16Z

Awesome work, I'll check this out tomorrow!

coreylowman

This is looking great! Really coming along. Just a few changes that I think would help. Let me know if you want me to help and I can throw some PRs into your branch

coreylowman · 2023-03-26T18:06:52Z

src/nn/mod.rs

@@ -256,6 +259,7 @@ pub mod modules {
        TransformerEncoder, TransformerEncoderBlock,
    };
    pub use super::unbiased_linear::UnbiasedLinear;
+    pub use super::upscale::Upscale2D;


You'll need to add to the builders mod below as well

I did this for ConvTrans2D, but neither upscale has a builder.

Ah yeah, but currently builders re-exports modules if there's no existing builder for it. The intent is someone can either do use dfdx::modules::* or dfdx::builders::* and get everything either way.

coreylowman · 2023-03-26T18:07:47Z

src/nn/upscale.rs

+
+#[cfg(feature = "nightly")]
+#[derive(Debug, Default, Clone)]
+pub struct Upscale2DBy<const H: usize, const W: usize = H, M: UpscaleMethod = NearestNeighbor>(M);


src/tensor_ops/convtrans2d/cuda_kernel.rs

coreylowman · 2023-03-26T18:12:43Z

src/tensor_ops/upscale2d/upscale2d.cu

+    float h_scale = ((float)(inp_strides[1]/inp_strides[2]))/(out_strides[1]/out_strides[2]);
+    float w_scale = ((float)(inp_strides[2]/inp_strides[3]))/(out_strides[2]/out_strides[3]);


Have you tried this with broadcasted tensors? Should this be using the shape instead of strides?

It should be yes.

This should be fixed.

coreylowman · 2023-03-26T18:14:33Z

src/tensor_ops/upscale2d/upscale2d.cu

+    while (oh_s*h_scale < y) {
+        oh_s++;
+    }


Is this size_t oh_s = y / h_scale + 1?

~~I dont think so if h_scale divides y.~~

src/nn/upscale.rs

src/tensor_ops/mod.rs

coreylowman · 2023-03-26T18:25:11Z

src/tensor_ops/convtrans2d/mod.rs

+    /// print(x.grad)
+    /// print(torch.swapaxes(w.grad, 0, 1))
+    /// ```
+    fn convtrans2d_test() {


any thoughts on more easily writing tests & increasing code coverage for complex operations like conv2d & conv transpose? They are some of the more complicated ones to test (along with transformers), and the current method we have for testing leaves a lot to be desired

I think as long as the tests have a tensor with all dimensions of input and output >1 and both the output and grads match with pytorch, I think its fine. Were you thinking more along the lines of tests that don't rely on pytorch/ are randomly generated?

Yeah exactly, and in a way where we can test larger tensors. Mostly thinking about testing conv2d in cuda. The kernels are fairly complex, and some issues might not be caught unless we have fairly large sizes on things.

Co-authored-by: Corey Lowman <coreylowman@users.noreply.github.com>

coreylowman · 2023-03-29T13:47:34Z

@opfromthestart looks like this contains the changes for leakyrelu/prelu, can you remove those so they stay separate?

coreylowman

This looks good to me - once the prelu and leakyrelu are removed i'll merge! Will probably be a bunch of other updates to apply to this separately to this pr including:

Updating conv transpose kernels & launching based on recent updates to conv2d ones
Making both conv transpose & upscale compatible with runtime dimensions

coreylowman · 2023-03-30T12:35:55Z

Awesome work @opfromthestart!! Super great contribution 🚀

opfromthestart added 16 commits March 19, 2023 00:55

PReLU forward works-ish

f1ce9b5

Doctest works now

068183c

Merge branch 'coreylowman:main' into main

ef62baa

Better-ish implementation

f2fe0c4

PReLU forward and backward now working properly

Merge branch 'main' of github.com:opfromthestart/dfdx into main

94f7a20

Now has actual layers to use

f3ef318

fmt, clippy, tests

b132fcb

Cleaning

a42e0d2

Remove one unneeded generic

c248c7e

Cuda maybe working (idk)

4caa8d8

Actually runs now

LeakyReLU now should work

d9c1346

Changed some defaults to match Pytorch

Fix test

641f429

Merge branch 'main' of github.com:coreylowman/dfdx into real_origin/head

22ef1f9

just a normal pull

Base things

8cf7ac0

Upscale operation now added

7d11365

test with uneven upsize

92d92c6

coreylowman reviewed Mar 23, 2023

View reviewed changes

opfromthestart added 6 commits March 23, 2023 11:35

Interpolation is now a trait bound (fmt)

f71b82e

Uses trait bounds instead of individually names types. Formatting

Added test for convtrans2d

2bc6c82

Convtrans tests and impls

a3bb955

ConvTrans and Upscale modules

14307e7

Format & 4d

9778a5d

Upscale binilear works

2bab567

This should be the last non-fix commit

opfromthestart changed the title ~~[WIP] Upscale2D and ConvTrans2d~~ Upscale2D and ConvTrans2d Mar 24, 2023

Fix of things not being found

1c4d081

put imports under flag

a9e1838

coreylowman requested changes Mar 26, 2023

View reviewed changes

opfromthestart and others added 12 commits March 27, 2023 21:20

Fix 1

95b5ad5

Co-authored-by: Corey Lowman <coreylowman@users.noreply.github.com>

Fix 2

17b1803

Co-authored-by: Corey Lowman <coreylowman@users.noreply.github.com>

Fix 3

872357a

Co-authored-by: Corey Lowman <coreylowman@users.noreply.github.com>

Fix 4

31c207b

Co-authored-by: Corey Lowman <coreylowman@users.noreply.github.com>

Fixes for the rest

45cb9f6

Formatting

b635dfe

Merge branch 'real_origin/head' into conv2dtranspose

c2ddb6f

Merge branch 'coreylowman:main' into conv2dtranspose

0d01b52

Fix no-std Vec

40b56ad

Clippy fixes and own error

c6b0266

fmt

3a5d35b

Another clippy fix

69e93e8

coreylowman approved these changes Mar 29, 2023

View reviewed changes

opfromthestart added 3 commits March 29, 2023 18:21

remove prelu

2424b57

unused imports

6bee6f7

Forating

d9397b8

coreylowman merged commit ec72be1 into coreylowman:main Mar 30, 2023

		float h_scale = ((float)(inp_strides[1]/inp_strides[2]))/(out_strides[1]/out_strides[2]);
		float w_scale = ((float)(inp_strides[2]/inp_strides[3]))/(out_strides[2]/out_strides[3]);

Uh oh!

Upscale2D and ConvTrans2d #603

Upscale2D and ConvTrans2d #603

Uh oh!

Conversation

opfromthestart commented Mar 23, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

coreylowman commented Mar 23, 2023

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

opfromthestart Mar 23, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

opfromthestart commented Mar 23, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

opfromthestart commented Mar 24, 2023

Uh oh!

coreylowman commented Mar 24, 2023

Uh oh!

coreylowman left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

opfromthestart Mar 28, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

coreylowman commented Mar 29, 2023

Uh oh!

coreylowman left a comment

Choose a reason for hiding this comment

Uh oh!

coreylowman commented Mar 30, 2023

Uh oh!

Uh oh!

opfromthestart commented Mar 23, 2023 •

edited

Loading

opfromthestart Mar 23, 2023 •

edited

Loading

opfromthestart commented Mar 23, 2023 •

edited

Loading

opfromthestart Mar 28, 2023 •

edited

Loading