When having emitDecoratorMetadata enabled and @swc/core installed the resulting bundle that gets produced by tsup will for unknown reason not work properly when debugging in VSCode

I have made a simple reproduction repo: https://github.com/favna/improved-palm-tree
Steps to reproduce:

1. Clone repo
2. Go into the with-swc and without-swc folders either sequentially or in different terminal tabs
3. Run yarn to install modules
4. Run yarn build to compile TS code
5. In both folders open src/index.ts and set a VSCode breakpoint on line 2
6. Go to the debugger tab in VSCode, here you will be able to select 2 predefined debug configs (configured in .vscode/launch.json):
   
7. First launch the version without swc, you should now get stopped on the breakpoint:
   
8. Kill it and launch the version with swc, you are not stopped on the breakpoint and the process exits, detaching the debugger

This is really quite annoying because being able to just set breakpoints in TS code is super super useful, while on the flip side using @swc/core is AFAIK a must-have when emitDecoratorMetadata is also required for the project.