You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Account for LinkedIn Spark versioning (#551)
### What changes are proposed in this pull request, and why are they necessary?
In `TransportUDFTransformer` we have a function which decides the Scala version to use for a given Spark version. In LinkedIn we have changed the versioning scheme we use internally for Spark 3.5+, where the major version of the internal Spark distribution denotes the full OSS Spark version. E.g. For Spark 3.5.2, this would be 352.x.y (where x and y are internal versions). This PR accounts for this new versioning format.
### How was this patch tested?
Existing unit tests.
Tested the regex separately via regex101
<img width="231" alt="image" src="https://github.com/user-attachments/assets/779eb267-b839-4329-8925-a86900b7467f" />
Register "passthrough" UDFs with correct ordinal return type (#541)
* register spark groot udf with correct return type
* test showcase
* extend + refactor unit test
---------
Co-authored-by: Limian Zhang <raylmzhang@gmail.com>