这是indexloc提供的服务，不要输入任何密码

[FEATURE] Add html preview of dataset #80

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Open

KorlaMarch wants to merge 62 commits into tuplex:master from KorlaMarch:jupyter_html

Collaborator

KorlaMarch commented Jan 28, 2022

This PR is still work-in-progress. It's not a complete feature yet!
Currently the preview is done by __html__() function in Dataset.py, which is a pure python implementation using data from collect(). The problem is that it is really slow on the large dataset

I am in the middle of adding takeLast() operator to C++ backend, which will speed up the computation (hopefully!).

KorlaMarch force-pushed the jupyter_html branch from 11d5441 to a577dcb Compare

February 16, 2022 04:43

KorlaMarch added 7 commits

February 22, 2022 21:44


          Modify dataset

1f1d7bb


          Add in takeLast operator

0b1e767


          (wip) add reverse limit in partition

eafb76d


          Remove row count

cb47a4d


          refactor TakeOperator

d879bcd


          Add unit tests

a6f31dd


          add bottom limit to transform stage (wip)

89cee2e

KorlaMarch force-pushed the jupyter_html branch from a8c3b2e to 89cee2e Compare

February 23, 2022 02:53


          more physical stage update (wip)

07b87fd

LeonhardFS reviewed

View reviewed changes

tuplex/core/include/DataSet.h Outdated Show resolved Hide resolved

LeonhardFS reviewed

View reviewed changes

tuplex/core/include/Partition.h Outdated Show resolved Hide resolved

LeonhardFS reviewed

View reviewed changes

tuplex/core/include/Partition.h Outdated Show resolved Hide resolved

LeonhardFS reviewed

View reviewed changes

tuplex/core/include/Partition.h Show resolved Hide resolved

LeonhardFS reviewed

View reviewed changes

tuplex/core/include/logical/TakeOperator.h Outdated Show resolved Hide resolved

KorlaMarch added 2 commits

March 4, 2022 11:02


          Quick push

b2beb88


          Rework LocalBackend and TransformTask to support top and bottom limit

3e1d243

LeonhardFS reviewed

View reviewed changes

tuplex/core/include/Executor.h Outdated Show resolved Hide resolved

LeonhardFS reviewed

View reviewed changes

tuplex/core/include/Executor.h Outdated Show resolved Hide resolved

LeonhardFS reviewed

View reviewed changes

tuplex/core/include/EmptyDataset.h Outdated Show resolved Hide resolved

LeonhardFS reviewed

View reviewed changes

tuplex/core/include/ee/local/LocalBackend.h Outdated Show resolved Hide resolved

LeonhardFS reviewed

View reviewed changes

tuplex/core/include/ee/local/LocalBackend.h Outdated Show resolved Hide resolved

LeonhardFS reviewed

View reviewed changes

tuplex/core/include/DataSet.h Outdated Show resolved Hide resolved

LeonhardFS reviewed

View reviewed changes

tuplex/core/include/logical/TakeOperator.h Outdated Show resolved Hide resolved

LeonhardFS reviewed

View reviewed changes

tuplex/core/include/logical/TakeOperator.h Outdated Show resolved Hide resolved

LeonhardFS reviewed

View reviewed changes

tuplex/core/include/logical/TakeOperator.h Outdated Show resolved Hide resolved

LeonhardFS reviewed

View reviewed changes

tuplex/core/include/physical/ResultSet.h Outdated Show resolved Hide resolved

LeonhardFS reviewed

View reviewed changes

tuplex/core/include/physical/StageBuilder.h Outdated Show resolved Hide resolved

LeonhardFS reviewed

View reviewed changes

tuplex/core/include/physical/StageBuilder.h Outdated Show resolved Hide resolved

LeonhardFS reviewed

View reviewed changes

tuplex/core/include/physical/TransformStage.h Outdated Show resolved Hide resolved

KorlaMarch added 5 commits

April 20, 2022 11:49


          Add file testcases

72b6580


          Python Dataset Debug

172d6b5


          Remove showHTMLPreview from Dataset in C++

a2d4178


          Separate out partition utils

993937d


          Fix Azure pipeline failing

6f528f8

KorlaMarch force-pushed the jupyter_html branch from a072e40 to 6f528f8 Compare

April 20, 2022 15:50

KorlaMarch changed the title ~~[FEATURE] Add html preview of dataset (wip)~~ [FEATURE] Add html preview of dataset

LeonhardFS and others added 5 commits

April 20, 2022 11:51


          Merge branch 'jupyter_html' of github.com:KorlaMarch/tuplex into jupy…

4a328cc

…ter_html


          Minor Debug in Python lib

816567f


          Remove column counts

4b2e2af


          Merge branch 'tuplex:master' into jupyter_html

01e12ba


          Fix CI not running core tests

a935f1e

LeonhardFS reviewed

View reviewed changes

tuplex/test/core/TakeTest.cc Outdated Show resolved Hide resolved

LeonhardFS reviewed

View reviewed changes

tuplex/test/core/TakeTest.cc Show resolved Hide resolved

KorlaMarch and others added 4 commits

May 13, 2022 17:22


          Speed up tests

359ffed


          Merge branch 'master' into jupyter_html

9fc4382


          Disable the limit merge test

d36e4b2


          Fix the wrapper test

2ed45d5

KorlaMarch mentioned this pull request

[BUG] take() with limit return too many items in exception case #103

Closed

KorlaMarch requested a review from LeonhardFS

May 18, 2022 19:37

Collaborator Author

KorlaMarch commented May 18, 2022

test is passing now, so this is ready to merge whenever you're ready @LeonhardFS

LeonhardFS and others added 9 commits

May 19, 2022 21:28


          Merge branch 'jupyter_html' of github.com:KorlaMarch/tuplex into jupy…

588382a

…ter_html


          Merge branch 'master' of github.com:tuplex/tuplex into jupyter_html

28a9064


          Fix Typo (and rerun CI)


          Add logging after load and transform task

2099d7a


          Fix missing completed work issue

809b87d


          Merge branch 'master' into jupyter_html

1088bc3


          Resolve merge conflict

bfb56a3


          Resolve merge conflict

71e0fe5


          update partition grouping when trim partitions

ee5ff30

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet