tuplex · LeonhardFS · Oct 22, 2023 · Oct 14, 2023 · Oct 14, 2023 · Oct 14, 2023
diff --git a/README.md b/README.md
@@ -2,8 +2,7 @@
 
 [![Build Status](https://dev.azure.com/leonhardspiegelberg/Tuplex%20-%20Open%20Source/_apis/build/status/tuplex.tuplex?branchName=master)](https://dev.azure.com/leonhardspiegelberg/Tuplex%20-%20Open%20Source/_build/latest?definitionId=2&branchName=master)
 [![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)
-![Supported python versions](https://img.shields.io/badge/python-3.7%20%7C%203.8%20%7C%203.9-blue)
-[![Gitter](https://badges.gitter.im/tuplex/community.svg)](https://gitter.im/tuplex/community?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge)
+![Supported python versions](https://img.shields.io/badge/python-3.8%20%7C%203.9%20%7C%203.10%20%7C%203.11-blue)
 [![PyPi Downloads](https://img.shields.io/pypi/dm/tuplex)](https://img.shields.io/pypi/dm/tuplex)
 
 [Website](https://tuplex.cs.brown.edu/) [Documentation](https://tuplex.cs.brown.edu/python-api.html)
@@ -62,7 +61,7 @@ pip install tuplex
 
 ### Building
 
-Tuplex is available for MacOS and Linux. The current version has been tested under MacOS 10.13-12.0 and Ubuntu 18.04/20.04/22.04 LTS.
+Tuplex is available for MacOS and Linux. The current version has been tested under MacOS 10.13+ and Ubuntu 20.04/22.04 LTS.
 To install Tuplex, simply install the dependencies first and then build the package.
 
 #### MacOS build from source
@@ -74,10 +73,10 @@ python3 setup.py install --user
 ```
 
 #### Ubuntu build from source
-To faciliate installing the dependencies for Ubuntu, we do provide two scripts (`scripts/ubuntu1804/install_reqs.sh` for Ubuntu 18.04, or `scripts/ubuntu2004/install_reqs.sh` for Ubuntu 20.04). To create an up to date version of Tuplex, simply run
+To faciliate installing the dependencies for Ubuntu, we do provide two scripts (`scripts/ubuntu2004/install_reqs.sh` for Ubuntu 20.04, or `scripts/ubuntu2204/install_reqs.sh` for Ubuntu 22.04). To create an up to date version of Tuplex, simply run
 ```
-./scripts/ubuntu1804/install_reqs.sh
-python3 -m pip install 'cloudpickle<2.0' numpy
+./scripts/ubuntu2204/install_reqs.sh
+python3 -m pip install cloudpickle numpy
 python3 setup.py install --user
 ```
 
@@ -94,17 +93,17 @@ The python package corresponding to Tuplex can be then found in `build/dist/pyth
 
 To customize the cmake build, the following options are available to be passed via `-D<option>=<value>`:
 
-| option | values | description |
-| ------ | ------ | ----------- |
+| option | values                                                                  | description |
+| ------ |-------------------------------------------------------------------------| ----------- |
 | `CMAKE_BUILD_TYPE` | `Release` (default), `Debug`, `RelWithDebInfo`, `tsan`, `asan`, `ubsan` | select compile mode. Tsan/Asan/Ubsan correspond to Google Sanitizers. |
-| `BUILD_WITH_AWS` | `ON` (default), `OFF` | build with AWS SDK or not. On Ubuntu this will build the Lambda executor. |
-| `BUILD_WITH_ORC` | `ON`, `OFF` (default) | build with ORC file format support. |
-| `BUILD_NATIVE` | `ON`, `OFF` (default) | build with `-march=native` to target platform architecture. |
-| `SKIP_AWS_TESTS` | `ON` (default), `OFF` | skip aws tests, helpful when no AWS credentials/AWS Tuplex chain is setup. |
-| `GENERATE_PDFS` | `ON`, `OFF` (default) | output in Debug mode PDF files if graphviz is installed (e.g., `brew install graphviz`) for ASTs of UDFs, query plans, ...|
-| `PYTHON3_VERSION` | `3.7`, ... | when trying to select a python3 version to build against, use this by specifying `major.minor`. To specify the python executable, use the options provided by [cmake](https://cmake.org/cmake/help/git-stage/module/FindPython3.html). |
-| `LLVM_ROOT_DIR` | e.g. `/usr/lib/llvm-9` | specify which LLVM version to use |
-| `BOOST_DIR` | e.g. `/opt/boost` | specify which Boost version to use. Note that the python component of boost has to be built against the python version used to build Tuplex |
+| `BUILD_WITH_AWS` | `ON` (default), `OFF`                                                   | build with AWS SDK or not. On Ubuntu this will build the Lambda executor. |
+| `BUILD_WITH_ORC` | `ON`, `OFF` (default)                                                   | build with ORC file format support. |
+| `BUILD_NATIVE` | `ON`, `OFF` (default)                                                   | build with `-march=native` to target platform architecture. |
+| `SKIP_AWS_TESTS` | `ON` (default), `OFF`                                                   | skip aws tests, helpful when no AWS credentials/AWS Tuplex chain is setup. |
+| `GENERATE_PDFS` | `ON`, `OFF` (default)                                                   | output in Debug mode PDF files if graphviz is installed (e.g., `brew install graphviz`) for ASTs of UDFs, query plans, ...|
+| `PYTHON3_VERSION` | `3.8`, ...                                                              | when trying to select a python3 version to build against, use this by specifying `major.minor`. To specify the python executable, use the options provided by [cmake](https://cmake.org/cmake/help/git-stage/module/FindPython3.html). |
+| `LLVM_ROOT_DIR` | e.g. `/usr/lib/llvm-9`                                                  | specify which LLVM version to use |
+| `BOOST_DIR` | e.g. `/opt/boost`                                                       | specify which Boost version to use. Note that the python component of boost has to be built against the python version used to build Tuplex |
 
 For example, to create a debug build which outputs PDFs use the following snippet:
 
@@ -134,4 +133,4 @@ series = {SIGMOD/PODS '21}
 ```
 
 ---
-(c) 2017-2022 Tuplex contributors
+(c) 2017-2023 Tuplex contributors
diff --git a/azure-pipelines.yml b/azure-pipelines.yml
@@ -13,22 +13,22 @@ pr:
     - master
 
 pool:
-  vmImage: 'ubuntu-18.04'
+  vmImage: 'ubuntu-22.04'
 jobs:
   - job: 'tuplex'
     timeoutInMinutes: 180
     steps:
-      - script: sudo bash scripts/ubuntu1804/install_mongodb.sh
+      - script: sudo bash scripts/azure/install_mongodb.sh
         displayName: 'Install MongoDB'
-      - script: sudo bash scripts/install_azure_ci_reqs.sh
+      - script: sudo bash scripts/azure/install_azure_ci_reqs.sh
         displayName: 'Install required packages'
-      - script: sudo apt-get install -y python3-setuptools ninja-build && sudo apt-get remove -y python-pexpect python3-pexpect && sudo python3.7 -m pip install --upgrade pip && sudo python3.7 -m pip uninstall -y pygments && sudo python3.7 -m pip install pytest pygments>=2.4.1 MarkupSafe==2.0 pexpect setuptools astor PyYAML jupyter nbformat pymongo eventlet==0.30.0 gunicorn pymongo && jupyter --version
+      - script: sudo apt-get install -y python3-setuptools ninja-build && python3 -m pip install pytest pygments>=2.4.1 MarkupSafe==2.0 pexpect setuptools astor PyYAML jupyter nbformat pymongo eventlet==0.30.0 gunicorn pymongo && jupyter --version
         displayName: 'Install python dependencies'
       - script: cd tuplex/python && python3 -m pip install -r requirements.txt && python3 mongodb_test.py && pkill mongod || true
         displayName: 'Test local MongoDB'
-      - script: TUPLEX_BUILD_ALL=1 CMAKE_ARGS="-DBUILD_WITH_ORC=ON -DLLVM_ROOT_DIR=/usr/lib/llvm-9 -DCMAKE_BUILD_TYPE=Release -DBUILD_FOR_CI=ON" python3 setup.py install --user
+      - script: TUPLEX_BUILD_ALL=1 CMAKE_ARGS="-DBUILD_WITH_ORC=ON -DLLVM_ROOT_DIR=/usr/lib/llvm-16 -DCMAKE_BUILD_TYPE=Release -DBUILD_FOR_CI=ON" python3 setup.py install --user
         displayName: 'Build Tuplex'
-      - script: cd build/temp.linux-x86_64-3.7 && ctest --timeout 180 --output-on-failure
+      - script: cd build/temp.linux-x86_64-3.10 && ctest --timeout 180 --output-on-failure
         displayName: 'C++ tests'
-      - script: cd build/temp.linux-x86_64-3.7/dist/python && python3.7 -m pytest -x --full-trace -l --log-cli-level=DEBUG --capture=tee-sys
+      - script: cd build/temp.linux-x86_64-3.10/dist/python && python3.10 -m pytest -x --full-trace -l --log-cli-level=DEBUG --capture=tee-sys
         displayName: 'Python tests'
diff --git a/scripts/install_azure_ci_reqs.sh → scripts/azure/install_azure_ci_reqs.sh b/scripts/install_azure_ci_reqs.sh → scripts/azure/install_azure_ci_reqs.sh
@@ -1,9 +1,12 @@
 #!/usr/bin/env bash
-# (c) Tuplex team 2017-2022
-# auto-generated on 2022-09-20 14:15:28.013499
+# (c) Tuplex team 2017-2023
+# auto-generated on 2023-10-16 22:23:55.272907
 # install all dependencies required to compile tuplex + whatever is needed for profiling
 # everything will be installed to /opt by default
 
+
+set -euxo pipefail
+
 # need to run this with root privileges
 if [[ $(id -u) -ne 0 ]]; then
   echo "Please run this script with root privileges"
@@ -12,7 +15,20 @@ fi
 
 export DEBIAN_FRONTEND=noninteractive
 
-""" + workdir_setup() + """
+PREFIX=${PREFIX:-/opt}
+WORKDIR=${WORKDIR:-/tmp}
+
+echo ">> Installing packages into ${PREFIX}"
+mkdir -p $PREFIX && chmod 0755 $PREFIX
+mkdir -p $PREFIX/sbin
+mkdir -p $PREFIX/bin
+mkdir -p $PREFIX/share
+mkdir -p $PREFIX/include
+mkdir -p $PREFIX/lib
+
+echo ">> Files will be downloaded to ${WORKDIR}/tuplex-downloads"
+WORKDIR=$WORKDIR/tuplex-downloads
+mkdir -p $WORKDIR
 
 PYTHON_EXECUTABLE=${PYTHON_EXECUTABLE:-python3}
 PYTHON_BASENAME="$(basename -- $PYTHON_EXECUTABLE)"
@@ -36,31 +52,20 @@ WORKDIR=$WORKDIR/tuplex-downloads
 mkdir -p $WORKDIR
 
 export DEBIAN_FRONTEND=noninteractive
-# add recent python3.7 package, confer https://linuxize.com/post/how-to-install-python-3-7-on-ubuntu-18-04/
-apt install -y software-properties-common \
-&& add-apt-repository -y ppa:deadsnakes/ppa \
-&& apt-get update
-apt-get install -y build-essential autoconf automake libtool software-properties-common wget libedit-dev libz-dev \
+apt-get install -qy build-essential autoconf automake libtool software-properties-common wget libedit-dev libz-dev \
   python3-yaml pkg-config libssl-dev libcurl4-openssl-dev curl \
-  uuid-dev git python3.7 python3.7-dev python3-pip libffi-dev \
+  uuid-dev git python3.11 python3.11-dev python3-pip libffi-dev \
   doxygen doxygen-doc doxygen-latex doxygen-gui graphviz \
-  gcc-7 g++-7 libgflags-dev libncurses-dev \
-  awscli openjdk-8-jdk libyaml-dev libmagic-dev ninja-build
-# LLVM 9 packages (prob not all of them needed, but here for complete install)
+  libgflags-dev libncurses-dev \
+  awscli openjdk-11-jdk libyaml-dev libmagic-dev ninja-build
+# LLVM packages (prob not all of them needed, but here for complete install)
 wget https://apt.llvm.org/llvm.sh && chmod +x llvm.sh \
-&& ./llvm.sh 9 && rm -rf llvm.sh
-# set gcc-7 as default
-update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-7 70 --slave /usr/bin/g++ g++ /usr/bin/g++-7
-# set python3.7 as default
-update-alternatives --install /usr/bin/python3 python3 /usr/bin/python3.7 70 --slave /usr/bin/python3m python3m /usr/bin/python3.7m
-# upgrade pip
-python3.7 -m pip install --upgrade pip
-
+&& ./llvm.sh 16 && rm -rf llvm.sh
 
 # fetch recent cmake & install
 CMAKE_VER_MAJOR=3
-CMAKE_VER_MINOR=23
-CMAKE_VER_PATCH=3
+CMAKE_VER_MINOR=27
+CMAKE_VER_PATCH=5
 CMAKE_VER="${CMAKE_VER_MAJOR}.${CMAKE_VER_MINOR}"
 CMAKE_VERSION="${CMAKE_VER}.${CMAKE_VER_PATCH}"
 URL=https://github.com/Kitware/CMake/releases/download/v${CMAKE_VERSION}/cmake-${CMAKE_VERSION}-linux-x86_64.tar.gz
@@ -88,29 +93,29 @@ pushd ${WORKDIR}/boost && wget https://boostorg.jfrog.io/artifactory/main/releas
 mkdir -p ${WORKDIR}/yamlcpp && cd ${WORKDIR}/yamlcpp \
 && git clone https://github.com/jbeder/yaml-cpp.git yaml-cpp \
 && cd yaml-cpp \
-&& git checkout tags/yaml-cpp-0.6.3 \
+&& git checkout tags/0.8.0 \
 && mkdir build && cd build \
-&& cmake -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=${prefix} -DYAML_CPP_BUILD_TESTS=OFF -DBUILD_SHARED_LIBS=OFF -DCMAKE_CXX_FLAGS="-fPIC" .. \
+&& cmake -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=${PREFIX} -DYAML_CPP_BUILD_TESTS=OFF -DBUILD_SHARED_LIBS=OFF -DCMAKE_CXX_FLAGS="-fPIC" .. \
 && make -j$(nproc) && make install
 mkdir -p ${WORKDIR}/celero && cd ${WORKDIR}/celero \
 &&  git clone https://github.com/DigitalInBlue/Celero.git celero && cd celero \
 && git checkout tags/v2.8.3 \
 && mkdir build && cd build \
-&& cmake -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=${PREFIX} -DBUILD_SHARED_LIBS=OFF -DCMAKE_CXX_FLAGS="-fPIC -std=c++11" .. \
+&& cmake -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=/opt -DBUILD_SHARED_LIBS=OFF -DCMAKE_CXX_FLAGS="-fPIC -std=c++11" .. \
 && make -j$(nproc) && make install
 mkdir -p ${WORKDIR}/antlr && cd ${WORKDIR}/antlr \
-&& curl -O https://www.antlr.org/download/antlr-4.8-complete.jar \
-&& cp antlr-4.8-complete.jar ${PREFIX}/lib/ \
-&& curl -O https://www.antlr.org/download/antlr4-cpp-runtime-4.8-source.zip \
-&& unzip antlr4-cpp-runtime-4.8-source.zip -d antlr4-cpp-runtime \
-&& rm antlr4-cpp-runtime-4.8-source.zip \
+&& curl -O https://www.antlr.org/download/antlr-4.13.1-complete.jar \
+&& cp antlr-4.13.1-complete.jar ${PREFIX}/lib/ \
+&& curl -O https://www.antlr.org/download/antlr4-cpp-runtime-4.13.1-source.zip \
+&& unzip antlr4-cpp-runtime-4.13.1-source.zip -d antlr4-cpp-runtime \
+&& rm antlr4-cpp-runtime-4.13.1-source.zip \
 && cd antlr4-cpp-runtime \
 && mkdir build && cd build && cmake -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=${PREFIX} .. \
 && make -j$(nproc) && make install
 mkdir -p ${WORKDIR}/aws && cd ${WORKDIR}/aws \
 &&  git clone --recurse-submodules https://github.com/aws/aws-sdk-cpp.git \
-&& cd aws-sdk-cpp && git checkout tags/1.9.320 && mkdir build && cd build \
-&& cmake -DCMAKE_BUILD_TYPE=Release -DUSE_OPENSSL=ON -DENABLE_TESTING=OFF -DENABLE_UNITY_BUILD=ON -DCPP_STANDARD=14 -DBUILD_SHARED_LIBS=OFF -DBUILD_ONLY="s3;core;lambda;transfer" -DCMAKE_INSTALL_PREFIX=${PREFIX} .. \
+&& cd aws-sdk-cpp && git checkout tags/1.11.164 && mkdir build && cd build \
+&& cmake -DCMAKE_BUILD_TYPE=Release -DUSE_OPENSSL=ON -DENABLE_TESTING=OFF -DENABLE_UNITY_BUILD=ON -DCPP_STANDARD=17 -DBUILD_SHARED_LIBS=OFF -DBUILD_ONLY="s3;core;lambda;transfer" -DCMAKE_INSTALL_PREFIX=${PREFIX} .. \
 && make -j$(nproc) \
 && make install
 
@@ -120,23 +125,18 @@ cd ${WORKDIR}/aws \
 && git clone https://github.com/awslabs/aws-lambda-cpp.git \
 && cd aws-lambda-cpp \
 && git fetch && git fetch --tags \
-&& git checkout v0.2.6 \
+&& git checkout v0.2.8 \
 && mkdir build \
 && cd build \
 && cmake -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=${PREFIX} .. \
 && make -j$(nproc) && make install
 mkdir -p ${WORKDIR}/pcre2 && cd ${WORKDIR}/pcre2 \
-&& curl -LO https://github.com/PhilipHazel/pcre2/releases/download/pcre2-10.39/pcre2-10.39.zip \
-&& unzip pcre2-10.39.zip \
-&& rm pcre2-10.39.zip \
-&& cd pcre2-10.39 \
+&& curl -LO https://github.com/PhilipHazel/pcre2/releases/download/pcre2-10.42/pcre2-10.42.zip \
+&& unzip pcre2-10.42.zip \
+&& rm pcre2-10.42.zip \
+&& cd pcre2-10.42 \
 && ./configure CFLAGS="-O2 -fPIC" --prefix=${PREFIX} --enable-jit=auto --disable-shared \
 && make -j$(nproc) && make install
-mkdir -p ${WORKDIR}/protobuf && cd ${WORKDIR}/protobuf \
-&& curl -OL https://github.com/protocolbuffers/protobuf/releases/download/v21.5/protobuf-cpp-3.21.5.tar.gz \
-&& tar xf protobuf-cpp-3.21.5.tar.gz \
-&& cd protobuf-3.21.5 \
-&& ./autogen.sh && ./configure "CFLAGS=-fPIC" "CXXFLAGS=-fPIC" \
-&& make -j$(nproc) && make install && ldconfig
-pip3 install 'cloudpickle<2.0.0' cython numpy
+mkdir -p ${WORKDIR}/protobuf && cd ${WORKDIR}/protobuf && git clone -b v24.3 https://github.com/protocolbuffers/protobuf.git && cd protobuf && git submodule update --init --recursive && mkdir build && cd build && cmake -DCMAKE_CXX_FLAGS="-fPIC" -DCMAKE_CXX_STANDARD=17 -Dprotobuf_BUILD_TESTS=OFF .. && make -j$(nproc) && make install && ldconfig
+pip3 install 'cloudpickle>2.0.0' cython numpy
 echo ">>> installing reqs done."
diff --git a/scripts/azure/install_mongodb.sh b/scripts/azure/install_mongodb.sh
@@ -0,0 +1,16 @@
+#!/usr/bin/env bash
+#(c) 2017-2023 Tuplex team
+
+set -euxo pipefail
+
+apt-get update -y
+
+apt install -y wget curl gnupg2 software-properties-common apt-transport-https ca-certificates lsb-release
+
+curl -fsSL https://www.mongodb.org/static/pgp/server-6.0.asc| gpg --dearmor -o /etc/apt/trusted.gpg.d/mongodb-6.gpg
+
+echo "deb [ arch=amd64,arm64 ] https://repo.mongodb.org/apt/ubuntu $(lsb_release -cs)/mongodb-org/6.0 multiverse" | tee /etc/apt/sources.list.d/mongodb-org-6.0.list
+
+apt update -y
+
+apt install -y mongodb-org
diff --git a/scripts/deprecated/build_colab_wheel.sh b/scripts/deprecated/build_colab_wheel.sh
diff --git a/scripts/deprecated/copy_to_docker.sh b/scripts/deprecated/copy_to_docker.sh
diff --git a/scripts/deprecated/count-lines.sh b/scripts/deprecated/count-lines.sh