[2021-03-31 14:16:43.294] [global] [error] could not find backend, defaulting to local
[2021-03-31 14:16:43.294] [local ee] [info] loaded runtime library from/usr/local/lib/python3.6/site-packages/Tuplex-0.1.7-py3.6.egg/tuplex/libexec/tuplex_runtime.so
[2021-03-31 14:16:43.294] [local ee] [info] initializing LLVM backend
[2021-03-31 14:16:43.294] [local ee] [warning] init JIT compiler also only in local mode
[2021-03-31 14:16:43.294] [LLVM] [info] compiling code for skylake-avx512
[2021-03-31 14:16:43.295] [memory] [info] allocated bitmap managed memory region (60.00 GB, 32.00 MB block size)
[2021-03-31 14:16:43.295] [memory] [info] allocated bitmap managed memory region (60.00 GB, 32.00 MB block size)
[2021-03-31 14:16:43.295] [memory] [info] allocated bitmap managed memory region (60.00 GB, 32.00 MB block size)
[2021-03-31 14:16:43.295] [local execution engine] [info] started local executor E/1 (60.00 GB, 32.00 MB default partition size)
[2021-03-31 14:16:43.295] [E/1] [info] starting detached process queue
[2021-03-31 14:16:43.295] [local execution engine] [info] started local executor E/2 (60.00 GB, 32.00 MB default partition size)
[2021-03-31 14:16:43.295] [E/2] [info] starting detached process queue
[2021-03-31 14:16:43.295] [E/1] [info] initialized runtime memory (4.00 MB)
[2021-03-31 14:16:43.295] [E/2] [info] initialized runtime memory (4.00 MB)
[2021-03-31 14:16:43.295] [local execution engine] [info] started local executor E/3 (60.00 GB, 32.00 MB default partition size)
[2021-03-31 14:16:43.295] [E/3] [info] starting detached process queue
[2021-03-31 14:16:43.295] [memory] [info] allocated bitmap managed memory region (60.00 GB, 32.00 MB block size)
[2021-03-31 14:16:43.295] [local execution engine] [info] started driver (60.00 GB, 32.00 MB default partition size)
overflow will be cached at file:///results/scratch/driver
[2021-03-31 14:16:43.295] [E/3] [info] initialized runtime memory (4.00 MB)
[2021-03-31 14:16:43.296] [fileinputoperator] [info] found 23 files (5.89 GB) to process.
[2021-03-31 14:16:43.296] [global] [info] sampled file:///data/flights_small/flights_on_time_performance_2019_11.csv on 256.00 KB
[2021-03-31 14:16:43.429] [fileinputoperator] [info] found 1 file (742.61 KB) to process.
[2021-03-31 14:16:43.429] [global] [info] sampled file:///code/benchmark/flights/data/GlobalAirportDatabase.txt on 256.00 KB
[2021-03-31 14:16:43.579] [fileinputoperator] [info] found 1 file (81.54 KB) to process.
[2021-03-31 14:16:43.579] [global] [info] sampled file:///code/benchmark/flights/data/L_CARRIER_HISTORY.csv on 81.55 KB
[2021-03-31 14:16:43.627] [global] [info] performing static typing for UDF in operator withColumn
[2021-03-31 14:16:43.634] [global] [info] performing static typing for UDF in operator withColumn
[2021-03-31 14:16:43.639] [global] [info] performing static typing for UDF in operator withColumn
[2021-03-31 14:16:43.644] [global] [info] performing static typing for UDF in operator withColumn
[2021-03-31 14:16:43.672] [global] [info] performing static typing for UDF in operator withColumn
[2021-03-31 14:16:43.690] [global] [info] performing static typing for UDF in operator withColumn
[2021-03-31 14:16:43.702] [global] [info] performing static typing for UDF in operator withColumn
[2021-03-31 14:16:43.707] [global] [info] performing static typing for UDF in operator withColumn
[2021-03-31 14:16:43.716] [global] [info] performing static typing for UDF in operator withColumn
[2021-03-31 14:16:43.727] [global] [info] performing static typing for UDF in operator withColumn
[2021-03-31 14:16:43.835] [global] [info] performing static typing for UDF in operator map
[2021-03-31 14:16:43.849] [global] [info] performing static typing for UDF in operator map
[2021-03-31 14:16:43.864] [global] [info] performing static typing for UDF in operator map
[2021-03-31 14:16:43.879] [global] [info] performing static typing for UDF in operator map
[2021-03-31 14:16:43.879] [global] [error] contains an assign operator - can't break down filter
[2021-03-31 14:16:44.000] [logical planner] [info] logical optimization took 0.150237ms
output type of specialized pipeline is: (Option[str],Option[str],Option[str],Option[i64],Option[f64],Option[f64])
is this the most outer stage?: 0
need to upgrade output type to (Option[str],Option[str],Option[str],Option[i64],Option[f64],Option[f64])
[2021-03-31 14:16:44.000] [codegen] [info] generating pipeline for (Option[str],Option[str],Option[str],Option[i64],Option[f64],Option[f64]) -> (Option[str],Option[str],Option[str],Option[i64],Option[f64],Option[f64]) (2 operators pipelined)
[2021-03-31 14:16:44.001] [codegen] [info] generating lambda function for (Option[str]) -> Option[str]
[2021-03-31 14:16:44.001] [codegen] [info] generating lambda function for (Option[str]) -> Option[str]
output type of specialized pipeline is: (Option[str],Option[str],Option[str],Option[i64],Option[f64],Option[f64])
is this the most outer stage?: 0
need to upgrade output type to (Option[str],Option[str],Option[str],Option[i64],Option[f64],Option[f64])
[2021-03-31 14:16:44.022] [codegen] [info] generating pipeline for (Option[str],Option[str],Option[str],Option[i64],Option[f64],Option[f64]) -> (Option[str],Option[str],Option[str],Option[i64],Option[f64],Option[f64]) (2 operators pipelined)
[2021-03-31 14:16:44.022] [codegen] [info] generating lambda function for (Option[str]) -> Option[str]
[2021-03-31 14:16:44.023] [codegen] [info] generating lambda function for (Option[str]) -> Option[str]
output type of specialized pipeline is: (Option[str],Option[str],str,i64,Option[i64])
is this the most outer stage?: 0
need to upgrade output type to (Option[str],Option[str],str,i64,Option[i64])
[2021-03-31 14:16:44.043] [codegen] [info] generating pipeline for (Option[str],Option[str]) -> (Option[str],Option[str],str,i64,Option[i64]) (3 operators pipelined)
[2021-03-31 14:16:44.043] [codegen] [info] generating lambda function for ((Option[str],Option[str])) -> str
[2021-03-31 14:16:44.044] [codegen] [info] generating lambda function for ((Option[str],Option[str],str)) -> i64
[2021-03-31 14:16:44.045] [codegen] [info] generating function extractDefunctYear for ((Option[str],Option[str],str,i64)) -> Option[i64]
output type of specialized pipeline is: (str)
is this the most outer stage?: 1
[2021-03-31 14:16:44.065] [codegen] [info] generating pipeline for (Option[i64],Option[i64],Option[i64],Option[i64],Option[str],Option[i64],Option[str],Option[str],Option[str],Option[str],Option[i64],Option[f64],Option[f64],Option[f64],Option[i64],Option[f64],Option[f64],Option[str],Option[f64],Option[f64],Option[f64],Option[f64],Option[f64],Option[f64],Option[f64],Option[f64],Option[f64],Option[f64],Option[f64],Option[f64]) -> (str) (151 operators pipelined)
outSchema is different than last operator's schema:
outSchema: (str)
last Op: (str,Option[str],Option[i64],Option[i64],Option[i64],Option[i64],Option[i64],str,str,Option[str],Option[f64],Option[f64],Option[i64],str,str,Option[str],Option[f64],Option[f64],Option[i64],f64,Option[str],boolean,boolean,Option[str],Option[str],i64,i64,i64,i64,i64,i64,i64,i64,i64,i64,i64,i64,i64,Option[i64])
[2021-03-31 14:16:44.066] [codegen] [info] generating lambda function for ((Option[i64],Option[i64],Option[i64],Option[i64],Option[str],Option[i64],Option[str],Option[str],Option[str],Option[str],Option[i64],Option[f64],Option[f64],Option[f64],Option[i64],Option[f64],Option[f64],Option[str],Option[f64],Option[f64],Option[f64],Option[f64],Option[f64],Option[f64],Option[f64],Option[f64],Option[f64],Option[f64],Option[f64],Option[f64])) -> str
[2021-03-31 14:16:44.067] [codegen] [info] generating lambda function for ((Option[i64],Option[i64],Option[i64],Option[i64],Option[str],Option[i64],Option[str],Option[str],Option[str],Option[str],Option[i64],Option[f64],Option[f64],Option[f64],Option[i64],Option[f64],Option[f64],Option[str],Option[f64],Option[f64],Option[f64],Option[f64],Option[f64],Option[f64],Option[f64],Option[f64],Option[f64],Option[f64],Option[f64],Option[f64],str)) -> str
[2021-03-31 14:16:44.069] [codegen] [info] generating lambda function for ((Option[i64],Option[i64],Option[i64],Option[i64],Option[str],Option[i64],Option[str],Option[str],Option[str],Option[str],Option[i64],Option[f64],Option[f64],Option[f64],Option[i64],Option[f64],Option[f64],Option[str],Option[f64],Option[f64],Option[f64],Option[f64],Option[f64],Option[f64],Option[f64],Option[f64],Option[f64],Option[f64],Option[f64],Option[f64],str,str)) -> str
[2021-03-31 14:16:44.070] [codegen] [info] generating lambda function for ((Option[i64],Option[i64],Option[i64],Option[i64],Option[str],Option[i64],Option[str],Option[str],Option[str],Option[str],Option[i64],Option[f64],Option[f64],Option[f64],Option[i64],Option[f64],Option[f64],Option[str],Option[f64],Option[f64],Option[f64],Option[f64],Option[f64],Option[f64],Option[f64],Option[f64],Option[f64],Option[f64],Option[f64],Option[f64],str,str,str)) -> str
[2021-03-31 14:16:44.071] [codegen] [info] generating lambda function for (Option[i64]) -> Option[str]
[2021-03-31 14:16:44.072] [codegen] [info] generating lambda function for (Option[i64]) -> Option[str]
[2021-03-31 14:16:44.073] [codegen] [info] generating function cleanCode for ((Option[i64],Option[i64],Option[i64],Option[i64],Option[str],Option[i64],Option[str],Option[str],Option[str],Option[str],Option[str],Option[f64],Option[f64],Option[f64],Option[str],Option[f64],Option[f64],Option[str],Option[f64],Option[f64],Option[f64],Option[f64],Option[f64],Option[f64],Option[f64],Option[f64],Option[f64],Option[f64],Option[f64],Option[f64],str,str,str,str)) -> Option[str]
[2021-03-31 14:16:44.075] [codegen] [info] generating lambda function for (Option[f64]) -> boolean
[2021-03-31 14:16:44.076] [codegen] [info] generating lambda function for (Option[f64]) -> boolean
[2021-03-31 14:16:44.077] [codegen] [info] generating function divertedUDF for ((Option[i64],Option[i64],Option[i64],Option[i64],Option[str],Option[i64],Option[str],Option[str],Option[str],Option[str],Option[str],Option[f64],Option[f64],Option[f64],Option[str],Option[f64],boolean,Option[str],boolean,Option[f64],Option[f64],Option[f64],Option[f64],Option[f64],Option[f64],Option[f64],Option[f64],Option[f64],Option[f64],Option[f64],str,str,str,str)) -> Option[str]
[2021-03-31 14:16:44.078] [codegen] [info] generating function fillInTimesUDF for ((Option[i64],Option[i64],Option[i64],Option[i64],Option[str],Option[i64],Option[str],Option[str],Option[str],Option[str],Option[str],Option[f64],Option[f64],Option[f64],Option[str],Option[f64],boolean,Option[str],boolean,Option[f64],Option[f64],Option[f64],Option[f64],Option[f64],Option[f64],Option[f64],Option[f64],Option[f64],Option[f64],Option[f64],str,str,str,str,Option[str])) -> Option[f64]
[2021-03-31 14:16:44.079] [codegen] [info] generating function filterDefunctFlights for ((Option[i64],Option[i64],Option[i64],Option[i64],Option[i64],Option[str],Option[str],Option[str],Option[str],Option[str],Option[f64],Option[f64],Option[f64],Option[str],Option[f64],boolean,Option[str],boolean,Option[f64],Option[f64],Option[f64],Option[f64],Option[f64],Option[f64],Option[f64],Option[f64],Option[f64],Option[f64],Option[f64],str,str,str,str,Option[str],Option[str],Option[str],str,i64,Option[i64])) -> boolean
[2021-03-31 14:16:44.081] [codegen] [info] generating lambda function for (Option[f64]) -> f64
[2021-03-31 14:16:44.082] [codegen] [info] generating lambda function for (str) -> str
[2021-03-31 14:16:44.083] [codegen] [info] generating lambda function for (Option[f64]) -> i64
[2021-03-31 14:16:44.085] [codegen] [info] generating lambda function for (Option[f64]) -> i64
[2021-03-31 14:16:44.086] [codegen] [info] generating lambda function for (Option[f64]) -> i64
[2021-03-31 14:16:44.087] [codegen] [info] generating lambda function for (Option[f64]) -> i64
[2021-03-31 14:16:44.089] [codegen] [info] generating lambda function for (Option[f64]) -> i64
[2021-03-31 14:16:44.090] [codegen] [info] generating lambda function for (Option[f64]) -> i64
[2021-03-31 14:16:44.091] [codegen] [info] generating lambda function for (Option[f64]) -> i64
[2021-03-31 14:16:44.093] [codegen] [info] generating lambda function for (Option[f64]) -> i64
[2021-03-31 14:16:44.094] [codegen] [info] generating lambda function for (Option[f64]) -> i64
[2021-03-31 14:16:44.095] [codegen] [info] generating lambda function for (Option[f64]) -> i64
[2021-03-31 14:16:44.096] [codegen] [info] generating lambda function for (Option[f64]) -> i64
[2021-03-31 14:16:44.098] [codegen] [info] generating lambda function for (Option[f64]) -> i64
[2021-03-31 14:16:44.099] [codegen] [info] generating lambda function for ((Option[i64],Option[i64],Option[i64],Option[i64],Option[i64],Option[str],Option[str],Option[str],i64,i64,i64,Option[str],i64,boolean,Option[str],boolean,i64,i64,i64,f64,i64,i64,i64,i64,i64,Option[f64],Option[f64],str,str,str,str,Option[str],Option[str],Option[str],str,i64,Option[i64],Option[str],Option[str],Option[str],Option[i64],Option[f64],Option[f64],Option[str],Option[str],Option[str],Option[i64],Option[f64],Option[f64])) -> (str,Option[str],Option[i64],Option[i64],Option[i64],Option[i64],Option[i64],str,str,Option[str],Option[f64],Option[f64],Option[i64],str,str,Option[str],Option[f64],Option[f64],Option[i64],f64,Option[str],boolean,boolean,Option[str],Option[str],i64,i64,i64,i64,i64,i64,i64,i64,i64,i64,i64,i64,i64,Option[i64])
[2021-03-31 14:16:44.138] [global] [info] Stage3 depends on: Stage2 Stage1 Stage0 
[2021-03-31 14:16:44.138] [global] [info] Stage2 depends on: 
[2021-03-31 14:16:44.140] [global] [info] provided option to explicitly merge bad rows in order back, however rows will be hashed. Disabling option. To silence this warning, set
tuplex.optimizer.mergeExceptionsInOrder=false
[2021-03-31 14:16:44.187] [global] [info] Optimization via LLVM passes took 0.047333 ms
[2021-03-31 14:16:44.213] [global] [info] Compiled code paths for stage 2 in 0.03 ms
[2021-03-31 14:16:44.213] [global] [info] [Transform Stage] Stage 2 compiled to x86 in 0.0747719s
[2021-03-31 14:16:44.213] [local ee] [info] split /code/benchmark/flights/data/L_CARRIER_HISTORY.csv into 1 task
[2021-03-31 14:16:44.216] [driver] [info] [Task Finished] Transform to in-memory hash table in 0.002508s (1928 normal rows, 0 exceptions, 1726 buckets)
[2021-03-31 14:16:44.216] [global] [info] [Transform Stage] Stage 2 completed 1 load&transform tasks in 0.00262773s
[2021-03-31 14:16:44.216] [global] [info] [Transform Stage] Stage 2 total wall clock time: 0.00250755s, 1928 input rows, time to process 1 row via fast path: 0.0013006ms
[2021-03-31 14:16:44.216] [global] [info] [Transform Stage] Stage 2 completed 1 sink tasks in 4.513e-06s
[2021-03-31 14:16:44.216] [global] [info] [Transform Stage] Stage 2 took 0.0774265s
[2021-03-31 14:16:44.216] [global] [info] Stage1 depends on: 
[2021-03-31 14:16:44.217] [global] [info] provided option to explicitly merge bad rows in order back, however rows will be hashed. Disabling option. To silence this warning, set
tuplex.optimizer.mergeExceptionsInOrder=false
[2021-03-31 14:16:44.240] [global] [info] Optimization via LLVM passes took 0.023549 ms
[2021-03-31 14:16:44.251] [global] [info] Compiled code paths for stage 1 in 0.01 ms
[2021-03-31 14:16:44.251] [global] [info] [Transform Stage] Stage 1 compiled to x86 in 0.0351731s
[2021-03-31 14:16:44.251] [local ee] [info] split /code/benchmark/flights/data/GlobalAirportDatabase.txt into 1 task
[2021-03-31 14:16:44.262] [E/1] [info] [Task Finished] Transform to in-memory hash table in 0.011099s (3562 normal rows, 0 exceptions, 3512 buckets)
[2021-03-31 14:16:44.262] [global] [info] [Transform Stage] Stage 1 completed 1 load&transform tasks in 0.0112552s
[2021-03-31 14:16:44.262] [global] [info] [Transform Stage] Stage 1 total wall clock time: 0.0110993s, 9300 input rows, time to process 1 row via fast path: 0.00119347ms
[2021-03-31 14:16:44.262] [global] [info] [Transform Stage] Stage 1 completed 1 sink tasks in 2.301e-06s
[2021-03-31 14:16:44.262] [global] [info] [Transform Stage] Stage 1 took 0.0464527s
[2021-03-31 14:16:44.262] [global] [info] Stage0 depends on: 
[2021-03-31 14:16:44.263] [global] [info] provided option to explicitly merge bad rows in order back, however rows will be hashed. Disabling option. To silence this warning, set
tuplex.optimizer.mergeExceptionsInOrder=false
[2021-03-31 14:16:44.287] [global] [info] Optimization via LLVM passes took 0.023481 ms
[2021-03-31 14:16:44.297] [global] [info] Compiled code paths for stage 0 in 0.01 ms
[2021-03-31 14:16:44.297] [global] [info] [Transform Stage] Stage 0 compiled to x86 in 0.0349443s
[2021-03-31 14:16:44.297] [local ee] [info] split /code/benchmark/flights/data/GlobalAirportDatabase.txt into 1 task
[2021-03-31 14:16:44.308] [E/2] [info] [Task Finished] Transform to in-memory hash table in 0.011079s (3562 normal rows, 0 exceptions, 3512 buckets)
[2021-03-31 14:16:44.308] [global] [info] [Transform Stage] Stage 0 completed 1 load&transform tasks in 0.0112292s
[2021-03-31 14:16:44.308] [global] [info] [Transform Stage] Stage 0 total wall clock time: 0.0110793s, 9300 input rows, time to process 1 row via fast path: 0.00119133ms
[2021-03-31 14:16:44.309] [global] [info] [Transform Stage] Stage 0 completed 1 sink tasks in 2.229e-06s
[2021-03-31 14:16:44.309] [global] [info] [Transform Stage] Stage 0 took 0.0462082s
[2021-03-31 14:16:44.608] [global] [info] Optimization via LLVM passes took 0.278529 ms
[2021-03-31 14:16:44.704] [global] [info] Compiled code paths for stage 3 in 0.10 ms
[2021-03-31 14:16:44.704] [global] [info] [Transform Stage] Stage 3 compiled to x86 in 0.39591s
[2021-03-31 14:16:44.705] [local ee] [info] split /data/flights_small/flights_on_time_performance_2019_11.csv into 4 tasks
[2021-03-31 14:16:44.705] [local ee] [info] split /data/flights_small/flights_on_time_performance_2019_10.csv into 4 tasks
[2021-03-31 14:16:44.705] [local ee] [info] split /data/flights_small/flights_on_time_performance_2019_09.csv into 4 tasks
[2021-03-31 14:16:44.705] [local ee] [info] split /data/flights_small/flights_on_time_performance_2019_08.csv into 4 tasks
[2021-03-31 14:16:44.705] [local ee] [info] split /data/flights_small/flights_on_time_performance_2019_07.csv into 4 tasks
[2021-03-31 14:16:44.705] [local ee] [info] split /data/flights_small/flights_on_time_performance_2019_06.csv into 4 tasks
[2021-03-31 14:16:44.705] [local ee] [info] split /data/flights_small/flights_on_time_performance_2019_05.csv into 4 tasks
[2021-03-31 14:16:44.705] [local ee] [info] split /data/flights_small/flights_on_time_performance_2019_04.csv into 4 tasks
[2021-03-31 14:16:44.705] [local ee] [info] split /data/flights_small/flights_on_time_performance_2019_03.csv into 4 tasks
[2021-03-31 14:16:44.705] [local ee] [info] split /data/flights_small/flights_on_time_performance_2019_02.csv into 3 tasks
[2021-03-31 14:16:44.705] [local ee] [info] split /data/flights_small/flights_on_time_performance_2019_01.csv into 3 tasks
[2021-03-31 14:16:44.705] [local ee] [info] split /data/flights_small/flights_on_time_performance_2018_12.csv into 4 tasks
[2021-03-31 14:16:44.705] [local ee] [info] split /data/flights_small/flights_on_time_performance_2018_11.csv into 3 tasks
[2021-03-31 14:16:44.705] [local ee] [info] split /data/flights_small/flights_on_time_performance_2018_10.csv into 4 tasks
[2021-03-31 14:16:44.705] [local ee] [info] split /data/flights_small/flights_on_time_performance_2018_09.csv into 3 tasks
[2021-03-31 14:16:44.705] [local ee] [info] split /data/flights_small/flights_on_time_performance_2018_08.csv into 4 tasks
[2021-03-31 14:16:44.705] [local ee] [info] split /data/flights_small/flights_on_time_performance_2018_07.csv into 4 tasks
[2021-03-31 14:16:44.705] [local ee] [info] split /data/flights_small/flights_on_time_performance_2018_06.csv into 4 tasks
[2021-03-31 14:16:44.705] [local ee] [info] split /data/flights_small/flights_on_time_performance_2018_05.csv into 4 tasks
[2021-03-31 14:16:44.705] [local ee] [info] split /data/flights_small/flights_on_time_performance_2018_04.csv into 4 tasks
[2021-03-31 14:16:44.705] [local ee] [info] split /data/flights_small/flights_on_time_performance_2018_03.csv into 4 tasks
[2021-03-31 14:16:44.705] [local ee] [info] split /data/flights_small/flights_on_time_performance_2018_02.csv into 3 tasks
[2021-03-31 14:16:44.705] [local ee] [info] split /data/flights_small/flights_on_time_performance_2018_01.csv into 3 tasks
[2021-03-31 14:16:45.431] [E/3] [info] [Task Finished] Transform to mem in 0.725565s (148707 normal rows, 0 exceptions)
[2021-03-31 14:16:45.434] [driver] [info] [Task Finished] Transform to mem in 0.729246s (149938 normal rows, 0 exceptions)
[2021-03-31 14:16:45.439] [E/2] [info] [Task Finished] Transform to mem in 0.734249s (150364 normal rows, 0 exceptions)
[2021-03-31 14:16:45.475] [E/1] [info] [Task Finished] Transform to mem in 0.769399s (157470 normal rows, 0 exceptions)
[2021-03-31 14:16:46.155] [E/3] [info] [Task Finished] Transform to mem in 0.723829s (149472 normal rows, 0 exceptions)
[2021-03-31 14:16:46.159] [driver] [info] [Task Finished] Transform to mem in 0.724233s (148599 normal rows, 0 exceptions)
[2021-03-31 14:16:46.166] [E/2] [info] [Task Finished] Transform to mem in 0.726325s (150234 normal rows, 0 exceptions)
[2021-03-31 14:16:46.411] [E/1] [info] [Task Finished] Transform to mem in 0.936041s (192166 normal rows, 0 exceptions)
[2021-03-31 14:16:46.879] [E/3] [info] [Task Finished] Transform to mem in 0.724255s (150752 normal rows, 0 exceptions)
[2021-03-31 14:16:46.888] [E/2] [info] [Task Finished] Transform to mem in 0.722000s (150014 normal rows, 0 exceptions)
[2021-03-31 14:16:46.890] [driver] [info] [Task Finished] Transform to mem in 0.730865s (149789 normal rows, 0 exceptions)
[2021-03-31 14:16:47.189] [E/1] [info] [Task Finished] Transform to mem in 0.778733s (159520 normal rows, 0 exceptions)
[2021-03-31 14:16:47.608] [E/3] [info] [Task Finished] Transform to mem in 0.729216s (149718 normal rows, 0 exceptions)
[2021-03-31 14:16:47.616] [driver] [info] [Task Finished] Transform to mem in 0.726318s (149809 normal rows, 0 exceptions)
[2021-03-31 14:16:47.616] [E/2] [info] [Task Finished] Transform to mem in 0.728192s (149499 normal rows, 0 exceptions)
[2021-03-31 14:16:48.226] [E/1] [info] [Task Finished] Transform to mem in 1.036794s (213804 normal rows, 0 exceptions)
[2021-03-31 14:16:48.335] [E/3] [info] [Task Finished] Transform to mem in 0.727353s (150460 normal rows, 0 exceptions)
[2021-03-31 14:16:48.341] [driver] [info] [Task Finished] Transform to mem in 0.725466s (149055 normal rows, 0 exceptions)
[2021-03-31 14:16:48.343] [E/2] [info] [Task Finished] Transform to mem in 0.727004s (149667 normal rows, 0 exceptions)
[2021-03-31 14:16:49.061] [E/3] [info] [Task Finished] Transform to mem in 0.725812s (148341 normal rows, 0 exceptions)
[2021-03-31 14:16:49.066] [E/2] [info] [Task Finished] Transform to mem in 0.723360s (149485 normal rows, 0 exceptions)
[2021-03-31 14:16:49.070] [driver] [info] [Task Finished] Transform to mem in 0.728442s (149179 normal rows, 0 exceptions)
[2021-03-31 14:16:49.268] [E/1] [info] [Task Finished] Transform to mem in 1.041541s (214046 normal rows, 0 exceptions)
[2021-03-31 14:16:49.792] [E/2] [info] [Task Finished] Transform to mem in 0.725194s (148852 normal rows, 0 exceptions)
[2021-03-31 14:16:49.794] [driver] [info] [Task Finished] Transform to mem in 0.724090s (149082 normal rows, 0 exceptions)
[2021-03-31 14:16:49.992] [E/1] [info] [Task Finished] Transform to mem in 0.724735s (150335 normal rows, 0 exceptions)
[2021-03-31 14:16:50.001] [E/3] [info] [Task Finished] Transform to mem in 0.939644s (193813 normal rows, 0 exceptions)
[2021-03-31 14:16:50.524] [driver] [info] [Task Finished] Transform to mem in 0.730383s (150598 normal rows, 0 exceptions)
[2021-03-31 14:16:50.717] [E/1] [info] [Task Finished] Transform to mem in 0.724497s (148500 normal rows, 0 exceptions)
[2021-03-31 14:16:50.727] [E/2] [info] [Task Finished] Transform to mem in 0.935741s (192117 normal rows, 0 exceptions)
[2021-03-31 14:16:50.728] [E/3] [info] [Task Finished] Transform to mem in 0.726982s (150602 normal rows, 0 exceptions)
[2021-03-31 14:16:51.325] [driver] [info] [Task Finished] Transform to mem in 0.800794s (166012 normal rows, 0 exceptions)
[2021-03-31 14:16:51.447] [E/1] [info] [Task Finished] Transform to mem in 0.729888s (150222 normal rows, 0 exceptions)
[2021-03-31 14:16:51.448] [E/2] [info] [Task Finished] Transform to mem in 0.721112s (149253 normal rows, 0 exceptions)
[2021-03-31 14:16:51.454] [E/3] [info] [Task Finished] Transform to mem in 0.726225s (149403 normal rows, 0 exceptions)
[2021-03-31 14:16:52.167] [E/1] [info] [Task Finished] Transform to mem in 0.720555s (149045 normal rows, 0 exceptions)
[2021-03-31 14:16:52.172] [E/2] [info] [Task Finished] Transform to mem in 0.723851s (150202 normal rows, 0 exceptions)
[2021-03-31 14:16:52.227] [driver] [info] [Task Finished] Transform to mem in 0.902208s (187016 normal rows, 0 exceptions)
[2021-03-31 14:16:52.614] [E/3] [info] [Task Finished] Transform to mem in 1.159278s (237051 normal rows, 0 exceptions)
[2021-03-31 14:16:52.891] [E/1] [info] [Task Finished] Transform to mem in 0.723465s (149583 normal rows, 0 exceptions)
[2021-03-31 14:16:52.903] [E/2] [info] [Task Finished] Transform to mem in 0.730848s (150451 normal rows, 0 exceptions)
[2021-03-31 14:16:53.342] [E/3] [info] [Task Finished] Transform to mem in 0.728479s (152368 normal rows, 0 exceptions)
[2021-03-31 14:16:53.620] [driver] [info] [Task Finished] Transform to mem in 1.392551s (287414 normal rows, 0 exceptions)
[2021-03-31 14:16:53.624] [E/1] [info] [Task Finished] Transform to mem in 0.733214s (159972 normal rows, 0 exceptions)
[2021-03-31 14:16:53.625] [E/2] [info] [Task Finished] Transform to mem in 0.721614s (149298 normal rows, 0 exceptions)
[2021-03-31 14:16:54.072] [E/3] [info] [Task Finished] Transform to mem in 0.730047s (150041 normal rows, 0 exceptions)
[2021-03-31 14:16:54.344] [driver] [info] [Task Finished] Transform to mem in 0.723625s (153103 normal rows, 0 exceptions)
[2021-03-31 14:16:54.346] [E/1] [info] [Task Finished] Transform to mem in 0.722095s (149132 normal rows, 0 exceptions)
[2021-03-31 14:16:54.796] [E/3] [info] [Task Finished] Transform to mem in 0.723810s (149649 normal rows, 0 exceptions)
[2021-03-31 14:16:55.054] [E/2] [info] [Task Finished] Transform to mem in 1.428908s (302144 normal rows, 0 exceptions)
[2021-03-31 14:16:55.069] [driver] [info] [Task Finished] Transform to mem in 0.725614s (152111 normal rows, 0 exceptions)
[2021-03-31 14:16:55.080] [E/1] [info] [Task Finished] Transform to mem in 0.733561s (160333 normal rows, 0 exceptions)
[2021-03-31 14:16:55.633] [E/3] [info] [Task Finished] Transform to mem in 0.837388s (173647 normal rows, 0 exceptions)
[2021-03-31 14:16:55.781] [E/2] [info] [Task Finished] Transform to mem in 0.727119s (150674 normal rows, 0 exceptions)
[2021-03-31 14:16:55.802] [driver] [info] [Task Finished] Transform to mem in 0.732840s (159090 normal rows, 0 exceptions)
[2021-03-31 14:16:56.371] [E/3] [info] [Task Finished] Transform to mem in 0.737350s (166361 normal rows, 0 exceptions)
[2021-03-31 14:16:56.487] [E/1] [info] [Task Finished] Transform to mem in 1.406654s (295598 normal rows, 0 exceptions)
[2021-03-31 14:16:56.509] [E/2] [info] [Task Finished] Transform to mem in 0.727740s (150449 normal rows, 0 exceptions)
[2021-03-31 14:16:56.526] [driver] [info] [Task Finished] Transform to mem in 0.724172s (148760 normal rows, 0 exceptions)
[2021-03-31 14:16:57.218] [E/1] [info] [Task Finished] Transform to mem in 0.731281s (160502 normal rows, 0 exceptions)
[2021-03-31 14:16:57.230] [E/2] [info] [Task Finished] Transform to mem in 0.721414s (148915 normal rows, 0 exceptions)
[2021-03-31 14:16:57.250] [driver] [info] [Task Finished] Transform to mem in 0.724153s (155035 normal rows, 0 exceptions)
[2021-03-31 14:16:57.344] [E/3] [info] [Task Finished] Transform to mem in 0.973324s (200968 normal rows, 0 exceptions)
[2021-03-31 14:16:57.968] [E/2] [info] [Task Finished] Transform to mem in 0.737489s (165946 normal rows, 0 exceptions)
[2021-03-31 14:16:57.980] [driver] [info] [Task Finished] Transform to mem in 0.729667s (151409 normal rows, 0 exceptions)
[2021-03-31 14:16:58.066] [E/3] [info] [Task Finished] Transform to mem in 0.721773s (148951 normal rows, 0 exceptions)
[2021-03-31 14:16:58.201] [E/1] [info] [Task Finished] Transform to mem in 0.983032s (201934 normal rows, 0 exceptions)
[2021-03-31 14:16:58.715] [driver] [info] [Task Finished] Transform to mem in 0.734945s (165562 normal rows, 0 exceptions)
[2021-03-31 14:16:58.797] [E/3] [info] [Task Finished] Transform to mem in 0.730624s (150595 normal rows, 0 exceptions)
[2021-03-31 14:16:58.849] [E/2] [info] [Task Finished] Transform to mem in 0.881811s (180829 normal rows, 0 exceptions)
[2021-03-31 14:16:58.923] [E/1] [info] [Task Finished] Transform to mem in 0.721809s (150101 normal rows, 0 exceptions)
[2021-03-31 14:16:59.534] [E/3] [info] [Task Finished] Transform to mem in 0.737130s (166720 normal rows, 0 exceptions)
[2021-03-31 14:16:59.547] [driver] [info] [Task Finished] Transform to mem in 0.832163s (171108 normal rows, 0 exceptions)
[2021-03-31 14:16:59.580] [E/2] [info] [Task Finished] Transform to mem in 0.730853s (151952 normal rows, 0 exceptions)
[2021-03-31 14:16:59.645] [E/1] [info] [Task Finished] Transform to mem in 0.721968s (148896 normal rows, 0 exceptions)
[2021-03-31 14:17:00.267] [E/3] [info] [Task Finished] Transform to mem in 0.732994s (150051 normal rows, 0 exceptions)
[2021-03-31 14:17:00.286] [driver] [info] [Task Finished] Transform to mem in 0.738889s (170336 normal rows, 0 exceptions)
[2021-03-31 14:17:00.312] [E/2] [info] [Task Finished] Transform to mem in 0.731137s (151891 normal rows, 0 exceptions)
[2021-03-31 14:17:00.366] [E/1] [info] [Task Finished] Transform to mem in 0.721095s (149675 normal rows, 0 exceptions)
[2021-03-31 14:17:01.011] [driver] [info] [Task Finished] Transform to mem in 0.724762s (150764 normal rows, 0 exceptions)
[2021-03-31 14:17:01.038] [E/2] [info] [Task Finished] Transform to mem in 0.726694s (149107 normal rows, 0 exceptions)
[2021-03-31 14:17:01.067] [E/3] [info] [Task Finished] Transform to mem in 0.800543s (164890 normal rows, 0 exceptions)
[2021-03-31 14:17:01.467] [E/1] [info] [Task Finished] Transform to mem in 1.100878s (241985 normal rows, 0 exceptions)
[2021-03-31 14:17:01.754] [driver] [info] [Task Finished] Transform to mem in 0.742988s (169851 normal rows, 0 exceptions)
[2021-03-31 14:17:01.763] [E/2] [info] [Task Finished] Transform to mem in 0.725100s (149447 normal rows, 0 exceptions)
[2021-03-31 14:17:02.393] [E/3] [info] [Task Finished] Transform to mem in 1.325737s (274592 normal rows, 0 exceptions)
[2021-03-31 14:17:02.393] [global] [info] [Transform Stage] Stage 3 completed 86 load&transform tasks in 17.6886s
[2021-03-31 14:17:02.393] [global] [info] [Transform Stage] Stage 3 total wall clock time: 68.5548s, 14009720 input rows, time to process 1 row via fast path: 0.00489337ms

[2021-03-31 14:17:02.748] [global] [info] Writing 14304381 rows as output to file (2.78 GB)
[2021-03-31 14:17:03.325] [driver] [info] [Task Finished] write to file in 0.576230s (3357303 rows, 668.88 MB)
[2021-03-31 14:17:03.381] [E/3] [info] [Task Finished] write to file in 0.632969s (3671189 rows, 730.69 MB)
[2021-03-31 14:17:03.384] [E/2] [info] [Task Finished] write to file in 0.635488s (3645801 rows, 726.29 MB)
[2021-03-31 14:17:03.827] [E/1] [info] [Task Finished] write to file in 1.079019s (3630088 rows, 723.06 MB)
[2021-03-31 14:17:03.827] [global] [info] writing output took 1.434336s
[2021-03-31 14:17:03.827] [global] [info] [Transform Stage] Stage 3 completed 86 sink tasks in 1.43439s
[2021-03-31 14:17:03.828] [global] [info] [Transform Stage] Stage 3 took 19.519s
[2021-03-31 14:17:03.828] [global] [info] Query Execution took 19.992s. (planning: 0.302834s, execution: 19.6892s)
>>> running Tuplex on ['/data/flights_small/flights_on_time_performance_2019_11.csv', '/data/flights_small/flights_on_time_performance_2019_10.csv', '/data/flights_small/flights_on_time_performance_2019_09.csv', '/data/flights_small/flights_on_time_performance_2019_08.csv', '/data/flights_small/flights_on_time_performance_2019_07.csv', '/data/flights_small/flights_on_time_performance_2019_06.csv', '/data/flights_small/flights_on_time_performance_2019_05.csv', '/data/flights_small/flights_on_time_performance_2019_04.csv', '/data/flights_small/flights_on_time_performance_2019_03.csv', '/data/flights_small/flights_on_time_performance_2019_02.csv', '/data/flights_small/flights_on_time_performance_2019_01.csv', '/data/flights_small/flights_on_time_performance_2018_12.csv', '/data/flights_small/flights_on_time_performance_2018_11.csv', '/data/flights_small/flights_on_time_performance_2018_10.csv', '/data/flights_small/flights_on_time_performance_2018_09.csv', '/data/flights_small/flights_on_time_performance_2018_08.csv', '/data/flights_small/flights_on_time_performance_2018_07.csv', '/data/flights_small/flights_on_time_performance_2018_06.csv', '/data/flights_small/flights_on_time_performance_2018_05.csv', '/data/flights_small/flights_on_time_performance_2018_04.csv', '/data/flights_small/flights_on_time_performance_2018_03.csv', '/data/flights_small/flights_on_time_performance_2018_02.csv', '/data/flights_small/flights_on_time_performance_2018_01.csv']
Welcome to

  _____            _
 |_   _|   _ _ __ | | _____  __
   | || | | | '_ \| |/ _ \ \/ /
   | || |_| | |_) | |  __/>  <
   |_| \__,_| .__/|_|\___/_/\_\ 0.1.7
            |_|
    
using Python 3.6.9 (default, Mar 31 2021, 05:47:10) 
[GCC 10.2.0] on linux
Tuplex startup time: 0.280620813369751
Tuplex job time: 20.541371822357178 s
{'tuplex.useLLVMOptimizer': True, 'tuplex.autoUpcast': False, 'tuplex.allowUndefinedBehavior': False, 'tuplex.optimizer.codeStats': False, 'tuplex.optimizer.generateParser': False, 'tuplex.optimizer.nullValueOptimization': False, 'tuplex.optimizer.filterPushdown': True, 'tuplex.optimizer.sharedObjectPropagation': True, 'tuplex.interleaveIO': True, 'tuplex.resolveWithInterpreterOnly': False, 'tuplex.csv.selectionPushdown': True, 'tuplex.webui.enable': False, 'tuplex.executorCount': 3, 'tuplex.csv.maxDetectionRows': 10000, 'tuplex.webui.port': 5000, 'tuplex.webui.mongodb.port': 27017, 'tuplex.webui.exceptionDisplayLimit': 5, 'tuplex.normalcaseThreshold': 0.9, 'tuplex.csv.comments': ['#', '~'], 'tuplex.csv.maxDetectionMemory': '256KB', 'tuplex.csv.quotechar': '"', 'tuplex.csv.separators': [',', ';', '|', '\t'], 'tuplex.driverMemory': '60G', 'tuplex.env.hostname': '6bcf8f6a24af', 'tuplex.env.mode': 'file', 'tuplex.env.user': 'root', 'tuplex.executorMemory': '60G', 'tuplex.inputSplitSize': '64MB', 'tuplex.logDir': '.', 'tuplex.optimizer.mergeExceptionsInOrder': 'False', 'tuplex.optimizer.operatorReordering': 'false', 'tuplex.optionalThreshold': '0.7', 'tuplex.partitionSize': '32MB', 'tuplex.readBufferSize': '128KB', 'tuplex.runTimeLibrary': '/usr/local/lib/python3.6/site-packages/Tuplex-0.1.7-py3.6.egg/tuplex/libexec/tuplex_runtime', 'tuplex.runTimeMemory': '64MB', 'tuplex.runTimeMemoryBlockSize': '4MB', 'tuplex.scratchDir': '/results/scratch', 'tuplex.webui.mongodb.path': '/data/tuplex_history', 'tuplex.webui.mongodb.url': 'localhost', 'tuplex.webui.url': 'localhost'}
{"logical_optimization_time_s":0.150237,"llvm_optimization_time_s":0.278529,"llvm_compilation_time_s":0.143317,"total_compilation_time_s":0.540791,"sampling_time_s":0.46644,"stages":[{"stage_no":2,"fast_path_wall_time_s":0.00250755,"fast_path_time_s":0.002638,"fast_path_per_row_time_ns":1300.6,"slow_path_wall_time_s":0,"slow_path_time_s":0,"slow_path_per_row_time_ns":0},{"stage_no":1,"fast_path_wall_time_s":0.0110993,"fast_path_time_s":0.0112664,"fast_path_per_row_time_ns":1193.47,"slow_path_wall_time_s":0,"slow_path_time_s":0,"slow_path_per_row_time_ns":0},{"stage_no":0,"fast_path_wall_time_s":0.0110793,"fast_path_time_s":0.0112508,"fast_path_per_row_time_ns":1191.33,"slow_path_wall_time_s":0,"slow_path_time_s":0,"slow_path_per_row_time_ns":0},{"stage_no":3,"fast_path_wall_time_s":68.5548,"fast_path_time_s":17.6887,"fast_path_per_row_time_ns":4893.37,"slow_path_wall_time_s":0,"slow_path_time_s":0,"slow_path_per_row_time_ns":0}]}
{"webui.enable": false, "executorMemory": "60G", "executorCount": "3", "driverMemory": "60G", "partitionSize": "32MB", "runTimeMemory": "64MB", "inputSplitSize": "64MB", "useLLVMOptimizer": true, "optimizer.nullValueOptimization": false, "csv.selectionPushdown": true, "optimizer.generateParser": false, "optimizer.mergeExceptionsInOrder": false, "optimizer.filterPushdown": true, "scratchDir": "/results/scratch"}
{"startupTime": 0.280620813369751, "jobTime": 20.541371822357178, "io_load": 0.3212554454803467, "compute_and_write": 20.22011637687683}
[2021-03-31 14:17:03.867] [local execution engine] [info] Stopped local executor E/1
[2021-03-31 14:17:03.902] [local execution engine] [info] Stopped local executor E/2
[2021-03-31 14:17:03.937] [local execution engine] [info] Stopped local executor E/3
[GLOBAL] Local engine terminated.
[GLOBAL] releasing 4 active partitions
