[2021-03-28 20:25:27.468] [global] [error] could not find backend, defaulting to local
[2021-03-28 20:25:27.469] [local ee] [info] loaded runtime library from/usr/local/lib/python3.6/site-packages/Tuplex-0.1.7-py3.6.egg/tuplex/libexec/tuplex_runtime.so
[2021-03-28 20:25:27.469] [local ee] [info] initializing LLVM backend
[2021-03-28 20:25:27.469] [local ee] [warning] init JIT compiler also only in local mode
[2021-03-28 20:25:27.469] [LLVM] [info] compiling code for skylake-avx512
[2021-03-28 20:25:27.470] [memory] [info] allocated bitmap managed memory region (6.00 GB, 32.00 MB block size)
[2021-03-28 20:25:27.470] [memory] [info] allocated bitmap managed memory region (6.00 GB, 32.00 MB block size)
[2021-03-28 20:25:27.470] [memory] [info] allocated bitmap managed memory region (6.00 GB, 32.00 MB block size)
[2021-03-28 20:25:27.470] [memory] [info] allocated bitmap managed memory region (6.00 GB, 32.00 MB block size)
[2021-03-28 20:25:27.470] [memory] [info] allocated bitmap managed memory region (6.00 GB, 32.00 MB block size)
[2021-03-28 20:25:27.470] [memory] [info] allocated bitmap managed memory region (6.00 GB, 32.00 MB block size)
[2021-03-28 20:25:27.470] [memory] [info] allocated bitmap managed memory region (6.00 GB, 32.00 MB block size)
[2021-03-28 20:25:27.470] [memory] [info] allocated bitmap managed memory region (6.00 GB, 32.00 MB block size)
[2021-03-28 20:25:27.470] [memory] [info] allocated bitmap managed memory region (6.00 GB, 32.00 MB block size)
[2021-03-28 20:25:27.470] [memory] [info] allocated bitmap managed memory region (6.00 GB, 32.00 MB block size)
[2021-03-28 20:25:27.470] [memory] [info] allocated bitmap managed memory region (6.00 GB, 32.00 MB block size)
[2021-03-28 20:25:27.470] [memory] [info] allocated bitmap managed memory region (6.00 GB, 32.00 MB block size)
[2021-03-28 20:25:27.470] [memory] [info] allocated bitmap managed memory region (6.00 GB, 32.00 MB block size)
[2021-03-28 20:25:27.470] [memory] [info] allocated bitmap managed memory region (6.00 GB, 32.00 MB block size)
[2021-03-28 20:25:27.470] [memory] [info] allocated bitmap managed memory region (6.00 GB, 32.00 MB block size)
[2021-03-28 20:25:27.470] [local execution engine] [info] started local executor E/1 (6.00 GB, 32.00 MB default partition size)
[2021-03-28 20:25:27.470] [E/1] [info] starting detached process queue
[2021-03-28 20:25:27.470] [local execution engine] [info] started local executor E/2 (6.00 GB, 32.00 MB default partition size)
[2021-03-28 20:25:27.470] [E/2] [info] starting detached process queue
[2021-03-28 20:25:27.470] [E/1] [info] initialized runtime memory (4.00 MB)
[2021-03-28 20:25:27.470] [E/2] [info] initialized runtime memory (4.00 MB)
[2021-03-28 20:25:27.470] [local execution engine] [info] started local executor E/3 (6.00 GB, 32.00 MB default partition size)
[2021-03-28 20:25:27.470] [E/3] [info] starting detached process queue
[2021-03-28 20:25:27.470] [local execution engine] [info] started local executor E/4 (6.00 GB, 32.00 MB default partition size)
[2021-03-28 20:25:27.470] [E/3] [info] initialized runtime memory (4.00 MB)
[2021-03-28 20:25:27.470] [E/4] [info] starting detached process queue
[2021-03-28 20:25:27.470] [local execution engine] [info] started local executor E/5 (6.00 GB, 32.00 MB default partition size)
[2021-03-28 20:25:27.470] [E/4] [info] initialized runtime memory (4.00 MB)
[2021-03-28 20:25:27.470] [E/5] [info] starting detached process queue
[2021-03-28 20:25:27.470] [local execution engine] [info] started local executor E/6 (6.00 GB, 32.00 MB default partition size)
[2021-03-28 20:25:27.470] [E/5] [info] initialized runtime memory (4.00 MB)
[2021-03-28 20:25:27.470] [local execution engine] [info] started local executor E/7 (6.00 GB, 32.00 MB default partition size)
[2021-03-28 20:25:27.470] [E/6] [info] starting detached process queue
[2021-03-28 20:25:27.470] [E/7] [info] starting detached process queue
[2021-03-28 20:25:27.470] [local execution engine] [info] started local executor E/8 (6.00 GB, 32.00 MB default partition size)
[2021-03-28 20:25:27.470] [E/6] [info] initialized runtime memory (4.00 MB)
[2021-03-28 20:25:27.470] [E/8] [info] starting detached process queue
[2021-03-28 20:25:27.470] [local execution engine] [info] started local executor E/9 (6.00 GB, 32.00 MB default partition size)
[2021-03-28 20:25:27.470] [E/8] [info] initialized runtime memory (4.00 MB)
[2021-03-28 20:25:27.470] [E/9] [info] starting detached process queue
[2021-03-28 20:25:27.470] [E/7] [info] initialized runtime memory (4.00 MB)
[2021-03-28 20:25:27.470] [local execution engine] [info] started local executor E/10 (6.00 GB, 32.00 MB default partition size)
[2021-03-28 20:25:27.470] [E/10] [info] starting detached process queue
[2021-03-28 20:25:27.471] [local execution engine] [info] started local executor E/11 (6.00 GB, 32.00 MB default partition size)
[2021-03-28 20:25:27.470] [E/9] [info] initialized runtime memory (4.00 MB)
[2021-03-28 20:25:27.471] [E/10] [info] initialized runtime memory (4.00 MB)
[2021-03-28 20:25:27.471] [E/11] [info] starting detached process queue
[2021-03-28 20:25:27.471] [local execution engine] [info] started local executor E/12 (6.00 GB, 32.00 MB default partition size)
[2021-03-28 20:25:27.471] [E/11] [info] initialized runtime memory (4.00 MB)
[2021-03-28 20:25:27.471] [E/12] [info] starting detached process queue
[2021-03-28 20:25:27.471] [local execution engine] [info] started local executor E/13 (6.00 GB, 32.00 MB default partition size)
[2021-03-28 20:25:27.471] [E/12] [info] initialized runtime memory (4.00 MB)
[2021-03-28 20:25:27.471] [E/13] [info] starting detached process queue
[2021-03-28 20:25:27.471] [local execution engine] [info] started local executor E/14 (6.00 GB, 32.00 MB default partition size)
[2021-03-28 20:25:27.471] [E/13] [info] initialized runtime memory (4.00 MB)
[2021-03-28 20:25:27.471] [E/14] [info] starting detached process queue
[2021-03-28 20:25:27.471] [local execution engine] [info] started local executor E/15 (6.00 GB, 32.00 MB default partition size)
[2021-03-28 20:25:27.471] [E/14] [info] initialized runtime memory (4.00 MB)
[2021-03-28 20:25:27.471] [E/15] [info] starting detached process queue
[2021-03-28 20:25:27.471] [memory] [info] allocated bitmap managed memory region (10.00 GB, 32.00 MB block size)
[2021-03-28 20:25:27.471] [local execution engine] [info] started driver (10.00 GB, 32.00 MB default partition size)
overflow will be cached at file:///tmp/driver
[2021-03-28 20:25:27.471] [E/15] [info] initialized runtime memory (4.00 MB)
[2021-03-28 20:25:27.471] [fileinputoperator] [info] found 23 files (5.89 GB) to process.
[2021-03-28 20:25:27.472] [global] [info] sampled file:///data/flights_small/flights_on_time_performance_2019_11.csv on 256.00 KB
[2021-03-28 20:25:27.619] [global] [info] performing static typing for UDF in operator withColumn
[2021-03-28 20:25:27.626] [global] [info] performing static typing for UDF in operator withColumn
[2021-03-28 20:25:27.631] [global] [info] performing static typing for UDF in operator withColumn
[2021-03-28 20:25:27.637] [global] [info] performing static typing for UDF in operator withColumn
[2021-03-28 20:25:27.664] [global] [info] performing static typing for UDF in operator withColumn
[2021-03-28 20:25:27.682] [global] [info] performing static typing for UDF in operator withColumn
[2021-03-28 20:25:27.694] [global] [info] performing static typing for UDF in operator withColumn
[2021-03-28 20:25:27.694] [fileinputoperator] [info] found 1 file (81.54 KB) to process.
[2021-03-28 20:25:27.694] [global] [info] sampled file:///code/benchmark/flights/data/L_CARRIER_HISTORY.csv on 81.55 KB
[2021-03-28 20:25:27.739] [global] [info] performing static typing for UDF in operator withColumn
[2021-03-28 20:25:27.748] [global] [info] performing static typing for UDF in operator withColumn
[2021-03-28 20:25:27.759] [global] [info] performing static typing for UDF in operator withColumn
[2021-03-28 20:25:27.760] [fileinputoperator] [info] found 1 file (742.61 KB) to process.
[2021-03-28 20:25:27.760] [global] [info] sampled file:///code/benchmark/flights/data/GlobalAirportDatabase.txt on 256.00 KB
[2021-03-28 20:25:28.038] [global] [info] performing static typing for UDF in operator map
[2021-03-28 20:25:28.052] [global] [info] performing static typing for UDF in operator map
[2021-03-28 20:25:28.067] [global] [info] performing static typing for UDF in operator map
[2021-03-28 20:25:28.081] [global] [info] performing static typing for UDF in operator map
[2021-03-28 20:25:28.082] [global] [error] contains an assign operator - can't break down filter
[2021-03-28 20:25:28.203] [logical planner] [info] logical optimization took 0.149888ms
output type of specialized pipeline is: (Option[str],Option[str],Option[str],Option[i64],Option[f64],Option[f64])
is this the most outer stage?: 0
need to upgrade output type to (Option[str],Option[str],Option[str],Option[i64],Option[f64],Option[f64])
[2021-03-28 20:25:28.210] [codegen] [info] generating pipeline for (Option[str],Option[str],str,i64,f64,f64) -> (Option[str],Option[str],Option[str],Option[i64],Option[f64],Option[f64]) (2 operators pipelined)
outSchema is different than last operator's schema:
outSchema: (Option[str],Option[str],Option[str],Option[i64],Option[f64],Option[f64])
last Op: (Option[str],Option[str],Option[str],i64,f64,f64)
[2021-03-28 20:25:28.210] [codegen] [info] generating lambda function for (Option[str]) -> Option[str]
[2021-03-28 20:25:28.211] [codegen] [info] generating lambda function for (str) -> Option[str]
[2021-03-28 20:25:28.212] [codegen] [info] generating lambda function for (Option[str]) -> Option[str]
[2021-03-28 20:25:28.213] [codegen] [info] generating lambda function for (Option[str]) -> Option[str]
output type of specialized pipeline is: (Option[str],Option[str],Option[str],Option[i64],Option[f64],Option[f64])
is this the most outer stage?: 0
need to upgrade output type to (Option[str],Option[str],Option[str],Option[i64],Option[f64],Option[f64])
[2021-03-28 20:25:28.242] [codegen] [info] generating pipeline for (Option[str],Option[str],str,i64,f64,f64) -> (Option[str],Option[str],Option[str],Option[i64],Option[f64],Option[f64]) (2 operators pipelined)
outSchema is different than last operator's schema:
outSchema: (Option[str],Option[str],Option[str],Option[i64],Option[f64],Option[f64])
last Op: (Option[str],Option[str],Option[str],i64,f64,f64)
[2021-03-28 20:25:28.243] [codegen] [info] generating lambda function for (Option[str]) -> Option[str]
[2021-03-28 20:25:28.243] [codegen] [info] generating lambda function for (str) -> Option[str]
[2021-03-28 20:25:28.244] [codegen] [info] generating lambda function for (Option[str]) -> Option[str]
[2021-03-28 20:25:28.245] [codegen] [info] generating lambda function for (Option[str]) -> Option[str]
[2021-03-28 20:25:28.267] [global] [info] performing static typing for UDF in operator withColumn
[2021-03-28 20:25:28.268] [global] [info] performing static typing for UDF in operator withColumn
[2021-03-28 20:25:28.269] [global] [info] performing static typing for UDF in operator withColumn
output type of specialized pipeline is: (Option[str],Option[str],str,i64,Option[i64])
is this the most outer stage?: 0
need to upgrade output type to (Option[str],Option[str],str,i64,Option[i64])
[2021-03-28 20:25:28.269] [codegen] [info] generating pipeline for (str,str) -> (Option[str],Option[str],str,i64,Option[i64]) (3 operators pipelined)
outSchema is different than last operator's schema:
outSchema: (Option[str],Option[str],str,i64,Option[i64])
last Op: (str,str,str,i64,Option[i64])
[2021-03-28 20:25:28.270] [codegen] [info] generating lambda function for ((str,str)) -> str
[2021-03-28 20:25:28.270] [codegen] [info] generating lambda function for ((str,str,str)) -> i64
[2021-03-28 20:25:28.271] [codegen] [info] generating function extractDefunctYear for ((str,str,str,i64)) -> Option[i64]
[2021-03-28 20:25:28.272] [codegen] [info] generating lambda function for ((Option[str],Option[str])) -> str
[2021-03-28 20:25:28.273] [codegen] [info] generating lambda function for ((Option[str],Option[str],str)) -> i64
[2021-03-28 20:25:28.273] [codegen] [info] generating function extractDefunctYear for ((Option[str],Option[str],str,i64)) -> Option[i64]
[2021-03-28 20:25:28.713] [global] [info] performing static typing for UDF in operator withColumn
[2021-03-28 20:25:28.717] [global] [info] performing static typing for UDF in operator withColumn
[2021-03-28 20:25:28.722] [global] [info] performing static typing for UDF in operator withColumn
[2021-03-28 20:25:28.727] [global] [info] performing static typing for UDF in operator withColumn
[2021-03-28 20:25:28.740] [global] [info] performing static typing for UDF in operator withColumn
[2021-03-28 20:25:28.754] [global] [info] performing static typing for UDF in operator withColumn
start checking retyping here!!!
[2021-03-28 20:25:28.759] [global] [info] performing static typing for UDF in operator withColumn
[2021-03-28 20:25:29.073] [global] [info] performing static typing for UDF in operator map
[2021-03-28 20:25:29.073] [global] [info] performing static typing for UDF in operator select
output type of specialized pipeline is: (str,str,i64,i64,i64,i64,i64,str,str,str,Option[f64],Option[f64],Option[i64],str,str,str,Option[f64],Option[f64],Option[i64],f64,Option[str],boolean,boolean,Option[str],Option[str],i64,i64,i64,i64,i64,i64,i64,i64,i64,i64,i64,i64,i64,Option[i64])
is this the most outer stage?: 1
[2021-03-28 20:25:29.074] [codegen] [info] generating pipeline for (i64,i64,i64,i64,str,i64,str,str,str,str,i64,f64,f64,f64,i64,f64,f64,null,f64,f64,f64,f64,f64,Option[f64],Option[f64],Option[f64],Option[f64],Option[f64],null,null) -> (str,str,i64,i64,i64,i64,i64,str,str,str,Option[f64],Option[f64],Option[i64],str,str,str,Option[f64],Option[f64],Option[i64],f64,Option[str],boolean,boolean,Option[str],Option[str],i64,i64,i64,i64,i64,i64,i64,i64,i64,i64,i64,i64,i64,Option[i64]) (151 operators pipelined)
[2021-03-28 20:25:29.074] [codegen] [info] generating lambda function for ((i64,i64,i64,i64,str,i64,str,str,str,str,i64,f64,f64,f64,i64,f64,f64,null,f64,f64,f64,f64,f64,Option[f64],Option[f64],Option[f64],Option[f64],Option[f64],null,null)) -> str
[2021-03-28 20:25:29.075] [codegen] [info] generating lambda function for ((i64,i64,i64,i64,str,i64,str,str,str,str,i64,f64,f64,f64,i64,f64,f64,null,f64,f64,f64,f64,f64,Option[f64],Option[f64],Option[f64],Option[f64],Option[f64],null,null,str)) -> str
[2021-03-28 20:25:29.076] [codegen] [info] generating lambda function for ((i64,i64,i64,i64,str,i64,str,str,str,str,i64,f64,f64,f64,i64,f64,f64,null,f64,f64,f64,f64,f64,Option[f64],Option[f64],Option[f64],Option[f64],Option[f64],null,null,str,str)) -> str
[2021-03-28 20:25:29.077] [codegen] [info] generating lambda function for ((i64,i64,i64,i64,str,i64,str,str,str,str,i64,f64,f64,f64,i64,f64,f64,null,f64,f64,f64,f64,f64,Option[f64],Option[f64],Option[f64],Option[f64],Option[f64],null,null,str,str,str)) -> str
[2021-03-28 20:25:29.078] [codegen] [info] generating lambda function for (i64) -> Option[str]
[2021-03-28 20:25:29.079] [codegen] [info] generating lambda function for (i64) -> Option[str]
[2021-03-28 20:25:29.080] [codegen] [info] generating function cleanCode for ((i64,i64,i64,i64,str,i64,str,str,str,str,Option[str],f64,f64,f64,Option[str],f64,f64,null,f64,f64,f64,f64,f64,Option[f64],Option[f64],Option[f64],Option[f64],Option[f64],null,null,str,str,str,str)) -> Option[str]
[2021-03-28 20:25:29.081] [codegen] [info] generating lambda function for (f64) -> boolean
[2021-03-28 20:25:29.082] [codegen] [info] generating lambda function for (f64) -> boolean
[2021-03-28 20:25:29.083] [codegen] [info] generating function divertedUDF for ((i64,i64,i64,i64,str,i64,str,str,str,str,Option[str],f64,f64,f64,Option[str],f64,boolean,Option[str],boolean,f64,f64,f64,f64,Option[f64],Option[f64],Option[f64],Option[f64],Option[f64],null,null,str,str,str,str)) -> Option[str]
[2021-03-28 20:25:29.084] [codegen] [info] generating function fillInTimesUDF for ((i64,i64,i64,i64,str,i64,str,str,str,str,Option[str],f64,f64,f64,Option[str],f64,boolean,Option[str],boolean,f64,f64,f64,f64,Option[f64],Option[f64],Option[f64],Option[f64],Option[f64],null,null,str,str,str,str,Option[str])) -> f64
[2021-03-28 20:25:29.085] [codegen] [info] generating function filterDefunctFlights for ((i64,i64,i64,i64,i64,str,str,str,str,Option[str],f64,f64,f64,Option[str],f64,boolean,Option[str],boolean,f64,f64,f64,f64,Option[f64],Option[f64],Option[f64],Option[f64],Option[f64],null,null,str,str,str,str,Option[str],str,Option[str],str,i64,Option[i64])) -> boolean
[2021-03-28 20:25:29.087] [codegen] [info] generating lambda function for (f64) -> f64
[2021-03-28 20:25:29.088] [codegen] [info] generating lambda function for (str) -> str
[2021-03-28 20:25:29.088] [codegen] [info] generating lambda function for (f64) -> i64
[2021-03-28 20:25:29.090] [codegen] [info] generating lambda function for (f64) -> i64
[2021-03-28 20:25:29.091] [codegen] [info] generating lambda function for (f64) -> i64
[2021-03-28 20:25:29.092] [codegen] [info] generating lambda function for (Option[f64]) -> i64
[2021-03-28 20:25:29.093] [codegen] [info] generating lambda function for (f64) -> i64
[2021-03-28 20:25:29.094] [codegen] [info] generating lambda function for (f64) -> i64
[2021-03-28 20:25:29.095] [codegen] [info] generating lambda function for (Option[f64]) -> i64
[2021-03-28 20:25:29.096] [codegen] [info] generating lambda function for (Option[f64]) -> i64
[2021-03-28 20:25:29.097] [codegen] [info] generating lambda function for (Option[f64]) -> i64
[2021-03-28 20:25:29.099] [codegen] [info] generating lambda function for (f64) -> i64
[2021-03-28 20:25:29.100] [codegen] [info] generating lambda function for (f64) -> i64
[2021-03-28 20:25:29.101] [codegen] [info] generating lambda function for (Option[f64]) -> i64
[2021-03-28 20:25:29.102] [codegen] [info] generating lambda function for ((i64,i64,i64,i64,i64,str,str,Option[str],i64,i64,i64,Option[str],i64,boolean,Option[str],boolean,i64,i64,i64,f64,i64,i64,i64,i64,i64,null,null,str,str,str,str,Option[str],str,Option[str],str,i64,Option[i64],str,Option[str],Option[str],Option[i64],Option[f64],Option[f64],str,Option[str],Option[str],Option[i64],Option[f64],Option[f64])) -> (str,str,i64,i64,i64,i64,i64,str,str,str,Option[f64],Option[f64],Option[i64],str,str,str,Option[f64],Option[f64],Option[i64],f64,Option[str],boolean,boolean,Option[str],Option[str],i64,i64,i64,i64,i64,i64,i64,i64,i64,i64,i64,i64,i64,Option[i64])
[2021-03-28 20:25:29.111] [codegen] [info] generating lambda function for ((Option[i64],Option[i64],Option[i64],Option[i64],Option[str],Option[i64],Option[str],Option[str],Option[str],Option[str],Option[i64],Option[f64],Option[f64],Option[f64],Option[i64],Option[f64],Option[f64],Option[str],Option[f64],Option[f64],Option[f64],Option[f64],Option[f64],Option[f64],Option[f64],Option[f64],Option[f64],Option[f64],Option[f64],Option[f64])) -> str
[2021-03-28 20:25:29.112] [codegen] [info] generating lambda function for ((Option[i64],Option[i64],Option[i64],Option[i64],Option[str],Option[i64],Option[str],Option[str],Option[str],Option[str],Option[i64],Option[f64],Option[f64],Option[f64],Option[i64],Option[f64],Option[f64],Option[str],Option[f64],Option[f64],Option[f64],Option[f64],Option[f64],Option[f64],Option[f64],Option[f64],Option[f64],Option[f64],Option[f64],Option[f64],str)) -> str
[2021-03-28 20:25:29.114] [codegen] [info] generating lambda function for ((Option[i64],Option[i64],Option[i64],Option[i64],Option[str],Option[i64],Option[str],Option[str],Option[str],Option[str],Option[i64],Option[f64],Option[f64],Option[f64],Option[i64],Option[f64],Option[f64],Option[str],Option[f64],Option[f64],Option[f64],Option[f64],Option[f64],Option[f64],Option[f64],Option[f64],Option[f64],Option[f64],Option[f64],Option[f64],str,str)) -> str
[2021-03-28 20:25:29.115] [codegen] [info] generating lambda function for ((Option[i64],Option[i64],Option[i64],Option[i64],Option[str],Option[i64],Option[str],Option[str],Option[str],Option[str],Option[i64],Option[f64],Option[f64],Option[f64],Option[i64],Option[f64],Option[f64],Option[str],Option[f64],Option[f64],Option[f64],Option[f64],Option[f64],Option[f64],Option[f64],Option[f64],Option[f64],Option[f64],Option[f64],Option[f64],str,str,str)) -> str
[2021-03-28 20:25:29.116] [codegen] [info] generating lambda function for (Option[i64]) -> Option[str]
[2021-03-28 20:25:29.117] [codegen] [info] generating lambda function for (Option[i64]) -> Option[str]
[2021-03-28 20:25:29.118] [codegen] [info] generating function cleanCode for ((Option[i64],Option[i64],Option[i64],Option[i64],Option[str],Option[i64],Option[str],Option[str],Option[str],Option[str],Option[str],Option[f64],Option[f64],Option[f64],Option[str],Option[f64],Option[f64],Option[str],Option[f64],Option[f64],Option[f64],Option[f64],Option[f64],Option[f64],Option[f64],Option[f64],Option[f64],Option[f64],Option[f64],Option[f64],str,str,str,str)) -> Option[str]
[2021-03-28 20:25:29.119] [codegen] [info] generating lambda function for (Option[f64]) -> boolean
[2021-03-28 20:25:29.120] [codegen] [info] generating lambda function for (Option[f64]) -> boolean
[2021-03-28 20:25:29.121] [codegen] [info] generating function divertedUDF for ((Option[i64],Option[i64],Option[i64],Option[i64],Option[str],Option[i64],Option[str],Option[str],Option[str],Option[str],Option[str],Option[f64],Option[f64],Option[f64],Option[str],Option[f64],boolean,Option[str],boolean,Option[f64],Option[f64],Option[f64],Option[f64],Option[f64],Option[f64],Option[f64],Option[f64],Option[f64],Option[f64],Option[f64],str,str,str,str)) -> Option[str]
[2021-03-28 20:25:29.123] [codegen] [info] generating function fillInTimesUDF for ((Option[i64],Option[i64],Option[i64],Option[i64],Option[str],Option[i64],Option[str],Option[str],Option[str],Option[str],Option[str],Option[f64],Option[f64],Option[f64],Option[str],Option[f64],boolean,Option[str],boolean,Option[f64],Option[f64],Option[f64],Option[f64],Option[f64],Option[f64],Option[f64],Option[f64],Option[f64],Option[f64],Option[f64],str,str,str,str,Option[str])) -> Option[f64]
[2021-03-28 20:25:29.124] [codegen] [info] generating function filterDefunctFlights for ((Option[i64],Option[i64],Option[i64],Option[i64],Option[i64],Option[str],Option[str],Option[str],Option[str],Option[str],Option[f64],Option[f64],Option[f64],Option[str],Option[f64],boolean,Option[str],boolean,Option[f64],Option[f64],Option[f64],Option[f64],Option[f64],Option[f64],Option[f64],Option[f64],Option[f64],Option[f64],Option[f64],str,str,str,str,Option[str],Option[str],Option[str],str,i64,Option[i64])) -> boolean
[2021-03-28 20:25:29.126] [codegen] [info] generating lambda function for (Option[f64]) -> f64
[2021-03-28 20:25:29.127] [codegen] [info] generating lambda function for (str) -> str
[2021-03-28 20:25:29.128] [codegen] [info] generating lambda function for (Option[f64]) -> i64
[2021-03-28 20:25:29.129] [codegen] [info] generating lambda function for (Option[f64]) -> i64
[2021-03-28 20:25:29.131] [codegen] [info] generating lambda function for (Option[f64]) -> i64
[2021-03-28 20:25:29.132] [codegen] [info] generating lambda function for (Option[f64]) -> i64
[2021-03-28 20:25:29.133] [codegen] [info] generating lambda function for (Option[f64]) -> i64
[2021-03-28 20:25:29.135] [codegen] [info] generating lambda function for (Option[f64]) -> i64
[2021-03-28 20:25:29.136] [codegen] [info] generating lambda function for (Option[f64]) -> i64
[2021-03-28 20:25:29.137] [codegen] [info] generating lambda function for (Option[f64]) -> i64
[2021-03-28 20:25:29.138] [codegen] [info] generating lambda function for (Option[f64]) -> i64
[2021-03-28 20:25:29.140] [codegen] [info] generating lambda function for (Option[f64]) -> i64
[2021-03-28 20:25:29.141] [codegen] [info] generating lambda function for (Option[f64]) -> i64
[2021-03-28 20:25:29.142] [codegen] [info] generating lambda function for (Option[f64]) -> i64
[2021-03-28 20:25:29.143] [codegen] [info] generating lambda function for ((Option[i64],Option[i64],Option[i64],Option[i64],Option[i64],Option[str],Option[str],Option[str],i64,i64,i64,Option[str],i64,boolean,Option[str],boolean,i64,i64,i64,f64,i64,i64,i64,i64,i64,Option[f64],Option[f64],str,str,str,str,Option[str],Option[str],Option[str],str,i64,Option[i64],Option[str],Option[str],Option[str],Option[i64],Option[f64],Option[f64],Option[str],Option[str],Option[str],Option[i64],Option[f64],Option[f64])) -> (str,Option[str],Option[i64],Option[i64],Option[i64],Option[i64],Option[i64],str,str,Option[str],Option[f64],Option[f64],Option[i64],str,str,Option[str],Option[f64],Option[f64],Option[i64],f64,Option[str],boolean,boolean,Option[str],Option[str],i64,i64,i64,i64,i64,i64,i64,i64,i64,i64,i64,i64,i64,Option[i64])
[2021-03-28 20:25:29.198] [global] [info] Stage3 depends on: Stage2 Stage1 Stage0 
[2021-03-28 20:25:29.198] [global] [info] Stage2 depends on: 
[2021-03-28 20:25:29.202] [global] [info] provided option to explicitly merge bad rows in order back, however rows will be hashed. Disabling option. To silence this warning, set
tuplex.optimizer.mergeExceptionsInOrder=false
[2021-03-28 20:25:29.324] [global] [info] Optimization via LLVM passes took 0.122571 ms
[2021-03-28 20:25:29.389] [global] [info] Compiled code paths for stage 2 in 0.07 ms
[2021-03-28 20:25:29.389] [global] [info] [Transform Stage] Stage 2 compiled to x86 in 0.191627s
[2021-03-28 20:25:29.390] [local ee] [info] split /code/benchmark/flights/data/L_CARRIER_HISTORY.csv into 1 task
[2021-03-28 20:25:29.393] [E/1] [info] [Task Finished] Transform to in-memory hash table in 0.003491s (1928 normal rows, 0 exceptions, 1726 buckets)
[2021-03-28 20:25:29.393] [global] [info] [Transform Stage] Stage 2 completed 1 load&transform tasks in 0.00366972s
[2021-03-28 20:25:29.393] [global] [info] [Transform Stage] Stage 2 total wall clock time: 0.00349124s, 1928 input rows, time to process 1 row via fast path: 0.00181081ms
[2021-03-28 20:25:29.393] [global] [info] [Transform Stage] Stage 2 completed 1 sink tasks in 3.062e-06s
[2021-03-28 20:25:29.393] [global] [info] [Transform Stage] Stage 2 took 0.195324s
[2021-03-28 20:25:29.393] [global] [info] Stage1 depends on: 
[2021-03-28 20:25:29.397] [global] [info] provided option to explicitly merge bad rows in order back, however rows will be hashed. Disabling option. To silence this warning, set
tuplex.optimizer.mergeExceptionsInOrder=false
[2021-03-28 20:25:29.496] [global] [info] Optimization via LLVM passes took 0.099493 ms
[2021-03-28 20:25:29.542] [global] [info] Compiled code paths for stage 1 in 0.05 ms
[2021-03-28 20:25:29.542] [global] [info] [Transform Stage] Stage 1 compiled to x86 in 0.149044s
[2021-03-28 20:25:29.542] [local ee] [info] split /code/benchmark/flights/data/GlobalAirportDatabase.txt into 1 task
[2021-03-28 20:25:29.552] [E/1] [info] [Task Finished] Transform to in-memory hash table in 0.009479s (3562 normal rows, 0 exceptions, 3512 buckets)
[2021-03-28 20:25:29.552] [global] [info] [Transform Stage] Stage 1 completed 1 load&transform tasks in 0.00963617s
[2021-03-28 20:25:29.552] [global] [info] [Transform Stage] Stage 1 total wall clock time: 0.00947938s, 9300 input rows, time to process 1 row via fast path: 0.00101929ms
[2021-03-28 20:25:29.552] [global] [info] [Transform Stage] Stage 1 completed 1 sink tasks in 2.548e-06s
[2021-03-28 20:25:29.552] [global] [info] [Transform Stage] Stage 1 took 0.158707s
[2021-03-28 20:25:29.552] [global] [info] Stage0 depends on: 
[2021-03-28 20:25:29.555] [global] [info] provided option to explicitly merge bad rows in order back, however rows will be hashed. Disabling option. To silence this warning, set
tuplex.optimizer.mergeExceptionsInOrder=false
[2021-03-28 20:25:29.655] [global] [info] Optimization via LLVM passes took 0.099910 ms
[2021-03-28 20:25:29.702] [global] [info] Compiled code paths for stage 0 in 0.05 ms
[2021-03-28 20:25:29.702] [global] [info] [Transform Stage] Stage 0 compiled to x86 in 0.149663s
[2021-03-28 20:25:29.702] [local ee] [info] split /code/benchmark/flights/data/GlobalAirportDatabase.txt into 1 task
[2021-03-28 20:25:29.710] [E/1] [info] [Task Finished] Transform to in-memory hash table in 0.008623s (3562 normal rows, 0 exceptions, 3512 buckets)
[2021-03-28 20:25:29.710] [global] [info] [Transform Stage] Stage 0 completed 1 load&transform tasks in 0.0087915s
[2021-03-28 20:25:29.710] [global] [info] [Transform Stage] Stage 0 total wall clock time: 0.00862263s, 9300 input rows, time to process 1 row via fast path: 0.000927164ms
[2021-03-28 20:25:29.710] [global] [info] [Transform Stage] Stage 0 completed 1 sink tasks in 2.473e-06s
[2021-03-28 20:25:29.710] [global] [info] [Transform Stage] Stage 0 took 0.158488s
[2021-03-28 20:25:30.630] [global] [info] Optimization via LLVM passes took 0.873421 ms
[2021-03-28 20:25:30.946] [global] [info] Compiled code paths for stage 3 in 0.32 ms
[2021-03-28 20:25:30.946] [global] [info] [Transform Stage] Stage 3 compiled to x86 in 1.23582s
[2021-03-28 20:25:30.946] [local ee] [info] split /data/flights_small/flights_on_time_performance_2019_11.csv into 4 tasks
[2021-03-28 20:25:30.946] [local ee] [info] split /data/flights_small/flights_on_time_performance_2019_10.csv into 4 tasks
[2021-03-28 20:25:30.946] [local ee] [info] split /data/flights_small/flights_on_time_performance_2019_09.csv into 4 tasks
[2021-03-28 20:25:30.946] [local ee] [info] split /data/flights_small/flights_on_time_performance_2019_08.csv into 4 tasks
[2021-03-28 20:25:30.946] [local ee] [info] split /data/flights_small/flights_on_time_performance_2019_07.csv into 4 tasks
[2021-03-28 20:25:30.946] [local ee] [info] split /data/flights_small/flights_on_time_performance_2019_06.csv into 4 tasks
[2021-03-28 20:25:30.946] [local ee] [info] split /data/flights_small/flights_on_time_performance_2019_05.csv into 4 tasks
[2021-03-28 20:25:30.947] [local ee] [info] split /data/flights_small/flights_on_time_performance_2019_04.csv into 4 tasks
[2021-03-28 20:25:30.947] [local ee] [info] split /data/flights_small/flights_on_time_performance_2019_03.csv into 4 tasks
[2021-03-28 20:25:30.947] [local ee] [info] split /data/flights_small/flights_on_time_performance_2019_02.csv into 3 tasks
[2021-03-28 20:25:30.947] [local ee] [info] split /data/flights_small/flights_on_time_performance_2019_01.csv into 3 tasks
[2021-03-28 20:25:30.947] [local ee] [info] split /data/flights_small/flights_on_time_performance_2018_12.csv into 4 tasks
[2021-03-28 20:25:30.947] [local ee] [info] split /data/flights_small/flights_on_time_performance_2018_11.csv into 3 tasks
[2021-03-28 20:25:30.947] [local ee] [info] split /data/flights_small/flights_on_time_performance_2018_10.csv into 4 tasks
[2021-03-28 20:25:30.947] [local ee] [info] split /data/flights_small/flights_on_time_performance_2018_09.csv into 3 tasks
[2021-03-28 20:25:30.947] [local ee] [info] split /data/flights_small/flights_on_time_performance_2018_08.csv into 4 tasks
[2021-03-28 20:25:30.947] [local ee] [info] split /data/flights_small/flights_on_time_performance_2018_07.csv into 4 tasks
[2021-03-28 20:25:30.947] [local ee] [info] split /data/flights_small/flights_on_time_performance_2018_06.csv into 4 tasks
[2021-03-28 20:25:30.947] [local ee] [info] split /data/flights_small/flights_on_time_performance_2018_05.csv into 4 tasks
[2021-03-28 20:25:30.947] [local ee] [info] split /data/flights_small/flights_on_time_performance_2018_04.csv into 4 tasks
[2021-03-28 20:25:30.947] [local ee] [info] split /data/flights_small/flights_on_time_performance_2018_03.csv into 4 tasks
[2021-03-28 20:25:30.947] [local ee] [info] split /data/flights_small/flights_on_time_performance_2018_02.csv into 3 tasks
[2021-03-28 20:25:30.947] [local ee] [info] split /data/flights_small/flights_on_time_performance_2018_01.csv into 3 tasks
[2021-03-28 20:25:31.521] [E/10] [info] [Task Finished] Transform to mem in 0.573660s (147124 normal rows, 2886 exceptions)
[2021-03-28 20:25:31.523] [E/11] [info] [Task Finished] Transform to mem in 0.575809s (147343 normal rows, 3389 exceptions)
[2021-03-28 20:25:31.523] [E/9] [info] [Task Finished] Transform to mem in 0.575936s (147676 normal rows, 1031 exceptions)
[2021-03-28 20:25:31.525] [E/2] [info] [Task Finished] Transform to mem in 0.577771s (147634 normal rows, 1820 exceptions)
[2021-03-28 20:25:31.526] [driver] [info] [Task Finished] Transform to mem in 0.579360s (146336 normal rows, 3459 exceptions)
[2021-03-28 20:25:31.528] [E/7] [info] [Task Finished] Transform to mem in 0.580696s (147013 normal rows, 1583 exceptions)
[2021-03-28 20:25:31.528] [E/4] [info] [Task Finished] Transform to mem in 0.580940s (148705 normal rows, 1224 exceptions)
[2021-03-28 20:25:31.529] [E/5] [info] [Task Finished] Transform to mem in 0.581752s (149150 normal rows, 1076 exceptions)
[2021-03-28 20:25:31.531] [E/13] [info] [Task Finished] Transform to mem in 0.583822s (147491 normal rows, 2223 exceptions)
[2021-03-28 20:25:31.531] [E/15] [info] [Task Finished] Transform to mem in 0.584086s (146867 normal rows, 2625 exceptions)
[2021-03-28 20:25:31.534] [E/12] [info] [Task Finished] Transform to mem in 0.586797s (147356 normal rows, 2414 exceptions)
[2021-03-28 20:25:31.534] [E/6] [info] [Task Finished] Transform to mem in 0.587393s (148648 normal rows, 1689 exceptions)
[2021-03-28 20:25:31.561] [E/8] [info] [Task Finished] Transform to mem in 0.614355s (156053 normal rows, 1402 exceptions)
[2021-03-28 20:25:31.571] [E/3] [info] [Task Finished] Transform to mem in 0.624135s (156932 normal rows, 2574 exceptions)
[2021-03-28 20:25:31.696] [E/14] [info] [Task Finished] Transform to mem in 0.748594s (190258 normal rows, 1898 exceptions)
[2021-03-28 20:25:31.770] [E/1] [info] [Task Finished] Transform to mem in 0.823098s (208988 normal rows, 4803 exceptions)
[2021-03-28 20:25:32.098] [E/10] [info] [Task Finished] Transform to mem in 0.577473s (147152 normal rows, 3302 exceptions)
[2021-03-28 20:25:32.099] [E/9] [info] [Task Finished] Transform to mem in 0.575784s (146306 normal rows, 3351 exceptions)
[2021-03-28 20:25:32.099] [E/4] [info] [Task Finished] Transform to mem in 0.571569s (144695 normal rows, 4754 exceptions)
[2021-03-28 20:25:32.100] [E/11] [info] [Task Finished] Transform to mem in 0.577595s (145306 normal rows, 3738 exceptions)
[2021-03-28 20:25:32.104] [E/13] [info] [Task Finished] Transform to mem in 0.573720s (145230 normal rows, 3609 exceptions)
[2021-03-28 20:25:32.106] [driver] [info] [Task Finished] Transform to mem in 0.579405s (146257 normal rows, 2081 exceptions)
[2021-03-28 20:25:32.109] [E/12] [info] [Task Finished] Transform to mem in 0.575536s (146366 normal rows, 3947 exceptions)
[2021-03-28 20:25:32.110] [E/15] [info] [Task Finished] Transform to mem in 0.578698s (146022 normal rows, 3033 exceptions)
[2021-03-28 20:25:32.110] [E/7] [info] [Task Finished] Transform to mem in 0.582276s (146143 normal rows, 3024 exceptions)
[2021-03-28 20:25:32.141] [E/8] [info] [Task Finished] Transform to mem in 0.579306s (145442 normal rows, 5129 exceptions)
[2021-03-28 20:25:32.149] [E/3] [info] [Task Finished] Transform to mem in 0.577758s (146071 normal rows, 2428 exceptions)
[2021-03-28 20:25:32.272] [E/14] [info] [Task Finished] Transform to mem in 0.576755s (146274 normal rows, 4308 exceptions)
[2021-03-28 20:25:32.274] [E/5] [info] [Task Finished] Transform to mem in 0.745287s (187821 normal rows, 5945 exceptions)
[2021-03-28 20:25:32.278] [E/6] [info] [Task Finished] Transform to mem in 0.744132s (187623 normal rows, 4462 exceptions)
[2021-03-28 20:25:32.353] [E/2] [info] [Task Finished] Transform to mem in 0.828029s (209176 normal rows, 4857 exceptions)
[2021-03-28 20:25:32.406] [E/1] [info] [Task Finished] Transform to mem in 0.636366s (161835 normal rows, 4138 exceptions)
[2021-03-28 20:25:32.672] [E/9] [info] [Task Finished] Transform to mem in 0.573277s (146745 normal rows, 2502 exceptions)
[2021-03-28 20:25:32.674] [E/13] [info] [Task Finished] Transform to mem in 0.569732s (144188 normal rows, 4844 exceptions)
[2021-03-28 20:25:32.677] [E/4] [info] [Task Finished] Transform to mem in 0.577339s (146560 normal rows, 2826 exceptions)
[2021-03-28 20:25:32.679] [driver] [info] [Task Finished] Transform to mem in 0.572878s (144986 normal rows, 5198 exceptions)
[2021-03-28 20:25:32.680] [E/10] [info] [Task Finished] Transform to mem in 0.581770s (146881 normal rows, 3309 exceptions)
[2021-03-28 20:25:32.683] [E/15] [info] [Task Finished] Transform to mem in 0.573068s (144926 normal rows, 4651 exceptions)
[2021-03-28 20:25:32.686] [E/7] [info] [Task Finished] Transform to mem in 0.576080s (144158 normal rows, 6268 exceptions)
[2021-03-28 20:25:32.729] [E/3] [info] [Task Finished] Transform to mem in 0.580191s (150131 normal rows, 2104 exceptions)
[2021-03-28 20:25:32.818] [E/11] [info] [Task Finished] Transform to mem in 0.717278s (182052 normal rows, 4932 exceptions)
[2021-03-28 20:25:32.848] [E/5] [info] [Task Finished] Transform to mem in 0.573835s (147089 normal rows, 2183 exceptions)
[2021-03-28 20:25:32.858] [E/14] [info] [Task Finished] Transform to mem in 0.586114s (157646 normal rows, 1769 exceptions)
[2021-03-28 20:25:32.860] [E/6] [info] [Task Finished] Transform to mem in 0.581375s (147935 normal rows, 2049 exceptions)
[2021-03-28 20:25:32.930] [E/2] [info] [Task Finished] Transform to mem in 0.577603s (151381 normal rows, 1619 exceptions)
[2021-03-28 20:25:32.982] [E/1] [info] [Task Finished] Transform to mem in 0.576020s (147761 normal rows, 1365 exceptions)
[2021-03-28 20:25:33.032] [E/12] [info] [Task Finished] Transform to mem in 0.922268s (230201 normal rows, 6819 exceptions)
[2021-03-28 20:25:33.248] [E/8] [info] [Task Finished] Transform to mem in 1.107013s (280278 normal rows, 7103 exceptions)
[2021-03-28 20:25:33.250] [E/13] [info] [Task Finished] Transform to mem in 0.576169s (147922 normal rows, 1714 exceptions)
[2021-03-28 20:25:33.257] [E/4] [info] [Task Finished] Transform to mem in 0.579682s (150687 normal rows, 1323 exceptions)
[2021-03-28 20:25:33.262] [E/15] [info] [Task Finished] Transform to mem in 0.578714s (147946 normal rows, 2721 exceptions)
[2021-03-28 20:25:33.268] [driver] [info] [Task Finished] Transform to mem in 0.589876s (158753 normal rows, 1398 exceptions)
[2021-03-28 20:25:33.274] [E/7] [info] [Task Finished] Transform to mem in 0.588464s (156854 normal rows, 1998 exceptions)
[2021-03-28 20:25:33.348] [E/10] [info] [Task Finished] Transform to mem in 0.668177s (172209 normal rows, 1332 exceptions)
[2021-03-28 20:25:33.406] [E/11] [info] [Task Finished] Transform to mem in 0.588356s (162164 normal rows, 3490 exceptions)
[2021-03-28 20:25:33.426] [E/5] [info] [Task Finished] Transform to mem in 0.577969s (146575 normal rows, 3847 exceptions)
[2021-03-28 20:25:33.436] [E/14] [info] [Task Finished] Transform to mem in 0.577298s (145394 normal rows, 3360 exceptions)
[2021-03-28 20:25:33.513] [E/2] [info] [Task Finished] Transform to mem in 0.582269s (155776 normal rows, 4384 exceptions)
[2021-03-28 20:25:33.557] [E/1] [info] [Task Finished] Transform to mem in 0.574830s (146157 normal rows, 2732 exceptions)
[2021-03-28 20:25:33.609] [E/12] [info] [Task Finished] Transform to mem in 0.577132s (151708 normal rows, 3129 exceptions)
[2021-03-28 20:25:33.634] [E/6] [info] [Task Finished] Transform to mem in 0.774390s (196326 normal rows, 4630 exceptions)
[2021-03-28 20:25:33.812] [E/9] [info] [Task Finished] Transform to mem in 1.140240s (297336 normal rows, 4447 exceptions)
[2021-03-28 20:25:33.835] [E/13] [info] [Task Finished] Transform to mem in 0.584977s (159355 normal rows, 6330 exceptions)
[2021-03-28 20:25:33.836] [E/15] [info] [Task Finished] Transform to mem in 0.574152s (146104 normal rows, 2822 exceptions)
[2021-03-28 20:25:33.840] [E/4] [info] [Task Finished] Transform to mem in 0.583188s (149378 normal rows, 2019 exceptions)
[2021-03-28 20:25:33.850] [E/3] [info] [Task Finished] Transform to mem in 1.121415s (290677 normal rows, 4780 exceptions)
[2021-03-28 20:25:33.864] [E/7] [info] [Task Finished] Transform to mem in 0.589040s (162883 normal rows, 2436 exceptions)
[2021-03-28 20:25:33.932] [E/10] [info] [Task Finished] Transform to mem in 0.583494s (148352 normal rows, 2228 exceptions)
[2021-03-28 20:25:33.973] [driver] [info] [Task Finished] Transform to mem in 0.704473s (178286 normal rows, 2542 exceptions)
[2021-03-28 20:25:33.981] [E/11] [info] [Task Finished] Transform to mem in 0.574699s (147282 normal rows, 2806 exceptions)
[2021-03-28 20:25:34.025] [E/14] [info] [Task Finished] Transform to mem in 0.589631s (164202 normal rows, 2216 exceptions)
[2021-03-28 20:25:34.033] [E/8] [info] [Task Finished] Transform to mem in 0.785693s (198072 normal rows, 3847 exceptions)
[2021-03-28 20:25:34.091] [E/5] [info] [Task Finished] Transform to mem in 0.665606s (168740 normal rows, 2366 exceptions)
[2021-03-28 20:25:34.096] [E/2] [info] [Task Finished] Transform to mem in 0.583244s (149776 normal rows, 2157 exceptions)
[2021-03-28 20:25:34.133] [E/1] [info] [Task Finished] Transform to mem in 0.576007s (147097 normal rows, 1795 exceptions)
[2021-03-28 20:25:34.194] [E/12] [info] [Task Finished] Transform to mem in 0.585200s (147798 normal rows, 2252 exceptions)
[2021-03-28 20:25:34.225] [E/6] [info] [Task Finished] Transform to mem in 0.590344s (164752 normal rows, 4593 exceptions)
[2021-03-28 20:25:34.394] [E/9] [info] [Task Finished] Transform to mem in 0.581319s (148621 normal rows, 3259 exceptions)
[2021-03-28 20:25:34.406] [E/13] [info] [Task Finished] Transform to mem in 0.570083s (144647 normal rows, 5022 exceptions)
[2021-03-28 20:25:34.417] [E/4] [info] [Task Finished] Transform to mem in 0.576984s (147721 normal rows, 3018 exceptions)
[2021-03-28 20:25:34.433] [E/3] [info] [Task Finished] Transform to mem in 0.582054s (147977 normal rows, 1128 exceptions)
[2021-03-28 20:25:34.469] [E/15] [info] [Task Finished] Transform to mem in 0.632804s (158507 normal rows, 6380 exceptions)
[2021-03-28 20:25:34.523] [E/10] [info] [Task Finished] Transform to mem in 0.591741s (162807 normal rows, 5984 exceptions)
[2021-03-28 20:25:34.548] [driver] [info] [Task Finished] Transform to mem in 0.574889s (144142 normal rows, 5294 exceptions)
[2021-03-28 20:25:34.741] [E/7] [info] [Task Finished] Transform to mem in 0.877722s (234581 normal rows, 6943 exceptions)
[2021-03-28 20:25:35.031] [E/11] [info] [Task Finished] Transform to mem in 1.050238s (266739 normal rows, 7818 exceptions)
[2021-03-28 20:25:35.031] [global] [info] [Transform Stage] Stage 3 completed 86 load&transform tasks in 4.0848s
[2021-03-28 20:25:35.031] [global] [info] [Transform Stage] Stage 3 total wall clock time: 54.6063s, 14009720 input rows, time to process 1 row via fast path: 0.00389774ms
[2021-03-28 20:25:35.031] [global] [info] Exception details: 
+------------+-------------------------------------+--------+
| OperatorID | Exception                           | Count  |
+------------+-------------------------------------+--------+
| 100000     | tuplex.internal.BadParseStringInput | 288283 |
+------------+-------------------------------------+--------+
[2021-03-28 20:25:35.053] [local ee] [info] compiled pure python pipeline in 0.021619s
[2021-03-28 20:25:35.053] [local ee] [info] creating hybrid intermediates took 0.000016s
[2021-03-28 20:25:35.054] [local ee] [info] Created 86 resolve tasks in 0.000704s
[2021-03-28 20:25:35.054] [local ee] [info] 86/86 tasks require executing the slow path.
[2021-03-28 20:25:35.128] [E/15] [info] [Task Finished] Resolve in 0.074030s
[2021-03-28 20:25:35.130] [E/8] [info] [Task Finished] Resolve in 0.076309s
[2021-03-28 20:25:35.130] [E/12] [info] [Task Finished] Resolve in 0.076391s
[2021-03-28 20:25:35.130] [E/10] [info] [Task Finished] Resolve in 0.076694s
[2021-03-28 20:25:35.131] [E/6] [info] [Task Finished] Resolve in 0.077458s
[2021-03-28 20:25:35.131] [E/2] [info] [Task Finished] Resolve in 0.077880s
[2021-03-28 20:25:35.132] [E/9] [info] [Task Finished] Resolve in 0.078518s
[2021-03-28 20:25:35.133] [E/3] [info] [Task Finished] Resolve in 0.079166s
[2021-03-28 20:25:35.133] [driver] [info] [Task Finished] Resolve in 0.079225s
[2021-03-28 20:25:35.133] [E/14] [info] [Task Finished] Resolve in 0.079549s
[2021-03-28 20:25:35.134] [E/11] [info] [Task Finished] Resolve in 0.080479s
[2021-03-28 20:25:35.134] [E/1] [info] [Task Finished] Resolve in 0.080487s
[2021-03-28 20:25:35.134] [E/4] [info] [Task Finished] Resolve in 0.080871s
[2021-03-28 20:25:35.138] [E/13] [info] [Task Finished] Resolve in 0.084538s
[2021-03-28 20:25:35.152] [E/5] [info] [Task Finished] Resolve in 0.098757s
[2021-03-28 20:25:35.169] [E/7] [info] [Task Finished] Resolve in 0.115299s
[2021-03-28 20:25:35.209] [E/15] [info] [Task Finished] Resolve in 0.081242s
[2021-03-28 20:25:35.209] [E/2] [info] [Task Finished] Resolve in 0.077976s
[2021-03-28 20:25:35.211] [E/8] [info] [Task Finished] Resolve in 0.081066s
[2021-03-28 20:25:35.212] [E/10] [info] [Task Finished] Resolve in 0.081927s
[2021-03-28 20:25:35.212] [E/6] [info] [Task Finished] Resolve in 0.081212s
[2021-03-28 20:25:35.213] [E/3] [info] [Task Finished] Resolve in 0.079740s
[2021-03-28 20:25:35.213] [E/11] [info] [Task Finished] Resolve in 0.078683s
[2021-03-28 20:25:35.214] [driver] [info] [Task Finished] Resolve in 0.080874s
[2021-03-28 20:25:35.214] [E/12] [info] [Task Finished] Resolve in 0.083966s
[2021-03-28 20:25:35.214] [E/9] [info] [Task Finished] Resolve in 0.081972s
[2021-03-28 20:25:35.217] [E/1] [info] [Task Finished] Resolve in 0.082870s
[2021-03-28 20:25:35.219] [E/14] [info] [Task Finished] Resolve in 0.085925s
[2021-03-28 20:25:35.244] [E/13] [info] [Task Finished] Resolve in 0.105537s
[2021-03-28 20:25:35.244] [E/4] [info] [Task Finished] Resolve in 0.109528s
[2021-03-28 20:25:35.259] [E/7] [info] [Task Finished] Resolve in 0.089727s
[2021-03-28 20:25:35.268] [E/5] [info] [Task Finished] Resolve in 0.115884s
[2021-03-28 20:25:35.286] [E/15] [info] [Task Finished] Resolve in 0.077510s
[2021-03-28 20:25:35.290] [E/8] [info] [Task Finished] Resolve in 0.078714s
[2021-03-28 20:25:35.292] [E/9] [info] [Task Finished] Resolve in 0.077614s
[2021-03-28 20:25:35.292] [E/2] [info] [Task Finished] Resolve in 0.082997s
[2021-03-28 20:25:35.293] [driver] [info] [Task Finished] Resolve in 0.079135s
[2021-03-28 20:25:35.293] [E/6] [info] [Task Finished] Resolve in 0.080842s
[2021-03-28 20:25:35.295] [E/3] [info] [Task Finished] Resolve in 0.082596s
[2021-03-28 20:25:35.296] [E/10] [info] [Task Finished] Resolve in 0.084212s
[2021-03-28 20:25:35.298] [E/14] [info] [Task Finished] Resolve in 0.078679s
[2021-03-28 20:25:35.299] [E/1] [info] [Task Finished] Resolve in 0.082420s
[2021-03-28 20:25:35.300] [E/11] [info] [Task Finished] Resolve in 0.087080s
[2021-03-28 20:25:35.316] [E/12] [info] [Task Finished] Resolve in 0.101964s
[2021-03-28 20:25:35.320] [E/4] [info] [Task Finished] Resolve in 0.075858s
[2021-03-28 20:25:35.321] [E/13] [info] [Task Finished] Resolve in 0.077304s
[2021-03-28 20:25:35.363] [E/15] [info] [Task Finished] Resolve in 0.076761s
[2021-03-28 20:25:35.366] [E/8] [info] [Task Finished] Resolve in 0.076672s
[2021-03-28 20:25:35.371] [E/9] [info] [Task Finished] Resolve in 0.078844s
[2021-03-28 20:25:35.374] [E/2] [info] [Task Finished] Resolve in 0.081148s
[2021-03-28 20:25:35.375] [driver] [info] [Task Finished] Resolve in 0.081675s
[2021-03-28 20:25:35.378] [E/10] [info] [Task Finished] Resolve in 0.081646s
[2021-03-28 20:25:35.378] [E/11] [info] [Task Finished] Resolve in 0.078483s
[2021-03-28 20:25:35.379] [E/14] [info] [Task Finished] Resolve in 0.080669s
[2021-03-28 20:25:35.381] [E/6] [info] [Task Finished] Resolve in 0.087583s
[2021-03-28 20:25:35.383] [E/3] [info] [Task Finished] Resolve in 0.087349s
[2021-03-28 20:25:35.387] [E/1] [info] [Task Finished] Resolve in 0.087689s
[2021-03-28 20:25:35.392] [E/7] [info] [Task Finished] Resolve in 0.133217s
[2021-03-28 20:25:35.397] [E/12] [info] [Task Finished] Resolve in 0.080988s
[2021-03-28 20:25:35.423] [E/5] [info] [Task Finished] Resolve in 0.154954s
[2021-03-28 20:25:35.429] [E/4] [info] [Task Finished] Resolve in 0.108821s
[2021-03-28 20:25:35.446] [E/8] [info] [Task Finished] Resolve in 0.079288s
[2021-03-28 20:25:35.450] [E/9] [info] [Task Finished] Resolve in 0.079022s
[2021-03-28 20:25:35.456] [E/15] [info] [Task Finished] Resolve in 0.092322s
[2021-03-28 20:25:35.457] [E/10] [info] [Task Finished] Resolve in 0.078817s
[2021-03-28 20:25:35.457] [E/14] [info] [Task Finished] Resolve in 0.078634s
[2021-03-28 20:25:35.459] [driver] [info] [Task Finished] Resolve in 0.084671s
[2021-03-28 20:25:35.464] [E/6] [info] [Task Finished] Resolve in 0.083447s
[2021-03-28 20:25:35.471] [E/7] [info] [Task Finished] Resolve in 0.079470s
[2021-03-28 20:25:35.473] [E/11] [info] [Task Finished] Resolve in 0.094168s
[2021-03-28 20:25:35.473] [E/12] [info] [Task Finished] Resolve in 0.076025s
[2021-03-28 20:25:35.476] [E/1] [info] [Task Finished] Resolve in 0.088893s
[2021-03-28 20:25:35.479] [E/13] [info] [Task Finished] Resolve in 0.157936s
[2021-03-28 20:25:35.490] [E/3] [info] [Task Finished] Resolve in 0.107219s
[2021-03-28 20:25:35.502] [E/5] [info] [Task Finished] Resolve in 0.078369s
[2021-03-28 20:25:35.519] [E/4] [info] [Task Finished] Resolve in 0.090357s
[2021-03-28 20:25:35.527] [E/2] [info] [Task Finished] Resolve in 0.152814s
[2021-03-28 20:25:35.527] [E/8] [info] [Task Finished] Resolve in 0.081159s
[2021-03-28 20:25:35.531] [E/9] [info] [Task Finished] Resolve in 0.081673s
[2021-03-28 20:25:35.532] [E/10] [info] [Task Finished] Resolve in 0.075127s
[2021-03-28 20:25:35.536] [E/15] [info] [Task Finished] Resolve in 0.080135s
[2021-03-28 20:25:35.547] [E/6] [info] [Task Finished] Resolve in 0.082835s
[2021-03-28 20:25:35.550] [E/14] [info] [Task Finished] Resolve in 0.092700s
[2021-03-28 20:25:35.554] [driver] [info] [Task Finished] Resolve in 0.094740s
[2021-03-28 20:25:35.601] [E/7] [info] [Task Finished] Resolve in 0.129778s
[2021-03-28 20:25:35.620] [E/11] [info] [Task Finished] Resolve in 0.147512s
[2021-03-28 20:25:35.620] [local ee] [info] slow path resolved 288283/288283 exceptions in 0.589089s
[2021-03-28 20:25:35.620] [local ee] [info] slow path for Stage 3: total wall clock time: 7.60432s, time to process 1 row via slow path: 0.026378ms
[2021-03-28 20:25:35.620] [global] [info] [Transform Stage] Stage 3 completed 86 resolve tasks in 0.58912s

[2021-03-28 20:25:36.026] [global] [info] Writing 14304381 rows as output to file (2.78 GB)
[2021-03-28 20:25:36.358] [E/12] [info] [Task Finished] write to file in 0.331672s (924895 rows, 183.72 MB)
[2021-03-28 20:25:36.358] [E/8] [info] [Task Finished] write to file in 0.331695s (892594 rows, 178.31 MB)
[2021-03-28 20:25:36.361] [E/7] [info] [Task Finished] write to file in 0.335490s (913340 rows, 182.57 MB)
[2021-03-28 20:25:36.362] [E/15] [info] [Task Finished] write to file in 0.336537s (941522 rows, 187.20 MB)
[2021-03-28 20:25:36.367] [E/4] [info] [Task Finished] write to file in 0.340812s (952475 rows, 189.54 MB)
[2021-03-28 20:25:36.367] [E/3] [info] [Task Finished] write to file in 0.340991s (962345 rows, 191.30 MB)
[2021-03-28 20:25:36.368] [E/13] [info] [Task Finished] write to file in 0.341815s (968506 rows, 193.28 MB)
[2021-03-28 20:25:36.369] [E/2] [info] [Task Finished] write to file in 0.342737s (971702 rows, 193.69 MB)
[2021-03-28 20:25:36.369] [driver] [info] [Task Finished] write to file in 0.343191s (937607 rows, 187.01 MB)
[2021-03-28 20:25:36.370] [E/10] [info] [Task Finished] write to file in 0.343737s (943257 rows, 187.98 MB)
[2021-03-28 20:25:36.370] [E/5] [info] [Task Finished] write to file in 0.344242s (972313 rows, 193.26 MB)
[2021-03-28 20:25:36.373] [E/11] [info] [Task Finished] write to file in 0.346683s (984982 rows, 195.76 MB)
[2021-03-28 20:25:36.375] [E/14] [info] [Task Finished] write to file in 0.348979s (1010746 rows, 200.70 MB)
[2021-03-28 20:25:36.384] [E/6] [info] [Task Finished] write to file in 0.358613s (1023547 rows, 203.63 MB)
[2021-03-28 20:25:36.430] [E/1] [info] [Task Finished] write to file in 0.404384s (904550 rows, 180.95 MB)
[2021-03-28 20:25:36.430] [global] [info] writing output took 0.809999s
[2021-03-28 20:25:36.430] [global] [info] [Transform Stage] Stage 3 completed 86 sink tasks in 0.810065s
[2021-03-28 20:25:36.430] [global] [info] [Transform Stage] Stage 3 took 6.71986s
[2021-03-28 20:25:36.430] [global] [info] Query Execution took 8.39203s. (planning: 1.15953s, execution: 7.23249s)
>>> running Tuplex on ['/data/flights_small/flights_on_time_performance_2019_11.csv', '/data/flights_small/flights_on_time_performance_2019_10.csv', '/data/flights_small/flights_on_time_performance_2019_09.csv', '/data/flights_small/flights_on_time_performance_2019_08.csv', '/data/flights_small/flights_on_time_performance_2019_07.csv', '/data/flights_small/flights_on_time_performance_2019_06.csv', '/data/flights_small/flights_on_time_performance_2019_05.csv', '/data/flights_small/flights_on_time_performance_2019_04.csv', '/data/flights_small/flights_on_time_performance_2019_03.csv', '/data/flights_small/flights_on_time_performance_2019_02.csv', '/data/flights_small/flights_on_time_performance_2019_01.csv', '/data/flights_small/flights_on_time_performance_2018_12.csv', '/data/flights_small/flights_on_time_performance_2018_11.csv', '/data/flights_small/flights_on_time_performance_2018_10.csv', '/data/flights_small/flights_on_time_performance_2018_09.csv', '/data/flights_small/flights_on_time_performance_2018_08.csv', '/data/flights_small/flights_on_time_performance_2018_07.csv', '/data/flights_small/flights_on_time_performance_2018_06.csv', '/data/flights_small/flights_on_time_performance_2018_05.csv', '/data/flights_small/flights_on_time_performance_2018_04.csv', '/data/flights_small/flights_on_time_performance_2018_03.csv', '/data/flights_small/flights_on_time_performance_2018_02.csv', '/data/flights_small/flights_on_time_performance_2018_01.csv']
Welcome to

  _____            _
 |_   _|   _ _ __ | | _____  __
   | || | | | '_ \| |/ _ \ \/ /
   | || |_| | |_) | |  __/>  <
   |_| \__,_| .__/|_|\___/_/\_\ 0.1.7
            |_|
    
using Python 3.6.9 (default, Mar 28 2021, 12:48:06) 
[GCC 10.2.0] on linux
Tuplex startup time: 0.2866373062133789
Tuplex job time: 8.96846890449524 s
{'tuplex.useLLVMOptimizer': True, 'tuplex.autoUpcast': True, 'tuplex.allowUndefinedBehavior': False, 'tuplex.optimizer.codeStats': False, 'tuplex.optimizer.generateParser': True, 'tuplex.optimizer.nullValueOptimization': True, 'tuplex.optimizer.filterPushdown': True, 'tuplex.optimizer.sharedObjectPropagation': True, 'tuplex.interleaveIO': True, 'tuplex.resolveWithInterpreterOnly': False, 'tuplex.csv.selectionPushdown': True, 'tuplex.webui.enable': False, 'tuplex.executorCount': 15, 'tuplex.csv.maxDetectionRows': 10000, 'tuplex.webui.port': 5000, 'tuplex.webui.mongodb.port': 27017, 'tuplex.webui.exceptionDisplayLimit': 5, 'tuplex.normalcaseThreshold': 0.9, 'tuplex.csv.comments': ['#', '~'], 'tuplex.csv.maxDetectionMemory': '256KB', 'tuplex.csv.quotechar': '"', 'tuplex.csv.separators': [',', ';', '|', '\t'], 'tuplex.driverMemory': '10G', 'tuplex.env.hostname': 'c415128f6054', 'tuplex.env.mode': 'file', 'tuplex.env.user': 'root', 'tuplex.executorMemory': '6G', 'tuplex.inputSplitSize': '64MB', 'tuplex.logDir': '.', 'tuplex.optimizer.mergeExceptionsInOrder': 'true', 'tuplex.optimizer.operatorReordering': 'false', 'tuplex.optionalThreshold': '0.7', 'tuplex.partitionSize': '32MB', 'tuplex.readBufferSize': '128KB', 'tuplex.runTimeLibrary': '/usr/local/lib/python3.6/site-packages/Tuplex-0.1.7-py3.6.egg/tuplex/libexec/tuplex_runtime', 'tuplex.runTimeMemory': '8MB', 'tuplex.runTimeMemoryBlockSize': '4MB', 'tuplex.scratchDir': '/tmp', 'tuplex.webui.mongodb.path': '/data/tuplex_history', 'tuplex.webui.mongodb.url': 'localhost', 'tuplex.webui.url': 'localhost'}
{"logical_optimization_time_s":0.149888,"llvm_optimization_time_s":0.873421,"llvm_compilation_time_s":0.473631,"total_compilation_time_s":1.72614,"sampling_time_s":0.515684,"stages":[{"stage_no":2,"fast_path_wall_time_s":0.00349124,"fast_path_time_s":0.00368161,"fast_path_per_row_time_ns":1810.81,"slow_path_wall_time_s":0,"slow_path_time_s":0,"slow_path_per_row_time_ns":0},{"stage_no":1,"fast_path_wall_time_s":0.00947938,"fast_path_time_s":0.00964933,"fast_path_per_row_time_ns":1019.29,"slow_path_wall_time_s":0,"slow_path_time_s":0,"slow_path_per_row_time_ns":0},{"stage_no":0,"fast_path_wall_time_s":0.00862263,"fast_path_time_s":0.00881151,"fast_path_per_row_time_ns":927.164,"slow_path_wall_time_s":0,"slow_path_time_s":0,"slow_path_per_row_time_ns":0},{"stage_no":3,"fast_path_wall_time_s":54.6063,"fast_path_time_s":4.08484,"fast_path_per_row_time_ns":3897.74,"slow_path_wall_time_s":7.60432,"slow_path_time_s":0.589089,"slow_path_per_row_time_ns":26378}]}
{"webui.enable": false, "executorCount": 15, "executorMemory": "6G", "driverMemory": "10G", "partitionSize": "32MB", "inputSplitSize": "64MB", "runTimeMemory": "8MB", "useLLVMOptimizer": true, "autoUpcast": true, "optimizer.generateParser": true, "optimizer.nullValueOptimization": true, "csv.selectionPushdown": true, "resolveWithInterpreterOnly": false}
{"startupTime": 0.2866373062133789, "jobTime": 8.96846890449524}
[GLOBAL] releasing 12 active partitions
[2021-03-28 20:25:36.469] [local execution engine] [info] Stopped local executor E/1
[GLOBAL] releasing 11 active partitions
[2021-03-28 20:25:36.488] [local execution engine] [info] Stopped local executor E/2
[GLOBAL] releasing 11 active partitions
[2021-03-28 20:25:36.507] [local execution engine] [info] Stopped local executor E/3
[GLOBAL] releasing 12 active partitions
[2021-03-28 20:25:36.526] [local execution engine] [info] Stopped local executor E/4
[GLOBAL] releasing 12 active partitions
[2021-03-28 20:25:36.546] [local execution engine] [info] Stopped local executor E/5
[GLOBAL] releasing 13 active partitions
[2021-03-28 20:25:36.564] [local execution engine] [info] Stopped local executor E/6
[GLOBAL] releasing 14 active partitions
[2021-03-28 20:25:36.584] [local execution engine] [info] Stopped local executor E/7
[GLOBAL] releasing 10 active partitions
[2021-03-28 20:25:36.606] [local execution engine] [info] Stopped local executor E/8
[GLOBAL] releasing 11 active partitions
[2021-03-28 20:25:36.625] [local execution engine] [info] Stopped local executor E/9
[GLOBAL] releasing 14 active partitions
[2021-03-28 20:25:36.645] [local execution engine] [info] Stopped local executor E/10
[GLOBAL] releasing 14 active partitions
[2021-03-28 20:25:36.666] [local execution engine] [info] Stopped local executor E/11
[GLOBAL] releasing 11 active partitions
[2021-03-28 20:25:36.689] [local execution engine] [info] Stopped local executor E/12
[GLOBAL] releasing 12 active partitions
[2021-03-28 20:25:36.707] [local execution engine] [info] Stopped local executor E/13
[GLOBAL] releasing 11 active partitions
[2021-03-28 20:25:36.726] [local execution engine] [info] Stopped local executor E/14
[GLOBAL] releasing 12 active partitions
[2021-03-28 20:25:36.745] [local execution engine] [info] Stopped local executor E/15
[GLOBAL] Local engine terminated.
[GLOBAL] releasing 17 active partitions
