[2021-03-25 10:31:18.830] [global] [error] could not find backend, defaulting to local
[2021-03-25 10:31:18.831] [local ee] [info] loaded runtime library from/usr/local/lib/python3.6/site-packages/Tuplex-0.1.7-py3.6.egg/tuplex/libexec/tuplex_runtime.so
[2021-03-25 10:31:18.831] [local ee] [info] initializing LLVM backend
[2021-03-25 10:31:18.831] [local ee] [warning] init JIT compiler also only in local mode
[2021-03-25 10:31:18.831] [LLVM] [info] compiling code for skylake-avx512
[2021-03-25 10:31:18.832] [memory] [info] allocated bitmap managed memory region (6.00 GB, 32.00 MB block size)
[2021-03-25 10:31:18.832] [memory] [info] allocated bitmap managed memory region (6.00 GB, 32.00 MB block size)
[2021-03-25 10:31:18.832] [memory] [info] allocated bitmap managed memory region (6.00 GB, 32.00 MB block size)
[2021-03-25 10:31:18.832] [memory] [info] allocated bitmap managed memory region (6.00 GB, 32.00 MB block size)
[2021-03-25 10:31:18.832] [memory] [info] allocated bitmap managed memory region (6.00 GB, 32.00 MB block size)
[2021-03-25 10:31:18.832] [memory] [info] allocated bitmap managed memory region (6.00 GB, 32.00 MB block size)
[2021-03-25 10:31:18.832] [memory] [info] allocated bitmap managed memory region (6.00 GB, 32.00 MB block size)
[2021-03-25 10:31:18.832] [memory] [info] allocated bitmap managed memory region (6.00 GB, 32.00 MB block size)
[2021-03-25 10:31:18.832] [memory] [info] allocated bitmap managed memory region (6.00 GB, 32.00 MB block size)
[2021-03-25 10:31:18.832] [memory] [info] allocated bitmap managed memory region (6.00 GB, 32.00 MB block size)
[2021-03-25 10:31:18.832] [memory] [info] allocated bitmap managed memory region (6.00 GB, 32.00 MB block size)
[2021-03-25 10:31:18.832] [memory] [info] allocated bitmap managed memory region (6.00 GB, 32.00 MB block size)
[2021-03-25 10:31:18.832] [memory] [info] allocated bitmap managed memory region (6.00 GB, 32.00 MB block size)
[2021-03-25 10:31:18.832] [memory] [info] allocated bitmap managed memory region (6.00 GB, 32.00 MB block size)
[2021-03-25 10:31:18.832] [memory] [info] allocated bitmap managed memory region (6.00 GB, 32.00 MB block size)
[2021-03-25 10:31:18.832] [local execution engine] [info] started local executor E/1 (6.00 GB, 32.00 MB default partition size)
[2021-03-25 10:31:18.832] [E/1] [info] starting detached process queue
[2021-03-25 10:31:18.832] [local execution engine] [info] started local executor E/2 (6.00 GB, 32.00 MB default partition size)
[2021-03-25 10:31:18.832] [E/2] [info] starting detached process queue
[2021-03-25 10:31:18.832] [E/1] [info] initialized runtime memory (4.00 MB)
[2021-03-25 10:31:18.832] [local execution engine] [info] started local executor E/3 (6.00 GB, 32.00 MB default partition size)
[2021-03-25 10:31:18.832] [E/2] [info] initialized runtime memory (4.00 MB)
[2021-03-25 10:31:18.832] [E/3] [info] starting detached process queue
[2021-03-25 10:31:18.832] [local execution engine] [info] started local executor E/4 (6.00 GB, 32.00 MB default partition size)
[2021-03-25 10:31:18.832] [E/3] [info] initialized runtime memory (4.00 MB)
[2021-03-25 10:31:18.832] [E/4] [info] starting detached process queue
[2021-03-25 10:31:18.832] [local execution engine] [info] started local executor E/5 (6.00 GB, 32.00 MB default partition size)
[2021-03-25 10:31:18.832] [E/5] [info] starting detached process queue
[2021-03-25 10:31:18.832] [E/4] [info] initialized runtime memory (4.00 MB)
[2021-03-25 10:31:18.832] [local execution engine] [info] started local executor E/6 (6.00 GB, 32.00 MB default partition size)
[2021-03-25 10:31:18.832] [E/5] [info] initialized runtime memory (4.00 MB)
[2021-03-25 10:31:18.832] [E/6] [info] starting detached process queue
[2021-03-25 10:31:18.832] [local execution engine] [info] started local executor E/7 (6.00 GB, 32.00 MB default partition size)
[2021-03-25 10:31:18.832] [E/7] [info] starting detached process queue
[2021-03-25 10:31:18.832] [E/6] [info] initialized runtime memory (4.00 MB)
[2021-03-25 10:31:18.832] [E/7] [info] initialized runtime memory (4.00 MB)
[2021-03-25 10:31:18.832] [local execution engine] [info] started local executor E/8 (6.00 GB, 32.00 MB default partition size)
[2021-03-25 10:31:18.832] [E/8] [info] starting detached process queue
[2021-03-25 10:31:18.832] [local execution engine] [info] started local executor E/9 (6.00 GB, 32.00 MB default partition size)
[2021-03-25 10:31:18.832] [E/8] [info] initialized runtime memory (4.00 MB)
[2021-03-25 10:31:18.832] [E/9] [info] starting detached process queue
[2021-03-25 10:31:18.832] [local execution engine] [info] started local executor E/10 (6.00 GB, 32.00 MB default partition size)
[2021-03-25 10:31:18.832] [E/10] [info] starting detached process queue
[2021-03-25 10:31:18.832] [E/9] [info] initialized runtime memory (4.00 MB)
[2021-03-25 10:31:18.832] [local execution engine] [info] started local executor E/11 (6.00 GB, 32.00 MB default partition size)
[2021-03-25 10:31:18.832] [E/10] [info] initialized runtime memory (4.00 MB)
[2021-03-25 10:31:18.832] [E/11] [info] starting detached process queue
[2021-03-25 10:31:18.833] [local execution engine] [info] started local executor E/12 (6.00 GB, 32.00 MB default partition size)
[2021-03-25 10:31:18.833] [E/11] [info] initialized runtime memory (4.00 MB)
[2021-03-25 10:31:18.833] [E/12] [info] starting detached process queue
[2021-03-25 10:31:18.833] [E/12] [info] initialized runtime memory (4.00 MB)
[2021-03-25 10:31:18.833] [local execution engine] [info] started local executor E/13 (6.00 GB, 32.00 MB default partition size)
[2021-03-25 10:31:18.833] [E/13] [info] starting detached process queue
[2021-03-25 10:31:18.833] [local execution engine] [info] started local executor E/14 (6.00 GB, 32.00 MB default partition size)
[2021-03-25 10:31:18.833] [E/13] [info] initialized runtime memory (4.00 MB)
[2021-03-25 10:31:18.833] [E/14] [info] starting detached process queue
[2021-03-25 10:31:18.833] [local execution engine] [info] started local executor E/15 (6.00 GB, 32.00 MB default partition size)
[2021-03-25 10:31:18.833] [E/15] [info] starting detached process queue
[2021-03-25 10:31:18.833] [E/14] [info] initialized runtime memory (4.00 MB)
[2021-03-25 10:31:18.833] [memory] [info] allocated bitmap managed memory region (10.00 GB, 32.00 MB block size)
[2021-03-25 10:31:18.833] [E/15] [info] initialized runtime memory (4.00 MB)
[2021-03-25 10:31:18.833] [local execution engine] [info] started driver (10.00 GB, 32.00 MB default partition size)
overflow will be cached at file:///tmp/driver
[2021-03-25 10:31:18.833] [fileinputoperator] [info] found 1 file (10.61 GB) to process.
[2021-03-25 10:31:18.833] [global] [info] sampled file:///data/zillow/Zdirty/zillow_dirty_synthetic@10G.csv on 256.00 KB
[2021-03-25 10:31:18.968] [global] [info] performing static typing for UDF in operator withColumn
[2021-03-25 10:31:18.984] [global] [info] performing static typing for UDF in operator withColumn
[2021-03-25 10:31:18.992] [global] [info] performing static typing for UDF in operator withColumn
[2021-03-25 10:31:19.020] [global] [info] performing static typing for UDF in operator withColumn
[2021-03-25 10:31:19.040] [global] [info] performing static typing for UDF in operator withColumn
[2021-03-25 10:31:19.054] [global] [info] performing static typing for UDF in operator withColumn
[2021-03-25 10:31:19.086] [global] [info] performing static typing for UDF in operator withColumn
[2021-03-25 10:31:19.103] [global] [info] performing static typing for UDF in operator map
[2021-03-25 10:31:19.106] [global] [info] performing static typing for UDF in operator map
[2021-03-25 10:31:19.108] [global] [info] performing static typing for UDF in operator map
[2021-03-25 10:31:19.111] [global] [info] performing static typing for UDF in operator map
[2021-03-25 10:31:19.127] [logical planner] [info] logical optimization took 0.020593ms
output type of specialized pipeline is: (str)
is this the most outer stage?: 1
[2021-03-25 10:31:19.127] [codegen] [info] generating pipeline for (Option[str],Option[str],Option[str],Option[str],Option[f64],Option[str],Option[str],Option[str]) -> (str) (13 operators pipelined)
outSchema is different than last operator's schema:
outSchema: (str)
last Op: (Option[str],str,Option[str],str,Option[str],i64,f64,i64,str,str,i64)
[2021-03-25 10:31:19.127] [codegen] [info] generating function extractBd for ((Option[str],Option[str],Option[str],Option[str],Option[f64],Option[str],Option[str],Option[str])) -> i64
[2021-03-25 10:31:19.128] [codegen] [info] generating lambda function for ((Option[str],Option[str],Option[str],Option[str],Option[f64],Option[str],Option[str],Option[str],i64)) -> boolean
[2021-03-25 10:31:19.129] [codegen] [info] generating function extractType for ((Option[str],Option[str],Option[str],Option[str],Option[f64],Option[str],Option[str],Option[str],i64)) -> str
[2021-03-25 10:31:19.129] [codegen] [info] generating lambda function for ((Option[str],Option[str],Option[str],Option[str],Option[f64],Option[str],Option[str],Option[str],i64,str)) -> boolean
[2021-03-25 10:31:19.130] [codegen] [info] generating lambda function for ((Option[str],Option[str],Option[str],Option[str],Option[f64],Option[str],Option[str],Option[str],i64,str)) -> str
[2021-03-25 10:31:19.130] [codegen] [info] generating lambda function for (Option[str]) -> str
[2021-03-25 10:31:19.131] [codegen] [info] generating function extractBa for ((Option[str],Option[str],str,Option[str],Option[f64],Option[str],Option[str],Option[str],i64,str,str)) -> f64
[2021-03-25 10:31:19.132] [codegen] [info] generating function extractSqft for ((Option[str],Option[str],str,Option[str],Option[f64],Option[str],Option[str],Option[str],i64,str,str,f64)) -> i64
[2021-03-25 10:31:19.133] [codegen] [info] generating function extractOffer for ((Option[str],Option[str],str,Option[str],Option[f64],Option[str],Option[str],Option[str],i64,str,str,f64,i64)) -> str
[2021-03-25 10:31:19.134] [codegen] [info] generating function extractPrice for ((Option[str],Option[str],str,Option[str],Option[f64],Option[str],Option[str],Option[str],i64,str,str,f64,i64,str)) -> i64
[2021-03-25 10:31:19.135] [codegen] [info] generating lambda function for ((Option[str],Option[str],str,Option[str],Option[f64],i64,Option[str],Option[str],i64,str,str,f64,i64,str)) -> boolean
[2021-03-25 10:31:19.136] [codegen] [info] generating lambda function for ((Option[str],Option[str],str,Option[str],Option[f64],i64,Option[str],Option[str],i64,str,str,f64,i64,str)) -> boolean
[2021-03-25 10:31:19.136] [codegen] [info] generating lambda function for ((Option[str],Option[str],str,Option[str],Option[f64],i64,Option[str],Option[str],i64,str,str,f64,i64,str)) -> (Option[str],str,Option[str],str,Option[str],i64,f64,i64,str,str,i64)
[2021-03-25 10:31:19.162] [global] [info] Stage0 depends on: 
[2021-03-25 10:31:19.308] [global] [info] Optimization via LLVM passes took 0.138081 ms
[2021-03-25 10:31:19.370] [global] [info] Compiled code paths for stage 0 in 0.06 ms
[2021-03-25 10:31:19.370] [global] [info] [Transform Stage] Stage 0 compiled to x86 in 0.207795s
[2021-03-25 10:31:19.370] [local ee] [info] split /data/zillow/Zdirty/zillow_dirty_synthetic@10G.csv into 169 tasks
[2021-03-25 10:31:19.633] [E/3] [info] [Task Finished] Transform to mem in 0.262332s (10378 normal rows, 82714 exceptions)
[2021-03-25 10:31:19.633] [E/9] [info] [Task Finished] Transform to mem in 0.262525s (10243 normal rows, 83069 exceptions)
[2021-03-25 10:31:19.633] [driver] [info] [Task Finished] Transform to mem in 0.262761s (10252 normal rows, 83089 exceptions)
[2021-03-25 10:31:19.633] [E/2] [info] [Task Finished] Transform to mem in 0.262993s (10386 normal rows, 82651 exceptions)
[2021-03-25 10:31:19.633] [E/12] [info] [Task Finished] Transform to mem in 0.263000s (10505 normal rows, 82883 exceptions)
[2021-03-25 10:31:19.634] [E/4] [info] [Task Finished] Transform to mem in 0.263071s (10396 normal rows, 83077 exceptions)
[2021-03-25 10:31:19.634] [E/14] [info] [Task Finished] Transform to mem in 0.263139s (10542 normal rows, 82535 exceptions)
[2021-03-25 10:31:19.634] [E/6] [info] [Task Finished] Transform to mem in 0.263229s (10406 normal rows, 82712 exceptions)
[2021-03-25 10:31:19.634] [E/7] [info] [Task Finished] Transform to mem in 0.263321s (10215 normal rows, 82926 exceptions)
[2021-03-25 10:31:19.634] [E/10] [info] [Task Finished] Transform to mem in 0.263445s (10359 normal rows, 82920 exceptions)
[2021-03-25 10:31:19.634] [E/13] [info] [Task Finished] Transform to mem in 0.263448s (10514 normal rows, 82793 exceptions)
[2021-03-25 10:31:19.634] [E/5] [info] [Task Finished] Transform to mem in 0.263649s (10365 normal rows, 82838 exceptions)
[2021-03-25 10:31:19.634] [E/15] [info] [Task Finished] Transform to mem in 0.263803s (10531 normal rows, 82507 exceptions)
[2021-03-25 10:31:19.634] [E/8] [info] [Task Finished] Transform to mem in 0.263967s (10575 normal rows, 82724 exceptions)
[2021-03-25 10:31:19.635] [E/1] [info] [Task Finished] Transform to mem in 0.264322s (10532 normal rows, 83146 exceptions)
[2021-03-25 10:31:19.637] [E/11] [info] [Task Finished] Transform to mem in 0.266280s (10512 normal rows, 83014 exceptions)
[2021-03-25 10:31:19.892] [E/3] [info] [Task Finished] Transform to mem in 0.259458s (10378 normal rows, 82790 exceptions)
[2021-03-25 10:31:19.892] [E/12] [info] [Task Finished] Transform to mem in 0.258911s (10514 normal rows, 82719 exceptions)
[2021-03-25 10:31:19.893] [E/4] [info] [Task Finished] Transform to mem in 0.259476s (10388 normal rows, 82649 exceptions)
[2021-03-25 10:31:19.893] [E/2] [info] [Task Finished] Transform to mem in 0.259600s (10288 normal rows, 83066 exceptions)
[2021-03-25 10:31:19.893] [E/9] [info] [Task Finished] Transform to mem in 0.260175s (10373 normal rows, 83047 exceptions)
[2021-03-25 10:31:19.893] [E/13] [info] [Task Finished] Transform to mem in 0.259337s (10311 normal rows, 82729 exceptions)
[2021-03-25 10:31:19.893] [E/6] [info] [Task Finished] Transform to mem in 0.259636s (10510 normal rows, 82396 exceptions)
[2021-03-25 10:31:19.893] [driver] [info] [Task Finished] Transform to mem in 0.260127s (10510 normal rows, 82547 exceptions)
[2021-03-25 10:31:19.893] [E/7] [info] [Task Finished] Transform to mem in 0.259681s (10288 normal rows, 83177 exceptions)
[2021-03-25 10:31:19.894] [E/10] [info] [Task Finished] Transform to mem in 0.259652s (10514 normal rows, 82639 exceptions)
[2021-03-25 10:31:19.894] [E/8] [info] [Task Finished] Transform to mem in 0.259356s (10317 normal rows, 83176 exceptions)
[2021-03-25 10:31:19.894] [E/14] [info] [Task Finished] Transform to mem in 0.260191s (10373 normal rows, 83312 exceptions)
[2021-03-25 10:31:19.894] [E/5] [info] [Task Finished] Transform to mem in 0.259715s (10450 normal rows, 83093 exceptions)
[2021-03-25 10:31:19.895] [E/1] [info] [Task Finished] Transform to mem in 0.260272s (10522 normal rows, 82584 exceptions)
[2021-03-25 10:31:19.895] [E/15] [info] [Task Finished] Transform to mem in 0.261009s (10461 normal rows, 82531 exceptions)
[2021-03-25 10:31:19.898] [E/11] [info] [Task Finished] Transform to mem in 0.261542s (10323 normal rows, 82590 exceptions)
[2021-03-25 10:31:20.152] [E/12] [info] [Task Finished] Transform to mem in 0.259345s (10480 normal rows, 82481 exceptions)
[2021-03-25 10:31:20.153] [E/4] [info] [Task Finished] Transform to mem in 0.259562s (10317 normal rows, 83180 exceptions)
[2021-03-25 10:31:20.153] [E/9] [info] [Task Finished] Transform to mem in 0.259526s (10304 normal rows, 82568 exceptions)
[2021-03-25 10:31:20.153] [E/6] [info] [Task Finished] Transform to mem in 0.259677s (10425 normal rows, 82382 exceptions)
[2021-03-25 10:31:20.153] [E/10] [info] [Task Finished] Transform to mem in 0.259447s (10303 normal rows, 82500 exceptions)
[2021-03-25 10:31:20.153] [E/14] [info] [Task Finished] Transform to mem in 0.259291s (10421 normal rows, 82435 exceptions)
[2021-03-25 10:31:20.153] [E/13] [info] [Task Finished] Transform to mem in 0.259846s (10434 normal rows, 83448 exceptions)
[2021-03-25 10:31:20.153] [E/2] [info] [Task Finished] Transform to mem in 0.260178s (10522 normal rows, 82568 exceptions)
[2021-03-25 10:31:20.153] [E/3] [info] [Task Finished] Transform to mem in 0.261183s (10431 normal rows, 83410 exceptions)
[2021-03-25 10:31:20.154] [E/5] [info] [Task Finished] Transform to mem in 0.259851s (10379 normal rows, 83209 exceptions)
[2021-03-25 10:31:20.154] [driver] [info] [Task Finished] Transform to mem in 0.260411s (10353 normal rows, 83264 exceptions)
[2021-03-25 10:31:20.154] [E/8] [info] [Task Finished] Transform to mem in 0.260341s (10428 normal rows, 83672 exceptions)
[2021-03-25 10:31:20.155] [E/7] [info] [Task Finished] Transform to mem in 0.261078s (10536 normal rows, 82412 exceptions)
[2021-03-25 10:31:20.155] [E/1] [info] [Task Finished] Transform to mem in 0.260387s (10537 normal rows, 82448 exceptions)
[2021-03-25 10:31:20.156] [E/15] [info] [Task Finished] Transform to mem in 0.260861s (10295 normal rows, 82359 exceptions)
[2021-03-25 10:31:20.161] [E/11] [info] [Task Finished] Transform to mem in 0.262710s (10441 normal rows, 83814 exceptions)
[2021-03-25 10:31:20.411] [E/12] [info] [Task Finished] Transform to mem in 0.259421s (10421 normal rows, 82390 exceptions)
[2021-03-25 10:31:20.412] [E/6] [info] [Task Finished] Transform to mem in 0.259221s (10285 normal rows, 82239 exceptions)
[2021-03-25 10:31:20.412] [E/14] [info] [Task Finished] Transform to mem in 0.259307s (10413 normal rows, 82527 exceptions)
[2021-03-25 10:31:20.413] [E/4] [info] [Task Finished] Transform to mem in 0.260006s (10366 normal rows, 83177 exceptions)
[2021-03-25 10:31:20.413] [E/9] [info] [Task Finished] Transform to mem in 0.260083s (10550 normal rows, 82536 exceptions)
[2021-03-25 10:31:20.413] [E/13] [info] [Task Finished] Transform to mem in 0.260018s (10387 normal rows, 83218 exceptions)
[2021-03-25 10:31:20.413] [E/10] [info] [Task Finished] Transform to mem in 0.260138s (10438 normal rows, 83705 exceptions)
[2021-03-25 10:31:20.414] [E/3] [info] [Task Finished] Transform to mem in 0.260059s (10207 normal rows, 82548 exceptions)
[2021-03-25 10:31:20.414] [E/2] [info] [Task Finished] Transform to mem in 0.260348s (10550 normal rows, 82245 exceptions)
[2021-03-25 10:31:20.414] [driver] [info] [Task Finished] Transform to mem in 0.259894s (10399 normal rows, 82487 exceptions)
[2021-03-25 10:31:20.414] [E/5] [info] [Task Finished] Transform to mem in 0.260211s (10516 normal rows, 83753 exceptions)
[2021-03-25 10:31:20.414] [E/8] [info] [Task Finished] Transform to mem in 0.259861s (10372 normal rows, 82987 exceptions)
[2021-03-25 10:31:20.415] [E/1] [info] [Task Finished] Transform to mem in 0.259519s (10225 normal rows, 82594 exceptions)
[2021-03-25 10:31:20.416] [E/7] [info] [Task Finished] Transform to mem in 0.261589s (10541 normal rows, 82369 exceptions)
[2021-03-25 10:31:20.417] [E/15] [info] [Task Finished] Transform to mem in 0.261228s (10536 normal rows, 83589 exceptions)
[2021-03-25 10:31:20.423] [E/11] [info] [Task Finished] Transform to mem in 0.261679s (10387 normal rows, 82745 exceptions)
[2021-03-25 10:31:20.671] [E/12] [info] [Task Finished] Transform to mem in 0.259972s (10384 normal rows, 82992 exceptions)
[2021-03-25 10:31:20.672] [E/14] [info] [Task Finished] Transform to mem in 0.259694s (10217 normal rows, 82839 exceptions)
[2021-03-25 10:31:20.672] [E/6] [info] [Task Finished] Transform to mem in 0.260144s (10527 normal rows, 82110 exceptions)
[2021-03-25 10:31:20.672] [E/9] [info] [Task Finished] Transform to mem in 0.259658s (10380 normal rows, 82714 exceptions)
[2021-03-25 10:31:20.673] [E/13] [info] [Task Finished] Transform to mem in 0.259416s (10413 normal rows, 82733 exceptions)
[2021-03-25 10:31:20.673] [E/4] [info] [Task Finished] Transform to mem in 0.260446s (10536 normal rows, 83591 exceptions)
[2021-03-25 10:31:20.673] [E/10] [info] [Task Finished] Transform to mem in 0.260201s (10527 normal rows, 82391 exceptions)
[2021-03-25 10:31:20.674] [E/3] [info] [Task Finished] Transform to mem in 0.260131s (10212 normal rows, 82864 exceptions)
[2021-03-25 10:31:20.674] [driver] [info] [Task Finished] Transform to mem in 0.260008s (10380 normal rows, 82666 exceptions)
[2021-03-25 10:31:20.674] [E/5] [info] [Task Finished] Transform to mem in 0.260116s (10395 normal rows, 82884 exceptions)
[2021-03-25 10:31:20.674] [E/2] [info] [Task Finished] Transform to mem in 0.260681s (10541 normal rows, 83305 exceptions)
[2021-03-25 10:31:20.675] [E/8] [info] [Task Finished] Transform to mem in 0.260688s (10537 normal rows, 82571 exceptions)
[2021-03-25 10:31:20.675] [E/1] [info] [Task Finished] Transform to mem in 0.259902s (10211 normal rows, 82855 exceptions)
[2021-03-25 10:31:20.678] [E/15] [info] [Task Finished] Transform to mem in 0.261020s (10403 normal rows, 82568 exceptions)
[2021-03-25 10:31:20.679] [E/7] [info] [Task Finished] Transform to mem in 0.262702s (10524 normal rows, 83296 exceptions)
[2021-03-25 10:31:20.685] [E/11] [info] [Task Finished] Transform to mem in 0.261999s (10382 normal rows, 82778 exceptions)
[2021-03-25 10:31:20.931] [E/12] [info] [Task Finished] Transform to mem in 0.259857s (10541 normal rows, 82755 exceptions)
[2021-03-25 10:31:20.931] [E/14] [info] [Task Finished] Transform to mem in 0.259249s (10215 normal rows, 82861 exceptions)
[2021-03-25 10:31:20.932] [E/9] [info] [Task Finished] Transform to mem in 0.259765s (10386 normal rows, 82689 exceptions)
[2021-03-25 10:31:20.933] [E/6] [info] [Task Finished] Transform to mem in 0.260092s (10532 normal rows, 83104 exceptions)
[2021-03-25 10:31:20.933] [E/13] [info] [Task Finished] Transform to mem in 0.259988s (10365 normal rows, 82841 exceptions)
[2021-03-25 10:31:20.933] [E/10] [info] [Task Finished] Transform to mem in 0.259275s (10215 normal rows, 82938 exceptions)
[2021-03-25 10:31:20.933] [E/4] [info] [Task Finished] Transform to mem in 0.259660s (10575 normal rows, 82687 exceptions)
[2021-03-25 10:31:20.934] [driver] [info] [Task Finished] Transform to mem in 0.259945s (10406 normal rows, 82684 exceptions)
[2021-03-25 10:31:20.934] [E/5] [info] [Task Finished] Transform to mem in 0.260019s (10359 normal rows, 82956 exceptions)
[2021-03-25 10:31:20.935] [E/8] [info] [Task Finished] Transform to mem in 0.259770s (10252 normal rows, 83088 exceptions)
[2021-03-25 10:31:20.935] [E/2] [info] [Task Finished] Transform to mem in 0.260337s (10531 normal rows, 82508 exceptions)
[2021-03-25 10:31:20.935] [E/3] [info] [Task Finished] Transform to mem in 0.261073s (10512 normal rows, 83031 exceptions)
[2021-03-25 10:31:20.936] [E/1] [info] [Task Finished] Transform to mem in 0.260605s (10505 normal rows, 82881 exceptions)
[2021-03-25 10:31:20.939] [E/15] [info] [Task Finished] Transform to mem in 0.260750s (10378 normal rows, 82714 exceptions)
[2021-03-25 10:31:20.940] [E/7] [info] [Task Finished] Transform to mem in 0.260850s (10396 normal rows, 83080 exceptions)
[2021-03-25 10:31:20.946] [E/11] [info] [Task Finished] Transform to mem in 0.261714s (10542 normal rows, 82507 exceptions)
[2021-03-25 10:31:21.190] [E/12] [info] [Task Finished] Transform to mem in 0.258988s (10243 normal rows, 83097 exceptions)
[2021-03-25 10:31:21.191] [E/14] [info] [Task Finished] Transform to mem in 0.259926s (10514 normal rows, 82752 exceptions)
[2021-03-25 10:31:21.192] [E/10] [info] [Task Finished] Transform to mem in 0.259562s (10288 normal rows, 83067 exceptions)
[2021-03-25 10:31:21.192] [E/9] [info] [Task Finished] Transform to mem in 0.260040s (10378 normal rows, 82785 exceptions)
[2021-03-25 10:31:21.192] [E/13] [info] [Task Finished] Transform to mem in 0.259709s (10510 normal rows, 82546 exceptions)
[2021-03-25 10:31:21.193] [E/4] [info] [Task Finished] Transform to mem in 0.259864s (10514 normal rows, 82716 exceptions)
[2021-03-25 10:31:21.193] [E/6] [info] [Task Finished] Transform to mem in 0.260944s (10373 normal rows, 83092 exceptions)
[2021-03-25 10:31:21.194] [driver] [info] [Task Finished] Transform to mem in 0.260126s (10353 normal rows, 82638 exceptions)
[2021-03-25 10:31:21.194] [E/5] [info] [Task Finished] Transform to mem in 0.260103s (10408 normal rows, 83285 exceptions)
[2021-03-25 10:31:21.194] [E/8] [info] [Task Finished] Transform to mem in 0.259667s (10510 normal rows, 82399 exceptions)
[2021-03-25 10:31:21.194] [E/2] [info] [Task Finished] Transform to mem in 0.259566s (10288 normal rows, 83178 exceptions)
[2021-03-25 10:31:21.195] [E/1] [info] [Task Finished] Transform to mem in 0.259934s (10311 normal rows, 82729 exceptions)
[2021-03-25 10:31:21.196] [E/3] [info] [Task Finished] Transform to mem in 0.260924s (10514 normal rows, 82630 exceptions)
[2021-03-25 10:31:21.200] [E/15] [info] [Task Finished] Transform to mem in 0.261088s (10435 normal rows, 83133 exceptions)
[2021-03-25 10:31:21.201] [E/7] [info] [Task Finished] Transform to mem in 0.261130s (10476 normal rows, 82536 exceptions)
[2021-03-25 10:31:21.209] [E/11] [info] [Task Finished] Transform to mem in 0.262104s (10317 normal rows, 83177 exceptions)
[2021-03-25 10:31:21.450] [E/12] [info] [Task Finished] Transform to mem in 0.259830s (10522 normal rows, 82577 exceptions)
[2021-03-25 10:31:21.451] [E/14] [info] [Task Finished] Transform to mem in 0.259806s (10304 normal rows, 82572 exceptions)
[2021-03-25 10:31:21.453] [E/13] [info] [Task Finished] Transform to mem in 0.260313s (10317 normal rows, 83176 exceptions)
[2021-03-25 10:31:21.453] [E/10] [info] [Task Finished] Transform to mem in 0.260720s (10450 normal rows, 83434 exceptions)
[2021-03-25 10:31:21.453] [E/9] [info] [Task Finished] Transform to mem in 0.261026s (10480 normal rows, 82448 exceptions)
[2021-03-25 10:31:21.454] [E/6] [info] [Task Finished] Transform to mem in 0.260368s (10304 normal rows, 82556 exceptions)
[2021-03-25 10:31:21.454] [E/4] [info] [Task Finished] Transform to mem in 0.261496s (10522 normal rows, 82561 exceptions)
[2021-03-25 10:31:21.455] [E/5] [info] [Task Finished] Transform to mem in 0.260506s (10399 normal rows, 82383 exceptions)
[2021-03-25 10:31:21.455] [driver] [info] [Task Finished] Transform to mem in 0.261068s (10434 normal rows, 83503 exceptions)
[2021-03-25 10:31:21.455] [E/8] [info] [Task Finished] Transform to mem in 0.261232s (10379 normal rows, 83265 exceptions)
[2021-03-25 10:31:21.456] [E/2] [info] [Task Finished] Transform to mem in 0.261277s (10536 normal rows, 82412 exceptions)
[2021-03-25 10:31:21.457] [E/1] [info] [Task Finished] Transform to mem in 0.261329s (10290 normal rows, 82497 exceptions)
[2021-03-25 10:31:21.458] [E/3] [info] [Task Finished] Transform to mem in 0.262351s (10441 normal rows, 83672 exceptions)
[2021-03-25 10:31:21.462] [E/15] [info] [Task Finished] Transform to mem in 0.261748s (10421 normal rows, 82438 exceptions)
[2021-03-25 10:31:21.463] [E/7] [info] [Task Finished] Transform to mem in 0.261819s (10379 normal rows, 83209 exceptions)
[2021-03-25 10:31:21.471] [E/11] [info] [Task Finished] Transform to mem in 0.262242s (10537 normal rows, 82444 exceptions)
[2021-03-25 10:31:21.710] [E/12] [info] [Task Finished] Transform to mem in 0.259674s (10295 normal rows, 82316 exceptions)
[2021-03-25 10:31:21.712] [E/14] [info] [Task Finished] Transform to mem in 0.261040s (10441 normal rows, 83796 exceptions)
[2021-03-25 10:31:21.712] [E/13] [info] [Task Finished] Transform to mem in 0.259694s (10421 normal rows, 82414 exceptions)
[2021-03-25 10:31:21.713] [E/6] [info] [Task Finished] Transform to mem in 0.259360s (10285 normal rows, 82265 exceptions)
[2021-03-25 10:31:21.714] [E/9] [info] [Task Finished] Transform to mem in 0.260463s (10550 normal rows, 82481 exceptions)
[2021-03-25 10:31:21.714] [E/10] [info] [Task Finished] Transform to mem in 0.260839s (10366 normal rows, 83213 exceptions)
[2021-03-25 10:31:21.715] [E/5] [info] [Task Finished] Transform to mem in 0.260231s (10413 normal rows, 82527 exceptions)
[2021-03-25 10:31:21.715] [E/4] [info] [Task Finished] Transform to mem in 0.261062s (10438 normal rows, 83738 exceptions)
[2021-03-25 10:31:21.715] [driver] [info] [Task Finished] Transform to mem in 0.260455s (10387 normal rows, 83171 exceptions)
[2021-03-25 10:31:21.716] [E/8] [info] [Task Finished] Transform to mem in 0.260633s (10524 normal rows, 82267 exceptions)
[2021-03-25 10:31:21.718] [E/2] [info] [Task Finished] Transform to mem in 0.261945s (10233 normal rows, 82572 exceptions)
[2021-03-25 10:31:21.718] [E/1] [info] [Task Finished] Transform to mem in 0.261514s (10516 normal rows, 83709 exceptions)
[2021-03-25 10:31:21.719] [E/3] [info] [Task Finished] Transform to mem in 0.260918s (10387 normal rows, 82531 exceptions)
[2021-03-25 10:31:21.723] [E/15] [info] [Task Finished] Transform to mem in 0.261045s (10384 normal rows, 82987 exceptions)
[2021-03-25 10:31:21.724] [E/7] [info] [Task Finished] Transform to mem in 0.261621s (10541 normal rows, 82314 exceptions)
[2021-03-25 10:31:21.732] [E/11] [info] [Task Finished] Transform to mem in 0.261225s (10205 normal rows, 82633 exceptions)
[2021-03-25 10:31:21.970] [E/12] [info] [Task Finished] Transform to mem in 0.260740s (10534 normal rows, 83592 exceptions)
[2021-03-25 10:31:21.972] [E/14] [info] [Task Finished] Transform to mem in 0.260175s (10406 normal rows, 82739 exceptions)
[2021-03-25 10:31:21.973] [E/13] [info] [Task Finished] Transform to mem in 0.260582s (10387 normal rows, 82968 exceptions)
[2021-03-25 10:31:21.974] [E/6] [info] [Task Finished] Transform to mem in 0.260324s (10527 normal rows, 82155 exceptions)
[2021-03-25 10:31:21.974] [E/9] [info] [Task Finished] Transform to mem in 0.259767s (10217 normal rows, 82840 exceptions)
[2021-03-25 10:31:21.975] [E/10] [info] [Task Finished] Transform to mem in 0.261217s (10536 normal rows, 83542 exceptions)
[2021-03-25 10:31:21.975] [E/5] [info] [Task Finished] Transform to mem in 0.260410s (10380 normal rows, 82709 exceptions)
[2021-03-25 10:31:21.976] [E/4] [info] [Task Finished] Transform to mem in 0.260329s (10395 normal rows, 82771 exceptions)
[2021-03-25 10:31:21.976] [E/8] [info] [Task Finished] Transform to mem in 0.259935s (10220 normal rows, 82876 exceptions)
[2021-03-25 10:31:21.977] [driver] [info] [Task Finished] Transform to mem in 0.261944s (10537 normal rows, 82395 exceptions)
[2021-03-25 10:31:21.978] [E/2] [info] [Task Finished] Transform to mem in 0.260938s (10515 normal rows, 83284 exceptions)
[2021-03-25 10:31:21.980] [E/1] [info] [Task Finished] Transform to mem in 0.261369s (10403 normal rows, 82663 exceptions)
[2021-03-25 10:31:21.980] [E/3] [info] [Task Finished] Transform to mem in 0.261266s (10398 normal rows, 82849 exceptions)
[2021-03-25 10:31:21.985] [E/15] [info] [Task Finished] Transform to mem in 0.261936s (10537 normal rows, 82629 exceptions)
[2021-03-25 10:31:21.986] [E/7] [info] [Task Finished] Transform to mem in 0.261442s (10211 normal rows, 82855 exceptions)
[2021-03-25 10:31:21.996] [E/11] [info] [Task Finished] Transform to mem in 0.263907s (10524 normal rows, 83295 exceptions)
[2021-03-25 10:31:22.230] [E/12] [info] [Task Finished] Transform to mem in 0.259321s (10403 normal rows, 82520 exceptions)
[2021-03-25 10:31:22.232] [E/14] [info] [Task Finished] Transform to mem in 0.259918s (10382 normal rows, 82801 exceptions)
[2021-03-25 10:31:22.233] [E/13] [info] [Task Finished] Transform to mem in 0.260180s (10541 normal rows, 82781 exceptions)
[2021-03-25 10:31:22.233] [E/6] [info] [Task Finished] Transform to mem in 0.259745s (10215 normal rows, 82861 exceptions)
[2021-03-25 10:31:22.235] [E/5] [info] [Task Finished] Transform to mem in 0.259475s (10365 normal rows, 82842 exceptions)
[2021-03-25 10:31:22.235] [E/9] [info] [Task Finished] Transform to mem in 0.261295s (10532 normal rows, 83051 exceptions)
[2021-03-25 10:31:22.235] [E/10] [info] [Task Finished] Transform to mem in 0.259867s (10386 normal rows, 82742 exceptions)
[2021-03-25 10:31:22.236] [E/4] [info] [Task Finished] Transform to mem in 0.259919s (10575 normal rows, 82639 exceptions)
[2021-03-25 10:31:22.448] [E/8] [info] [Task Finished] Transform to mem in 0.472241s (18692 normal rows, 150477 exceptions)
[2021-03-25 10:31:22.448] [global] [info] [Transform Stage] Stage 0 completed 169 load&transform tasks in 3.07825s
[2021-03-25 10:31:22.448] [global] [info] [Transform Stage] Stage 0 total wall clock time: 44.2702s, 56312200 input rows, time to process 1 row via fast path: 0.000786156ms
[2021-03-25 10:31:22.449] [global] [info] Exception details: 
+------------+------------+----------+
| OperatorID | Exception  | Count    |
+------------+------------+----------+
| 100001     | ValueError | 14067100 |
+------------+------------+----------+

[2021-03-25 10:31:22.487] [global] [info] Writing 1768060 rows as output to file (269.16 MB)
[2021-03-25 10:31:22.502] [E/5] [info] [Task Finished] write to file in 0.014747s (50018 rows, 7.62 MB)
[2021-03-25 10:31:22.519] [E/10] [info] [Task Finished] write to file in 0.032147s (114650 rows, 17.45 MB)
[2021-03-25 10:31:22.519] [E/7] [info] [Task Finished] write to file in 0.032215s (114344 rows, 17.41 MB)
[2021-03-25 10:31:22.519] [E/12] [info] [Task Finished] write to file in 0.032230s (114349 rows, 17.41 MB)
[2021-03-25 10:31:22.519] [E/13] [info] [Task Finished] write to file in 0.032280s (114504 rows, 17.43 MB)
[2021-03-25 10:31:22.519] [E/9] [info] [Task Finished] write to file in 0.032297s (114506 rows, 17.43 MB)
[2021-03-25 10:31:22.519] [E/14] [info] [Task Finished] write to file in 0.032335s (114658 rows, 17.45 MB)
[2021-03-25 10:31:22.519] [E/4] [info] [Task Finished] write to file in 0.032346s (114512 rows, 17.43 MB)
[2021-03-25 10:31:22.519] [E/6] [info] [Task Finished] write to file in 0.032370s (114500 rows, 17.43 MB)
[2021-03-25 10:31:22.519] [E/8] [info] [Task Finished] write to file in 0.032376s (114504 rows, 17.43 MB)
[2021-03-25 10:31:22.520] [driver] [info] [Task Finished] write to file in 0.032455s (114525 rows, 17.44 MB)
[2021-03-25 10:31:22.520] [E/3] [info] [Task Finished] write to file in 0.032515s (114644 rows, 17.45 MB)
[2021-03-25 10:31:22.520] [E/2] [info] [Task Finished] write to file in 0.032526s (114524 rows, 17.44 MB)
[2021-03-25 10:31:22.520] [E/11] [info] [Task Finished] write to file in 0.032575s (114523 rows, 17.43 MB)
[2021-03-25 10:31:22.520] [E/15] [info] [Task Finished] write to file in 0.032661s (114661 rows, 17.45 MB)
[2021-03-25 10:31:22.526] [E/1] [info] [Task Finished] write to file in 0.038804s (114638 rows, 17.45 MB)
[2021-03-25 10:31:22.526] [global] [info] writing output took 0.077318s
[2021-03-25 10:31:22.526] [global] [info] [Transform Stage] Stage 0 completed 169 sink tasks in 0.0774052s
[2021-03-25 10:31:22.526] [global] [info] [Transform Stage] Stage 0 took 3.36366s
[2021-03-25 10:31:22.526] [global] [info] During execution 14067100 exceptions occurred:
+- csv
|- withColumn
   +------------+----------+
   | ValueError | 14067100 |
   +------------+----------+
|- filter
|- withColumn
|- filter
|- withColumn
|- mapColumn
|- withColumn
|- withColumn
|- withColumn
|- withColumn
|- filter
|- select
+- csv
[2021-03-25 10:31:22.526] [global] [info] Query Execution took 3.42258s. (planning: 0.0588672s, execution: 3.36371s)
Welcome to

  _____            _
 |_   _|   _ _ __ | | _____  __
   | || | | | '_ \| |/ _ \ \/ /
   | || |_| | |_) | |  __/>  <
   |_| \__,_| .__/|_|\___/_/\_\ 0.1.7
            |_|
    
using Python 3.6.9 (default, Mar 23 2021, 11:08:57) 
[GCC 10.2.0] on linux
>>> running tuplex on ['/data/zillow/Zdirty/zillow_dirty_synthetic@10G.csv']
>>> bug in generated parser, force to false
Tuplex startup time: 0.006096601486206055
>>> running pipeline without any resolvers enabled...
Tuplex job time: 3.6942131519317627 s
{'tuplex.useLLVMOptimizer': True, 'tuplex.autoUpcast': False, 'tuplex.allowUndefinedBehavior': False, 'tuplex.optimizer.codeStats': False, 'tuplex.optimizer.generateParser': False, 'tuplex.optimizer.nullValueOptimization': False, 'tuplex.optimizer.filterPushdown': True, 'tuplex.optimizer.sharedObjectPropagation': True, 'tuplex.interleaveIO': True, 'tuplex.resolveWithInterpreterOnly': False, 'tuplex.csv.selectionPushdown': True, 'tuplex.webui.enable': False, 'tuplex.executorCount': 15, 'tuplex.csv.maxDetectionRows': 10000, 'tuplex.webui.port': 5000, 'tuplex.webui.mongodb.port': 27017, 'tuplex.webui.exceptionDisplayLimit': 5, 'tuplex.normalcaseThreshold': 0.9, 'tuplex.csv.comments': ['#', '~'], 'tuplex.csv.maxDetectionMemory': '256KB', 'tuplex.csv.quotechar': '"', 'tuplex.csv.separators': [',', ';', '|', '\t'], 'tuplex.driverMemory': '10G', 'tuplex.env.hostname': '4e1bb76d17c5', 'tuplex.env.mode': 'file', 'tuplex.env.user': 'root', 'tuplex.executorMemory': '6G', 'tuplex.inputSplitSize': '64MB', 'tuplex.logDir': '.', 'tuplex.optimizer.mergeExceptionsInOrder': 'False', 'tuplex.optimizer.operatorReordering': 'false', 'tuplex.optionalThreshold': '0.7', 'tuplex.partitionSize': '32MB', 'tuplex.readBufferSize': '128KB', 'tuplex.runTimeLibrary': '/usr/local/lib/python3.6/site-packages/Tuplex-0.1.7-py3.6.egg/tuplex/libexec/tuplex_runtime', 'tuplex.runTimeMemory': '64MB', 'tuplex.runTimeMemoryBlockSize': '4MB', 'tuplex.scratchDir': '/tmp', 'tuplex.webui.mongodb.path': '/data/tuplex_history', 'tuplex.webui.mongodb.url': 'localhost', 'tuplex.webui.url': 'localhost'}
{"logical_optimization_time_s":0.0205931,"llvm_optimization_time_s":0.138081,"llvm_compilation_time_s":0.0622385,"total_compilation_time_s":0.207793,"sampling_time_s":0.11032,"stages":[{"stage_no":0,"fast_path_wall_time_s":44.2702,"fast_path_time_s":3.07829,"fast_path_per_row_time_ns":786.156,"slow_path_wall_time_s":0,"slow_path_time_s":0,"slow_path_per_row_time_ns":0}]}
{"webui.enable": false, "executorMemory": "6G", "executorCount": 15, "driverMemory": "10G", "partitionSize": "32MB", "runTimeMemory": "64MB", "inputSplitSize": "64MB", "useLLVMOptimizer": true, "optimizer.nullValueOptimization": false, "csv.selectionPushdown": true, "optimizer.generateParser": false, "optimizer.mergeExceptionsInOrder": false, "optimizer.filterPushdown": true, "resolveWithInterpreterOnly": false}
{"startupTime": 0.006096601486206055, "jobTime": 3.6942131519317627, "mode": "plain", "mergeExceptionsInOrder": false}
[GLOBAL] releasing 10 active partitions
[2021-03-25 10:31:22.550] [local execution engine] [info] Stopped local executor E/1
[GLOBAL] releasing 10 active partitions
[2021-03-25 10:31:22.563] [local execution engine] [info] Stopped local executor E/2
[GLOBAL] releasing 10 active partitions
[2021-03-25 10:31:22.575] [local execution engine] [info] Stopped local executor E/3
[GLOBAL] releasing 11 active partitions
[2021-03-25 10:31:22.587] [local execution engine] [info] Stopped local executor E/4
[GLOBAL] releasing 11 active partitions
[2021-03-25 10:31:22.600] [local execution engine] [info] Stopped local executor E/5
[GLOBAL] releasing 11 active partitions
[2021-03-25 10:31:22.613] [local execution engine] [info] Stopped local executor E/6
[GLOBAL] releasing 10 active partitions
[2021-03-25 10:31:22.626] [local execution engine] [info] Stopped local executor E/7
[GLOBAL] releasing 12 active partitions
[2021-03-25 10:31:22.637] [local execution engine] [info] Stopped local executor E/8
[GLOBAL] releasing 11 active partitions
[2021-03-25 10:31:22.651] [local execution engine] [info] Stopped local executor E/9
[GLOBAL] releasing 11 active partitions
[2021-03-25 10:31:22.664] [local execution engine] [info] Stopped local executor E/10
[GLOBAL] releasing 10 active partitions
[2021-03-25 10:31:22.677] [local execution engine] [info] Stopped local executor E/11
[GLOBAL] releasing 11 active partitions
[2021-03-25 10:31:22.689] [local execution engine] [info] Stopped local executor E/12
[GLOBAL] releasing 11 active partitions
[2021-03-25 10:31:22.702] [local execution engine] [info] Stopped local executor E/13
[GLOBAL] releasing 11 active partitions
[2021-03-25 10:31:22.714] [local execution engine] [info] Stopped local executor E/14
[GLOBAL] releasing 10 active partitions
[2021-03-25 10:31:22.727] [local execution engine] [info] Stopped local executor E/15
[GLOBAL] Local engine terminated.
[GLOBAL] releasing 11 active partitions
