[BUG] Tuplex fails to parse/run functions defined within shell #107
Description
After defining the following function in a python3.6 shell:
def count(L):
d = {}
for x in L:
if x not in d.keys():
d[x] = 0
d[x] += 1
return drunning:
c = Context()
c.parallelize([[1, 1, 1], [1, 2, 1, 2]]).map(count).collect()I get the following error:
Traceback (most recent call last):
File "/usr/lib/python3.6/code.py", line 91, in runcode
exec(code, self.locals)
File "console-9", line 1, in <module>
File "/home/rgoyal6/tuplex/tuplex/build/dist/python/tuplex/dataset.py", line 64, in map
code = get_udf_source(ftor)
File "/home/rgoyal6/tuplex/tuplex/build/dist/python/tuplex/utils/reflection.py", line 207, in get_source
return get_function_code(f)
File "/home/rgoyal6/tuplex/tuplex/build/dist/python/tuplex/utils/reflection.py", line 156, in get_function_code
return extract_function_code(function_name, dill.source.getsource(f))
File "/home/rgoyal6/.local/lib/python3.6/site-packages/dill-0.3.4-py3.6.egg/dill/source.py", line 354, in getsource
lines, lnum = getsourcelines(object, enclosing=enclosing)
File "/home/rgoyal6/.local/lib/python3.6/site-packages/dill-0.3.4-py3.6.egg/dill/source.py", line 325, in getsourcelines
code, n = getblocks(object, lstrip=lstrip, enclosing=enclosing, locate=True)
File "/home/rgoyal6/.local/lib/python3.6/site-packages/dill-0.3.4-py3.6.egg/dill/source.py", line 251, in getblocks
lines, lnum = findsource(object)
File "/home/rgoyal6/.local/lib/python3.6/site-packages/dill-0.3.4-py3.6.egg/dill/source.py", line 144, in findsource
raise IOError('source code not available')
OSError: source code not available