Iterator built-in functions #25

yunzhi-jake · 2021-09-01T22:34:41Z

Add the following built-in functions in UDF:
iterable can be a list, homogenous tuple, string, range or iterator object.
(i) iter(iterable) : Returns an iterator.
(ii) reversed(seq) : seq can be a list, homogenous tuple, string or range object.
(iii) enumerate(iterable[, start]) : Start=0 by default if not provided. Returns an iterator with yieldType=(I64, iter(iterable).yieldType).
(iv) zip(*iterables): Currently at least one iterable must be provided. Returns an iterator with yieldType=(iter(iterable_1).yieldType, iter(iterable_2).yieldType, ..., iter(iterable_N).yieldType).
(v) next(iterator[, default]): Returns the next item from iterator. If iterator is exhausted, returns default if default is provided, otherwise raise StopIteration.

Reference: Built-in Functions -- Python 3.9.7 documentation
Using unsupported types in the functions above (dictionary as iterable or seq, non-homogenous tuple as iterable or seq, default type in next call different from iterator.yieldType) or returning an iterator from UDF is resolved through fallback mode. Error handling codes are adapted from ASTHelper and now refactoring into IFailable.
Add support for using iterators (generated from iter, enumerate or zip) as testlist in for loops (i.e. for i in iteratorType: ...).

Fix getListType for list of tuples

Code refactoring of error handling for unsupported types

…ins multiple elements "for a, b in (t1, t2), (t3, t4)" should work now

1. Update visit Ncall, declareVariables, assignToSingleVariable to make iterator related calls work 2. Fix codegen for list of tuples 3. Add support for for loops with iterator as expression

…to generators

ptr should not be overwritten with nullptr

yunzhi-jake · 2021-09-08T15:30:17Z

Fixed len([]) causing segfault in UDF

tuplex/codegen/include/ASTAnnotation.h

LeonhardFS · 2021-09-10T01:34:24Z

tuplex/codegen/include/ASTAnnotation.h

    std::vector<python::Type> types;

+    ///! iterator-specific info
+    IteratorInfo *iteratorInfo;


maybe use std::shared_ptr etc. for memory mangement? Avoid potentially also issues with weak vs. deep copy.

i.e., is the memory of IteratorInfo etc. released properly?

Refactored to std::shared_ptr.

LeonhardFS · 2021-09-10T01:36:14Z

tuplex/codegen/include/AnnotatedAST.h

            AnnotatedAST& removeParameterTypes();

            /*!
             * throw exception for unsupported types


so this function basically checks the error in IFailable and throws an exception? If so, this should be in the description.

Added in comments.

LeonhardFS · 2021-09-10T01:36:42Z

tuplex/codegen/include/BlockGeneratorVisitor.h

            // simplify interfaces a bit
            inline codegen::SerializableValue load(llvm::IRBuilder<>& builder) const {
-                assert(ptr && sizePtr);
+                // assert(ptr && sizePtr);


can we remove this? or why is it commented?

maybe keep this assertion if it's not an iterator, otherwise check that it's an iterator? i.e. update the assertion rather than removing it altogether

Kept the assertion.
Changed to use dummy LLVM type for iterators during declareVariables() so iterator's slot ptr won't become nullptr.
Comments were added in corresponding places.

LeonhardFS · 2021-09-10T01:37:22Z

tuplex/codegen/include/BlockGeneratorVisitor.h


            inline void store(llvm::IRBuilder<>& builder, const codegen::SerializableValue& val) {
-                assert(ptr && sizePtr);
+                // assert(sizePtr);


same regarding it, yet I think leaving the iterator slot may not have ptr yet comment is good.

Maybe add to description when this is the case?

Kept the assertion.
Changed to use dummy LLVM type for iterators during declareVariables() so iterator's slot ptr won't become nullptr.
Comments were added in corresponding places.

tuplex/codegen/include/FunctionRegistry.h

LeonhardFS · 2021-09-10T01:38:30Z

tuplex/codegen/include/FunctionRegistry.h

        private:
            LLVMEnvironment& _env;
            bool _sharedObjectPropagation;
+            IteratorContextProxy *_icp;


std::shared_ptr? std::unique_ptr? how is the memory here managed?

Changed to std::shared_ptr.

tuplex/test/core/IteratorTest.cc

tuplex/test/core/LoopTest.cc

LeonhardFS

This is great work! A very large PR bringing iterator functionality and improved loops to the project. Let's get it merged and released soon.

…r[ListType]

…t ptr being nullptr

…ltBlockAddress Use curr*sign(step) < end*sign(step) to check if curr is within a range, instead of checking if curr*step < end*step

yunzhi-jake · 2021-09-24T07:37:55Z

Added support for reversed() built-in function.

LeonhardFS · 2021-09-24T13:32:48Z

tuplex/codegen/include/BlockGeneratorVisitor.h

        // store current iteration ending block and loop ending block for for and while loops
        std::deque<llvm::BasicBlock*> _loopBlockStack;

+        std::shared_ptr<IteratorContextProxy> _icp;


let's try to avoid acronyms here, just call the variable _iteratorContentProxy. Then it's super clear what it is :)

Refactored variable name in BlockGeneratorVisitor.h and FunctionRegistry.h.

LeonhardFS · 2021-09-24T13:35:51Z

tuplex/codegen/src/FunctionRegistry.cc

+                return createNextCall(lfb, builder, argsType, retType, args, iteratorInfo);
+            }
+
+            Logger::instance().defaultLogger().error("unsupported symbol encountered");


maybe add in error message what symbol, i.e.

"unsupported symbol '" + symbol + "' encountered in createIteratorRelatedSymbolCall"

LeonhardFS

great work! ready to get merged to master.

yunzhi-jake added 16 commits September 1, 2021 03:49

Add iterator type and related functions for typing

5b53d3e

Add symbol for iterator-related functions (iter, zip, enumerate, next)

324ec46

Add iterator-specific annotation

cd324b8

Add iterator-related helping functions in LLVMEnvironment

2c7b9b8

Fix getListType for list of tuples

Code refactoring of error handling for unsupported types

0328da6

Annotate iterator-related AST nodes with iterator-specific info

1a9906c

Code refactoring of error handling for unsupported types

Add iterator core class

890a2f3

Add functions for creating iterator-related calls in FunctionRegistry

8024249

Fix the case when expression (also called testlist) in for loop conta…

9949335

…ins multiple elements "for a, b in (t1, t2), (t3, t4)" should work now

Update BlockGeneratorVisitor

bcd9c89

1. Update visit Ncall, declareVariables, assignToSingleVariable to make iterator related calls work 2. Fix codegen for list of tuples 3. Add support for for loops with iterator as expression

Merge branch 'tuplex:master' into generators

eeb71af

Use fallback mode for mixed AST node types in for loop exprlist for now

5527e7e

Fix iterator not working under if statement

02028df

Add tests for using iterator as testlist in for loops

06f78d6

Add tests about iterators

d6bc445

Merge branch 'generators' of https://github.com/yunzhi-jake/tuplex in…

21bd899

…to generators

yunzhi-jake changed the title ~~Iterators built-in functions~~ Iterator built-in functions Sep 1, 2021

yunzhi-jake added 5 commits September 1, 2021 18:49

Delete extra spaces

59b8ff4

Deal with testlist separately in visitFor_stmt

dd50fac

Fix len() call on EMPTYLIST

e85466c

Loop variable i of type "unsigned long" causes problem

f9f8212

Fix empty tuple as input not working for UDF

9fbbe79

ptr should not be overwritten with nullptr

LeonhardFS reviewed Sep 10, 2021

View reviewed changes

tuplex/codegen/include/ASTAnnotation.h Outdated Show resolved Hide resolved

LeonhardFS reviewed Sep 10, 2021

View reviewed changes

tuplex/codegen/include/FunctionRegistry.h Show resolved Hide resolved

LeonhardFS reviewed Sep 10, 2021

View reviewed changes

LeonhardFS reviewed Sep 21, 2021

View reviewed changes

tuplex/test/core/IteratorTest.cc Show resolved Hide resolved

LeonhardFS reviewed Sep 21, 2021

View reviewed changes

tuplex/test/core/LoopTest.cc Show resolved Hide resolved

LeonhardFS requested changes Sep 21, 2021

View reviewed changes

yunzhi-jake added 19 commits September 21, 2021 17:07

Add comments about IteratorInfo struct

319564e

Use std::shared_ptr for iteratorInfo annotation

e149108

Refactored variable names related to compile errors in IFailable

75a45b1

Update logic about iterator variable slot

7df9846

Use std::shared_ptr for IteratorContextProxy instance

e27bd63

Rename function

135b41d

Adjust if statement and add comments

6317cb7

Fix some logic when for loop expresssion(testlist) is of type Iterato…

a41576e

…r[ListType]

Fix bug in UnrollLoopsVisitor. Add tests

c5833df

Add test

528c269

Refactor

f438904

Use 8 bytes for bool

431d3db

Add error message

9a4ecd5

Use dummy LLVM type during variable declaration to avoid variable slo…

2d953f4

…t ptr being nullptr

Refactor

64d1da1

Refactor

d5a0616

Update logic for function createOrGetUpdateIteratorIndexFunctionDefau…

e8f522f

…ltBlockAddress Use curr*sign(step) < end*sign(step) to check if curr is within a range, instead of checking if curr*step < end*step

Add symbol "reversed"

b370e75

Add support for reversed() function

01e45fa

LeonhardFS reviewed Sep 24, 2021

View reviewed changes

yunzhi-jake added 3 commits September 24, 2021 12:02

Refactor and add comments.

16c4059

Rename test

6e607f9

Add comment

4549eb9

LeonhardFS approved these changes Oct 1, 2021

View reviewed changes

LeonhardFS merged commit b0a7d35 into tuplex:master Oct 1, 2021

Iterator built-in functions #25

Iterator built-in functions #25

Uh oh!

Conversation

yunzhi-jake commented Sep 1, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

yunzhi-jake commented Sep 8, 2021

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yunzhi-jake Sep 23, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

LeonhardFS left a comment

Choose a reason for hiding this comment

Uh oh!

yunzhi-jake commented Sep 24, 2021

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yunzhi-jake Sep 24, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

LeonhardFS left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

yunzhi-jake commented Sep 1, 2021 •

edited

Loading

yunzhi-jake Sep 23, 2021 •

edited

Loading

yunzhi-jake Sep 24, 2021 •

edited

Loading