Summary

The current indentation handling model employed by the YAML lexer is not sophisticated enough to handle YAML compact forms, which allow a new collection to start without explicit indentation.

For more context, normally YAML requires a new collection, whether a sequence or a mapping, to start with an indent:

a:
  b:
    - x
  d: e

In contrast, compact forms allow a collection to start on the same line as its parent collection entry:

- a: b
  c: d

This breaks the current assumption held by the YAML lexer, which expects all nested collection to begin and end with indent and dedent tokens.

For example, given this nested compact sequence, the lexer currently only detects a single indent token preceding the second line - token, which results in the following token stream:

- - - - -
    -
-

# Whitespace tokens emitted for brevity
DASH:1
DASH:1
DASH:1
DASH:1
DASH:1
INDENT:5
DASH:1
DEDENT:1
DASH:1

Given the above stream, the YAML parser doesn't have enough information to correctly identify that the second level - token belongs to the third degree nested sequence, and that the final - belongs to the top level collection, purely based on indentation.

Similarly, in dealing with compact mapping, one can't confidentally tell that the following nested mapping is wrongly indented given only one indent token (unless we count the number of characters):

- a: b
    c: d

Additionally, a side effect of exposing indentation information to the parser is that the lexer must also emit newline token for when newline happens without changing indent level. It's extremely annoying to work with in the parser, even though newlines only matters in some context.

To deal with the above describe problems, this PR introduces a new scope concept to the lexer. In short, any token appears further to the right of a scope's boundary belong to that scope. For example:

a: [1, 2
 ]

Afther the first : token, the lexer enters a block mapping scope, with its border at column 2. Any token appearing beyound column 2 belong to this scope.

Now, instead of emitting indent and dedent tokens, the YAML lexer now emits MAPPING_START, MAPPING_END, SEQUENCE_START, SEQUENCE_END, FLOW_START, FLOW_END specially to signal to the parser the when to start new collection. All disambiguation logic, like distinguishing between a flow collection ([1, 2, 3]) and the start of a new mapping (plain token followed by :: [1, 2, 3]:) is handled at the lexer.

Sorry that this takes a while to get out. It doesn't help that the YAML spec has so many edge cases.

After this, the next major blocker would be in dealing with YAML properties, as it has quite a challenging indentation rule.

Test Plan

This PR adds more tests for the YAML lexer. Most notable ones are

• Correctly identifies mis-indented sequence as a single plain token. For example:

  - a
   - b
   - c
   - d

• Limits the impact of missing closing indicators for flow collections to only the closest scope. For example, in the below document, the lexer can identify that the [1, 2, 3] collection misses a closing bracket and correctly insert a FLOW_END token before emitting the next mapping key

  a:
    b: [1, 2, 3
  c:

• Correctly handles sequence nested under mapping without indentation. For example, the following document is equivalent to {"a": ["b"]}

  a:
  - b

• Correctly handles nested compact forms:

  - a: b
    c: d