Seth ideas

• Virtual Batching ideas (re eval effects of virtual loss [Data] effect of virtual_loss #427)
  • Only add X from batch of Y to the tree, put the rest in NNCache to use if they get needed later (this is basically a different version of Speculative Execution
• Supervised Eval Doc
  • Rate cut in SL experiments
• SL training with rate cut and more steps...
• Train some smaller distilled models and test on time parity

Ideas inspired by @lightvector and KataGo

• NNCache
  • Turning off Tree Reuse
• Ownernship head
• Score distribution Head
• Score Maximization ("Score Utility")
• Playout oscillation
• Forking games (early for diversity, late for Komi, ...)

Ideas inspired by LZ

• SWA: Initial Proof Of Concept in Oneoff SWA script #283 but more work needed
• Visits "timemanagement" (stopping when 2nd move can't overtake first)

Ideas from AG/AGZ/AZ papers

• Gating: Write up in "Rough Gating" work required #459
• Playing anything early within 1% of best

Ideas from elsewhere

• Changing value to 0.95 (AKA 1 - false positive) for selfplay with resign (idea from lightvector)
• Active learning (Needs more details) @brilee
  (https://medium.com/oracledevs/lessons-from-alpha-zero-part-6-hyperparameter-tuning-b1cfcbe4ca9a))
• Cyclic LR

Done

• Slow window, not super successful (inspired by Oracle's Connect Four work)
• Q=0 (Sorta tested in V11)
• FPU (Handled by Support init-to-parent, init-to-loss, and FPU #629)