(For educational purposes) generative model for bioRxiv titles.

Roadmap

• Implement attention https://arxiv.org/abs/1706.03762
• Load data from https://huggingface.co/datasets/laion/biorXiv_metadata
• tokenize data
• Pytorch data loader
• Token embedder and position embedder
• generate mode
• Implement train step and eval step (Overfit on one batch with a small model)
• Train small v0 model on CPU
• Scale up model and train v1 on Colab GPU