全部动态

I didn’t expect I’d be so impressed by Grok Bot. People told me to use it and I had low expectations, but been the most useful tool in the past week. I have few bots, from learning Japanese, librarian, community building assistant to AI animation intern. A favorite AI tool atm

Super excited to share some of the work I’ve been doing on supporting 1M token context training on SkyRL for GLM 5.3 Flash, 64x higher than the out of the box Megatron baseline! 🚀 This work was done in collaboration with the amazing folks at @LilaSciences! 🤩 Getting there took key algorithmic and systems optimizations in four main areas: 1) DeepSeek Sparse Attention: run DSA on a sparse kernel; chunk and shard its quadratic indexer. 2) Activation recompute and offloading: recompute activations in backward and offload saved inputs to CPU. 3) Context parallelism: split each sequence across GPUs, with new CP support for KDA and a fix for DSA. 4) Avoiding unnecessary precision upcasts: keep FP32 copies out of memory and off the network. For more details on our implementations and our results, check out our full technical blog here: https://blog.skyrl.ai/blog/1m-context-glm-5-3-flash! This work benefited from the contributions of many people along the way. Big thanks to @pcmoritz, @erictang000, @sumanthrh, @CyrusHakha, @richliaw from @anyscalecompute and Keith Truongcao, Tushar Khot, Vivek Subramanian, Semih Kara from Lila Sciences for all the help and guidance!