Hugging Face Transformers Adds UltraBERT with Layer-Specific Rotary Embeddings

Hugging Face integrated UltraBERT into its Transformers library. The model uses two distinct rotary embedding theta values for local and global attention layers, managed via a ModuleDict keyed by embedding type. The integration includes config, pretrained weights, and support for Masked Language Modeling and Sequence Classification.

• UltraBERT uses separate rotary embeddings for local and global attention
• Implementation relies on a ModuleDict keyed as local and global
• Model ships with config and pretrained weights in Transformers
• MLM and Sequence Classification heads are supported