Databricks Runtime adds NEAREST BY: native SQL vector search at scale

Databricks Runtime introduced NEAREST BY, a native SQL join for batch top-k vector search that runs inside the analytical engine instead of a separate vector-serving endpoint. It combines a fused Photon operator, a specialized GEMM kernel and an approximate index in a liquid-clustered Delta table, with APPROX and EXACT modes.

• NEAREST BY returns up to k nearest rows per join row using cosine, L2 or inner product
• Search runs inside the Photon engine, removing the external vector endpoint dependency
• The index lives in a liquid-clustered Delta table; APPROX and EXACT set the search mode
• Databricks cites matching 100M daily transactions against 140M merchant embeddings