PGX Inc. has released version 1.0 of pgx-bm25, an open-source database extension that integrates Okapi BM25 ranked full-text search directly into PostgreSQL as a native index access method. Installed as bm25_native, the extension supports PostgreSQL 17 and PostgreSQL 18 under the PostgreSQL License. It provides relevance-ranked document retrieval inside the core database engine, eliminating the operational overhead of running external search clusters or third-party background services.
Native Storage and System Recovery
The extension stores all index data inside the index relation's own dedicated pages. As a result, bm25_native indexes automatically benefit from PostgreSQL's core infrastructure, including write-ahead logging (WAL), crash recovery guarantees, and physical replication across database nodes. Routine database maintenance routines such as VACUUM manage the index alongside standard engine tables. Written entirely in C against official server headers, pgx-bm25 builds using standard PGXS tooling with only a C compiler and pg_config required. Text analysis uses PostgreSQL's native Snowball dictionaries, which can be configured per index to handle language stemming.
Block-Max WAND and Ordered Scans
Searches run directly as ordered index scans, removing the need for a separate Sort node in query execution plans. To handle large document sets efficiently, pgx-bm25 uses the block-max WAND algorithm for top-N queries, preventing the engine from scoring every matching document when only a limited set of results is requested. Index configuration options—such as the BM25 parameters k1 and b, as well as field weight boosts—can be adjusted dynamically with ALTER INDEX commands and take effect on the next scan without requiring a full REINDEX.
Query Building and Upgrade Paths
In addition to plain text searches, the extension offers builder functions that output jsonb query trees. This structure allows developers to safely construct complex boolean conditions—such as must, should, and must_not directives—passing user inputs as parameter values rather than concatenated search syntax. The project includes full regression, TAP, UBSan, and AddressSanitizer test suites, along with automated verification for crash recovery and replica consistency. An on-disk compatibility contract ensures additive format updates do not force index rebuilds, and a dedicated bm25_upgrade() function allows in-place migrations when breaking format updates occur.
AI-generated editorial illustration; not a documentary photograph.
Source: pgx-bm25 1.0: BM25 ranked full-text search as a native PostgreSQL index
SIGNAL TITANS
Leave a Reply