Sign inSign up

reveever/outline

By reveever

•Updated 6 months ago

Outline (BUSL) fork with per-character CJK tokenization for full-text search across CJK content

Image
0

489

reveever/outline repository overview

⁠Outline — CJK Search Fork

A fork of outline/outline⁠ that adds per-character CJK tokenization to PostgreSQL full-text search, fixing the long-standing issue where Chinese / Japanese / Korean queries miss matches because the default english tokenizer treats CJK runs as a single opaque token.

This is the runtime image — pull this one to deploy.

⁠What's different from upstream

  • Adds a searchVectorCJK tsvector column on documents, with a GIN index.
  • Adds an outline_split_cjk(text) SQL helper that pads each CJK character (Han / Hiragana / Katakana / Hangul) with spaces so PostgreSQL tokenizes per-character.
  • Rewrites documents_search_trigger() to maintain both the original searchVector and the new searchVectorCJK, so the rest of the codebase keeps working.
  • Search code in PostgresSearchProvider reads from searchVectorCJK and applies the same per-character splitting to user queries, keeping query and index in sync.
  • Single character searches like 项目 or 计划 now match documents where those characters appear, with no ICU/zhparser/jieba extension required.

Latin / mixed-script content is unaffected — it still goes through the standard english tokenizer.

⁠Tags

  • vX.Y.Z-cjk.N — pinned to upstream Outline vX.Y.Z, fork iteration N. Example: 1.7.0-cjk.1.
  • X.Y — latest patch on a minor release.
  • latest — most recent published tag.

⁠Deploy

Same as upstream — replace the image in your docker-compose.yml:

services:
  outline:
    image: reveever/outline:latest
    # ...env, volumes, depends_on as in upstream docs

The first run after upgrading from upstream Outline will execute a migration that adds the new column, index, helper function, and trigger, then backfills searchVectorCJK for all existing documents (UPDATE documents SET "title" = "title"). On large datasets this can take several minutes.

⁠Source & license

⁠Architectures

linux/amd64 only.

⁠Verifying CJK search after deploy

-- searchVectorCJK column + index exist
\d documents

-- trigger maintains both vectors
SELECT pg_get_functiondef('documents_search_trigger'::regproc);
-- body must assign to BOTH "searchVector" AND "searchVectorCJK"

Tag summary

Content type

Image

Digest

sha256:36e7e7395…

Size

259.5 MB

Last updated

6 months ago

docker pull reveever/outline