convmind/neural_es repository overview

⁠NeuralES (Neural Encoder Search)

This is a multi-purpose services for searching documents using various neural encoders in elastic search index. It is also designed convert raw objects (text, images etc) into the embedding space and store them in the elastic search index.

⁠Installation

pip install -r requirements.txt

To make a pacakge, run:

python setup.py sdist

⁠Overview

⁠Basic Functions

⁠Encoders Management

NeuralES uses ConvMind encoder python module (https://github.com/ConvMind/Encoders⁠) to encode diverse data types using neural encoders. The following is a list of Encoder IDs that the application clients can use:

English:

• Sentence-level:
  • en-RnnEncoder-1563406351
• Dialog-level:
  • coming soon.

Chinese:

• Coming soon

⁠Query

A search query contains the following arguments:

• bot_id
• query (a natural language input (e.g. dialog/question))
• context (additional meta features that will be used for filtering.)
• method: neural (default) vs bm25

NeuralES also supports auto-completion given some a prefix query.

• coming soon.

⁠Index Management

Currently, NeuralES supports the following operations

• add: add a list of documents to a existing or new index. Limited up to 5000 documents at each API call. For larger index, please use bulk_add.
• bulk_add: trigger a database pull and encode a (large) amount of questions from database. (see details below)
• remove: delete a index
• get_meta: return meta information of a index

⁠Database Schema:

{
    context: [turn_1, turn2], # proceeding dialog history,
    task_id: xxx # unique ID,
    status: enable, # deleted, dsbale, enable, in_review
    q: x, 
    q_paraphrases: [xx, xxx, xxx],
    a: [{act: utterance, value: hi, type: text}, 
        {act: page_navigation, value: xxx/a.html, type: navigation}] # same as our msg format
    updated_time: timestamp,
    user_id: tony # who created this entry
    filters: {
        expiration_date: null/date,
        tags: [x, y, z],
        category: FAQ # top-level folder
    },
    follow_up_questions: ?? // just learn from MS QnA bot. need to think more about it.
}

NES will always try to convert the external data into the following format:

{
    q: x, 
    a: [{act: utterance, value: hi, type: text}, 
        {act: page_navigation, value: xxx/a.html, type: navigation}] # same as our msg format
    a_key: xxhhxx # a unique hashkey of a. can be obtained via hashlib
    expire_time: xx
    creation_time: xx
    tags: [x, y, z]
    category: xxx,
    displayable: True/False # will it go to frequent_answers
}

⁠Tokenizer

⁠Chinese

• hanlp: hanlp默认分词
• hanlp_standard: 标准分词
• hanlp_index: 索引分词
• hanlp_nlp: NLP分词
• hanlp_n_short: N-最短路分词
• hanlp_dijkstra: 最短路分词
• hanlp_crf: CRF分词（已有最新方式）
• hanlp_speed: 极速词典分词

⁠Output Format for Document-based Answer Bot

[{act: utterance, value: hi, type: text}, 
 {act: span, value: i am a man, type: text, answer_start: 421, chunk_id: xxx, doc_id: xxx},
 {act: section, value: abstract, type: text, answer_start: 0, chunk_id: xxx, doc_id: xxx},
 {act: title, value: ConvMind Handbook, type: text, answer_start: 0, chunk_id: xxx, doc_id: xxx}]
 

answer_start is the character-level offset in the chunk (defined by chunk_id). The chunk_id is a chunk that comes from document (defined by doc_id)

⁠Configuration

⁠Training

• IDP_URL: str
• TASK_ID: str
• ENCODER_ID: str, encoder question, and indexing match data
• DB_ENCODER_ID: str, encoder for indexing relevance data
• REGION: US or CN
• MODEL_TYPE: soco
• DATA_SOURCE - mlab, idp, local, docs
• DATA_PATH: str - path to json data if running local
• USE_GPU: bool - if use GPU

⁠Inference

• REGION: str, US, CN
• PORT: int
• ES_URL: str
• ES_PORT: int

⁠How to user filters

###Current filterable fields (growing):

• expire_time
• tags
• category
• department (BOC)
• position (BOC)

###Data format:

1. Single filter, single value

   {filters: {term: {key: value}}}

2. Single filter, multiple value

   {filters: {terms: {key: [v1, v2]}}}

3. composed filters

   {filters: [{term: {key: value}}, {terms: {key: []}}]