sydrawat/biogpt repository overview

⁠BioGPT

BioGPT is a generative language model that has been pre-trained on large amounts of biomedical literature. It is a domain-specific variant of the GPT family of language models and is designed to generate fluent descriptions for biomedical terms. It has been shown to outperform previous models on a range of biomedical natural language processing tasks, including relation extraction and question-answering, making it a promising tool for biomedical researchers and practitioners.

⁠Implementation policy

This repository contains the implementation of BioGPT: Generative Pre-trained Transformer for Biomedical Text Generation and Mining⁠, written by:

• Renqian Luo
• Liai Sun
• Yingce Xia
• Tao Qin
• Sheng Zhang
• Hoifung Poon
• Tie-Yan Liu.

⁠News!

• BioGPT-Large model with 1.5B parameters is coming, currently available on PubMedQA task with SOTA performance of 81% accuracy. See Question Answering on PubMedQA⁠ for evaluation.

⁠Requirements and Installation

All you need is docker to be setup in your local machine, and you're good to go.

docker pull sydrawat/biogpt

To run the image in a container (in bash mode):

docker run --name mybiogpt -ti sydrawat/biogpt bash

To run the basic demo with the pre-trained BioGPT model from Microsoft (which comes bundles with this image):

# inside the running container
python3 biogpt-pt.py

⁠Pre-trained models

We provide our pre-trained BioGPT model checkpoints along with fine-tuned checkpoints for downstream tasks, available both through URL download as well as through the Hugging Face 🤗 Hub.

Download them and extract them to the checkpoints folder of this project.

For example:

mkdir checkpoints
cd checkpoints
wget https://msramllasc.blob.core.windows.net/modelrelease/BioGPT/checkpoints/Pre-trained-BioGPT.tgz
tar -zxvf Pre-trained-BioGPT.tgz

⁠Example Usage

Use pre-trained BioGPT model in your code:

import torch
from fairseq.models.transformer_lm import TransformerLanguageModel
m = TransformerLanguageModel.from_pretrained(
        "checkpoints/Pre-trained-BioGPT",
        "checkpoint.pt",
        "data",
        tokenizer='moses',
        bpe='fastbpe',
        bpe_codes="data/bpecodes",
        min_len=100,
        max_len_b=1024)
m.cuda()
# comment m.cuda() if you are not using the PyTorch with GPU support
src_tokens = m.encode("COVID-19 is")
generate = m.generate([src_tokens], beam=5)[0]
output = m.decode(generate[0]["tokens"])
print(output)

Use fine-tuned BioGPT model on KD-DTI for drug-target-interaction in your code:

import torch
from src.transformer_lm_prompt import TransformerLanguageModelPrompt
m = TransformerLanguageModelPrompt.from_pretrained(
        "checkpoints/RE-DTI-BioGPT",
        "checkpoint_avg.pt",
        "data/KD-DTI/relis-bin",
        tokenizer='moses',
        bpe='fastbpe',
        bpe_codes="data/bpecodes",
        max_len_b=1024,
        beam=1)
m.cuda()
# comment m.cuda() if you are not using the PyTorch with GPU support
src_text="" # input text, e.g., a PubMed abstract
src_tokens = m.encode(src_text)
generate = m.generate([src_tokens], beam=args.beam)[0]
output = m.decode(generate[0]["tokens"])
print(output)

For more downstream tasks, please see below.

⁠Downstream tasks

See corresponding folder in examples⁠:

⁠Relation Extraction on BC5CDR⁠

⁠Relation Extraction on KD-DTI⁠

⁠Relation Extraction on DDI⁠

⁠Document Classification on HoC⁠

⁠Question Answering on PubMedQA⁠

⁠Text Generation⁠

⁠Hugging Face 🤗 Usage

BioGPT has also been integrated into the Hugging Face transformers library, and model checkpoints are available on the Hugging Face Hub.

You can use this model directly with a pipeline for text generation. Since the generation relies on some randomness, we set a seed for reproducibility:

from transformers import pipeline, set_seed
from transformers import BioGptTokenizer, BioGptForCausalLM
model = BioGptForCausalLM.from_pretrained("microsoft/biogpt")
tokenizer = BioGptTokenizer.from_pretrained("microsoft/biogpt")
generator = pipeline('text-generation', model=model, tokenizer=tokenizer)
set_seed(42)
generator("COVID-19 is", max_length=20, num_return_sequences=5, do_sample=True)

Here is how to use this model to get the features of a given text in PyTorch:

from transformers import BioGptTokenizer, BioGptForCausalLM
tokenizer = BioGptTokenizer.from_pretrained("microsoft/biogpt")
model = BioGptForCausalLM.from_pretrained("microsoft/biogpt")
text = "Replace me by any text you'd like."
encoded_input = tokenizer(text, return_tensors='pt')
output = model(**encoded_input)

Beam-search decoding:

import torch
from transformers import BioGptTokenizer, BioGptForCausalLM, set_seed

tokenizer = BioGptTokenizer.from_pretrained("microsoft/biogpt")
model = BioGptForCausalLM.from_pretrained("microsoft/biogpt")

sentence = "COVID-19 is"
inputs = tokenizer(sentence, return_tensors="pt")

set_seed(42)

with torch.no_grad():
    beam_output = model.generate(**inputs,
                                 min_length=100,
                                 max_length=1024,
                                 num_beams=5,
                                 early_stopping=True
                                )
tokenizer.decode(beam_output[0], skip_special_tokens=True)

For more information, please see the documentation⁠ on the Hugging Face website.

⁠Demos

Check out these demos on Hugging Face Spaces:

• Text Generation with BioGPT-Large⁠
• Question Answering with BioGPT-Large-PubMedQA⁠

⁠Docker image

[Work in Progress] A complete BioGPT-ready Ubuntu docker image is in development. Once ready, it will be available on dockerhub. Uses will be able to use the biogpt-pt.py⁠ script to execute sample BioGPT models to return descriptive results on biomedical terms.

⁠License⁠

BioGPT is MIT-licensed. The license applies to the pre-trained models as well.

⁠Contributing⁠

This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit https://cla.opensource.microsoft.com⁠.

When you submit a pull request, a CLA bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA.

This project has adopted the Microsoft Open Source Code of Conduct⁠. For more information see the Code of Conduct FAQ⁠ or contact [email protected]⁠ with any additional questions or comments.

⁠Trademarks

This project may contain trademarks or logos for projects, products, or services. Authorized use of Microsoft trademarks or logos is subject to and must follow Microsoft's Trademark & Brand Guidelines⁠. Use of Microsoft trademarks or logos in modified versions of this project must not cause confusion or imply Microsoft sponsorship. Any use of third-party trademarks or logos are subject to those third-party's policies.