bytez/fine-tuned_stackoverflow repository overview

⁠Model

Model: fine-tuned/stackoverflow

Task: feature-extraction

⁠How to Run this Model

If you're just getting started, we recommend that you try out the Bytez Model Playground⁠ directly or use one of our Client Libraries⁠ to access the Bytez Inference API⁠.

You'll receive 100 free credits of inference each month!

Javascript⁠, Python⁠, and Julia⁠ are currently supported.

⁠Bytez Model Playground

You can play with models without having to write any code by visiting Bytez⁠

Models can also be explored:

If that's not your cup of tea, keep reading!

⁠Setup Requirements

1. Ensure Docker is installed.
2. This model container is free and open source. It requires a free API key, so it can alert you of analytics and when model upgrades are available. To get your key, visit Bytez⁠ and sign in.

Your API key will be front and center with a copy button, like you see in the image below:

⁠Running the Image

⁠Step 1: Pull the Docker Image

docker pull bytez/fine-tuned_stackoverflow

⁠Step 2: Start the container on port 8000

docker run -it \
 -e KEY=YOUR_BYTEZ_API_KEY_HERE \
 -e PORT=8000 \
 -p 8000:8000 \
 bytez/fine-tuned_stackoverflow

⁠Adjusting the port if desired

NOTE you can adjust the port if needed by the -e PORT= environment variable and the -p option.

e.g. if you want to start the container on port 80, you'd do this instead:

docker run -it \
 -e KEY=YOUR_BYTEZ_API_KEY_HERE \
 -e PORT=80 \
 -p 80:80 \
 bytez/fine-tuned_stackoverflow

⁠Step 3: Make an Inference

Send POST requests to the container and the model will reply.

curl --location 'http://0.0.0.0:8000/run' \
--header 'Content-Type: application/json' \
--data-raw '{
  "text": "Frank",
  "params": {}
}'

⁠Storing Weights Locally (Save them to your disk)

To ensure that weights are saved locally between runs, you can specify a directory for where you want weights to be stored.

For large models, this is highly recommeded, as download times can be hours for larger models.

This can be specified via the -v option

To do this, run the following command:

docker run -it \
 -v /PATH/TO/YOUR/CACHING/DIRECTORY/HERE:/server/model \
 -e HF_HOME=/server/model \
 -e KEY=YOUR_BYTEZ_API_KEY_HERE \
 -p 8000:8000 \
 -e PORT=8000 \
 bytez/fine-tuned_stackoverflow

Notice how in the command above we have -v /PATH/TO/YOUR/CACHING/DIRECTORY/HERE:/server/model and -e HF_HOME=/server/model

The -v /PATH/TO/YOUR/CACHING/DIRECTORY/HERE:/server/model says, mount the directory -v /PATH/TO/YOUR/CACHING/DIRECTORY/HERE to the docker container's filesystem at the directory /server/model

-e HF_HOME=/server/model allows the code to load the model from the directory in the docker container, i.e. from the /server/model directory.

On my machine, the command looks like this:

docker run -it \
 -v /home/inf3rnus/models:/server/model \
 -e HF_HOME=/server/model \
 -e KEY=YOUR_BYTEZ_API_KEY_HERE \
 -p 8000:8000 \
 -e PORT=8000 \
 bytez/fine-tuned_stackoverflow

⁠Running on GPU(s)

To run on GPU(s), make sure you have the latest drivers from Nvidia and CUDA installed.

Then, simply run the command from above, but with --gpus all added to the list of docker options.

docker run -it \
 --gpus all \
 -e KEY=YOUR_BYTEZ_API_KEY_HERE \
 -p 8000:8000 \
 -e PORT=8000 \
 bytez/fine-tuned_stackoverflow

⁠Local caching and running on GPU(s)

The two commands from above combined into one:

docker run -it \
 --gpus all \
 -v /PATH/TO/YOUR/CACHING/DIRECTORY/HERE:/server/model \
 -e HF_HOME=/server/model \
 -e KEY=YOUR_BYTEZ_API_KEY_HERE \
 -p 8000:8000 \
 -e PORT=8000 \
 bytez/fine-tuned_stackoverflow

⁠Additional environment variables

-e DEVICE="SOME_VALUE_HERE"

Defaults to auto

Can be:

-e DEVICE="cuda"

or

-e DEVICE="auto"

or

-e DEVICE="cpu"

auto will attempt to place the weights on the GPU if available, and then place them onto system RAM if there is not enough memory.

cuda will attempt to place the weights on the GPU

cpu will attempt to place the weights on the CPU

Allows you to specify with greater control which device you want to run the model on. Auto may split the model across system RAM and VRAM. You will often use this to attempt forcing the model to be loaded onto the GPU.

NOTE: Some models only exclusively work with auto, cuda, or cpu

⁠Have questions? Need help?

Hop into the Bytez discord⁠ for live support: the community is happy to help. If you don't have discord, email us⁠.

⁠Model Parameters

⁠LENGTH CONTROL

• max_length (int) (optional): The maximum length the generated tokens can have. Default: 20.
• max_new_tokens (int) (optional): The maximum numbers of tokens to generate, ignoring the number of tokens in the prompt.
• min_length (int) (optional): The minimum length of the sequence to be generated. Default: 0.
• min_new_tokens (int) (optional): The minimum numbers of tokens to generate, ignoring the number of tokens in the prompt.
• early_stopping (bool or str) (optional): Controls the stopping condition for beam-based methods. Default: False.
• max_time (float) (optional): The maximum amount of time for the computation to run.

⁠GENERATION STRATEGY

• do_sample (bool) (optional): Whether to use sampling or greedy decoding. Default: False.
• num_beams (int) (optional): Number of beams for beam search. Default: 1.
• num_beam_groups (int) (optional): Number of groups for diversity among beams. Default: 1.
• penalty_alpha (float) (optional): Balance model confidence and degeneration penalty.
• use_cache (bool) (optional): Whether to use cache for speeding up decoding. Default: True.

⁠LOGITS MANIPULATION

• temperature (float) (optional): The value used to modulate the next token probabilities. Default: 1.
• top_k (int) (optional): The number of highest probability vocabulary tokens to keep for top-k-filtering. Default: 50.
• top_p (float) (optional): If set to float < 1, only the smallest set of most probable tokens with probabilities that add up to top_p or higher are kept for generation. Default: 1.
• typical_p (float) (optional): Local typicality measures how similar the conditional probability of predicting a target token next is to the expected conditional probability of predicting a random token next, given the partial text already generated. If set to float < 1, the smallest set of the most locally typical tokens with probabilities that add up to typical_p or higher are kept for generation. See this paper⁠ for more details. Default: 1.
• epsilon_cutoff (float) (optional): If set to float strictly between 0 and 1, only tokens with a conditional probability greater than epsilon_cutoff will be sampled. In the paper, suggested values range from 3e-4 to 9e-4, depending on the size of the model. See Truncation Sampling as Language Model Desmoothing⁠ for more details. Default: 0.
• eta_cutoff (float) (optional): Eta sampling is a hybrid of locally typical sampling and epsilon sampling. If set to float strictly between 0 and 1, a token is only considered if it is greater than either eta_cutoff or sqrt(eta_cutoff) * exp(-entropy(softmax(next_token_logits))). The latter term is intuitively the expected next token probability, scaled by sqrt(eta_cutoff). In the paper, suggested values range from 3e-4 to 2e-3, depending on the size of the model. See Truncation Sampling as Language Model Desmoothing⁠ for more details. Default: 0.
• diversity_penalty (float) (optional): This value is subtracted from a beam's score if it generates a token same as any beam from other group at a particular time. Note that diversity_penalty is only effective if group beam search is enabled. Default: 0.
• repetition_penalty (float) (optional): The parameter for repetition penalty. 1.0 means no penalty. See this paper⁠ for more details. Default: 1.
• encoder_repetition_penalty (float) (optional): The paramater for encoder_repetition_penalty. An exponential penalty on sequences that are not in the original input. 1.0 means no penalty. Default: 1.
• length_penalty (float) (optional): Exponential penalty to the length that is used with beam-based generation. It is applied as an exponent to the sequence length, which in turn is used to divide the score of the sequence. Since the score is the log likelihood of the sequence (i.e. negative), length_penalty > 0.0 promotes longer sequences, while length_penalty < 0.0 encourages shorter sequences. Default: 1.
• no_repeat_ngram_size (int) (optional): If set to int > 0, all ngrams of that size can only occur once. Default: 0.
• bad_words_ids (List[List[int]]) (optional): List of list of token ids that are not allowed to be generated. Check [~generation.NoBadWordsLogitsProcessor] for further documentation and examples.
• force_words_ids (List[List[int]]) (optional): List of token ids that must be generated. If given a List[List[int]], this is treated as a simple list of words that must be included, the opposite to bad_words_ids. If given List[List[List[int]]], this triggers a disjunctive constraint⁠, where one can allow different forms of each word.
• renormalize_logits (bool) (optional): Whether to renormalize the logits after applying all the logits processors or warpers (including the custom ones). It's highly recommended to set this flag to True as the search algorithms suppose the score logits are normalized but some logit processors or warpers break the normalization. Default: false.
• constraints (List[Constraint]) (optional): Custom constraints that can be added to the generation to ensure that the output will contain the use of certain tokens as defined by Constraint objects, in the most sensible way possible.
• forced_bos_token_id (int) (optional): The id of the token to force as the first generated token after the decoder_start_token_id. Useful for multilingual models like mBART⁠ where the first generated token needs to be the target language token. Default: model.config.forced_bos_token_id.
• forced_eos_token_id (Union[int, List[int]]) (optional): The id of the token to force as the last generated token when max_length is reached. Optionally, use a list to set multiple end-of-sequence tokens. Default: model.config.forced_eos_token_id.
• remove_invalid_values (bool) (optional): Whether to remove possible nan and inf outputs of the model to prevent the generation method to crash. Note that using remove_invalid_values can slow down generation. Default: model.config.remove_invalid_values.
• exponential_decay_length_penalty (tuple(int, float)) (optional): This Tuple adds an exponentially increasing length penalty, after a certain amount of tokens have been generated. The tuple shall consist of: (start_index, decay_factor) where start_index indicates where penalty starts and decay_factor represents the factor of exponential decay
• suppress_tokens (List[int]) (optional): A list of tokens that will be suppressed at generation. The SupressTokens logit processor will set their log probs to -inf so that they are not sampled.
• begin_suppress_tokens (List[int]) (optional): A list of tokens that will be suppressed at the beginning of the generation. The SupressBeginTokens logit processor will set their log probs to -inf so that they are not sampled.
• forced_decoder_ids (List[List[int]]) (optional): A list of pairs of integers which indicates a mapping from generation indices to token indices that will be forced before sampling. For example, [[1, 123]] means the second generated token will always be a token of index 123.
• sequence_bias (Dict[Tuple[int], float]) (optional): Dictionary that maps a sequence of tokens to its bias term. Positive biases increase the odds of the sequence being selected, while negative biases do the opposite.
• guidance_scale (float) (optional): The guidance scale for classifier free guidance (CFG). CFG is enabled by setting guidance_scale > 1. Higher guidance scale encourages the model to generate samples that are more closely linked to the input prompt, usually at the expense of poorer quality.
• low_memory (bool) (optional): Switch to sequential beam search and sequential topk for contrastive search to reduce peak memory. Used with beam search and contrastive search.

⁠GENERATE PARAMETERS

• num_return_sequences (int) (optional): The number of independently computed returned sequences for each element in the batch. Default: 1.
• output_attentions (bool) (optional): Whether or not to return the attentions tensors of all attention layers. See attentions under returned tensors for more details. Default: false.
• output_hidden_states (bool) (optional): Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for more details. Default: false.
• output_scores (bool) (optional): Whether or not to return the prediction scores. See scores under returned tensors for more details. Default: false.
• output_logits (bool) (optional): Whether or not to return the unprocessed prediction logit scores. See logits under returned tensors for more details.
• return_dict_in_generate (bool) (optional): Whether or not to return a [~utils.ModelOutput] instead of a plain tuple. Default: false.

⁠SPECIAL TOKENS

• pad_token_id (int) (optional): The id of the padding token.
• bos_token_id (int) (optional): The id of the beginning-of-sequence token.
• eos_token_id (Union[int, List[int]]) (optional): The id of the end-of-sequence token. Optionally, use a list to set multiple end-of-sequence tokens.

⁠ENCODER DECODER GENERATION PARAMETERS

• encoder_no_repeat_ngram_size (int) (optional): If set to int > 0, all ngrams of that size that occur in the encoder_input_ids cannot occur in the decoder_input_ids. Default: 0.
• decoder_start_token_id (Union[int, List[int]]) (optional): If an encoder-decoder model starts decoding with a different token than bos, the id of that token or a list of length batch_size. Indicating a list enables different start ids for each element in the batch (e.g. multilingual models with different target languages in one batch)

⁠ASSISTANT GENERATION PARAMETERS

• num_assistant_tokens (int) (optional): Defines the number of speculative tokens that shall be generated by the assistant model before being checked by the target model at each iteration. Higher values for num_assistant_tokens make the generation more speculative: If the assistant model is performant larger speed-ups can be reached, if the assistant model requires lots of corrections, lower speed-ups are reached. Default: 5.
• num_assistant_tokens_schedule (str) (optional): Defines the schedule at which max assistant tokens shall be changed during inference. - heuristic: When all speculative tokens are correct, increase num_assistant_tokens by 2 else reduce by 1. num_assistant_tokens value is persistent over multiple generation calls with the same assistant model. - heuristic_transient: Same as heuristic but num_assistant_tokens is reset to its initial value after each generation call. - constant: num_assistant_tokens stays unchanged during generation Default: heuristic.

⁠CACHING PARAMETERS

• cache_implementation (str) (optional): Cache class that should be used when generating. Default: null.

⁠GENERATION KWARGS

• generation_kwargs (object) (optional): Additional generation kwargs will be forwarded to the generate function of the model. Kwargs that are not present in generate's signature will be used in the model forward pass.

Full parameter list available here⁠, courtesy of Hugging Face.

Using models locally offers enhanced privacy, control, and customization for your projects. Happy building!