[{"_1":2,"_160":-5,"_161":-5},"loaderData",{"_3":4,"_17":18,"_25":26,"_139":140},"root",{"_5":-5,"_6":-7,"_7":8,"_9":-7,"_10":11,"_12":-7,"_13":14,"_15":16},"currentUser","primaryEmail","siteVerification","6LeIxAcTAAAAAJcZVRqyHh71UMIEGNQ_MXjiZKhI","toast","enableDev",false,"incidentStatus","theme","system","contrast","normal","routes/_layout",{"_19":-5,"_20":21,"_22":21,"_23":11,"_24":21},"username","isAskGordonEnabled",true,"isUsingNewTemplate","showMinimalLanding","showHoneypotTrap","routes/_layout.r.$namespace.$reponame",{"_27":28,"_29":-5,"_30":11,"_31":11,"_32":33,"_34":35,"_36":-5,"_37":11,"_38":39,"_88":-5,"_89":90,"_128":-5,"_129":11,"_130":21,"_131":11,"_132":-5,"_133":11,"_134":-5,"_135":136,"_137":91,"_138":-7},"canonicalUrl","https://hub.docker.com/r/bytez/microsoft_phi-3-medium-128k-instruct/tags","mcpServer","canSeeDockerfile","canSeeAutobuilds","namespace","bytez","reponame","microsoft_phi-3-medium-128k-instruct","cloudBuildSource","isAutobuildsAvailable","repo",{"_40":41,"_42":43,"_45":46,"_47":48,"_49":11,"_50":51,"_52":53,"_54":55,"_57":35,"_32":33,"_58":59,"_60":44,"_61":62,"_63":64,"_65":66,"_67":-5,"_68":62,"_69":70,"_71":11,"_72":33,"_73":74,"_79":11,"_80":81,"_85":33,"_86":87},"categories",[],"content_types",[44],"image","date_registered","2024-06-29T18:08:23.090354Z","description","","is_private","last_updated","2024-08-22T23:02:35.221597Z","last_modified","2025-05-01T00:18:55.257231Z","media_types",[56],"application/vnd.docker.container.image.v1+json","name","pull_count",15605,"repository_type","star_count",0,"status",1,"status_description","active","source","collaborator_count","full_description","# Model\n\nModel: microsoft/Phi-3-medium-128k-instruct\n\nTask: text-generation\n\n# How to Run this Model\n\nIf you're just getting started, we recommend that you try out the [Bytez Model Playground](https://bytez.com/settings) directly or use one of our [Client Libraries](https://github.com/Bytez-com/docs/blob/main/README.md) to access the [Bytez Inference API](https://bytez.com/about).\n\nYou'll receive *100* free credits of inference each month!\n\n[Javascript](https://github.com/Bytez-com/docs/blob/main/javascript/readme.md), [Python](https://github.com/Bytez-com/docs/blob/main/python/readme.md), and [Julia](https://github.com/Bytez-com/docs/blob/main/julia/readme.md) are currently supported. \n\n## Bytez Model Playground\n

You can play with models without having to write any code by visiting Bytez

Models can also be explored:

\n\nIf that's not your cup of tea, keep reading!\n\n## Setup Requirements\n\n1. Ensure Docker is installed.\n2. This model container is free and open source. It requires a free API key, so it can alert you of analytics and when model upgrades are available. To get your key, visit [Bytez](https://bytez.com/settings) and sign in. \n\nYour API key will be front and center with a copy button, like you see in the image below:\n\n

\n\n## Running the Image\n\n### Step 1: Pull the Docker Image\n\n```bash\ndocker pull bytez/microsoft_phi-3-medium-128k-instruct\n```\n\n### Step 2: Start the container on port 8000\n\n```bash\ndocker run -it \\\n -e KEY=YOUR_BYTEZ_API_KEY_HERE \\\n -e PORT=8000 \\\n -p 8000:8000 \\\n bytez/microsoft_phi-3-medium-128k-instruct\n```\n\n### Adjusting the port if desired\n\nNOTE you can adjust the port if needed by the `-e PORT=` environment variable and the -p option.\n\ne.g. if you want to start the container on port 80, you'd do this instead:\n\n```bash\ndocker run -it \\\n -e KEY=YOUR_BYTEZ_API_KEY_HERE \\\n -e PORT=80 \\\n -p 80:80 \\\n bytez/microsoft_phi-3-medium-128k-instruct\n```\n\n### Step 3: Make an Inference\nSend POST requests to the container and the model will reply.\n\n```bash\ncurl --location 'http://0.0.0.0:8000/run' \\\n--header 'Content-Type: application/json' \\\n--data-raw '{\n \"text\": [\n {\n \"role\": \"system\",\n \"content\": \"You are a friendly chatbot who responds in the tone of a pirate\"\n },\n {\n \"role\": \"user\",\n \"content\": \"Once upon a time there was a \"\n }\n ],\n \"stream\": false,\n \"params\": {\n \"max_new_tokens\": 50,\n \"temperature\": 1.3,\n \"top_p\": 1.2\n }\n}'\n```\n\nYou can also pass `\"stream\": true` to receive a streamed response. Tokens are streamed one at a time with the response's encoding as \"text/event-stream; charset=utf-8\"\n\n## Storing Weights Locally (Save them to your disk)\n\nTo ensure that weights are saved locally between runs, you can specify a directory for where you want weights to be stored.\n\nFor large models, this is *highly recommeded*, as download times can be hours for larger models.\n\nThis can be specified via the `-v` option\n\nTo do this, run the following command:\n\n```bash\ndocker run -it \\\n -v /PATH/TO/YOUR/CACHING/DIRECTORY/HERE:/server/model \\\n -e HF_HOME=/server/model \\\n -e KEY=YOUR_BYTEZ_API_KEY_HERE \\\n -p 8000:8000 \\\n -e PORT=8000 \\\n bytez/microsoft_phi-3-medium-128k-instruct\n```\n\nNotice how in the command above we have `-v /PATH/TO/YOUR/CACHING/DIRECTORY/HERE:/server/model` and `-e HF_HOME=/server/model`\n\nThe `-v /PATH/TO/YOUR/CACHING/DIRECTORY/HERE:/server/model` says, mount the directory `-v /PATH/TO/YOUR/CACHING/DIRECTORY/HERE` to the docker container's filesystem at the directory `/server/model`\n\n`-e HF_HOME=/server/model` allows the code to load the model from the directory in the docker container, i.e. from the `/server/model` directory.\n\nOn my machine, the command looks like this:\n\n```bash\ndocker run -it \\\n -v /home/inf3rnus/models:/server/model \\\n -e HF_HOME=/server/model \\\n -e KEY=YOUR_BYTEZ_API_KEY_HERE \\\n -p 8000:8000 \\\n -e PORT=8000 \\\n bytez/microsoft_phi-3-medium-128k-instruct\n```\n\n## Running on GPU(s)\n\nTo run on GPU(s), make sure you have the latest drivers from Nvidia and CUDA installed.\n\nThen, simply run the command from above, but with `--gpus all` added to the list of docker options.\n\n```bash\ndocker run -it \\\n --gpus all \\\n -e KEY=YOUR_BYTEZ_API_KEY_HERE \\\n -p 8000:8000 \\\n -e PORT=8000 \\\n bytez/microsoft_phi-3-medium-128k-instruct\n```\n\n## Local caching and running on GPU(s)\n\nThe two commands from above combined into one:\n\n```bash\ndocker run -it \\\n --gpus all \\\n -v /PATH/TO/YOUR/CACHING/DIRECTORY/HERE:/server/model \\\n -e HF_HOME=/server/model \\\n -e KEY=YOUR_BYTEZ_API_KEY_HERE \\\n -p 8000:8000 \\\n -e PORT=8000 \\\n bytez/microsoft_phi-3-medium-128k-instruct\n```\n\n\n## Additional environment variables\n\n`-e DEVICE=\"SOME_VALUE_HERE\"`\n\nDefaults to `auto`\n\nCan be:\n\n`-e DEVICE=\"cuda\"`\n\nor\n\n`-e DEVICE=\"auto\"` \n\nor\n\n`-e DEVICE=\"cpu\"`\n\n`auto` will attempt to place the weights on the GPU if available, and then place them onto system RAM if there is not enough memory.\n\n`cuda` will attempt to place the weights on the GPU\n\n`cpu` will attempt to place the weights on the CPU\n\nAllows you to specify with greater control which device you want to run the model on. Auto may split the model across system RAM and VRAM. You will often use this to attempt forcing the model to be loaded onto the GPU.\n\nNOTE: Some models only exclusively work with `auto`, `cuda`, or `cpu`\n\n## Have questions? Need help?\n\nHop into the [Bytez discord](https://discord.com/login?redirect_to=%2Foauth2%2Fauthorize%3Fclient_id%3D1113179276284526672%26redirect_uri%3Dhttps%253A%252F%252Fapi.bytez.com%252Fdiscord%26response_type%3Dcode%26scope%3Didentify%2520email%2520guilds.join%2520guilds.members.read%2520role_connections.write) for live support: the community is happy to help. If you don't have discord, [email us](mailto:supoprt@bytez.com).\n\n## Model Parameters\n### LENGTH CONTROL\n- **max_length** (int) (optional): The maximum length the generated tokens can have. Default: 20.\n- **max_new_tokens** (int) (optional): The maximum numbers of tokens to generate, ignoring the number of tokens in the prompt.\n- **min_length** (int) (optional): The minimum length of the sequence to be generated. Default: 0.\n- **min_new_tokens** (int) (optional): The minimum numbers of tokens to generate, ignoring the number of tokens in the prompt.\n- **early_stopping** (bool or str) (optional): Controls the stopping condition for beam-based methods. Default: False.\n- **max_time** (float) (optional): The maximum amount of time for the computation to run.\n\n### GENERATION STRATEGY\n- **do_sample** (bool) (optional): Whether to use sampling or greedy decoding. Default: False.\n- **num_beams** (int) (optional): Number of beams for beam search. Default: 1.\n- **num_beam_groups** (int) (optional): Number of groups for diversity among beams. Default: 1.\n- **penalty_alpha** (float) (optional): Balance model confidence and degeneration penalty.\n- **use_cache** (bool) (optional): Whether to use cache for speeding up decoding. Default: True.\n\n### LOGITS MANIPULATION\n- **temperature** (float) (optional): The value used to modulate the next token probabilities. Default: 1.\n- **top_k** (int) (optional): The number of highest probability vocabulary tokens to keep for top-k-filtering. Default: 50.\n- **top_p** (float) (optional): If set to float < 1, only the smallest set of most probable tokens with probabilities that add up to `top_p` or higher are kept for generation. Default: 1.\n- **typical_p** (float) (optional): Local typicality measures how similar the conditional probability of predicting a target token next is to the expected conditional probability of predicting a random token next, given the partial text already generated. If set to float < 1, the smallest set of the most locally typical tokens with probabilities that add up to `typical_p` or higher are kept for generation. See [this paper](https://arxiv.org/pdf/2202.00666.pdf) for more details. Default: 1.\n- **epsilon_cutoff** (float) (optional): If set to float strictly between 0 and 1, only tokens with a conditional probability greater than `epsilon_cutoff` will be sampled. In the paper, suggested values range from 3e-4 to 9e-4, depending on the size of the model. See [Truncation Sampling as Language Model Desmoothing](https://arxiv.org/abs/2210.15191) for more details. Default: 0.\n- **eta_cutoff** (float) (optional): Eta sampling is a hybrid of locally typical sampling and epsilon sampling. If set to float strictly between 0 and 1, a token is only considered if it is greater than either `eta_cutoff` or `sqrt(eta_cutoff) * exp(-entropy(softmax(next_token_logits)))`. The latter term is intuitively the expected next token probability, scaled by `sqrt(eta_cutoff)`. In the paper, suggested values range from 3e-4 to 2e-3, depending on the size of the model. See [Truncation Sampling as Language Model Desmoothing](https://arxiv.org/abs/2210.15191) for more details. Default: 0.\n- **diversity_penalty** (float) (optional): This value is subtracted from a beam's score if it generates a token same as any beam from other group at a particular time. Note that `diversity_penalty` is only effective if `group beam search` is enabled. Default: 0.\n- **repetition_penalty** (float) (optional): The parameter for repetition penalty. 1.0 means no penalty. See [this paper](https://arxiv.org/pdf/1909.05858.pdf) for more details. Default: 1.\n- **encoder_repetition_penalty** (float) (optional): The paramater for encoder_repetition_penalty. An exponential penalty on sequences that are not in the original input. 1.0 means no penalty. Default: 1.\n- **length_penalty** (float) (optional): Exponential penalty to the length that is used with beam-based generation. It is applied as an exponent to the sequence length, which in turn is used to divide the score of the sequence. Since the score is the log likelihood of the sequence (i.e. negative), `length_penalty` > 0.0 promotes longer sequences, while `length_penalty` < 0.0 encourages shorter sequences. Default: 1.\n- **no_repeat_ngram_size** (int) (optional): If set to int > 0, all ngrams of that size can only occur once. Default: 0.\n- **bad_words_ids** (List[List[int]]) (optional): List of list of token ids that are not allowed to be generated. Check [`~generation.NoBadWordsLogitsProcessor`] for further documentation and examples.\n- **force_words_ids** (List[List[int]]) (optional): List of token ids that must be generated. If given a `List[List[int]]`, this is treated as a simple list of words that must be included, the opposite to `bad_words_ids`. If given `List[List[List[int]]]`, this triggers a [disjunctive constraint](https://github.com/huggingface/transformers/issues/14081), where one can allow different forms of each word.\n- **renormalize_logits** (bool) (optional): Whether to renormalize the logits after applying all the logits processors or warpers (including the custom ones). It's highly recommended to set this flag to `True` as the search algorithms suppose the score logits are normalized but some logit processors or warpers break the normalization. Default: false.\n- **constraints** (List[Constraint]) (optional): Custom constraints that can be added to the generation to ensure that the output will contain the use of certain tokens as defined by `Constraint` objects, in the most sensible way possible.\n- **forced_bos_token_id** (int) (optional): The id of the token to force as the first generated token after the `decoder_start_token_id`. Useful for multilingual models like [mBART](../model_doc/mbart) where the first generated token needs to be the target language token. Default: model.config.forced_bos_token_id.\n- **forced_eos_token_id** (Union[int, List[int]]) (optional): The id of the token to force as the last generated token when `max_length` is reached. Optionally, use a list to set multiple *end-of-sequence* tokens. Default: model.config.forced_eos_token_id.\n- **remove_invalid_values** (bool) (optional): Whether to remove possible *nan* and *inf* outputs of the model to prevent the generation method to crash. Note that using `remove_invalid_values` can slow down generation. Default: model.config.remove_invalid_values.\n- **exponential_decay_length_penalty** (tuple(int, float)) (optional): This Tuple adds an exponentially increasing length penalty, after a certain amount of tokens have been generated. The tuple shall consist of: `(start_index, decay_factor)` where `start_index` indicates where penalty starts and `decay_factor` represents the factor of exponential decay\n- **suppress_tokens** (List[int]) (optional): A list of tokens that will be suppressed at generation. The `SupressTokens` logit processor will set their log probs to `-inf` so that they are not sampled.\n- **begin_suppress_tokens** (List[int]) (optional): A list of tokens that will be suppressed at the beginning of the generation. The `SupressBeginTokens` logit processor will set their log probs to `-inf` so that they are not sampled.\n- **forced_decoder_ids** (List[List[int]]) (optional): A list of pairs of integers which indicates a mapping from generation indices to token indices that will be forced before sampling. For example, `[[1, 123]]` means the second generated token will always be a token of index 123.\n- **sequence_bias** (Dict[Tuple[int], float]) (optional): Dictionary that maps a sequence of tokens to its bias term. Positive biases increase the odds of the sequence being selected, while negative biases do the opposite.\n- **guidance_scale** (float) (optional): The guidance scale for classifier free guidance (CFG). CFG is enabled by setting `guidance_scale > 1`. Higher guidance scale encourages the model to generate samples that are more closely linked to the input prompt, usually at the expense of poorer quality.\n- **low_memory** (bool) (optional): Switch to sequential beam search and sequential topk for contrastive search to reduce peak memory. Used with beam search and contrastive search.\n\n### GENERATE PARAMETERS\n- **num_return_sequences** (int) (optional): The number of independently computed returned sequences for each element in the batch. Default: 1.\n- **output_attentions** (bool) (optional): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more details. Default: false.\n- **output_hidden_states** (bool) (optional): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more details. Default: false.\n- **output_scores** (bool) (optional): Whether or not to return the prediction scores. See `scores` under returned tensors for more details. Default: false.\n- **output_logits** (bool) (optional): Whether or not to return the unprocessed prediction logit scores. See `logits` under returned tensors for more details.\n- **return_dict_in_generate** (bool) (optional): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. Default: false.\n\n### SPECIAL TOKENS\n- **pad_token_id** (int) (optional): The id of the *padding* token.\n- **bos_token_id** (int) (optional): The id of the *beginning-of-sequence* token.\n- **eos_token_id** (Union[int, List[int]]) (optional): The id of the *end-of-sequence* token. Optionally, use a list to set multiple *end-of-sequence* tokens.\n\n### ENCODER DECODER GENERATION PARAMETERS\n- **encoder_no_repeat_ngram_size** (int) (optional): If set to int > 0, all ngrams of that size that occur in the `encoder_input_ids` cannot occur in the `decoder_input_ids`. Default: 0.\n- **decoder_start_token_id** (Union[int, List[int]]) (optional): If an encoder-decoder model starts decoding with a different token than *bos*, the id of that token or a list of length `batch_size`. Indicating a list enables different start ids for each element in the batch (e.g. multilingual models with different target languages in one batch)\n\n### ASSISTANT GENERATION PARAMETERS\n- **num_assistant_tokens** (int) (optional): Defines the number of _speculative tokens_ that shall be generated by the assistant model before being checked by the target model at each iteration. Higher values for `num_assistant_tokens` make the generation more _speculative_: If the assistant model is performant larger speed-ups can be reached, if the assistant model requires lots of corrections, lower speed-ups are reached. Default: 5.\n- **num_assistant_tokens_schedule** (str) (optional): Defines the schedule at which max assistant tokens shall be changed during inference. - `heuristic`: When all speculative tokens are correct, increase `num_assistant_tokens` by 2 else reduce by 1. `num_assistant_tokens` value is persistent over multiple generation calls with the same assistant model. - `heuristic_transient`: Same as `heuristic` but `num_assistant_tokens` is reset to its initial value after each generation call. - `constant`: `num_assistant_tokens` stays unchanged during generation Default: heuristic.\n\n### CACHING PARAMETERS\n- **cache_implementation** (str) (optional): Cache class that should be used when generating. Default: null.\n\n### GENERATION KWARGS\n- **generation_kwargs** (object) (optional): Additional generation kwargs will be forwarded to the `generate` function of the model. Kwargs that are not present in `generate`'s signature will be used in the model forward pass.\n\nFull parameter list available [here](https://github.com/huggingface/transformers/blob/v4.38.2/src/transformers/generation/configuration_utils.py), courtesy of Hugging Face.\n\n\nUsing models locally offers enhanced privacy, control, and customization for your projects. Happy building!","has_starred","hub_user","immutable_tags_settings",{"_75":11,"_76":77},"enabled","rules",[78],".*","is_automated","permissions",{"_82":21,"_83":11,"_84":11},"read","write","admin","user","storage_size",3474570430,"orgData","recentTags",[91],{"_92":44,"_93":94,"_95":96,"_97":98,"_99":100,"_101":102,"_50":117,"_118":94,"_119":33,"_120":56,"_57":121,"_122":123,"_124":108,"_125":117,"_126":66,"_127":21},"content_type","creator",4550410,"digest","sha256:5c9c621d16de86c9b33ddc0b7b8326bcbaf78e207370ef554e9be1fca8496276","full_size",3474396870,"id",710814706,"images",[103],{"_104":105,"_95":96,"_106":48,"_107":108,"_109":110,"_111":112,"_113":48,"_114":-5,"_115":98,"_63":66,"_116":-5},"architecture","amd64","features","last_pulled","2026-10-01T13:30:53.760810523Z","last_pushed","2024-08-22T23:02:35Z","os","linux","os_features","os_version","size","variant","2024-08-22T23:02:34.271268Z","last_updater","last_updater_username","media_type","latest","repository",24492579,"tag_last_pulled","tag_last_pushed","tag_status","v2","sbxKitMetadata","isPotentialSpam","isHubSbxKitsEnabled","showSbxLogoOverride","sbxTrademarkText","displaySecurityMetadata","tagAttestations","tagsHaveAttestations",[],"selectedTag","attestationsUrl","routes/_layout.r.$namespace.$reponame.tags",{"_38":141,"_148":149,"_135":158,"_159":11},{"_40":142,"_42":143,"_45":46,"_47":48,"_49":11,"_50":51,"_52":53,"_54":144,"_57":35,"_32":33,"_58":59,"_60":44,"_61":62,"_63":64,"_65":66,"_67":-5,"_68":62,"_69":70,"_71":11,"_72":33,"_73":145,"_79":11,"_80":147,"_85":33,"_86":87},[],[44],[56],{"_75":11,"_76":146},[78],{"_82":21,"_83":11,"_84":11},"tags",{"_150":64,"_151":-5,"_152":-5,"_153":154},"count","next","previous","results",[155],{"_92":44,"_93":94,"_95":96,"_97":98,"_99":100,"_101":156,"_50":117,"_118":94,"_119":33,"_120":56,"_57":121,"_122":123,"_124":108,"_125":117,"_126":66,"_127":21},[157],{"_104":105,"_95":96,"_106":48,"_107":108,"_109":110,"_111":112,"_113":48,"_114":-5,"_115":98,"_63":66,"_116":-5},[],"canDeleteTags","actionData","errors"]

bytez/microsoft_phi-3-medium-128k-instruct

Tags for bytez/microsoft_phi-3-medium-128k-instruct