# LMFlow | An Extensible Toolkit For Finetuning And Inference Of Large Foundation Models **URL:** \ **Category:** Tools & Scripts\ **Tags:** ai, gaming\ **Created:** [November 21, 2024, 9:53pm UTC](https://onehack.st/t/lmflow-an-extensible-toolkit-for-finetuning-and-inference-of-large-foundation-models/296119 "2024-11-21T21:53:00Z")\ **Posts on this page:** 1\ **Page:** 1
**Author:** ![TheStrength](https://onehack.st/user_avatar/onehack.st/thestrength/32/46753_2.png) [@TheStrength](https://onehack.st/u/TheStrength)\ **Post date:** [November 21, 2024, 9:53pm UTC](https://onehack.st/t/lmflow-an-extensible-toolkit-for-finetuning-and-inference-of-large-foundation-models/296119/1 "2024-11-21T21:53:00Z")
[![LMFlow](https://github.com/OptimalScale/LMFlow/raw/main/assets/logo.png)](https://github.com/OptimalScale/LMFlow/blob/main/assets/logo.png) # LMFlow #### **English** | [简体中文](https://github.com/OptimalScale/LMFlow/blob/main/readme/README_zh-hans.md) | [Español](https://github.com/OptimalScale/LMFlow/blob/main/readme/README_es.md) | [日本語](https://github.com/OptimalScale/LMFlow/blob/main/readme/README_jp.md) | [한국어](https://github.com/OptimalScale/LMFlow/blob/main/readme/README_ko.md) | [हिंदी](https://github.com/OptimalScale/LMFlow/blob/main/readme/README_hindi.md) [![Website](https://camo.githubusercontent.com/e3dd38cd9c763b15fe03d03514b51990bcae4cb5ae851dd72aeafcea4204b155/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f576562736974652d44656d6f2d3230423241412e737667)](https://lmflow.com/) [![Code License](https://camo.githubusercontent.com/cafdb4dc0564bd18688987de750459b6e1505e60fa26b3dc9181d4fed474947a/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f436f64652532304c6963656e73652d4170616368655f322e302d677265656e2e737667)](https://github.com/OptimalScale/LMFlow/blob/main/LICENSE) [![Python 3.9+](https://camo.githubusercontent.com/f50916cb35299eec933c47d8f5086933c5b9252b2b7831239062d9236eca8f78/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f507974686f6e2d332e392b2d626c75652e737667)](https://www.python.org/downloads/release/python-390/) [![Doc](https://camo.githubusercontent.com/fb03d30477d8cada89261b56ff1cbf7f0f8179f752e9a87b765cabcfb36d6ec9/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f576562736974652d446f632d6666363962342e737667)](https://optimalscale.github.io/LMFlow/) [![Embark](https://camo.githubusercontent.com/a5f79b96c522e3d99414f04206142898dd537e001cdf43a87e4f063fa9ca7f5d/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f446973636f72642d4c4d466c6f772d2532333732383964612e7376673f6c6f676f3d646973636f7264)](https://discord.gg/u9VJNpzhvA) [![slack badge](https://camo.githubusercontent.com/828f2bbff8a6178822547aaff39daa570a18e8f2dec86cfa2a1990874fbc31ff/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f536c61636b2d4a6f696e2d626c756576696f6c65743f6c6f676f3d736c61636b26616d70)](https://join.slack.com/t/lmflow/shared_invite/zt-1wju9nicy-woXbNtS~5MavHSAtiMxmxQ) [![WeChat badge](https://camo.githubusercontent.com/2b879449b0b83bfb5aae6211102b658d38134f30a68b1dce57c0aca603b9f553/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f5765436861742d4a6f696e2d627269676874677265656e3f6c6f676f3d77656368617426616d70)](https://ibb.co/ZhM4hhn) An extensible, convenient, and efficient toolbox for finetuning large machine learning models, designed to be user-friendly, speedy and reliable, and accessible to the entire community. [![LMFlow-features](https://github.com/OptimalScale/LMFlow/raw/main/assets/features.png)](https://github.com/OptimalScale/LMFlow/blob/main/assets/features.png) ## Latest News - [2024-07-01] 🏆 LMFlow receives the [**Best Demo Paper Award**](https://docs.google.com/presentation/d/1TVDooAZqkNObz5ysVhDFtqnnVHR-u8wqYvgix-gzPMs/edit#slide=id.g2e55907bbcc_0_70) at **NAACL 2024**! 🎉 - [2024-06-30] Expanding Optimization Options! We now support custom optimizer training with a variety of optimizers. Dive into the details and try out the new features with our updated script at [custom\_optimizers](https://github.com/OptimalScale/LMFlow/blob/main/scripts/run_finetune_with_custom_optim.sh). - [2024-04-25] 🚀 Support conversation template! We’ve preset the latest [Llama-3](https://huggingface.co/meta-llama/Meta-Llama-3-70B) and [Phi-3](https://huggingface.co/microsoft/Phi-3-mini-128k-instruct) conversation templates as well as some frequently used templates such as `chatml` (see all templates [here](https://optimalscale.github.io/LMFlow/examples/DATASETS.html#conversation-template)), and we are working on adding more preset templates. Adding corresponding `--conversation_template` in the shell script and you are all set! 🚀 - [2024-03-27] Support [LISA](https://arxiv.org/abs/2403.17919), enabling 7B training in 24G memory without offloading! - [2023-09-11] Support [speculative decoding](https://arxiv.org/abs/2211.17192). Check out [speculative\_decoding](https://github.com/OptimalScale/LMFlow/blob/main/scripts/speculative_decoding/README.md) for the usage and acceleration details. - [2023-08-14] Support long context inference with position interpolation (Linear & NTK scaling ) for LLaMA models. Check out [postion\_interpolation](https://github.com/OptimalScale/LMFlow/blob/main/readme/Position_Interpolation.md) for more details. - [2023-08-07] Support [Flash Attention-2](https://crfm.stanford.edu/2023/07/17/flash2.html). Check out [flash\_attention](https://github.com/OptimalScale/LMFlow/blob/main/readme/flash_attn2.md) for more details. More news… - `` ## Table of Contents - [LMFlow](https://github.com/OptimalScale/LMFlow#lmflow) - [Latest News](https://github.com/OptimalScale/LMFlow#latest-news) - [Table of Contents](https://github.com/OptimalScale/LMFlow#table-of-contents) - [Supported Models](https://github.com/OptimalScale/LMFlow#supported-models) - [Quick Start](https://github.com/OptimalScale/LMFlow#quick-start) - [Setup](https://github.com/OptimalScale/LMFlow#setup) - [Prepare Dataset](https://github.com/OptimalScale/LMFlow#prepare-dataset) - [Finetuning](https://github.com/OptimalScale/LMFlow#finetuning) - [Full Finetuning](https://github.com/OptimalScale/LMFlow#full-finetuning) - [LISA](https://github.com/OptimalScale/LMFlow#lisa) - [LoRA](https://github.com/OptimalScale/LMFlow#lora) - [Inference](https://github.com/OptimalScale/LMFlow#inference) - [Deployment](https://github.com/OptimalScale/LMFlow#deployment) - [Evaluation](https://github.com/OptimalScale/LMFlow#evaluation) - [Supported Features](https://github.com/OptimalScale/LMFlow#supported-features) - [Support](https://github.com/OptimalScale/LMFlow#support) - [License](https://github.com/OptimalScale/LMFlow#license) - [Citation](https://github.com/OptimalScale/LMFlow#citation) ## Supported Models | Model | Conversation Template (Details) | | --- | --- | | [DeepSeek](https://huggingface.co/deepseek-ai) | `deepseek` ([Link](https://optimalscale.github.io/LMFlow/examples/supported_conversation_template.html#deepseek)) | | [Gemma](https://huggingface.co/google) | `gemma` ([Link](https://optimalscale.github.io/LMFlow/examples/supported_conversation_template.html#gemma)) | | [InternLM2](https://huggingface.co/internlm) | `internlm2` ([Link](https://optimalscale.github.io/LMFlow/examples/supported_conversation_template.html#internlm2)) | | [LLaMA-2](https://huggingface.co/meta-llama) | `llama2` ([Link](https://optimalscale.github.io/LMFlow/examples/supported_conversation_template.html#llama-2)) | | [LLaMA-3](https://huggingface.co/meta-llama) | `llama3` ([Link](https://optimalscale.github.io/LMFlow/examples/supported_conversation_template.html#llama-3)) | | [Phi-3](https://huggingface.co/microsoft) | `phi3` ([Link](https://optimalscale.github.io/LMFlow/examples/supported_conversation_template.html#phi-3)) | | [Qwen1.5 | | | Qwen2]([https://huggingface.co/Qwen](https://huggingface.co/Qwen)) | `qwen2` ([Link](https://optimalscale.github.io/LMFlow/examples/supported_conversation_template.html#qwen-2)) | | [Yi](https://huggingface.co/01-ai) | `chatml` ([Link](https://optimalscale.github.io/LMFlow/examples/supported_conversation_template.html#yi)) | | [Yi-1.5](https://huggingface.co/01-ai) | `yi1_5` ([Link](https://optimalscale.github.io/LMFlow/examples/supported_conversation_template.html#yi-15)) | | [Zephyr](https://huggingface.co/HuggingFaceH4) | `zephyr` ([Link](https://optimalscale.github.io/LMFlow/examples/supported_conversation_template.html#zephyr)) | ## Quick Start ### Setup Our package has been tested on Linux OS (Ubuntu 20.04). Other OS platforms (MacOS, Windows) are not fully tested, where you may encounter unexpected errors. If you are using LMFlow for the first time, we recommend you to try on a Linux machine or Google Colab. CUDA versions 10.3-11.7 are supported in versions `v0.0.5` or older. For CUDA versions greater than 11.7, one can use our stable branch `>= v0.0.6`. git clone -b v0.0.9 [https://github.com/OptimalScale/LMFlow.git](https://github.com/OptimalScale/LMFlow.git) cd LMFlow conda create -n lmflow python=3.9 -y conda activate lmflow conda install mpi4py pip install -e . Tip We use WandB to track and visualize the training process by default. Before running the training scripts, users may need to log in to WandB using the command: wandb login For detailed instructions, refer to the [WandB Quickstart Guide](https://docs.wandb.ai/quickstart/). Step 1 (registration) and Step 2 (login using your WandB API key) should be sufficient to set up your environment. Disabling wandb ### Prepare Dataset Please refer to our [doc](https://optimalscale.github.io/LMFlow/examples/DATASETS.html). ### Finetuning #### Full Finetuning Full training updates all the parameters to finetune a language model. Here is an example to finetune a GPT-2 base model. cd data && ./download.sh alpaca && cd - bash ./scripts/run\_finetune.sh \ –model\_name\_or\_path gpt2 \ –dataset\_path data/alpaca/train\_conversation \ –output\_model\_path output\_models/finetuned\_gpt2 Tip For conversation dataset, specify a conversation template for better performance by adding `--conversation_template` to the command. Llama-3-8B conversation dataset example #### LISA [LISA](https://arxiv.org/abs/2403.17919) is a memory-efficient finetuning algorithm that allows tradeoff between memory and the number of randomly unfreezed layers. This script currently is only tested in single gpus. Please stay tuned for our latest updates 😄 cd data && ./download.sh alpaca && cd - bash ./scripts/run\_finetune\_with\_lisa.sh \ –model\_name\_or\_path meta-llama/Llama-2-7b-hf \ –dataset\_path data/alpaca/train\_conversation \ –output\_model\_path output\_models/finetuned\_llama2\_7b \ –lisa\_activated\_layers 1 \ –lisa\_interval\_steps 20 Tip Llama-2-7B conversation dataset example #### LoRA LoRA is a parameter-efficient finetuning algorithm and is more efficient than full finetuning. cd data && ./download.sh alpaca && cd - bash ./scripts/run\_finetune\_with\_lora.sh \ –model\_name\_or\_path facebook/galactica-1.3b \ –dataset\_path data/alpaca/train\_conversation \ –output\_lora\_path output\_models/finetuned\_galactica\_lora Tip Llama-2-7B conversation dataset exampleMerge LoRA Weight ### Inference After finetuning, you can run the following command to chat with the model. bash ./scripts/run\_chatbot.sh output\_models/finetuned\_gpt2 Tip We recommend using vLLM for faster inference. Faster inference using vLLM ### Deployment If you want to deploy your own model locally, we provide a gradio-based UI for building chatbots. Running the following command will launch the demo for robin-7b: pip install gradio python ./examples/chatbot\_gradio.py --deepspeed configs/ds\_config\_chatbot.json --model\_name\_or\_path YOUR-LLAMA --lora\_model\_path ./robin-7b --prompt\_structure “A chat between a curious human and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the human’s questions.###Human: {input\_text}###Assistant:” --end\_string “#” --max\_new\_tokens 200 ### Evaluation [LMFlow Benchmark](https://blog.gopenai.com/lmflow-benchmark-an-automatic-evaluation-framework-for-open-source-llms-ef5c6f142418) is an automatic evaluation framework for open-source large language models. We use negative log likelihood (NLL) as the metric to evaluate different aspects of a language model: chitchat, commonsense reasoning, and instruction following abilities. You can directly run the LMFlow benchmark evaluation to obtain the results to participate in the [LLM comparision](https://docs.google.com/spreadsheets/d/1JYh4_pxNzmNA9I0YM2epgRA7VXBIeIGS64gPJBg5NHA/edit?usp=sharing). For example, to run GPT2 XL, one may execute bash ./scripts/run\_benchmark.sh --model\_name\_or\_path gpt2-xl `--model_name_or_path` is required, you may fill in huggingface model name or local model path here. To check the evaluation results, you may check `benchmark.log` in `./output_dir/gpt2-xl_lmflow_chat_nll_eval`, `./output_dir/gpt2-xl_all_nll_eval` and `./output_dir/gpt2-xl_commonsense_qa_eval`. ## Supported Features Finetune Acceleration & Memory Optimization - ` ` `` - `` Inference Acceleration - ` ` Long ContextModel CustomizationMultimodalCustom Optimization - ||| | — | — | ||| ||| ||| ||| ||| ||| ||| ||| ||| ||| ||| ||| ||| ||| ||| ||| ||| ||| ||| ||| ||| ||| ||| ## Support If you need any help, please submit a Github issue. ## License The code included in this project is licensed under the [Apache 2.0 license](https://github.com/OptimalScale/LMFlow/blob/main/LICENSE). If you wish to use the codes and models included in this project for commercial purposes, please sign this [document](https://docs.google.com/forms/d/e/1FAIpQLSfJYcci6cbgpIvx_Fh1xDL6pNkzsjGDH1QIcm4cYk88K2tqkw/viewform?usp=pp_url) to obtain authorization. ## Citation If you find this repository useful, please consider giving ⭐ and citing our [paper](https://arxiv.org/abs/2306.12420): ```auto @article{diao2023lmflow, title={Lmflow: An extensible toolkit for finetuning and inference of large foundation models}, author={Diao, Shizhe and Pan, Rui and Dong, Hanze and Shum, Ka Shun and Zhang, Jipeng and Xiong, Wei and Zhang, Tong}, journal={arXiv preprint arXiv:2306.12420}, year={2023} } ``` ```auto @article{dong2023raft, title={Raft: Reward ranked finetuning for generative foundation model alignment}, author={Dong, Hanze and Xiong, Wei and Goyal, Deepanshu and Pan, Rui and Diao, Shizhe and Zhang, Jipeng and Shum, Kashun and Zhang, Tong}, journal={arXiv preprint arXiv:2304.06767}, year={2023} } ``` ```auto @article{pan2024lisa, title={LISA: Layerwise Importance Sampling for Memory-Efficient Large Language Model Fine-Tuning}, author={Pan, Rui and Liu, Xiang and Diao, Shizhe and Pi, Renjie and Zhang, Jipeng and Han, Chi and Zhang, Tong}, journal={arXiv preprint arXiv:2403.17919}, year={2024} } ``` ## GitHub: [https://github.com/OptimalScale/LMFlow](https://github.com/OptimalScale/LMFlow)