Fine Tuning, Prompt Tuning, and Prompt Engineering

 Differences Between Fine Tuning, Prompt Tuning, and Prompt Engineering

  1. Fine Tuning: This involves re-training a pre-trained model to adapt it for a specific task. It adjusts the model's weights through additional training, which can require significant computational resources. The entire model is updated to optimize performance for the new task.

  2. Prompt Tuning: Unlike fine tuning, prompt tuning involves adding a set of trainable parameters (or soft prompts) to the model’s input without altering the original model’s weights. This approach allows the model to adapt to new tasks while keeping the pre-existing weights fixed, making it less computationally intensive than fine tuning.

  3. Prompt Engineering: This method relies entirely on crafting effective input prompts for the model. It does not involve any computational training or modification of model parameters. Prompt engineering is about designing prompts that effectively guide the model to generate the desired output.

Two Approaches to Prompt Tuning:

  1. Soft Prompt Tuning: This approach uses gradient descent to optimize continuous embedding vectors that are attached to the language model’s input. While soft prompts can be effective, they are typically hard for humans to interpret, which can obscure understanding of the model’s decision-making processes. Additionally, these prompts are not reusable across different models and require access to the model's internal gradients, which can be computationally expensive.

  2. Reinforcement Learning (RL) Prompt Tuning: RL prompt tuning optimizes prompts using reinforcement learning techniques. In this method, an agent generates prompts by selecting tokens in a way that maximizes a reward signal derived from the model’s performance on a task. This approach does not require gradient information from the language model, making it suitable for models where gradient access is impractical. RL prompts are specifically tuned for input scenarios and are interpretable, allowing them to handle diverse textual styles. They are also flexible and can be adapted to different models, such as left-to-right models like GPT or masked models like BERT.

 Fine tuning adjusts the entire model to new tasks but requires substantial resources. Prompt tuning modifies input handling to adapt the model with less resource expenditure, and prompt engineering creatively utilizes designed inputs to steer model outputs without any training. 

PEFT: https://arxiv.org/abs/2104.08691 

RL-prompt: https://arxiv.org/abs/2205.12548 

Comments

Post a Comment

Popular posts from this blog

Activation Functions

Image
 Activation functions play a crucial role in neural networks, typically employed in hidden and output layers, but not in input layers. By default, the absence of an activation function implies a linear activation. Here's a closer look at several common types: Sigmoid: Characterized by its S-shaped curve, the sigmoid function outputs values between 0 and 1 for any input ranging from negative to positive infinity. While useful, it is prone to causing vanishing gradient issues due to its output range, and its outputs are not zero-centered. Tanh (Hyperbolic Tangent): Similar to the sigmoid in shape but outputs values from -1 to 1. It offers stronger gradients than sigmoid, making it more effective in some cases. However, it still suffers from vanishing gradient problems like its sigmoid counterpart. ReLU (Rectified Linear Unit): This function addresses some of the drawbacks of sigmoid and tanh by outputting the input directly if it is positive; otherwise, it outputs zero. Although i...
Read more

Efficiency in Large Language Model Training: LoRA, Qlora, and Galore

 Training large language models (LLMs) is a resource-intensive process, primarily due to the vast number of parameters involved. Various methods have been developed to improve the efficiency of this process, focusing on reducing memory usage without significantly sacrificing model performance. LoRA: Low-Rank Adaptation LoRA (Low-Rank Adaptation) is a technique that introduces two low-rank matrices, A and B, into the training process. This method allows the freezing of pre-trained model weights, thereby reducing the number of trainable parameters. By focusing only on adapting these low-rank matrices, LoRA enables finer model tuning while leveraging existing, well-optimized model architectures. QLoRa: Quantized Low-Rank Adaptation Building on the foundation of LoRA, QLoRa incorporates a 4-bit quantized pre-trained model with low-rank adapters. This approach aims to maintain the efficiency benefits of LoRA while further reducing the memory footprint through quantization. Galore: Gradi...
Read more