LoRA & QLoRA Architectural Policy
Low-Rank Adaptation (LoRA) freezes pre-trained foundational weights and injects trainable rank decomposition matrices into Transformer attention and feed-forward layers, drastically reducing trainable parameter counts and memory overhead.
LoRA (Low-Rank Adaptation) decomposes dense weight updates into two low-rank matrices (W = W₀ + (α/r) · BA). Instead of modifying 8 billion parameters, LoRA trains ~0.1% to 1.0% of the weights, enabling full domain adaptation on consumer GPUs or Apple Silicon while yielding lightweight adapter checkpoints under 150 MB.
The platform explicitly separates controls that novices often confuse:
• LoRA Rank (r): Dimensionality of adapter weight decomposition (e.g. 16, 32).
• Retrieval Candidate top_k: Number of vector search items fetched from the index (e.g. 20-50).
• Reranker Final k: Top items preserved after Lexical Re-Ranker scoring (e.g. 3-5).
• Generation Sampling top_k: Vocabulary distribution sampling truncation (e.g. 40).
What Are the Recommended Hyperparameters for LoRA?
LoRA Rank (r) = 16
Captures rich domain adaptation while keeping adapter weights compact (~40MB-120MB per model).
LoRA Alpha (α) = 32 (2 * r)
Maintains stable scaling ratio (alpha / r = 2.0) preventing gradient explosion during backpropagation.
Target Modules: All Linear Layers
Adapting attention projections (q, k, v, o) and MLP feed-forward projections (gate, up, down) produces significantly improved generalization in empirical benchmarks.
Learning Rate: 2e-4 (QLoRA) / 5e-5 (LoRA)
Cosine learning rate scheduler with 3% warmup steps to prevent loss spikes in early epochs.