"Question"	"Answer"	"Tag"
"State the FT09 module thesis in one sentence."	"Modern PEFT (DoRA + rsLoRA) closes roughly half the gap between vanilla LoRA and full fine-tuning, at only ~5–10% more VRAM and zero inference overhead after merging — making DoRA + rsLoRA the 2026 practical default, with GaLore as the bridge when true full-parameter FT is required."	c3::ft09::recall
"What does DoRA stand for, who authored it, and what is the arXiv id?"	"DoRA = Weight-Decomposed Low-Rank Adaptation. Authored by Liu et al. at NVIDIA. arXiv:2402.09353. ICML 2024 (Oral)."	c3::ft09::recall
"What is the structural deficit of LoRA that DoRA names and fixes?"	"LoRA couples magnitude and direction updates proportionally — when the low-rank product B·A shifts a column, both the column's magnitude and its direction move together in a fixed ratio. Full fine-tuning updates them independently; LoRA structurally cannot. This is roughly half of what full FT does, and the half LoRA cannot express at any rank."	c3::ft09::analysis
"Describe the DoRA decomposition. What gets adapted, and how?"	"DoRA rewrites the weight as m ⊙ (W₀ / ‖W₀‖) + ΔW_direction. It takes the frozen base W₀, normalizes it column-wise to get the pure direction, scales by a learned per-column magnitude vector m, and applies a LoRA-style low-rank update ONLY to the direction. Magnitude (the tiny vector m) and direction (the LoRA adapter) are adapted independently."	c3::ft09::analysis
"What does it mean that 'DoRA closes ~half the gap between LoRA and full FT', and at what cost?"	"DoRA recovers roughly half of the quality difference between vanilla LoRA and full-parameter fine-tuning. The cost is ~5–10% more VRAM than vanilla LoRA (the magnitude vectors and the decomposition overhead). After training, magnitude and direction are folded back into the merged weight, so there is ZERO inference overhead — a merged DoRA adapter is shape-identical and speed-identical to a merged LoRA."	c3::ft09::recall
"In which two regimes is DoRA's advantage over LoRA largest?"	"(1) Low ranks (r ≤ 16) — vanilla LoRA is most constrained here, and DoRA's decoupling buys the most expressive headroom exactly where cheap adapters need it. (2) Domain shifts — when the target distribution is far from pretraining (new language family, new modality, heavy style transfer), full FT updates magnitude and direction aggressively; DoRA can approximate that, LoRA cannot. DoRA's lead grows with the size of the distribution shift."	c3::ft09::application
"What is the high-rank paradox in vanilla LoRA?"	"Turning the rank up does not always help — vanilla LoRA often gets WORSE or stalls at high rank (r ≥ 64), despite having more parameters. The cause is the scaling factor α/r, which shrinks the effective update toward zero as rank grows; at rank 64 with α=16 the scale is 0.25, making the adapter numerically present but functionally inert."	c3::ft09::analysis
"What scaling factor does rsLoRA use, and why?"	"rsLoRA changes the scaling from α/r to α/√r. At rank 64 with α=16, the scale is 16/√64 = 2.0 instead of 16/64 = 0.25. This keeps the forward-pass magnitudes stable as rank climbs, so high-rank adapters actually train the way their parameter count suggests they should. 'rs' = rank-stabilized."	c3::ft09::recall
"Above what rank does rsLoRA become essential, and what is the rule of thumb?"	"Above roughly r=32, use rsLoRA. At low rank (r ≤ 16) rsLoRA barely differs from vanilla LoRA because the scaling discrepancy is small. At high rank (r ≥ 64) rsLoRA is essential; vanilla LoRA wastes the capacity. Below r=32 rsLoRA does no harm but offers little."	c3::ft09::application
"How do you enable rsLoRA in Hugging Face PEFT, and what does the flag do?"	"Set use_rslora=True in LoraConfig. The flag swaps the adapter scaling factor from α/r to α/√r. No other change to the training loop, optimizer, or export pipeline. One flag."	c3::ft09::recall
"Why are DoRA and rsLoRA complementary rather than competing?"	"They edit different parts of the adapter update. DoRA adds the magnitude/direction decomposition on top of the frozen base; rsLoRA swaps the scaling factor inside the direction-side LoRA adapter. They are orthogonal knobs. Empirically the combination (use_dora=True AND use_rslora=True) beats either alone, especially at high rank. You use BOTH."	c3::ft09::analysis
"Write the canonical LoraConfig for the 2026 DoRA + rsLoRA sweet spot."	"LoraConfig(r=64, lora_alpha=64, use_dora=True, use_rslora=True, lora_dropout=0.05, bias='none', task_type='CAUSAL_LM', target_modules=[q,k,v,o,gate,up,down]_proj). Two flags — use_dora=True and use_rslora=True — are the entire upgrade from the FT08 vanilla LoRA config to the 2026 default."	c3::ft09::application
"What is PiSSA, what does it change about LoRA, and what is its arXiv id?"	"PiSSA = Principal Singular values and Singular Vectors Adaptation (arXiv:2404.02948, NeurIPS 2024). It changes the INITIALIZATION: instead of LoRA's random-A / zero-B start, PiSSA runs a fast SVD on the frozen W₀, splits it into principal and residual components, and initializes the adapter from the principal part while freezing the residual as the new base. Result: faster convergence and stronger NLU quality at the same parameter count. Drop-in alternative initialization."	c3::ft09::recall
"What is the tradeoff of PiSSA's initialization, and when is PiSSA most often chosen?"	"PiSSA modifies the frozen base (it reshuffles which part is frozen and which is trained), so the merge-back workflow is slightly different — you must merge into the residual base, not the original. This is a one-time accounting cost, not a quality cost. PiSSA is most often chosen when convergence speed matters (short training budgets) and the task is NLU-heavy (classification, extraction, NLI) where the principal components carry the task signal."	c3::ft09::analysis
"What is GaLore, how does it work, and what is its arXiv id?"	"GaLore = Gradient Low-Rank Projection (arXiv:2403.03507, ICML 2024). It trains ALL parameters (full-parameter FT) but projects the GRADIENT into a low-rank subspace before the optimizer sees it. The weights are full-rank and fully updated; the optimizer state — the expensive part (Adam's first/second moment buffers) — lives in the low-rank subspace. The subspace is periodically recomputed via SVD of recent gradient history. Result: full-param learning at a fraction of the optimizer memory."	c3::ft09::recall
"What is the dominant memory cost during full-parameter training, and how does GaLore address it?"	"The optimizer state, not the weights or gradients. Adam keeps two fp32 buffers (first and second moment) per parameter — ~56 GB for a 7B model, the thing that makes full FT impossible on a 24 GB card. LoRA sidesteps it by training only the adapter. GaLore instead trains all parameters but compresses the optimizer state into a low-rank subspace (the gradient is projected before the optimizer sees it). GaLore fit a 7B full-param run on a single 24 GB RTX 4090."	c3::ft09::analysis
"What is the one-sentence use case for GaLore, and how does it relate to the DoRA decision?"	"'I want full-FT quality on a memory-limited node.' If the FT10 decision tree says you need full-parameter FT (large domain shift adapters cannot capture) but the GPU cannot afford vanilla full FT's optimizer memory, GaLore is the bridge. GaLore is NOT in competition with DoRA for the SFT-format-steering use case — DoRA wins there. GaLore is what you reach for when the decision says 'full FT' and the GPU says 'no.'"	c3::ft09::application
"What is QDoRA and why is it positioned as the leading quality/cost point in 2026 PEFT?"	"QDoRA = DoRA on top of a 4-bit quantized base (the DoRA analogue of QLoRA). It delivers full-FT-adjacent quality from a setup that fits a 7B-class model on a single consumer card. The magnitude/direction decomposition adds negligible memory on top of QLoRA, and the quality lift over QLoRA is the same DoRA lift over LoRA. On memory-constrained consumer hardware, QDoRA is the leading quality/cost point."	c3::ft09::recall
"Place VeRA, AdaLoRA, and MiSS on the standard-vs-niche axis, with a one-line verdict each."	"VeRA (shared frozen random matrices + per-layer scaling vectors; ~10× fewer params than LoRA) — NICHE/RESEARCH; useful only when adapter STORAGE (not training memory) is the bottleneck, quality not consistently there. AdaLOra (adaptive rank allocation during training) — LARGELY SUPERSEDED by DoRA; the problem it solved is better solved by DoRA's decomposition. MiSS (claims SOTA) — EXPERIMENTAL; not widely reproduced or adopted. Treat as research-awareness, not a production default."	c3::ft09::analysis
"What is VeRA's core mechanism, and what is its real niche?"	"VeRA uses a single pair of FROZEN, SHARED random matrices across ALL layers (identical for every layer) and learns only two small per-layer scaling vectors (length d and length r). This gives ~10× fewer trainable parameters than LoRA. Its real niche is when ADAPTER STORAGE is the bottleneck — e.g., thousands of adapters hot-swappable at inference — not when training memory is the bottleneck. Quality is not consistently competitive with LoRA/DoRA on standard tasks."	c3::ft09::recall
"Name the three FT09 anti-patterns."	"(1) Chasing novelty without measuring — switching the production stack to every new paper-claiming-a-win without a controlled A/B on your own held-out eval. (2) Ignoring the rank-stability issue at high rank — turning the rank up to 128 with vanilla LoRA (α/r scaling) and blaming the data when it doesn't help; the fix is use_rslora=True. (3) Full fine-tuning where DoRA would suffice — full FT costs 10–20× the memory/compute of DoRA, produces a non-swappable artifact, and on most steering tasks is within a couple points of DoRA."	c3::ft09::analysis
"Why is 'chasing novelty without measuring' the cardinal PEFT anti-pattern?"	"The PEFT field publishes a new method monthly; each claims a win on some benchmark. A single-paper claim is not the same as a default. DoRA earned default status through consistent gains across many tasks and models, REPRODUCED BY INDEPENDENT TEAMS. Switching your production stack to every new method is how you spend a quarter and ship nothing. Only adopt after measuring on YOUR held-out eval, YOUR data, with a controlled A/B. The lab is that discipline miniaturized."	c3::ft09::analysis
"A team reports 'we set rank=128 and it didn't help, our data must be bad.' Diagnose."	"This is the rank-stability anti-pattern. At rank 128 with α=16, vanilla LoRA scales the update by α/r = 16/128 = 0.125 — the adapter is numerically present but functionally inert, attenuated almost to zero. The data is probably fine. The fix is one flag: use_rslora=True, which switches the scaling to α/√r = 16/√128 ≈ 1.41, restoring a meaningful effective update. If you are above r=32 and not using rsLoRA, you are almost certainly wasting your rank budget."	c3::ft09::application
"Why does the FT09 lab hold base model, data, epochs, batch size, and LR constant across the LoRA and DoRA runs?"	"To isolate the effect of the PEFT method itself. If you also change the rank, the LR, or the dataset, you can no longer attribute the quality delta to DoRA — it is confounded with the other changes. The discipline of a PEFT A/B: change exactly one thing (the adapter method), hold everything else fixed. This is the only valid basis for adopting a new method; benchmark-shopping is the anti-pattern."	c3::ft09::application
"What is the 2026 practical PEFT decision shape, from default to escalation?"	"Start with DoRA + rsLoRA (the default). If the GPU is too small (< 12 GB), drop to QDoRA (DoRA on a 4-bit quantized base). If DoRA is measured-insufficient on a held-out eval (not vibes), the next step up is GaLore full-parameter FT — NOT vanilla LoRA at higher rank. PiSSA initialization is an optional add-on for faster convergence on NLU-heavy tasks. Vanilla LoRA and vanilla full FT are the methods you no longer START with in 2026."	c3::ft09::analysis