arXiv · Oct 16, 2026
Prefix-Tuning for Arbitrary Output Sequences on Pretrained Transformers
By Peter Cho-Ho Lam, Ziyi Wang, Zirui Zhou
Prefix-tuning is a parameter-efficient fine-tuning method for large language models (LLMs) that steers model outputs by prepending task-specific prefix representations to the input sequence. Despite its practical success, the theoretical understanding of the expressiveness of prefixes remains incomplete. In particular, the following question remains open: Given a pretrained autoregressive Transformer, for any input-output sequence pair (x,y), does there always exist a prefix S such that prepending S to x induces the model to generate the target output y? In this work, we prove that under the assumption that y is in the range of the pretrained Transformer, such a prefix S exists and can be explicitly computed by solving multiple linear systems. We further show that this assumption is necessary, in the sense that such a prefix may not exist when the assumption fails, thereby providing a complete answer to the above question. Our analysis is constructive and the obtained theoretical results hold under fairly general assumptions on model architecture. Extensive experiments on GPT-2 and Qwen3-4B demonstrate that our algorithm produces prefixes with strong control over model outputs.