BEGIN:VCALENDAR
VERSION:2.0
PRODID:-//CIG Group//Talk Calendar//EN
CALSCALE:GREGORIAN
METHOD:PUBLISH
BEGIN:VEVENT
UID:talk-266@theciggroup.net
DTSTAMP:20260928T222007Z
DTSTART:20261001T120000Z
DTEND:20261001T130000Z
SUMMARY:How Transformers Learn to Plan via Multi-Token Prediction
DESCRIPTION:While next-token prediction (NTP) has been the standard objective for training language models\, it often struggles to capture global structure in reasoning tasks. Multi-token prediction (MTP) has recently emerged as a promising alternative\, yet its underlying mechanisms remain poorly understood. In this paper\, we study how MTP facilitates reasoning\, with a focus on planning. Empirically\, we show that MTP consistently outperforms NTP on both synthetic graph path-finding tasks and more realistic reasoning benchmarks\, such as Countdown and boolean satisfiability problems. Theoretically\, we analyze a simplified two-layer Transformer on a star graph task. We prove that MTP induces a two-stage reverse reasoning process: the model first attends to the end node and then reconstructs the path by tracing intermediate nodes backward. This behavior arises from a gradient decoupling property of MTP\, which provides a cleaner training signal compared to NTP. Ultimately\, our results highlight how multi-token objectives inherently bias optimization toward robust and interpretable reasoning circuits.\n\nSpeaker: Prof Wei Huang\n\nJoin: https://us06web.zoom.us/j/89623178484?pwd=mq3uVw0wbavzb9t7g1pE4VpPlRYm7l.1
LOCATION:https://calendar.app.google/PjMbx5Het8obVJur9
STATUS:CONFIRMED
END:VEVENT
END:VCALENDAR