Benjamin V. Rozonoyer

2026

1. 

   ICML SPIGM

   Learned Relay Representations for Forward-Thinking Discrete Diffusion Models

   Benjamin Rozonoyer, Jacopo Minniti, Dhruvesh Patel, and 4 more authors

   arXiv preprint arXiv:2605.22967, 2026

   Accepted at the SPIGM and FoGen Workshops @ ICML 2026

   Abs Bib PDF Code Poster

   When Masked Diffusion Models (MDMs) generate sequences through iterative refinement, the rich internal computation over masked positions is discarded—forcing every subsequent refinement step to recompute the valuable internal information stored as model representations. To avoid a hard reset between denoising rounds, we propose Learned Relay Representations (Relay), a method that allows MDMs to be "forward-thinking" when denoising—explicitly learning how to propagate latent information for the benefit of future denoising steps. Relay introduces a differentiable per-token channel that passes information between forward passes and is trained via truncated backpropagation through time (BPTT). We show that this framework can be scaled to state-of-the-art Diffusion Language Models (DLMs), and is seamlessly compatible with techniques like block diffusion and KV caching. We first provide a thorough justification of the design choices in Relay on a challenging Sudoku-based planning task. We then scale Relay to Fast-dLLM v2, a state-of-the-art DLM, outperforming standard supervised finetuning on coding tasks while reducing the inference latency by up to 32%. Our empirical results demonstrate that state-of-the-art DLMs can be explicitly trained to relay latent information forward across decoding steps, advancing the performance-latency Pareto frontier.

   @article{rozonoyer2026relay,
     note = {Accepted at the SPIGM and FoGen Workshops @ ICML 2026},
     title = {Learned Relay Representations for Forward-Thinking Discrete Diffusion Models},
     author = {Rozonoyer, Benjamin and Minniti, Jacopo and Patel, Dhruvesh and Band, Neil and Bose, Avishek Joey and Rudner, Tim G. J. and McCallum, Andrew},
     journal = {arXiv preprint arXiv:2605.22967},
     doi = {10.48550/arXiv.2605.22967},
     eprint = {2605.22967},
     year = {2026}
   }

2. 

   ICML FoGen

   Autoregressive Ranking: Bridging the Gap Between Dual and Cross Encoders

   Benjamin Rozonoyer, Chong You, Michael Boratko, and 5 more authors

   arXiv preprint arXiv:2601.05588, 2026

   Accepted at the FoGen Workshop @ ICML 2026

   Abs Bib PDF Poster Slides

   Dual and cross encoders have long been mainstays of information retrieval (IR), but are being challenged by the emergent capabilities of LLMs. An LLM-based approach we term pointwise generative ranking - generating tokens the length of a single docID as opposed to a list in order to enable ranking via beam search - combines efficiency and expressivity benefits while leveraging the in-context capabilities of Causal Transformers. Although there is ample evidence to suggest that pretrained LLMs are well-suited for ranking, we find that the vast majority of LLM-based approaches rely on next-token prediction, a loss function which is fundamentally rank-agnostic (and especially so with pointwise supervision). In this paper, we first prove that the expressivity of pointwise generative ranking with multi-token docIDs is superior to that of dual encoders. We then propose SToICaL - a Simple Token-Item Calibrated Loss - which can incorporate rank-aware supervision at both the item and token levels within the pointwise setup. We run a suite of experiments on ranking tasks derived from WordNet (Fellbaum, 1998) and ESCI (Reddy et al., arXiv:2206.06588). Two variants of SToICaL successfully suppress the probability of invalid docID generations and improve on common ranking metrics beyond top-1 retrieval.

   @article{rozonoyer2026autoregressive,
     note = {Accepted at the FoGen Workshop @ ICML 2026},
     author = {Rozonoyer, Benjamin and You, Chong and Boratko, Michael and Jain, Himanshu and Gupta, Nilesh and Bhojanapalli, Srinadh and McCallum, Andrew and Yu, Felix},
     journal = {arXiv preprint arXiv:2601.05588},
     doi = {10.48550/arXiv.2601.05588},
     eprint = {2601.05588},
     year = {2026},
   }

2024

1. 

   NeurIPS

   Learning Representations for Hierarchies with Minimal Support

   Benjamin Rozonoyer, Michael Boratko, Dhruvesh Patel, and 4 more authors

   In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024

   Abs Bib PDF Supp Poster

   When training node embedding models to represent large directed graphs (digraphs), it is impossible to observe all entries of the adjacency matrix during training. As a consequence most methods employ sampling. For very large digraphs, however, this means many (most) entries may be unobserved during training. In general, observing every entry would be necessary to uniquely identify a graph, however if we know the graph has a certain property some entries can be omitted - for example, only half the entries would be required for a symmetric graph. In this work, we develop a novel framework to identify a subset of entries required to uniquely distinguish a graph among all transitively-closed DAGs. We give an explicit algorithm to compute the provably minimal set of entries, and demonstrate empirically that one can train node embedding models with greater efficiency and performance, provided the energy function has an appropriate inductive bias. We achieve robust performance on synthetic hierarchies and a larger real-world taxonomy, observing improved convergence rates in a resource-constrained setting while reducing the set of training examples by as much as 99%.

   @inproceedings{rozonoyerlearning,
     title = {Learning Representations for Hierarchies with Minimal Support},
     author = {Rozonoyer, Benjamin and Boratko, Michael and Patel, Dhruvesh and Zhao, Wenlong and Dasgupta, Shib Sankar and Le, Hung and McCallum, Andrew},
     booktitle = {The Thirty-eighth Annual Conference on Neural Information Processing Systems},
     year = {2024},
   }