Publications

2025

1. arXiv

   TwIST: Rigging the Lottery in Transformers with Independent Subnetwork Training

   Michael Menezes, Barbara Su, Xinze Feng, Yehya Farhat, Hamza Shili, and Anastasios Kyrillidis

   2025

   Abs PDF

   We introduce TwIST, a distributed training framework for efficient large language model (LLM) sparsification. TwIST trains multiple subnetworks in parallel, periodically aggregates their parameters, and resamples new subnetworks during training. This process identifies high-quality subnetworks ("golden tickets") without requiring post-training procedures such as calibration or Hessian-based recovery. As a result, TwIST enables zero-cost pruning at deployment time while achieving perplexity competitive with state-of-the-art post-training sparsification methods. The benefits are most pronounced under aggressive sparsity (e.g., 50%+), where TwIST significantly outperforms baseline methods; for example, reaching 23.14 PPL compared to 31.64 for the closest prior approach. Unlike unstructured pruning, TwIST produces structured, dense matrices that offer practical inference speedups and memory reductions on commodity hardware (e.g., CPUs) that do not support efficient sparse computation. TwIST provides an efficient training-time path to deployable sparse LLMs without additional fine-tuning or recovery overhead.

2. NeurIPS

   Learning to Specialize: Joint Gating-Expert Training for Adaptive MoEs in Decentralized Settings

   Yehya Farhat^*, Hamza ElMokhtar^* Shili, Fangshuo Liao^*, Chen Dun^*, Mirian Hipolito Garcia^*, Guoqing Zheng, Ahmed Hassan Awadallah, and 3 more authors

   2025

   Abs PDF

   Mixture-of-Experts (MoEs) achieve scalability by dynamically activating subsets of their components. Yet, understanding how expertise emerges through joint training of gating mechanisms and experts remains incomplete, especially in scenarios without clear task partitions. Motivated by inference costs and data heterogeneity, we study how joint training of gating functions and experts can dynamically allocate domain-specific expertise across multiple underlying data distributions. As an outcome of our framework, we develop an instance tailored specifically to decentralized training scenarios, introducing \textitDynamically Decentralized Orchestration of MoEs or \textttDDOME. \textttDDOME leverages heterogeneity emerging from distributional shifts across decentralized data sources to specialize experts dynamically. By integrating a pretrained common expert to inform a gating function, \textttDDOME achieves personalized expert subset selection on-the-fly, facilitating just-in-time personalization. We empirically validate \textttDDOME within a Federated Learning (FL) context: \textttDDOME attains from 4% up to an 24% accuracy improvement over state-of-the-art FL baselines in image and text classification tasks, while maintaining competitive zero-shot generalization capabilities. Furthermore, we provide theoretical insights confirming that the joint gating-experts training is critical for achieving meaningful expert specialization.

2024

1. arXiv

   On the Robustness of Decision-Focused Learning

   Yehya Farhat

   2024

   Abs PDF Code

   Decision-Focused Learning (DFL) is an emerging learning paradigm that tackles the task of training a machine learning (ML) model to predict missing parameters of an incomplete optimization problem in which the missing problem parameters are completed by a prediction. DFL trains an ML model in an end-to-end system by integrating the prediction and optimization tasks, providing better alignment of the training and testing objectives. DFL has shown a lot of promise and holds the capacity to revolutionize decision-making in many real-world applications. However, very little is known about the performance of these models under adversarial attacks. We adopt ten unique DFL methods and benchmark their performance under two distinctly focused attacks adapted towards Predict-then-Optimize-based problems. Our results lead us to hypothesize that the models’ robustness is highly correlated with the model’s ability to find predictions that lead to optimal solutions without deviating from the ground-truth label. Furthermore, we provide insight into how to target the models that do violate this condition and show how these models respond differently depending on the achieved optimality at the end of their training cycles

2023

1. M.Sc. Thesis

   End-to-End Decision-Focused Learning Using Learned Solvers

   Yehya Farhat

   Syracuse University, 2023

   Abs PDF Code Slides

   Achieving fusion of deep learning with combinatorial algorithms promises transformative changes to AI. Creating an impact in a real-world setting requires AI techniques to span a pipeline from data, to predictive models, to decisions. Aligning these components together requires careful consideration, as having these components trained separately does not account for the end goal of the model. This work surveys general frameworks for End-toEnd optimization learning; we focus on the integration of optimization methods within machine learning architectures. We discuss challenges and limitations associated with these methods and propose a novel approach to overcome the bottlenecks that arise.