Publications

2025

1. arXiv

   Learning to Specialize: Joint Gating-Expert Training for Adaptive MoEs in Decentralized Settings

   2025

   Abs PDF

   Mixture-of-Experts (MoEs) achieve scalability by dynamically activating subsets of their components. Yet, understanding how expertise emerges through joint training of gating mechanisms and experts remains incomplete, especially in scenarios without clear task partitions. Motivated by inference costs and data heterogeneity, we study how joint training of gating functions and experts can dynamically allocate domain-specific expertise across multiple underlying data distributions. As an outcome of our framework, we develop an instance tailored specifically to decentralized training scenarios, introducing \textitDynamically Decentralized Orchestration of MoEs or \textttDDOME. \textttDDOME leverages heterogeneity emerging from distributional shifts across decentralized data sources to specialize experts dynamically. By integrating a pretrained common expert to inform a gating function, \textttDDOME achieves personalized expert subset selection on-the-fly, facilitating just-in-time personalization. We empirically validate \textttDDOME within a Federated Learning (FL) context: \textttDDOME attains from 4% up to an 24% accuracy improvement over state-of-the-art FL baselines in image and text classification tasks, while maintaining competitive zero-shot generalization capabilities. Furthermore, we provide theoretical insights confirming that the joint gating-experts training is critical for achieving meaningful expert specialization.

2024

1. arXiv

   On the Robustness of Decision-Focused Learning

   Yehya Farhat

   2024

   Abs PDF Code

   Decision-Focused Learning (DFL) is an emerging learning paradigm that tackles the task of training a machine learning (ML) model to predict missing parameters of an incomplete optimization problem in which the missing problem parameters are completed by a prediction. DFL trains an ML model in an end-to-end system by integrating the prediction and optimization tasks, providing better alignment of the training and testing objectives. DFL has shown a lot of promise and holds the capacity to revolutionize decision-making in many real-world applications. However, very little is known about the performance of these models under adversarial attacks. We adopt ten unique DFL methods and benchmark their performance under two distinctly focused attacks adapted towards Predict-then-Optimize-based problems. Our results lead us to hypothesize that the models’ robustness is highly correlated with the model’s ability to find predictions that lead to optimal solutions without deviating from the ground-truth label. Furthermore, we provide insight into how to target the models that do violate this condition and show how these models respond differently depending on the achieved optimality at the end of their training cycles

2023

1. M.Sc. Thesis

   End-to-End Decision-Focused Learning Using Learned Solvers

   Yehya Farhat

   Syracuse University, 2023

   Abs PDF Code Slides

   Achieving fusion of deep learning with combinatorial algorithms promises transformative changes to AI. Creating an impact in a real-world setting requires AI techniques to span a pipeline from data, to predictive models, to decisions. Aligning these components together requires careful consideration, as having these components trained separately does not account for the end goal of the model. This work surveys general frameworks for End-toEnd optimization learning; we focus on the integration of optimization methods within machine learning architectures. We discuss challenges and limitations associated with these methods and propose a novel approach to overcome the bottlenecks that arise.