HISTORY近 30 天历史柱高表示当天去重热搜数量
09/08—10/07 有历史数据
- 01A Library for Learning Neural OperatorsWe present NeuralOperator, an open-source Python library for operator learning. Neural operators generalize neural networks to maps between function spaces instead of finite-dimensional Euclidean spaces. They can be trained and inferenced on input and output functions given at various discretizations, satisfying a discretization convergence properties. Part of the official PyTorch Ecosystem, NeuralOperator provides all the tools for training and deploying neural operator models, as well as develJean Kossaifi, Nikola Kovachki, Zongyi Li, David Pitt, Miguel Liu-Schiaffini, Robert J. George, Boris Bonev, Kamyar Azizzadenesheli, Julius Berner, Valentin Duruisseaux, Anima Anandkumar
- 02MarkDiffusion: An Open-Source Toolkit for Generative Watermarking of Latent Diffusion ModelsWe introduce MarkDiffusion, an open-source Python toolkit for generative watermarking of latent diffusion models. It comprises three key components: a unified implementation framework for streamlined watermarking algorithm integration and user-friendly interfaces; a mechanism visualization suite that intuitively presents embedded and extracted watermark patterns to aid public understanding; and a comprehensive evaluation module offering standard implementations of 24 tools for assessing detectabLeyi Pan, Sheng Guan, Zheyu Fu, Luyang Si, Huan Wang, Zian Wang, Hanqian Li, Xuming Hu, Irwin King, Philip S. Yu, Aiwei Liu, Lijie Wen
- 03OptunaHub: A Platform for Black-Box OptimizationBlack-box optimization (BBO) underpins advances in domains such as AutoML and Materials Informatics, yet implementations of algorithms and benchmarks remain fragmented across research communities. We introduce OptunaHub (https://hub.optuna.org/), a community-oriented, decentralized platform for distributing BBO components under a unified Optuna-compatible interface. OptunaHub enables independent publication, discovery, and reuse of optimization algorithms and benchmark problems through a lightweYoshihiko Ozaki, Shuhei Watanabe, Toshihiko Yanase
- 04Unveiling the Statistical Foundations of Chain-of-Thought Prompting MethodsChain-of-Thought (CoT) prompting and its variants have gained significant attention as effective methods for solving multi-step reasoning tasks with pretrained large language models (LLMs). However, their theoretical underpinnings remain insufficiently explored. We analyze CoT prompting from a statistical perspective, offering insights into why “pretrained LLMs + CoT prompting” performs well. Additionally, we examine the role of the transformer architecture and the inclusion of intermediate reasXinyang Hu, Fengzhuo Zhang, Siyu Chen, Zhuoran Yang
- 05Prob-GParareal: A Probabilistic Numerical Parallel-in-Time Solver for Differential EquationsWe introduce Prob-GParareal, a probabilistic extension of the GParareal algorithm designed to provide uncertainty quantification for the Parallel-in-Time (PinT) solution of (ordinary and partial) differential equations (ODEs, PDEs). The method employs Gaussian processes (GPs) to model the Parareal correction function, in line with GParareal, further enabling the propagation of numerical uncertainty across time and yielding probabilistic forecasts of the system's evolution. Furthermore, Prob-GParGuglielmo Gattiglio, Lyudmila Grigoryeva, Massimiliano Tamborrino
- 06From learnable objects to learnable random objectsWe consider the relationship between learnability of a "base class" of functions on a set $X$, and learnability of a class of statistical functions derived from the base class. For example, we refine results showing that learnability of a family $h_p: p \in \Theta$ of functions implies learnability of the family of functions $h_\mu(p) = \mathbb{E}_\mu[h_p]$, where $\mathbb{E}_\mu$ is the expectation with respect to $\mu$, and $\mu$ ranges over probability distributions on $X$. We will look at boAaron Anderson, Michael Benedikt
- 07A Theoretical Framework for Masked Pretraining (MPT)Recently, Masked Pretraining (MPT) based on reconstruction pretraining tasks has risen to a promising self-supervised learning paradigm across various domains and achieves remarkable performance in multiple downstream tasks. However, the theoretical understanding of the working mechanism behind MPT is still limited. In this paper, we introduce a new theoretical framework to analyze MPT and understand the crucial role of masking in extracting meaningful representations. We establish theoretical cQi Zhang, Runyu Zhou, Yifei Wang, Yisen Wang
- 08Robustness Against Weak or Invalid Instruments: Exploring Nonlinear Treatment Models with Machine LearningWe discuss causal inference for observational studies with possibly invalid instrumental variables. We propose a novel methodology called two-stage curvature identification (\texttt{TSCI}) by exploring the nonlinear treatment model with machine learning. The first-stage machine learning enables improving the instrumental variable's strength and adjusting for different forms of violating the instrumental variable assumptions. The success of \texttt{TSCI} requires the instrumental variable's effecZijian Guo, Mengchu Zheng, Peter Bühlmann
- 09Optimising Utility Functions in Multi-Objective Markov Decision ProcessesMulti-Objective Markov Decision Processes (MOMDPs) are among the most prevalent formal frameworks for addressing sequential decision-making problems involving multiple, potentially conflicting objectives. In most MOMDP approaches, a utility function is employed to aggregate these objectives into a single scalar criterion that encodes user preferences. Despite its widespread adoption, the theoretical foundations of MOMDPs remain incomplete in two main respects: first, there is no general characteManel Rodriguez-Soto
- 10Bayesian Transfer Learning for Artificially Intelligent Geospatial Systems: A Predictive Stacking ApproachBuilding artificially intelligent geospatial systems requires rapid delivery of spatial data analysis on massive scales with minimal human intervention. Depending on their intended use, learning about underlying spatial processes can also involve model assessment and uncertainty quantification. We devise transfer learning frameworks for deployment in artificially intelligent systems, where a massive data set is split into smaller data sets that stream into the analytical framework to propagate lLuca Presicce, Sudipto Banerjee
- 11Symmetric Rank-k MethodsThis paper proposes a novel class of block quasi-Newton methods for convex optimization which we call symmetric rank-$k$ (SR-$k$) methods. Each iteration of SR-$k$ incorporates the curvature information with $k$ Hessian-vector products achieved from the greedy or random strategy. We prove that SR-$k$ methods have the local superlinear convergence rate of $\mathcal{O}\big((1-k/d)^{t(t-1)/2}\big)$ for minimizing smooth and strongly convex functions, where $d$ is the problem dimension and $t$ is thChengchang Liu, Cheng chen, Luo Luo
- 12From Zipf's Law to Neural Scaling through Heaps' Law and Hilberg's HypothesisWe inspect the deductive connection between the neural scaling law and Zipf's law--two statements discussed in machine learning and quantitative linguistics. The neural scaling law describes how the cross entropy rate of a foundation model--such as a large language model--changes with respect to the amount of training tokens, parameters, and compute. By contrast, Zipf's law posits that the distribution of tokens exhibits a power law tail. Whereas similar claims have been made in more specific seŁukasz Dębowski
- 13Efficient Inference under Label Shift in Unsupervised Domain AdaptationIn many real-world applications, researchers aim to deploy models trained in a source domain to a target domain, where obtaining labeled data is often expensive, time-consuming, or even infeasible. While most existing literature assumes that the source and target data follow the same joint distribution, distribution shifts are common in practice. This paper considers a particular type of distribution shift, label shift, and develops an efficient inference procedure for general parameters charactSeong-ho Lee, Yanyuan Ma, Jiwei Zhao
- 14Adaptive Algorithms for Infinitely Many-Armed Bandits: A Unified FrameworkWe consider a bandit problem where the budget is smaller than the number of arms, which may be infinite. In this regime, the usual objective in the literature is to minimize simple regret. To analyze broad classes of distributions with potentially unbounded support, where simple regret may not be well-defined, we take a slightly different approach and seek to maximize the expected simple reward of the recommended arm, providing anytime guarantees. To that end, we introduce a distribution-free alEmmanuel Pilliat
- 15Gradient Estimation for Mixture Variational InferenceMixture distributions are expressive variational families for black-box VI, but their discrete component choices complicate gradient estimation. We systematize reparameterization-based estimators for mixtures in a common notation, giving self-contained derivations and extending several to new settings. In particular, we provide an elementary derivation of a single-sample post-stratified estimator---previously derived via transport equations---and prove a variance reduction relative to simple ranJavier Burroni, Daniel Sheldon
- 16torchsom: The Reference PyTorch Library for Self-Organizing MapsThis paper introduces torchsom, an open-source Python library that provides a reference implementation of the Self-Organizing Map (SOM) in PyTorch. This package offers three main features: (i) dimensionality reduction, (ii) clustering, and (iii) friendly data visualization. It relies on a PyTorch backend, enabling (i) fast and efficient training of SOMs through GPU acceleration, and (ii) easy and scalable integration with the PyTorch ecosystem. torchsom also follows the scikit-learn API for easeLouis Berthier, Ahmed Shokry, Maxime Moreaud, Guillaume Ramelet, Eric Moulines
- 17Pointwise Confidence Estimation in the Non-linear $\ell^2$-regularized Least SquaresWe consider a high-probability non-asymptotic confidence estimation in the $\ell^2$-regularized non-linear least-squares setting with fixed design. In particular, we study confidence estimation for local minimizers of the regularized training loss. We show a pointwise confidence bound, meaning that it holds for the prediction on any given fixed test input $x$. Importantly, the proposed confidence bound scales with similarity of the test input to the training data in the implicit feature space ofIlja Kuzborskij, Yasin Abbasi Yadkori
- 18Safe Learning Under Irreversible Dynamics via Asking for HelpMost learning algorithms with formal regret guarantees essentially rely on trying all possible behaviors, which is problematic when some errors cannot be recovered from. Instead, we allow the learning agent to ask for help from a mentor and to transfer knowledge between similar states. We show that this combination enables the agent to learn both safely and effectively. Under standard online learning assumptions, we provide an algorithm whose regret and number of mentor queries are both sublineaBenjamin Plaut, Juan Liévano-Karim, Hanlin Zhu, Stuart Russell
- 19AgentPEN: A Prediction-Explanation Network for Sequential Stock Movement via LLMs and Recurrent GenerationThe importance of explainability in stock prediction is increasingly recognized, especially for audit and regulatory purposes. Meanwhile, financial news corpora are often key drivers behind stock price fluctuations. However, the raw news data obtained is usually highly noisy, has a highly variable scope of influence in time and space, and is not precisely synchronized with stock price data. In this paper, we propose a prediction-explanation network called AgentPEN, which can provide clear explanShuqi Li, Mengyao Guo, Yunzhong Zheng, Siqi Li, Xin Gao, Rui Yan
- 20Feedback-Enhanced Online Multiple Testing with Applications to Conformal SelectionThis work studies online multiple testing with feedback, where decisions are made sequentially, and the true state of the hypothesis is revealed after decisions are made, either instantly or with a delay, and under either full or bandit feedback. We propose Generalized alpha-investing with feedback (GAIF) along with its adaptive variants, a feedback-enhanced framework that dynamically adjusts thresholds using revealed outcomes, ensuring finite-sample false discovery rate (FDR)/marginal FDR (mFDRLin Lu, Yuyang Huo, Haojie Ren, Zhaojun Wang, Changliang Zou
- 21Ehrenfeucht-Haussler Rank and Chain of ThoughtThe notion of rank of a Boolean function has been a cornerstone in PAC learning, enabling quasipolynomial-time learning algorithms for polynomial-size decision trees. We present a novel characterization of rank, grounded in the well-known Transformer architecture. We show that the rank of a function $f$ corresponds to the minimum number of Chain of Thought (CoT) iterations required by a single-layer Transformer with hard attention to compute $f$. Based on this characterization, we establish tighPablo Barceló, Alexander Kozachinskiy, Tomasz Steifer
- 22scikit-activeml: A Comprehensive and User-Friendly Active Learning Libraryscikit-activeml is a user-friendly open-source Python library for active learning on top of scikit-learn. Included are implementations of a large collection of query strategies, models, and visualization tools in pool- and stream-based active learning for classification or regression tasks with single or multiple annotators. The flexible design of the active learning cycle enables individual adaptations to a variety of learning scenarios. Our source code with comprehensive documentation is availMarek Herde, Minh Tuan Pham, Daniel Kottke, Alexander Benz, Lukas Lührs, Pascal Mergard, Christoph Sandrock, Jiaying Cheng, Atal Roghman, Mehmet Müjde, Lukas Rauch, Bernhard Sick
- 23Locally Private Estimation with Public FeaturesWe initiate the study of locally differentially private (LDP) learning with public features. We define semi-feature LDP, where some features are publicly available while the remaining ones, along with the label, require protection under local differential privacy. Under semi-feature LDP, we consider three fundamental estimation problems: non-parametric density estimation, classification, and regression. Given the smoothness assumption, we show that the minimax convergence rate is significantly iYuheng Ma, Hanfang Yang, Ke Jia
- 24Dimension Reduction for Derivative-Informed Operator Learning: An Analysis of Approximation ErrorsWe study the derivative-informed learning of nonlinear operators between infinite-dimensional Hilbert spaces. Such operators can arise as solution maps of partial differential equations, and their approximation by accurate surrogate models can accelerate simulation-intensive tasks of scientific and engineering interest, including inference, control, and uncertainty quantification. Since efficiently performing such tasks often requires an accurate representation of the operator's derivatives, weDingcheng Luo, Thomas O'Leary-Roseberry, Peng Chen, Omar Ghattas
- 25Domain Adaptation Targeting Heterogeneous and Imbalanced SubgroupsDomain adaptation enables generalizable and efficient data-driven research. However, existing work has largely focused on domain adaptation for some intrinsically homogeneous target cohort, overlooking inherent heterogeneity within the target, which can exacerbate biases and unfairness in the presence of subgroups with imbalanced sample sizes. We develop a novel domain adaptation framework that addresses a more complicated target dataset that consists of heterogeneous and data-sparse subgroups aDoudou Zhou, Mengyan Li, Yun Wang, Tianxi Cai, Molei Liu
- 26On the Effectiveness of the z-Transform Method in Quadratic OptimizationThe z-transform of a sequence is a classical tool used in signal processing, control theory, computer science, and electrical engineering. It allows one to study sequences from their generating functions, with many operations that can be equivalently defined on the original sequence and its z-transform. In particular, the z-transform method focuses on asymptotic behaviors and allows the use of Taylor expansions. We present a sequence of results of increasing significance and difficulty for lineaFrancis Bach
- 27Breaking the Curse of Dimensionality: Diffusion Models Efficiently Learn Low-Dimensional DistributionsDespite their empirical success across a wide range of generative tasks, the fundamental principles underlying the ability of diffusion models to learn data distributions are poorly understood. In this work, we develop a new mathematical framework that explains how diffusion models can effectively learn low-dimensional distributions from a finite number of training samples without suffering from the curse of dimensionality. Specifically, motivated by the intrinsic low-dimensional structure of imPeng Wang, Huijie Zhang, Zekai Zhang, Siyi Chen, Yi Ma, Qing Qu
- 28Consistency of Augmentation Graph and Network Approximability in Contrastive LearningContrastive learning leverages data augmentation to develop feature representation without relying on large labeled data sets. However, despite its empirical success, the theoretical foundations of contrastive learning remain incomplete, with many essential guarantees left unaddressed, particularly the realizability assumption concerning neural approximability of an optimal spectral contrastive loss solution. In this work, we overcome these limitations by analyzing pointwise and spectral consistChenghui Li, A. Martina Neuman
- 29Solving Nonlinear PDEs with Sparse Radial Basis Function NetworksWe propose a novel framework for solving nonlinear PDEs using sparse radial basis function (RBF) networks. Sparsity-promoting regularization is employed to prevent over-parameterization and reduce redundant features. This work is motivated by longstanding challenges in traditional RBF collocation methods, along with the limitations of physics-informed neural networks (PINNs) and Gaussian process (GP) approaches, aiming to blend their respective strengths in a unified framework. The theoretical fZihan Shao, Konstantin Pieper, Xiaochuan Tian
- 30Viscosity Convergence Analysis for Deep Q-NetworksDeep Q-Networks (DQNs) and related residual neural architectures are increasingly used for continuous-time reinforcement learning (CTRL), where optimal value functions solve fully nonlinear second-order Hamilton--Jacobi--Bellman (HJB) equations and may be non-smooth. In this regime, convergence should be analyzed in the viscosity-solution framework. We study a spatially-coupled monotone ResNet architecture whose one-step operator is a non-negative local aggregation, designed to satisfy monotonicQian Qi
- 31Clustering and Pruning in Causal Data FusionData fusion, the process of combining observational and experimental data, can enable the identification of causal effects that would otherwise remain non-identifiable. Although identification algorithms have been developed for specific scenarios, do-calculus remains the only general-purpose tool for causal data fusion, particularly when variables are present in some data sources but not others. However, approaches based on do-calculus may encounter computational challenges as the number of variOtto Tabell, Santtu Tikka, Juha Karvanen
- 32Leakage and Interpretability in Concept-Based ModelsConcept-based Models aim to improve interpretability by predicting high-level intermediate concepts, representing a promising approach for deployment in high-risk scenarios. However, they are known to suffer from information leakage, whereby models exploit unintended information encoded within the learned concepts. We introduce an information-theoretic framework to rigorously characterise and quantify leakage, and define two complementary measures: the concepts-task leakage (CTL) and interconcepEnrico Parisini, Tapabrata Chakraborti, Chris Harbron, Ben D. MacArthur, Christopher R.S. Banerji
- 33Resilience Beyond Stationary Client Unavailability: Unlocking Efficient and Unbiased Federated LearningDue to resource constraints or external and internal uncertainties, clients in real-world federated learning systems are often intermittently available edge devices. In highly dynamic environments, the parameter server lacks prior real-time knowledge of clients' availability, making it challenging to adapt traditional federated learning algorithms to be resilient to uncertainties in client availability. If not carefully addressed, complex client availability can introduce significant bias, potenMing Xiang, Stratis Ioannidis, Edmund Yeh, Carlee Joe-Wong, Lili Su
- 34Incorporating external data for analyzing randomized clinical trials: A transfer learning approachRandomized clinical trials are the gold standard for analyzing treatment effects. However, increasing costs and ethical concerns may limit trial recruitment, resulting in insufficient sample sizes and potentially invalid inference. Incorporating external trial data with similar characteristics (treatments, diseases, biomarkers, etc.) into the analysis appears promising for addressing these issues. Transfer learning, which in our context utilizes external trials as the source domain and current tYujia Gu, Hanzhong Liu, Wei Ma
- 35A Provably Convergent Plug-and-Play Framework for Stochastic Bilevel OptimizationBilevel optimization has recently attracted significant attention in machine learning due to its wide range of applications and advanced hierarchical optimization capabilities. In this paper, we propose a plug-and-play framework, named PnPBO, for developing and analyzing stochastic bilevel optimization methods. This framework integrates both modern unbiased and biased stochastic estimators into the single-loop bilevel optimization framework introduced in Dagréou et al. (2022), with several improTianshu Chu, Dachuan Xu, Wei Yao, Chengming Yu, Jin Zhang
- 36Particle Filter for Bayesian Inference on Privatized DataDifferential privacy is a probabilistic framework that protects privacy while preserving data utility. To protect the privacy of the individuals in the data set, differential privacy requires adding a precise amount of noise to a statistic of interest; however, this noise addition alters the resulting sampling distribution, making statistical inference challenging. One of the main differential privacy goals in Bayesian analysis is to make statistical inference based on the private posterior distYu-Wei Chen, Pranav Sanghi, Jordan Awan
- 37Test-time regression: a unifying framework for designing sequence models with associative memorySequence models lie at the heart of modern deep learning. However, rapid advancements have produced a diversity of seemingly unrelated architectures, such as Transformers and recurrent alternatives. In this paper, we introduce a unifying framework to understand and derive these sequence models, inspired by the empirical importance of associative recall, the capability to retrieve contextually relevant tokens. We formalize associative recall as a two-step process, memorization and retrieval, castKe Alexander Wang, Jiaxin Shi, Emily B. Fox
- 38Nonparametric Spectral Density Estimation using Interactive Mechanisms under Local Differential PrivacyWe study the problem of estimating the spectral density of a centered stationary Gaussian time series under local differential privacy constraints. Specifically, we propose new interactive privacy mechanisms for three tasks: recovering a single covariance coefficient, recovering the spectral density at a fixed frequency, and global recovery. Our approach achieves faster rates through a two-stage process: we first apply the Laplace mechanism to the truncated value, and then use the resulting privCristina Butucea, Karolina Klockmann, Tatyana Krivobokova
- 39Sublinear Variational Optimization of Gaussian Mixture Models with Millions to Billions of ParametersGaussian Mixture Models (GMMs) range among the most frequently used models in machine learning. However, training large, general GMMs becomes computationally prohibitive for data sets that have many data points $N$ of high-dimensionality $D$. For GMMs with arbitrary covariances, we here derive a highly efficient variational approximation, which is then integrated with mixtures of factor analyzers (MFAs). For GMMs with $C$ components, our proposed algorithm substantially reduces runtime complexitSebastian Salwig, Till Kahlke, Florian Hirschberger, Dennis Forster, Jörg Lücke
- 40Statistical Inference for High-dimensional Partially Linear Models via Debiased Rank LassoThis paper aims to develop tuning-free and robust regularized methods for partially linear models based on partial residual methods. In order to preserve the near-oracle rate of rank Lasso, we construct a new estimator via a data-splitting procedure and name the resulting estimator as data-splitting (DS) rank Lasso, which is based on partial prediction residuals. Thus, the proposed estimation procedure is distinguished from the traditional partial residual based methods, which are based on the mSongshan Yang, Delin Zhao, Runze Li
- 41Identifiability of the Instrumental Variable Model with the Treatment and Outcome Missing Not at RandomUnder the instrumental variable model, we can identify the local average treatment effect, also known as the complier average causal effect (CACE). In practice, however, the treatment and outcome are often missing. When they are missing not at random (MNAR), the underlying data distribution cannot be recovered, so the CACE is generally not identifiable without further assumptions. We study the conditions under which the CACE remains identifiable when data are MNAR. Searching exhaustively over miShuozhi Zuo, Peng Ding, Fan Yang
- 42Deep Neural Expected Shortfall Regression with Tail-RobustnessExpected shortfall (ES), also known as conditional value-at-risk, is a widely recognized risk measure that complements value-at-risk by capturing tail-related risks more effectively. Compared with quantile regression, which has been extensively developed and applied across disciplines, ES regression remains in its early stage, partly because the traditional empirical risk minimization framework is not directly applicable. In this paper, we develop a nonparametric framework for expected shortfallMyeonghun Yu, Kean Ming Tan, Huixia Judy Wang, Wen-Xin Zhou
- 43Optimal Convergence Rates for Neural OperatorsWe introduce the neural tangent kernel (NTK) regime for two-layer neural operators and analyze their generalization properties. For early-stopped gradient descent (GD), we derive fast convergence rates that are known to be minimax optimal within the framework of non-parametric regression in reproducing kernel Hilbert spaces (RKHS). We provide bounds on the number of hidden neurons and the number of second-stage samples necessary for generalization. To justify our NTK regime, we additionally showMike Nguyen, Nicole Mücke
- 44Have ASkotch: A Neat Solution for Large-Scale Kernel Ridge RegressionKernel ridge regression (KRR) is a fundamental computational tool, appearing in problems that range from computational chemistry to health analytics, with a particular interest due to its starring role in Gaussian process regression. However, full KRR solvers are challenging to scale to large datasets: both direct (e.g., Cholesky decomposition) and iterative methods (e.g., PCG) incur prohibitive computational and storage costs. The standard approach to scale KRR to large datasets chooses a set oPratik Rathore, Zachary Frangella, Jiaming Yang, Michał Dereziński, Madeleine Udell
- 45Pairwise Comparisons without Stochastic Transitivity: Model, Theory and ApplicationsMost statistical models for pairwise comparisons, including the Bradley-Terry (BT) and Thurstone models and many extensions, make a relatively strong assumption of stochastic transitivity. This assumption imposes the existence of an unobserved global ranking among all the players/teams/items and monotone constraints on the comparison probabilities implied by the global ranking. However, the stochastic transitivity assumption does not hold in many real-world scenarios of pairwise comparisons, espSze Ming Lee, Yunxiao Chen
- 46Canonical Correlation Analysis as Reduced Rank Regression in High DimensionsCanonical correlation analysis is a widespread technique for discovering linear relationships between two sets of variables. In high dimensions, however, standard estimates of the canonical directions cease to be consistent without assuming further structure. In this setting, a possible solution consists in leveraging the presumed sparsity of the solution: only a subset of the covariates span the canonical directions. While the last decade has seen a proliferation of sparse canonical correlationClaire Donnat, Elena Tuzhilina
- 47Bayesian Level Set ClusteringClassically, Bayesian clustering interprets each component of a mixture model as a cluster. The inferred clustering posterior is highly sensitive to any inaccuracies in the kernel within each component. As this kernel is made more flexible, problems arise in identifying the underlying clusters in the data. To address this pitfall, this article proposes a fundamentally different approach to Bayesian clustering that decouples the problems of clustering and flexible modeling of the data density f.David Buch, Miheer Dewaskar, David B. Dunson
- 48Differentially Private Synthetic Data Generation for Relational DatabasesExisting differentially private (DP) synthetic data generation mechanisms typically assume a single-source table. In practice, data is often distributed across multiple tables with relationships across tables. In this paper, we introduce the first-of-its-kind algorithm that can be combined with any existing DP mechanisms to generate synthetic relational databases. Our algorithm iteratively refines the relationship between individual synthetic tables to minimize their approximation errors in termKaveh Alim, Hao Wang, Ojas Gulati, Akash Srivastava, Navid Azizan
- 49A Neural Network Approach to Learning Solutions of a Class of Elliptic Variational InequalitiesWe develop a weak adversarial approach to solving obstacle problems using neural networks. By employing (generalised) regularised gap functions and their properties we rewrite the obstacle problem (which is an elliptic variational inequality) as a minmax problem, providing a natural formulation amenable to learning. Our approach, in contrast to much of the literature, does not require the elliptic operator to be symmetric. We provide an error analysis for suitable discretisations of the continuoAmal Alphonse, Michael Hintermüller, Alexander Kister, Chin Hang Lun, Clemens Sirotenko
- 50Impatient Bandits: Optimizing for the Long-Term Without DelayIncreasingly, recommender systems are tasked with improving users' long-term satisfaction. In this context, we study a content exploration task, which we formalize as a bandit problem with delayed rewards. There is an apparent trade-off in choosing the learning signal: waiting for the full reward to become available might take several weeks, slowing the rate of learning, whereas using short-term proxy rewards reflects the actual long-term goal only imperfectly. First, we develop a predictive modKelly W. Zhang, Thomas Baldwin-McDonald, Kamil Ciosek, Lucas Maystre, Daniel Russo
- 51Conditional Regression for the Nonlinear Single-Variable ModelRegressing a function $F$ on $\mathbb{R}^d$ without incurring the statistical and computational curse of dimensionality requires exploitable structure. Compositional models $F=f\circ g$ in which $g$ has a low-dimensional range include classical single- and multi-index models as well as certain neural networks; while the case of linear $g$ is well understood, substantially less is known for nonlinear $g$. We study the model $F(X)=f(\Pi_\gamma X)$, where $\Pi_\gamma$ is the closest-point coordinatYantao Wu, Mauro Maggioni
- 52Testability of Instrumental Variables in Additive Nonlinear, Non-Constant Effects ModelsWe address the issue of the testability of instrumental variables derived from observational data. Most existing testable implications are centered on scenarios where the treatment is a discrete variable, e.g., instrumental inequality (Pearl, 1995), or where the effect is assumed to be constant, e.g., instrumental variables condition based on the principle of independent mechanisms (Burauel, 2023). However, treatments can often be continuous variables, such as drug dosages or nutritional contentXichen Guo, Zheng Li, Biwei Huang, Yan Zeng, Zhi Geng, Feng Xie
- 53Sliced Wasserstein RegressionWhile statistical modeling of distributional data has gained increased attention, the case of multivariate distributions has been somewhat neglected despite its relevance in various applications. This is because the Wasserstein distance, commonly used in distributional data analysis, poses challenges for multivariate distributions. A promising alternative is the sliced Wasserstein distance, which offers a computationally simpler solution. We propose distributional regression models with multivarHan Chen, Yidong Zhou, Hans-Georg Müller
- 54Causal Falsification of Digital TwinsDigital twins are simulation-based models designed to predict how a real-world process will evolve in response to interventions. This modelling paradigm holds substantial promise in many applications, but rigorous procedures for assessing their accuracy are essential for safety-critical settings. We consider how to assess the accuracy of a digital twin using real-world data. We formulate this as a causal inference problem, which leads to a precise definition of what it means for a twin to be "coRob Cornish, Muhammad Faaiz Taufiq, Arnaud Doucet, Chris Holmes
- 55Bridging Rested and Restless Bandits with Graph-Triggering: Rising and RottingRested and Restless Bandits are two well-known bandit settings that are useful to model real-world sequential decision-making problems in which the expected reward of an arm evolves over time due to the actions we perform or due to the nature. In this work, we propose Graph-Triggered Bandits (GTBs), a unifying framework to generalize and extend rested and restless bandits. In this setting, the evolution of the arms' expected rewards is governed by a graph defined over the arms. An edge connectinGianmarco Genalti, Marco Mussi, Nicola Gatti, Marcello Restelli, Matteo Castiglioni, Alberto Maria Metelli
- 56Extrapolation-Aware Nonparametric Statistical InferenceWe define extrapolation as statistical inference on a conditional function (e.g., a conditional expectation or conditional quantile) evaluated outside the support of the conditioning variable. This type of extrapolation occurs in many data analysis applications and can invalidate the conclusions if not taken into account. While extrapolation is straightforward in parametric models, it becomes challenging in nonparametric models. In this work, we extend the nonparametric statistical model to explNiklas Pfister, Peter Bühlmann
- 57Online Generalized Sparse Regression: How Does Overparametrization Help?Regularized sparse regression has been extensively studied in the offline setting, but online formulation remains relatively under-explored. This gap stems from four key challenges: (i) the infeasibility of dynamically updating the regularization parameter in every online round, (ii) managing storage and memory complexity, (iii) enabling real-time computation via closed-form updates rather than solving full optimization problems at each round, and (iv) achieving optimal statistical guarantees unShuoguang Yang, Qiang Sun
- 58Simultaneous Identification of Sparse Structures and Communities in Heterogeneous Graphical ModelsExploring and detecting community structures hold significant importance in genetics, social sciences, biology, neuroscience and finance, among others. Graphical models are a useful and key tool for community detection through the exploration of sets of variables with group-like properties. In this paper, within the framework of Gaussian graphical models, we propose a decomposition of the underlying graphical structure into low-rank diagonal blocks and a sparse component. This new approach captuDapeng Shi, Tiandong Wang, Zhiliang Ying
- 59Better Simulations for Validating Causal Discovery with the DAG-Adaptation of the Onion MethodThe number of methods for learning causal models from data is growing rapidly, as evidenced by the exponential increase in causal discovery publications each year. Due to a lack of real-world datasets with known causal ground truth, most "causal discovery" or "causal structure learning" algorithms are primarily validated with simulation studies. However, the simulation study designs being used (i) vary from one study to another, (ii) have never been formally characterized, and (iii) have not beeBryan Andrews, Erich Kummerfeld
- 60Singular-limit analysis of gradient descent with noise injectionWe study the limiting dynamics of a large class of noisy gradient descent systems in the overparameterized regime. In this regime the zero-loss set of global minimizers of the loss is large, and when initialized in a neighbourhood of this zero-loss set a noisy gradient descent algorithm slowly evolves along this set. In some cases this slow evolution has been related to better generalization properties. We characterize this evolution for the broad class of noisy gradient descent systems in the lAnna Shalova, André Schlichting, Mark Peletier
- 61Information-Theoretic Safe Bayesian OptimizationWe consider a sequential decision making problem, where we aim to optimize an unknown function via noisy evaluations that do not violate an a-priori unknown (safety) constraint. As both the objective and the constraint are unknown, a common approach is to model them using a Gaussian process and restrict evaluations to those regions that are safe with high probability. Most current methods rely on a discretization of the domain and cannot be directly extended to the continuous case. Moreover, theAlessandro G. Bottero, Carlos E. Luis, Julia Vinogradska, Felix Berkenkamp, Jan Peters
- 62Dirichlet Active LearningThis work introduces Dirichlet Active Learning (DiAL), a Bayesian-inspired approach to the design of active learning algorithms. Our framework models feature-conditional class probabilities as a Dirichlet random field and lends observational strength between similar features in order to calibrate the random field. This random field can then be utilized in learning tasks: in particular, we can use current estimates of mean and variance to conduct classification and active learning in the contextKevin Miller, Ryan Murray
- 63Efficient Modeling of Surrogates to Improve Multi-source High-dimensional Integrative RegressionSurrogate variables play an important role in various fields due to the scarcity or absence of gold-standard labels. We develop a novel approach named SASH for Surrogate-Assisted and data-Shielding High-dimensional integrative regression. It is a semi-supervised approach that efficiently leverages sizable unlabeled samples with error-prone surrogate outcomes from multiple local sites to improve model estimation using the small gold-labeled sample. To facilitate stable and efficient knowledge extYue Liu, Molei Liu, Zijian Guo, Tianxi Cai
- 64torchgfn: A PyTorch GFlowNet LibraryThe growing popularity of generative flow networks (GFlowNets or GFNs) among a range of researchers with diverse backgrounds and areas of expertise necessitates a library that facilitates the testing of new features (e.g., training losses and training policies) against standard benchmark implementations, or on a set of common environments. We present torchgfn, a PyTorch library that aims to address this need. Its core contribution is a modular and decoupled architecture which treats environmentsJoseph D. Viviano, Omar G. Younis, Sanghyeok Choi, Victor Schmidt, Yoshua Bengio, Salem Lahlou
- 65Model-free generalized fiducial inferenceConformal prediction (CP) was developed to provide finite-sample probabilistic prediction guarantees. While CP algorithms are a relatively general purpose approach to uncertainty quantification, with finite-sample guarantees, they lack versatility. Namely, the CP approach does not prescribe how to quantify the degree to which a data set provides evidence in support of (or against) an arbitrary event from a general class of events. This paper uses tools from imprecise probability theory to buildJonathan P Williams
- 66Adversarial Rademacher Complexity of Deep Neural NetworksDeep neural networks (DNNs) are highly vulnerable to adversarial attacks. Ideally, a robust model should perform well on both perturbed training data and unseen perturbed test data. While DNNs can fit perturbed training data, generalizing to perturbed test data remains a significant challenge. This motivates the study of generalization guarantees from a learning theory perspective. This paper focuses on adversarial Rademacher complexity (ARC), first introduced by Khim and Loh (2018) and Yin et aJiancong Xiao, Yanbo Fan, Ruoyu Sun, Zhi-Quan Luo
- 67Keypoint-Guided Optimal Transport: Models, Algorithms, and ApplicationsExisting Optimal Transport (OT) methods mainly derive the optimal transport plan/matching under the criterion of transport cost/distance minimization, which may cause incorrect matching in some cases. In real applications, annotating a few matched keypoints across domains is reasonable or even effortless in annotation burden. It is valuable to investigate how to leverage the annotated keypoints to guide the correct matching in OT. In this paper, we propose a novel KeyPoint-Guided model by ReLatiXiang Gu, Yucheng Yang, Wei Zeng, Jian Sun, Zongben Xu
- 68The Role of Pseudo-Labels in Self-Training Linear Classifiers on High-Dimensional Gaussian Mixture DataSelf-training (ST) is a simple yet effective semi-supervised learning method. However, why and how ST improves generalization performance by using potentially erroneous pseudo-labels is still not well understood. To deepen the understanding of ST, we derive and analyze a sharp characterization of the behavior of iterative ST when training a linear classifier by minimizing the ridge-regularized convex loss on binary Gaussian mixtures, in the asymptotic limit where input dimension and data size diTakashi Takahashi
- 69Bridging Domain Invariance and Diversity: A Fine-Grained Risk Bound for Domain GeneralizationDomain-invariant representation learning and domain augmentation algorithms are two principal methodological paradigms for addressing domain generalization. They are widely employed in the machine learning literature to enhance domain invariance and domain diversity, respectively. However, existing risk bounds for domain generalization do not simultaneously capture the contributions of both approaches. This limitation arises because bounds derived directly in the original latent space are typicaXi Wang, Liang Bai, Xian Yang, Richard Yi Da Xu, Jiye Liang
- 70High-Dimensional Analysis of Gradient Flow for Extensive-Width Quadratic Neural NetworksWe study the high-dimensional training dynamics of a shallow neural network with quadratic activation in a teacher--student setup. We focus on the extensive-width regime, where the teacher and student network widths scale proportionally with the input dimension, and the sample size grows quadratically. This scaling aims to describe overparameterized neural networks in which feature learning still plays a central role. In the high-dimensional limit, we derive a dynamical characterization of the gSimon Martin, Giulio Biroli, Francis Bach
- 71Error Analyses of Auto-Regressive Video Diffusion ModelsAuto-Regressive Video Diffusion Models (AR-VDMs) have shown strong capabilities in generating long, photorealistic videos, but suffer from two key limitations: (i) history forgetting, where the model loses track of previously generated content, and (ii) temporal degradation, where frame quality deteriorates over time. Yet a rigorous theoretical analysis of these phenomena is lacking, and existing empirical understanding remains insufficiently grounded. In this paper, we introduce Meta-ARVDM, a uJing Wang, Fengzhuo Zhang, Xiaoli Li, Vincent Y.~ F. Tan, Tianyu Pang, Chao Du, Aixin Sun, Zhuoran Yang
- 72Near-optimal Delta-convex Estimation of Lipschitz FunctionsThis paper presents a tractable algorithm for estimating an unknown Lipschitz function from noisy observations and establishes an upper bound on its convergence rate. The approach extends max-affine methods from convex shape-restricted regression to the more general Lipschitz setting. A key component is a nonlinear feature expansion that maps max-affine functions into a subclass of delta-convex functions, which act as universal approximators of Lipschitz functions while preserving their LipschitGábor Balázs
- 73The Sample Complexity of Parameter-Free Stochastic Convex OptimizationWe study the sample complexity of stochastic convex optimization when problem parameters such as the distance to optimality and the Lipschitz constant are unknown. We pursue two strategies. First, we develop a reliable model selection method that avoids overfitting to the validation set. This method allows us to generically tune the learning rate of stochastic optimization methods to match the optimal known-parameter sample complexity up to $\log\log$ factors. Second, we develop a regularizationJared Lawrence, Ari Kalinsky, Hannah Bradfield, Yair Carmon, Oliver Hinder
- 74End-to-End Deep Learning for Predicting Metric Space-Valued OutputsMany modern applications involve predicting structured, non-Euclidean outputs such as probability distributions, networks, and symmetric positive-definite matrices. These outputs are naturally modeled as elements of general metric spaces, where classical regression techniques that rely on vector space structure no longer apply. We introduce E2M (End-to-End Metric regression), a deep learning framework for predicting metric space-valued outputs. E2M performs prediction via weighted Fréchet meansYidong Zhou, Su I Iao, Hans-Georg Müller
- 75Graph-based Clustering Revisited: A Relaxation of Kernel k-Means PerspectiveThe well-known graph-based clustering methods, including spectral clustering, symmetric non-negative matrix factorization, and doubly stochastic normalization, can be viewed as relaxations of the kernel k-means approach. However, we posit that these methods excessively relax their inherent low-rank, nonnegative, doubly stochastic, and orthonormal constraints to ensure numerical feasibility, potentially limiting their clustering efficacy. In this paper, guided by our systematic theoretical analysWenlong Lyu, Yuheng Jia, Hui Liu, Junhui Hou
- 76Learning to Play Two-Player Perfect-Information Games without KnowledgeThis paper introduces a set of techniques for learning game state evaluation functions through reinforcement learning. First, we generalize tree bootstrapping, i.e. learning the values of states encountered during search rather than restricting updates to states observed during matches, to the setting of reinforcement learning with non-linear function approximation. Second, we modifies Unbounded Best-First Minimax by extending best action sequences to terminal states. Third, we replace the tradiQuentin Cohen-Solal
- 77Doubly Debiased Robust Subsampling for Transfer LearningThis paper develops a general framework for doubly debiased robust subsampling for transfer learning. The setting arises when massive source datasets are computationally infeasible to use in full, while naive or heuristic subsampling leads to biased estimators that further inherit transfer bias under source-target distributional shifts. We resolve these challenges through two complementary debiasing mechanisms. Inverse probability weighting removes subsampling bias by ensuring that subsample-basTao Wang, Weng Kee Wong
- 78Abstract Gradient Training: A Unified Certification Framework for Data Poisoning, Unlearning, and Differential PrivacyThe impact of inference-time data perturbation (e.g., adversarial attacks) has been extensively studied in machine learning, leading to well-established certification techniques for adversarial robustness. In contrast, certifying models against training data perturbations remains a relatively under-explored area. These perturbations can arise in three critical contexts: adversarial data poisoning, where an adversary manipulates training samples to corrupt model performance; machine unlearning, wPhilip Sosnin, Matthew Wicker, Josh Collyer, Calvin Tsay
- 79Mixing times of data-augmentation Gibbs samplers for high-dimensional probit regressionWe investigate the convergence properties of popular data-augmentation samplers for Baye\-sian probit regression. Leveraging recent results on Gibbs samplers for log-concave targets, we provide simple and explicit non-asymptotic bounds on the associated mixing times (in Kullback-Leibler divergence). The bounds depend explicitly on the design matrix and the prior precision, while they hold uniformly over the vector of responses. We specialize the results for different regimes of statistical interFilippo Ascolani, Giacomo Zanella
- 80Underdamped Langevin MCMC with third order convergenceIn this paper, we propose a new numerical method for the underdamped Langevin diffusion (ULD) and present a non-asymptotic analysis of its sampling error in the 2-Wasserstein distance when the $d$-dimensional target distribution $p(x)\propto e^{-f(x)}$ is strongly log-concave and has varying degrees of smoothness. Precisely, under the assumptions that the gradient and Hessian of $f$ are Lipschitz continuous, our algorithm achieves a 2-Wasserstein error of $\varepsilon$ in $\mathcal{O}\big(\sqrt{Maximilian Scott, Dáire O'Kane, Andraž Jelinčič, James Foster
- 81Approximation-Free Differentiable Oblique Decision TreesDecision Trees (DTs) are widely used in safety-critical domains such as medical diagnosis, valued for their interpretability and effectiveness on tabular data. However, training accurate oblique DTs is challenging due to complex optimization landscapes and overfitting risks, particularly in regression. Recent advances have introduced differentiable formulations that enable gradient-based training and joint optimization of decision boundaries and leaf regressors. Yet, existing approaches typicallSubrat Prasad Panda, Blaise Genest, Arvind Easwaran
- 82Minimax Optimal Convergence of Gradient Descent in Logistic Regression via Large and Adaptive StepsizesWe study gradient descent (GD) for logistic regression on linearly separable data with stepsizes that adapt to the current risk, scaled by a constant hyperparameter \(\eta\). We show that after at most \(1/\gamma^2\) burn-in steps, GD achieves a risk upper bounded by \(\exp(-\Theta(\eta))\), where \(\gamma\) is the margin of the dataset. As \(\eta\) can be arbitrarily large, GD attains an arbitrarily small risk immediately after the burn-in steps, though the risk evolution may be non-monotonic.Ruiqi Zhang, Jingfeng Wu, Licong Lin, Peter L. Bartlett
- 83Adaptive Nonparametric Perturbations of Parametric Models with Generalized BayesParametric Bayesian modeling offers a powerful and flexible toolbox for machine learning. Yet the model, however detailed, may still be wrong, and this can make inferences untrustworthy. In this paper we introduce a new class of semiparametric corrections for parametric Bayesian models, when the target of inference is a functional of the true data distribution. Our starting point is a fully Bayesian modeling approach, which explicitly accounts for the possibility that the parametric model is wroBohan Wu, Eli N. Weinstein, Sohrab Salehi, Yixin Wang, David M. Blei
- 84Robust training of implicit generative models for multivariate and heavy-tailed distributions with an invariant statistical lossImplicit generative models are often trained adversarially, which can yield unstable dynamics and mode collapse. The invariant statistical loss (ISL) offers a fully sample-based alternative by comparing empirical ranks of real and generated samples. In this work, we formally characterize ISL as a proper divergence over continuous distributions and establish key regularity properties, showing that it is continuous and differentiable, thereby enabling stable gradient-based optimization without advJosé Manuel de Frutos, Manuel A. Vázquez, Pablo M. Olmos, Joaquín Míguez
- 85Gradient Span Algorithms Make Predictable Progress in High DimensionWe prove that all 'gradient span algorithms' have asymptotically deterministic behavior on scaled Gaussian random functions as the dimension tends to infinity. This is a functional generalization of similar results for random quadratic functions and spin glasses. They explain the counterintuitive phenomenon that different training runs of many large machine learning models result in approximately equal cost curves despite random initialization on a complicated non-convex landscape. This 'predictFelix Benning, Leif Döring
- 86py/cuTAGI: An Open-Source Library for Tractable Approximate Gaussian Inference in Bayesian Neural NetworksThis paper introduces pyTAGI, a Python wrapper, and cuTAGI, its high-performance C++/CUDA backend, implementing Tractable Approximate Gaussian Inference (TAGI) for neural networks. TAGI treats all network quantities as Gaussian random variables and derives closed-form expressions for prior/posterior expected values, variances, and covariances, enabling analytic Bayesian learning without relying on gradient descent or backpropagation. The libraries mimic PyTorch's sequential interface, allowing uLuong-Ha Nguyen, James-A. Goulet, Miquel Florensa-Montilla, Van-Dai Vuong
- 87Statistical Test for Attention in Transformers for Images and Time SeriesTransformer models have achieved exceptional performance in various domains, including computer vision and time-series analysis. Their core attention mechanism is widely used to interpret model decisions by assigning importance weights to input regions, such as image patches or time series intervals. However, the reliability of these interpretations remains a major concern. High-attention weights do not necessarily indicate genuinely significant features; they may instead be artifacts of the modTomohiro Shiraishi, Daiki Miwa, Teruyuki Katsuoka, Vo Nguyen Le Duy, Shuichi Nishino, Kouichi Taji, Ichiro Takeuchi
- 88Accelerating Constrained Sampling: A Large Deviations ApproachThe problem of sampling a target probability distribution on a constrained domain arises in many applications including machine learning. For constrained sampling, various Langevin algorithms such as projected Langevin Monte Carlo (PLMC), based on the discretization of reflected Langevin dynamics (RLD) and more generally skew-reflected non-reversible Langevin Monte Carlo (SRNLMC), based on the discretization of skew-reflected non-reversible Langevin dynamics (SRNLD), have been proposed and studiYingli Wang, Changwei Tu, Xiaoyu Wang, Lingjiong Zhu
- 89Learning general conditional independence structures via the neighbourhood latticeWe study the problem of learning multivariate dependencies in nonparametric and high-dimensional settings. This includes but is not limited to graphical models. Our approach effectively combines several features that are missing from previous work on this problem: We show how the entire dependence structure can be learned nonparametrically while simultaneously evading the curse of dimensionality and relaxing common assumptions such as faithfulness. To this end, we introduce and study the neighboArash A. Amini, Bryon Aragam, Qing Zhou
- 90Statistical guarantees for denoising reflected diffusion modelsIn recent years, denoising diffusion models have become a crucial area of research due to their abundance in the rapidly expanding field of generative AI. While recent statistical advances have delivered explanations for the generation ability of idealised denoising diffusion models for high-dimensional target data, implementations introduce thresholding procedures for the generating process to overcome issues arising from the unbounded state space of such models. This mismatch between theoreticAsbjørn Holk, Claudia Strauch, Lukas Trottner
- 91Vecchia-Inducing-Points Full-Scale Approximations for Gaussian ProcessesGaussian processes are flexible, probabilistic, non-parametric models widely used in machine learning and statistics. However, their scalability to large data sets is limited by computational constraints. To overcome these challenges, we propose Vecchia-inducing-points full-scale (VIF) approximations combining the strengths of global inducing points and local Vecchia approximations. Vecchia approximations excel in settings with low-dimensional inputs and moderately smooth covariance functions, wTim Gyger, Reinhard Furrer, Fabio Sigrist
- 92STDE++: Polynomial-Time Amortization for Linear Differential OperatorsOptimizing neural networks with losses that contain high-dimensional and high-order differential operators is expensive to evaluate with backpropagation due to $\mathcal{O}(d^{k})$ scaling of the derivative tensor size and the $\mathcal{O}(2^{k-1}L)$ scaling in the computation graph, where $d$ is the domain dimension, $L$ is the number of ops in the forward computation graph and $k$ is the derivative order. Previous works addressed the polynomial scaling in $d$ by amortizing the computation overZekun Shi, Zheyuan Hu, Min Lin, Kenji Kawaguchi
- 93The Within-Orbit Adaptive Leapfrog No-U-Turn SamplerLocally adapting parameters within Markov chain Monte Carlo methods while preserving reversibility is notoriously difficult. The success of the No-U-Turn Sampler (NUTS) largely stems from its clever local adaptation of the integration time in Hamiltonian Monte Carlo via a geometric U-turn condition. However, posterior distributions frequently exhibit multiscale geometries with extreme variations in scale, making it necessary to also adapt the leapfrog integrator's step size locally and dynamicalNawaf Bou-Rabee, Bob Carpenter, Tore Selland Kleppe, Sifan Liu
- 94Finite-Time Decoupled Convergence in Nonlinear Two-Time-Scale Stochastic ApproximationIn two-time-scale stochastic approximation (SA), two iterates are updated at varying speeds using different step sizes, with each update influencing the other. Previous studies on linear two-time-scale SA have shown that the convergence rates of the mean-square errors for these updates depend solely on their respective step sizes, a phenomenon termed decoupled convergence. However, achieving decoupled convergence in nonlinear SA remains less understood. Our research investigates the potential foYuze Han, Xiang Li, Zhihua Zhang
- 95Embedding Network Autoregression for Time Series Analysis and Causal Peer Effect InferenceWe propose an Embedding Network Autoregressive Model for multivariate networked longitudinal data. We assume the network is generated from a latent variable model, and these unobserved variables are included in a structural peer effect model or a time series network autoregressive model. This approach takes a unified view of two related yet different problems: (1) modeling and predicting multivariate networked time series data and (2) causal peer influence estimation in the presence of confoundiJae Ho Chang, Subhadeep Paul
- 96Three Types of Calibration using Properties and their Semantic and Formal RelationshipsFueled by discussions around "trustworthiness" and algorithmic fairness, calibration of predictive systems has regained scholars' attention. The vanilla definition and understanding of calibration is, simply put, on all days on which the rain probability has been predicted to be $p$, the actual frequency of rain days was $p$. However, the increased attention has led to an immense variety of new notions of "calibration". Some of the notions are incomparable, serve different purposes, or imply eacRabanus Derr, Jessie Finocchiaro, Robert C. Williamson
- 97Convergence of Decentralized Stochastic Subgradient-based Methods for Nonsmooth Nonconvex OptimizationIn this paper, we focus on the decentralized stochastic subgradient-based methods in minimizing nonsmooth nonconvex functions without Clarke regularity, especially in the decentralized training of nonsmooth neural networks. We propose a general framework that unifies various decentralized subgradient-based methods, such as decentralized stochastic subgradient descent (DSGD), DSGD with gradient-tracking technique (DSGD-T), and DSGD with momentum (DSGD-M). To establish the convergence properties oSiyuan Zhang, Nachuan Xiao, Xin Liu
- 98A Two-Timescale Primal-Dual Framework for Reinforcement Learning via Online Dual Variable GuidanceWe study reinforcement learning by combining recent advances in regularized linear programming formulations with the classical theory of stochastic approximation. Motivated by the challenge of designing algorithms that leverage off-policy data while maintaining on-policy exploration, we propose PGDA-RL, a novel primal-dual projected gradient descent-ascent algorithm for solving regularized Markov decision processes (MDPs). PGDA-RL integrates experience replay-based gradient estimation with a twoAxel F. Wolter, Tobias Sutter
- 99FLAGG: Flexible Autoregressive Graph GenerationThe Deep Graph Generation's panorama spans two extremes: one-shot and sequential models. The former generates nodes and edges jointly, while the latter samples them autoregressively. Each method performs better in different graph domains depending on size and topology, but neither is applicable to all graph categories. For instance, one-shot methods struggle with generating large graphs, while sequential methods underperform on smaller graphs. A possible way to overcome these limitations is to fSamuel Cognolato, Alessandro Sperduti, Luciano Serafini
- 100Nested Subspace Learning with FlagsMany machine learning methods look for low-dimensional representations of the data. The underlying subspace can be estimated by first choosing a dimension q and then optimizing a certain objective function over the space of q-dimensional subspaces (the Grassmannian). Trying different q generally yields non-nested subspaces, which raises an important issue of consistency between the data representations. In this paper, we propose a simple and easily implementable principle to enforce nestedness iTom Szwagier, Xavier Pennec
- 101A Unified Approach to Analysis and Design of Denoising Markov ModelsProbabilistic generative models based on measure transport, such as diffusion and flow-based models, are often formulated in the language of Markovian stochastic dynamics, where the choice of the underlying process impacts both algorithmic design choices and theoretical analysis. In this paper, we aim to establish a rigorous mathematical foundation for denoising Markov models, a broad class of generative models that postulate a forward process transitioning from the target distribution to a simpYinuo Ren, Grant M. Rotskoff, Lexing Ying
- 102Flavors of Margin: Implicit Bias of Steepest Descent in Homogeneous Neural NetworksWe study the implicit bias of the general family of steepest descent algorithms with infinitesimal learning rate in deep homogeneous neural networks. We show that: (a) an algorithm-dependent geometric margin starts increasing once the networks reach perfect training accuracy, and (b) any limit point of the training trajectory corresponds to a KKT point of the corresponding margin-maximization problem. We experimentally zoom into the trajectories of neural networks optimized with various steepestNikolaos Tsilivis, Eitan Gronich, Julia Kempe, Gal Vardi
- 103A Single-Loop Stochastic Proximal Quasi-Newton Method for Large-Scale Nonsmooth Convex OptimizationWe propose a new stochastic proximal quasi-Newton method for minimizing the sum of two convex functions in the particular context that one of the functions is the average of a large number of smooth functions and the other one is nonsmooth. The new method integrates a simple single-loop SVRG (L-SVRG) technique for sampling the gradient and a stochastic limited-memory BFGS (L-BFGS) scheme for approximating the Hessian of the smooth function components. The globally linear convergence rate of theYongcun Song, Zimeng Wang, Xiaoming Yuan, Hangrui Yue
- 104Statistical Learning Theory for Neural OperatorsWe present statistical convergence results for the learning of (possibly) non-linear mappings in infinite-dimensional spaces. Specifically, given a map $G_0:\mathcal X\to\mathcal Y$ between two separable Hilbert spaces, we analyze the problem of recovering $G_0$ from $n\in\mathbb{N}$ noisy input-output pairs $(x_i, y_i)_{i=1}^n$ with $y_i = G_0 (x_i)+\varepsilon_i$; here the $x_i\in\mathcal{X}$ represent randomly drawn "design" points, and the $\varepsilon_i$ are assumed to be either i.i.d. whitNiklas Reinhardt, Sven Wang, Jakob Zech
- 105Deconvolution in unlinked linear modelsUnlinked regression, in which covariates and responses are observed separately without known correspondence, has recently gained increasing attention. Deconvolution, on the other hand, is a fundamental and challenging problem in nonparametric statistics with the aim of estimating the distribution of a latent random variable $Z$ based on observations contaminated by some additive noise. The complexity of this task is heavily influenced by the smoothness of the noise distribution and often leads tBalabdaoui, Fadoua, Di Noia, Antonio, Durot, Cécile
- 106Spectral Truncation Kernels: Noncommutativity in C*-algebraic Kernel MachinesA central question in vector- and function-valued learning is how to design kernels that capture both local and non-local interactions while remaining computationally tractable. Existing operator-valued kernels offer only partial answers: separable kernels are efficient but fail to model interactions across the function domain, while commutative kernels capture only pointwise structure. To address this, we propose spectral truncation kernels, a new class of positive definite kernels for vector-Yuka Hashimoto, Ayoub Hafid, Masahiro Ikeda, Hachem Kadri
- 107High-dimensional Parameter Transfer With Fused-RegularizerParameter transfer aims to improve parameter estimation accuracy by leveraging knowledge from related sources. This paper studies the parameter transfer problem from heterogeneous sources for high-dimensional M-estimators. Specifically, we propose a novel one-step estimator with a fused-regularizer and a target-data-oriented constraint, which can robustly capture parameter knowledge from source data in the presence of different types of data distribution shifts. Nonasymptotic bound is provided fZelin He, Ying Sun, Jingyuan Liu, Runze Li
- 108Exogenous Randomness Empowering Random ForestsWe offer theoretical and empirical insights into the impact of exogenous randomness on the effectiveness of random forests with tree-building rules independent of training data. We formally introduce the concept of exogenous randomness which can come from feature subsampling or tie-breaking in tree-building processes. We develop non-asymptotic expansions for the mean squared error (MSE) for both individual trees and forests and establish sufficient and necessary conditions for their consistency.Tianxing Mei, Yingying Fan, Jinchi Lv
- 109Kernel Mean Embedding Deviation Subspace for Unsupervised Learning with Heterogeneous DataThis paper proposes a method for dimension reduction that preserves information in unsupervised learning with high-dimensional heterogeneous data, specifically targeting change point detection and clustering analysis. Our main strategy is to apply a Corrected Kernel Principal Component Analysis (CKPCA) method to construct the so-called kernel mean embedding deviation subspace. The approach efficiently identifies distributional changes in these dimension reduction subspaces for unsupervised dimenLuoyao Yu, Lixing Zhu, Ruoqing Zhu, Xuehu Zhu
- 110Deep Nonparametric Conditional Independence Tests for ImagesConditional independence tests (CITs) test for conditional dependence between random variables given a vector of conditioning or confounder variables. As existing CITs are limited in their applicability to complex, high-dimensional variables such as images, we introduce deep nonparametric CITs (DNCITs). The DNCITs combine embedding maps, which extract feature representations of high-dimensional variables, with nonparametric CITs applicable to these feature representations. For the embedding mapsMarco Simnacher, Xiangnan Xu, Hani Park, Christoph Lippert, Sonja Greven
- 111Semi-supervised learning for linear extremile regressionExtremile regression, as a least squares analog of quantile regression, is potentially a useful tool for modeling and understanding the extreme tails of a distribution. However, existing extremile regression methods, as nonparametric approaches, may face challenges in high-dimensional settings due to data sparsity, computational inefficiency, and the risk of overfitting. While linear regression, particularly in high-dimensional settings, serves as the foundation for many other statistical and maRong Jiang, Jiangfeng Wang, Keming Yu
- 112Transfer Learning via Regularized Random-effects Linear Discriminant AnalysisLinear discriminant analysis is a widely used method for classification. However, the high dimensionality of predictors combined with small sample sizes often results in large classification errors. To address this challenge, it is crucial to leverage data from related source models to enhance the classification performance of a target model. This paper proposes a transfer learning approach via regularized random-effects linear discriminant analysis, where the discriminant direction is estimatedHongzhe Zhang, Arnab Auddy, Hongzhe Li
- 113Cheap Bootstrap for Fast Uncertainty Quantification of Stochastic Gradient DescentStochastic gradient descent (SGD) or stochastic approximation has been widely used in model training and stochastic optimization. While there is a huge literature on analyzing its convergence, inference on the obtained solutions from SGD has only been recently studied, yet it is important due to the growing need for uncertainty quantification. We investigate two computationally cheap resampling-based methods to construct confidence intervals for SGD solutions. One uses multiple, but few, SGDs inHenry Lam, Zitong Wang
- 114A Natural Primal-Dual Hybrid Gradient Method for Adversarial Neural Network Training on Solving Partial Differential EquationWe propose a scalable preconditioned primal-dual hybrid gradient algorithm for solving partial differential equations (PDEs). We multiply the PDE with a dual test function to obtain an inf-sup problem whose loss functional involves lower-order differential operators. The Primal-Dual Hybrid Gradient (PDHG) algorithm is then leveraged for this saddle point problem. By introducing suitable precondition operators to the proximal steps in the PDHG algorithm, we obtain an alternative natural gradientShu Liu, Stanley Osher, Wuchen Li
- 115Generalized Resubstitution for Regression Error EstimationWe propose generalized resubstitution error estimators for regression. Each error estimator in this class corresponds to a choice of an empirical probability measure and a loss function. The standard empirical probability measure and the quadratic loss lead to the standard sum of squares error estimator. Other choices of empirical probability measure lead to more general estimators with superior bias and variance properties. We prove that these error estimators are consistent under broad assumptDiego Marcondes, Ulisses Braga-Neto
- 116Transfer Conformal Predictive Inference for RegressionConformal prediction, a powerful framework for constructing prediction intervals for response variables using any regression function estimators, often faces the challenge of producing overly broad intervals with limited target data. In this paper, we study the transfer learning problem in conformal prediction, aiming to improve the precision of the prediction interval of the target data with insufficient data by leveraging related auxiliary source datasets. Allowing for the potential non-exchanCe Zhang, Ting Li, Jinhan Xie, Linglong Kong, Bei Jiang
- 117Towards Convexity in Anomaly Detection: A New Formulation of SSLM with Unique Optimal SolutionsAn unsolved issue in widely used methods such as Support Vector Data Description (SVDD) and Small Sphere and Large Margin SVM (SSLM) for anomaly detection is their nonconvexity, which hampers the analysis of optimal solutions in a manner similar to SVMs and limits their applicability in large-scale scenarios. In this paper, we introduce a novel convex SSLM formulation which has been demonstrated to revert to a convex quadratic programming problem for hyperparameter values of interest. LeveragingHongying Liu, Hao Wang, Haoran Chu, Yibo Wu
- 118Node Regression on Latent Position Random Graphs via Local AveragingNode regression consists in predicting the value of a graph label at a node, given observations at the other nodes. We perform a theoretical study where the graph is generated by a Latent Position Model: each node has a latent position and the probability of connection depends on the distance between latent positions. We begin by studying the simplest estimator: averaging the label at all neighboring nodes. We show that in Latent Position Models this estimator tends to a Nadaraya-Watson estimatoMartin Gjorgjevski, Nicolas Keriven, Simon Barthelme, Yohann De Castro
- 119On the Relevance of Byzantine Robust Optimization Against Data PoisoningThe success of machine learning (ML) has been intimately linked with the availability of large amounts of data, typically collected from heterogeneous sources and processed on vast networks of computing devices (also called workers). Beyond accuracy, the use of ML in critical domains such as healthcare and autonomous driving calls for robustness against data poisoning and some faulty workers. The problem of Byzantine ML formalizes these robustness issues by considering a distributed ML environmeSadegh Farhadkhani, Rachid Guerraoui, Nirupam Gupta, Rafael Pinot
- 120Best Arm Identification with Minimal RegretMotivated by real-world applications that necessitate responsible experimentation, we introduce the problem of best arm identification (BAI) with minimal regret. This variant of the multi-armed bandit problem elegantly amalgamates two of its most ubiquitous objectives: regret minimization and BAI. More precisely, the agent's goal is to identify the best arm with a prescribed confidence level $\delta$, while minimizing the cumulative regret up to the stopping time. Focusing on single-parameter exJunwen Yang, Vincent Y. F. Tan, Tianyuan Jin
- 121Convergence of Noise-Free Sampling Algorithms with Regularized Wasserstein ProximalsIn this work, we investigate the convergence properties of the backward regularized Wasserstein proximal (BRWP) method for sampling a target distribution. The BRWP approach can be shown as a semi-implicit time discretization for a probability flow ODE with the score function whose density satisfies the Fokker-Planck equation of the overdamped Langevin dynamics. Specifically, the evolution of the density-hence the score function-is approximated via a kernel representation derived from the regularFuqun Han, Stanley Osher, Wuchen Li
- 122Almost Sure Convergence of Linear Temporal Difference Learning with Arbitrary FeaturesTemporal difference (TD) learning with linear function approximation (linear TD) is a classic and powerful prediction algorithm in reinforcement learning. While it is well-understood that linear TD converges almost surely to a unique point, this convergence traditionally requires the assumption that the features used by the approximator are linearly independent. However, this linear independence assumption does not hold in many practical scenarios. This work is the first to establish the almostJiuqi Wang, Shangtong Zhang
- 123Asymptotics of Stochastic Gradient Descent with Dropout Regularization in Linear ModelsThis paper proposes an asymptotic theory for online inference of the stochastic gradient descent (SGD) iterates with dropout regularization in linear regression. Specifically, we establish the geometric-moment contraction (GMC) for constant step-size SGD dropout iterates to show the existence of a unique stationary distribution of the dropout recursive function. Based on the GMC property, we use the functional dependence measure to provide quenched central limit theorems (CLT) for the gradient dJiaqi Li, Johannes Schmidt-Hieber, Wei Biao Wu
- 124Beyond Unconstrained Features: Neural Collapse for Shallow Neural Networks with General DataNeural collapse (${\cal NC}$) is a phenomenon that emerges at the terminal phase of the training (TPT) of deep neural networks (DNNs). The features of the data in the same class collapse to their respective sample means and the sample means exhibit a simplex equiangular tight frame (ETF). In the past few years, there has been a surge of works that focus on explaining why the ${\cal NC}$ occurs and how it affects generalization. Since the DNNs are notoriously difficult to analyze, most works mainWanli Hong, Shuyang Ling
- 125Differentially Private Estimation and Inference in High-Dimensional Regression with FDR ControlThis paper proposes new methodologies for conducting practical differentially private (DP) estimation and inference in high-dimensional linear regression. We first introduce a DP Bayesian Information Criterion (DP-BIC) for selecting the unknown sparsity parameter in differentially private sparse linear regression (DP-SLR), eliminating the need for prior knowledge of model sparsity, which is a requisite in the existing literature. Next, we develop the DP debiased algorithm that enables privacy-prZhanrui Cai, Sai Li, Xintao Xia, Linjun Zhang
- 126Demographic Parity in Regression and Classification Within the Unawareness FrameworkThis paper explores the theoretical foundations of fair regression under the constraint of demographic parity within the unawareness framework, where disparate treatment is prohibited, extending existing results where such treatment is permitted. Specifically, we aim to characterize the optimal fair regression function when minimizing the quadratic loss. Our results reveal that this function is given by the solution to a barycenter problem with optimal transport costs. Additionally, we study theVincent Divol, Solenne Gaucher
- 127Enhancing Accuracy in Generative Models via Knowledge TransferThis paper investigates the accuracy of generative models and the impact of knowledge transfer on their generation precision. Specifically, we examine a generative model for a target task, fine-tuned using a pre-trained model from a source task. Building on the "Shared Embedding" concept, which bridges the source and target tasks, we introduce a novel framework for transfer learning under distribution metrics such as the Kullback-Leibler divergence. This framework underscores the importance of lXinyu Tian, Xiaotong Shen
- 128Multi-relational Network Autoregression Model with Latent Group StructuresMulti-relational networks among entities are frequently observed in the era of big data. Quantifying the effects of multiple networks has attracted significant research interest recently. In this work, we model multiple network effects through an autoregressive framework for tensor-valued time series. To characterize the potential heterogeneity of the networks and handle the high dimensionality of the time series data simultaneously, we assume a separate group structure for entities in each netwYimeng Ren, Xuening Zhu, Ganggang Xu, Yanyuan Ma
- 129Limiting Over-Smoothing and Over-Squashing of Graph Message Passing by Deep Scattering TransformsGraph neural networks (GNNs) have become pivotal tools for processing graph-structured data, leveraging the message passing scheme as their core mechanism. However, traditional GNNs often grapple with issues such as instability, over-smoothing, and over-squashing, which can degrade performance and create a trade-off dilemma. In this paper, we introduce a discriminatively trained, multi-layer Deep Scattering Message Passing (DSMP) neural network designed to overcome these challenges. By harnessinYuanhong Jiang, Dongmian Zou, Xiaoqun Zhang, Yu Guang Wang
- 130A Fully Parameter-Free Second-Order Algorithm for Convex-Concave Minimax ProblemsIn this paper, we study second-order algorithms for the convex-concave minimax problem, which has attracted much attention in many fields such as machine learning in recent years. We propose a Lipschitz-free cubic regularization (LF-CR) algorithm for solving the convex-concave minimax optimization problem without knowing the Lipschitz constant. It can be shown that the iteration complexity of the LF-CR algorithm to obtain an $\epsilon$-optimal solution with respect to the restricted primal-dualJun-Lin Wang, Zi Xu, Hui-Ling Zhang
- 131Stochastic Differential Equations models for Least-Squares Stochastic Gradient DescentWe study the dynamics of a continuous-time model of stochastic gradient descent (SGD) for the least-square problem. Indeed, pursuing the work of, we analyze stochastic differential equations (SDEs) that model SGD either in the case of the training loss (finite samples) or the population one (online setting). A key qualitative feature of the dynamics is the existence of a perfect interpolator of the data, irrespective of the sample size. In both scenarios, we provide precise, non-asymptotic ratesAdrien Schertzer, Loucas Pillaud-Vivien
- 132Vector-Valued Gaussian Processes for Approximating Divergence- or Rotation-free Vector FieldsIn this paper, we discuss vector-valued Gaussian processes for the approximation of divergence- or rotation-free functions. We establish the theory for such Gaussian processes, then link the theory to multivariate approximation theory, and finally give error estimates for the predictive mean in various situations.Quoc Thong Le Gia, Ian Hugh Sloan, Holger Wendland
- 133Differentially Private Best-Arm IdentificationBest Arm Identification (BAI) problems are progressively used for data-sensitive applications, such as designing adaptive clinical trials, tuning hyper-parameters, and conducting user studies. Motivated by the data privacy concerns invoked by these applications, we study the problem of BAI with fixed confidence in both the local and central models, i.e. under $\epsilon$-local and $\epsilon$-global Differential Privacy (DP). First, to quantify the cost of privacy, we derive lower bounds on the saAchraf Azize, Marc Jourdan, Aymen Al Marjani, Debabrota Basu
- 134Corruptions of Supervised Learning Problems: Typology and MitigationsCorruption is notoriously widespread in data collection. Despite extensive research, the existing literature predominantly focuses on specific settings and learning scenarios, lacking a unified view of corruption modelization and mitigation. In this work, we develop a general theory of corruption, which incorporates all modifications to a supervised learning problem, including changes in model class and loss. Focusing on changes to the underlying probability distributions via Markov kernels, ourLaura Iacovissi, Nan Lu, Robert C. Williamson
- 135Nonparametric Partial Disentanglement via Mechanism Sparsity: Sparse Actions, Interventions and Sparse Temporal DependenciesThis work introduces a novel principle for disentanglement we call mechanism sparsity regularization, which applies when the latent factors of interest depend sparsely on observed auxiliary variables and/or past latent factors. We propose a representation learning method that induces disentanglement by simultaneously learning the latent factors and the sparse causal graphical model that explains them. We develop a nonparametric identifiability theory that formalizes this principle and shows thatSébastien Lachapelle, Pau Rodríguez López, Yash Sharma, Katie Everett, Rémi Le Priol, Alexandre Lacoste, Simon Lacoste-Julien
- 136A Mean-Field Analysis of Neural Stochastic Gradient Descent-Ascent for Functional Minimax OptimizationThis paper studies minimax optimization problems defined over infinite-dimensional function classes of over-parameterized two-layer neural networks. In particular, we consider the minimax optimization problem stemming from estimating linear functional equations defined by conditional expectations, where the objective functions are quadratic in the functional spaces. We address (i) the convergence of the stochastic gradient descent-ascent algorithm and (ii) the representation learning of the neurYuchen Zhu, Yufeng Zhang, Zhaoran Wang, Zhuoran Yang, Xiaohong Chen
- 137Minimax density estimation in the adversarial framework under local differential privacyWe consider the problem of nonparametric density estimation under privacy constraints in an adversarial framework. To this end, we study minimax rates over Sobolev spaces under local differential privacy. We first obtain a lower bound which allows us to quantify the impact of privacy compared with the classical framework. Next, we introduce a new Coordinate block privacy mechanism that guarantees local differential privacy, which, coupled with a projection estimator, achieves the minimax optimalMélisande Albert, Juliette Chevallier, Béatrice Laurent, Ousmane Sacko
- 138Approximations and Learning for Continuous State and Action MDPs under Average Cost CriteriaIn this paper, for Markov Decision Processes (MDPs) with standard Borel spaces, (i) we first provide a discretization based approximation method for MDPs with continuous spaces under average cost criteria, and provide error bounds for approximations when the dynamics are only weakly continuous (for asymptotic convergence of errors as the grid sizes vanish) or Wasserstein continuous (with a rate in approximation as the grid sizes vanish) under certain ergodicity assumptions. In particular, we relAli D. Kara, Serdar Yüksel
- 139Optimal Approximation and Generalization Errors for Deep Convolutional Neural NetworksThis paper focuses on approximation and learning performances of deep convolutional neural networks with zero-padding and max-pooling. We prove that, to approximate $r$-smooth function, the approximation rates of deep convolutional neural networks with depth $L$ are of order $ (L/\log L)^{-2r/d} $, which is optimal up to a logarithmic factor. Furthermore, we deduce almost optimal generalization errors for implementing empirical risk minimization over deep convolutional neural networks. Our theorJinxin Wang, Shao-Bo Lin
- 140Investigating the Histogram Loss in RegressionIt is becoming increasingly common in regression to train neural networks that model the entire distribution even if only the mean is required for prediction. This additional modeling often comes with performance gains, and the reasons behind the improvement are not fully known. This paper investigates a recent approach to regression, the histogram loss, which involves learning the conditional distribution of the target variable by minimizing the cross-entropy between a target distribution and aEhsan Imani, Kai Luedemann, Sam Scholnick-Hughes, Esraa Elelimy, Martha White
- 141Bayes-Optimal Fair Classification with Linear Disparity Constraints via Pre-, In-, and Post-processingMachine learning algorithms may have disparate impacts on protected groups. To address this, we develop methods for Bayes-optimal fair classification, aiming to minimize classification error subject to given group fairness constraints. We introduce the notion of linear disparity measures, which are linear functions of a probabilistic classifier; and bilinear disparity measures, which are also linear in the group-wise regression functions. We show that several popular disparity measures---the devXianli Zeng, Kevin Jiang, Guang Cheng, Edgar Dobriban
- 142Why "Classic" Transformers Are Shallow and A Depth-Enabling TechniqueSince its introduction in 2017, the Transformer has emerged as the leading neural network architecture, catalyzing revolutionary advancements in many AI disciplines. The key innovation in Transformer is a Self-Attention (SA) mechanism designed to capture contextual information. However, stacking up more layers of the same design has failed to produce trainable deeper Transformers. Thus far, various architectural modifications to the original design have been proposed to enable deeper depths forYueyao Yu, Yin Zhang
- 143Kernel-based Distributed Learning Beyond Least SquaresWe consider one-shot distributed learning problems in a reproducing kernel Hilbert space framework. Current results are limited to the least-squares loss and extensions beyond this meet with some significant technical challenges. We establish the optimal rate of distributed learning for some general class of convex loss functions satisfying mild assumptions, using a novel empirical process on the Bregman divergence induced by the loss, which is essential for carrying out a quadratic approximatioHeng Lian, Xu Guo
- 144A Convex Framework for Confounding Robust InferenceWe study policy evaluation of offline contextual bandits subject to unobserved confounders. Sensitivity analysis methods are commonly used to estimate the policy value under the worst-case confounding scenario within a given uncertainty set. However, existing work often resorts to some coarse relaxation of the uncertainty set for the sake of tractability, leading to overly conservative estimation of the policy value. In this paper, we propose a general estimator that provides a sharp lower boundKei Ishikawa, Niao He, Takafumi Kanamori
- 145Global Fr{\'{e}}chet Manifold Learning for Random Objects, With Application to Low-Dimensional Wasserstein Representations of Distributional DataWe study manifold learning with multidimensional scaling for samples of metric space valued data. By adopting a global version of ISOMAP we obtain low-dimensional Euclidean representations. A key innovation is that we demonstrate that global Fréchet regression can be utilized for mapping the elements of a convex set in the Euclidean representation space back to the metric space where the objects reside. We refer to this approach as Fréchet manifold learning and showcase it with one-dimensional dÁlvaro Gajardo, Hans-Georg Müller
- 146Probabilistic Rainfall Downscaling: Joint Generalized Neural Models with Censored Spatial Gaussian CopulaA novel approach for generating conditional probabilistic rainfall downscaling at finer scales from deterministic weather variables at coarser scales with temporal and spatial dependence is introduced. A two-step procedure is employed. Firstly, marginal location-specific distributions are jointly modelled conditional on the deterministic coarse weather variables. Secondly, a spatial dependency structure is learned to ensure spatial coherence among these distributions. To learn marginal distributDavid Huk, Rilwan A. Adewoyin, Ritabrata Dutta
- 147Sparse Topic Modeling via Spectral Decomposition and ThresholdingIn probabilistic Latent Semantic Indexing (pLSI), word frequencies across document corpora are modeled through a low-rank factorization of the expected document-term matrix into topic-word and topic-document components. In this paper, we study the estimation of the topic-word matrix under a sparsity structure motivated by Zipf's law: word frequencies within each topic exhibit a rapid empirical decay, with most probability mass concentrated on a small subset of words. Motivated by this observatioHuy Tran, Yating Liu, Claire Donnat
- 148Nonparametric generative modeling for time series via Schr{\"{o}}dinger bridgeWe propose a novel generative model for time series based on Schrödinger bridge (SB) approach. This consists in the entropic interpolation via optimal transport between a reference probability measure on path space and a target measure consistent with the joint data distribution of the time series. The solution is characterized by a stochastic differential equation on finite horizon with a path-dependent drift function, hence respec\-ting the temporal dynamics of the time series distribution. WeMohamed Hamdouche, Pierre Henry-Labordère, Huyên Pham
- 149Do We Need to Penalize Variance of Losses for Learning with Label Noise?Statistically consistent algorithms have been widely employed for dealing with noisy labels. Their objective functions are designed so that minimizing the expected risk on noisy data leads to the same minimizer as minimizing the expected risk on clean data. From the weak law of large numbers, penalizing the variance of losses would reduce the discrepancy between the average loss and the expected risk on the clean data when there is a finite training sample, and the estimation error in the model'Yexiong Lin, Yu Yao, Yuxuan Du, Jun Yu, Bo Han, Mingming Gong, Tongliang Liu
- 150Causal Influences over Social Learning NetworksThis paper investigates causal influences between agents linked by a social graph and interacting over time. In particular, the work examines the dynamics of social learning models and distributed decision-making protocols, and derives expressions that reveal the causal relations between pairs of agents and explain the flow of influence over the network. The results turn out to be dependent on the graph topology and the level of information that each agent has about the inference problem they arMert Kayaalp, Ali H. Sayed
- 151Neural Exploitation and Exploration of Contextual BanditsIn this paper, we study the neural exploration strategy for contextual bandits. The dilemma of exploitation and exploration widely exists in real-world applications such as recommender systems, online advertising, and clinical trials. Contextual bandits provide principled methods to solve this dilemma, including two prevalent techniques: Thompson Sampling (TS), and Upper Confidence Bound (UCB). Neural contextual bandits have been studied to adapt to the non-linear reward function, combined withYikun Ban, Yuchen Yan, Arindam Banerjee, Jingrui He
- 152Knowledge Cascade: Reverse Knowledge Distillation on Nonparametric Multivariate Functional EstimationAs machine learning models and datasets continue to grow, developing complex models has become increasingly computationally demanding. Knowledge distillation reduces deployment cost by compressing a large, well-trained teacher model into a compact student model, but it does not address settings where constructing the teacher itself is the bottleneck. Motivated by this challenge, we introduce Knowledge Cascade, a reverse knowledge distillation framework that uses information from a small, inexpenLuyang Fang, Haoran Lu, Yongkai Chen, Wenxuan Zhong, Ping Ma
- 153Inference with non-differentiable surrogate loss in a general high-dimensional classification frameworkPenalized empirical risk minimization with a surrogate loss function is often used to learn a high-dimensional linear decision rule in classification problems. Although much of the literature focus on the generalization error, there is a lack of inference procedures for identifying the driving factors of the estimated decision rule, especially when the surrogate loss is non-differentiable. We propose a kernel-smoothed decorrelated score to construct hypothesis tests and interval estimators for aMuxuan Liang, Yang Ning, Maureen A Smith, Ying-Qi Zhao
- 154A Functional-Space Mean-Field Theory of Partially-Trained Three-Layer Neural NetworksTo understand the training dynamics of neural networks, prior studies have considered the mean-field (MF) limit of two-layer NNs as the width tends to infinity, establishing theoretical guarantees for its convergence under gradient flow training as well as approximation and generalization capabilities. In this work, we study the infinite-width limit of a type of three-layer neural network where the first-layer weights are untrained. To rigorously define the limiting model, we extend the MF theorZhengdao Chen, Eric Vanden-Eijnden, Joan Bruna
- 155The Role of Contextual Information in Best Arm IdentificationWe study the best-arm identification problem with fixed confidence when contextual (covariate) information is available in stochastic bandits. In each round, we observe contextual information before selecting an arm. The distribution of the reward associated with the selected arm depends on the observed contextual information. We are interested in finding the arm with the maximum mean reward marginalized over the contextual distribution and not the mean reward conditioned on contexts. Our goal iMasahiro Kato, Kaito Ariu
- 156Transformers Can Overcome the Curse of Dimensionality: A Theoretical Study from an Approximation PerspectiveThe Transformer model is widely used in various application areas of machine learning, such as natural language processing. This paper investigates the approximation of the Hölder continuous function class $\mathcal{H}_{Q}^{\beta}\left([0,1]^{d\times n},\mathbb{R}^{d\times n}\right)$ by Transformers and constructs several Transformers that can overcome the curse of dimensionality. These Transformers consist of one self-attention layer with one head and the softmax function as the activation funcYuling Jiao, Yanming Lai, Yang Wang, Bokai Yan
- 157Online Bernstein-von Mises theoremOnline learning is an inferential paradigm in which parameters are updated incrementally from sequentially available data, in contrast to batch learning, where the entire dataset is processed at once. In this paper, we assume that mini-batches from the full dataset become available sequentially. The Bayesian framework, which updates beliefs about unknown parameters after observing each mini-batch, is naturally suited for online learning. At each step, we update the posterior distribution using tJeyong Lee, Junhyeok Choi, Minwoo Chae
- 158Covariate-dependent Hierarchical Dirichlet ProcessesBayesian hierarchical modeling is a natural framework to effectively integrate data and borrow information across groups. In this paper, we address problems related to density estimation and identifying clusters across related groups, by proposing a hierarchical Bayesian approach that incorporates additional covariate information. To achieve flexibility, our approach builds on ideas from Bayesian nonparametrics, combining the hierarchical Dirichlet process with dependent Dirichlet processes. TheHuizi Zhang, Sara Wade, Natalia Bochkina
- 159DCatalyst: A Unified Accelerated Framework for Decentralized OptimizationWe study decentralized optimization over a network of agents, modeled as an undirected graph and operating without a central server. The objective is to minimize a composite function $f+r$, where $f$ is a (strongly) convex function representing the average of the agents' losses, and $r$ is a convex, extended-value function (regularizer). We introduce DCatalyst, a unified black-box framework that injects Nesterov-type acceleration into decentralized optimization algorithms. At its core, DCatalystTIanyu Cao, Xiaokai Chen, Gesualdo Scutari
- 160Boosted Control Functions: Distribution Generalization and Invariance in Confounded ModelsModern machine learning methods and the availability of large-scale data have significantly advanced our ability to predict target quantities from large sets of covariates. However, these methods often struggle under distributional shifts, particularly in the presence of hidden confounding. While the impact of hidden confounding is well-studied in causal effect estimation, e.g., instrumental variables, its implications for prediction tasks under shifting distributions remain underexplored. ThisNicola Gnecco, Jonas Peters, Sebastian Engelke, Niklas Pfister
- 161Contrasting Local and Global Modeling with Machine Learning and Satellite Data: A Case Study Estimating Tree Canopy Height in African SavannasWhile advances in machine learning with satellite imagery (SatML) are facilitating environmental monitoring at a global scale, developing SatML models that are accurate and useful for local regions remains critical to understanding and acting on an ever-changing planet. As increasing attention and resources are being devoted to training SatML models with global data, it is important to understand when improvements in global models will make it easier to train or fine-tune models that are accuratEsther Rolf, Lucia Gordon, Milind Tambe, Andrew Davies
- 162A Symplectic Analysis of Alternating Mirror DescentMotivated by understanding the behavior of the Alternating Mirror Descent (AMD) algorithm for bilinear zero-sum games, we study the discretization of continuous-time Hamiltonian flow via the symplectic Euler method. We provide a framework for analysis using results from Hamiltonian dynamics and symplectic numerical integrators, with an emphasis on the existence and properties of a conserved quantity, the modified Hamiltonian (MH), for the symplectic Euler method. We compute the MH in closed-formJonas E. Katona, Xiuyuan Wang, Andre Wibisono
- 163Two-way Node Popularity Model for Directed and Bipartite NetworksThere has been increasing research attention on community detection in directed and bipartite networks. However, these studies often fail to consider the popularity of nodes in different communities, which is a common phenomenon in real-world networks. To address this issue, we propose a new probabilistic framework called the Two-Way Node Popularity Model (TNPM). The TNPM also accommodates edges from different distributions within a general sub-Gaussian family. We introduce the Delete-One-MethodBing-Yi Jing, Ting Li, Jiangzhou Wang, Ya Wang
- 164Convergence and complexity of block majorization-minimization for constrained block-Riemannian optimizationBlock majorization-minimization (BMM) is a simple iterative algorithm for nonconvex optimization that sequentially minimizes a majorizing surrogate of the objective function in each block coordinate while the other block coordinates are held fixed. We consider a family of BMM algorithms for minimizing nonsmooth nonconvex objectives, where each parameter block is constrained within a subset of a Riemannian manifold. We establish that this algorithm converges asymptotically to the set of stationarYuchen Li, Laura Balzano, Deanna Needell, Hanbaek Lyu
- 165Bayesian Inference of Contextual Bandit Policies via Empirical LikelihoodPolicy inference plays an essential role in the contextual bandit problem. In this paper, we use empirical likelihood to develop a Bayesian inference method for the joint analysis of multiple contextual bandit policies in finite sample regimes. The proposed inference method is robust to small sample sizes and is able to provide accurate uncertainty measurements for policy value evaluation. In addition, it allows for flexible inferences on policy comparison with full uncertainty quantification. WJiangrong Ouyang, Mingming Gong, Howard Bondell
- 166A causal fused lasso for interpretable heterogeneous treatment effects estimationWe propose a novel method for estimating heterogeneous treatment effects based on the fused lasso. By first ordering samples based on the propensity or prognostic score, we match units from the treatment and control groups. We then run the fused lasso to obtain piecewise constant treatment effects with respect to the ordering defined by the score. Similar to the existing methods based on discretizing the score, our methods yield interpretable subgroup effects. However, existing methods fixed theOscar Hernan Madrid Padilla, Yanzhen Chen, Carlos Misael Madrid Padilla, Gabriel Ruiz
- 167Unsupervised Feature Selection via Nonnegative Orthogonal Constrained Regularized MinimizationUnsupervised feature selection has drawn wide attention in the era of big data, since it serves as a fundamental technique for dimensionality reduction. However, many existing unsupervised feature selection models and solution methods are primarily designed for practical applications, and often lack rigorous theoretical support, such as convergence guarantees. In this paper, we first establish a novel unsupervised feature selection model based on regularized minimization with nonnegative orthogoYan Li, Defeng Sun, Liping Zhang
- 168Reparameterized Complex-valued Neurons Can Efficiently Learn More than Real-valued Neurons via Gradient DescentComplex-valued neural networks potentially possess better representations and performance than real-valued counterparts when dealing with some complicated tasks such as acoustic analysis, radar image classification, etc. Despite empirical successes, it remains unknown theoretically when and to what extent complex-valued neural networks outperform real-valued ones. We take one step in this direction by comparing the learnability of real-valued neurons and complex-valued neurons via gradient desceJin-Hui Wu, Shao-Qun Zhang, Yuan Jiang, Zhi-Hua Zhou
- 169Hierarchical Causal ModelsCausal questions often arise in settings where data are hierarchical: subunits are nested within units. Consider students in schools, cells in patients, or cities in states. In these settings, unit-level variables (e.g., a school's budget) may affect subunit-level outcomes (e.g., student test scores), and subunit-level characteristics may aggregate to influence unit-level outcomes. In this paper, we show how to analyze hierarchical data for causal inference. We introduce hierarchical causal modeEli N. Weinstein, David M. Blei
- 170Optimizing Attention with Mirror Descent: Generalized Max-Margin Token SelectionAttention mechanisms have revolutionized several domains of artificial intelligence, such as natural language processing and computer vision, by enabling models to selectively focus on relevant parts of the input data. While recent work has characterized the optimization dynamics of gradient descent (GD) in attention-based models and the structural properties of its preferred solutions, less is known about more general optimization algorithms such as mirror descent (MD). In this paper, we investAddison Kristanto Julistiono, Davoud Ataee Tarzanagh, Navid Azizan
- 171Adaptive Forward Stepwise: A Method for High Sparsity RegressionThis paper proposes a sparse regression method that continuously interpolates between Forward Stepwise selection (FS) and the LASSO. When tuned appropriately, our solutions are much sparser than typical LASSO fits but, unlike FS fits, benefit from the stabilizing effect of shrinkage. Our method, Adaptive Forward Stepwise Regression (AFS) addresses the need for sparser models with shrinkage. We show its connection with boosting via a soft-thresholding viewpoint and demonstrate the ease of adaptinIvy Zhang, Robert Tibshirani
- 172Optimization and Generalization of Gradient Descent for Shallow ReLU Networks with Minimal WidthUnderstanding the generalization and optimization of neural networks is a longstanding problem in modern learning theory. The prior analysis often leads to risk bounds of order $1/\sqrt{n}$ for ReLU networks, where $n$ is the sample size. In this paper, we present a general optimization and generalization analysis for gradient descent applied to shallow ReLU networks. We develop convergence rates of the order $1/T$ for gradient descent with $T$ iterations, and show that the gradient descent iterYunwen Lei, Puyu Wang, Yiming Ying, Ding-Xuan Zhou
- 173Finite Neural Networks as Mixtures of Gaussian Processes: From Provable Error Bounds to Prior SelectionInfinitely wide or deep neural networks (NNs) with independent and identically distributed (i.i.d.) parameters have been shown to be equivalent to Gaussian processes. Because of the favorable properties of Gaussian processes, this equivalence is commonly employed to analyze neural networks and has led to various breakthroughs over the years. However, neural networks and Gaussian processes are equivalent only in the limit; in the finite case there are currently no methods available to approximateSteven Adams, Andrea Patanè, Morteza Lahijanian, Luca Laurenti
- 174CHANI: Correlation-based Hawkes Aggregation of Neurons with bio-InspirationThe present work aims at proving mathematically that a neural network inspired by biology can learn a classification task thanks to local transformations only. In this purpose, we propose a spiking neural network named CHANI (Correlation-based Hawkes Aggregation of Neurons with bio-Inspiration), whose neurons activity is modeled by Hawkes processes. Synaptic weights are updated thanks to an expert aggregation algorithm, providing a local and simple learning rule. We were able to prove that our nSophie Jaffard, Samuel Vaiter, Patricia Reynaud-Bouret
- 175Persistence Diagrams Estimation of Multivariate Piecewise H{\"o}lder-continuous SignalsTo our knowledge, the analysis of convergence rates for persistence diagrams estimation from noisy signals has predominantly relied on lifting signal estimation results through sup-norm (or other functional norm) stability theorems. We believe that moving forward from this approach can lead to considerable gains. We illustrate it in the setting of nonparametric regression. From a minimax perspective, we examine the inference of persistence diagrams (for the sublevel sets filtration). We show thaHugo Henneuse
- 176Exploring Novel Uncertainty Quantification through Forward Intensity Function ModelingPredicting future time-to-event outcomes is a foundational task in statistical learning. While various methods exist for generating point predictions, quantifying the associated uncertainties poses a more substantial challenge. In this study, we introduce an innovative approach specifically designed to address this challenge, accommodating dynamic predictors that may manifest as stochastic processes. Our investigation harnesses the forward intensity function in a novel way, providing a fresh perYudong Wang, Zhi-Sheng Ye, Cheng Yong Tang
- 177Generative Bayesian Inference with GANsIn the absence of explicit or tractable likelihoods, Bayesians often resort to approximate Bayesian computation (ABC) for inference. Our work bridges ABC with deep neural implicit samplers based on generative adversarial networks (GANs) and adversarial variational Bayes. Both ABC and GANs compare aspects of observed and fake data to simulate from posteriors and likelihoods, respectively. We develop a Bayesian GAN (B-GAN) sampler that directly targets the posterior by solving an adversarial optimYuexi Wang, Veronika Rockova
- 178Communication-efficient Distributed Statistical Inference for Massive Data with Heterogeneous Auxiliary InformationHeterogeneous auxiliary information commonly arises in big data due to diverse study settings and privacy constraints. Excluding such indirect evidence often results in a substantial loss of statistical inference efficiency. This article proposes a novel framework for integrating a mixture of individual-level data and multiple external heterogeneous summary statistics by multiplying likelihood functions and confidence densities. Theoretically, we show that the proposed method possesses desirableMiaomiao Yu, Zhongfeng Jiang, Jiaxuan Li, Yong Zhou
- 179Decorrelated Local Linear Estimator: Inference for Non-linear Effects in High-dimensional Additive ModelsAdditive models play an essential role in studying non-linear relationships. Despite many recent advances in estimation, there is a lack of methods and theories for inference in high-dimensional additive models, including confidence interval construction and hypothesis testing. Motivated by inference for non-linear treatment effects, we consider the high-dimensional additive model and make inferences for the function derivative. We propose a novel decorrelated local linear estimator and establisZijian Guo, Wei Yuan, Cunhui Zhang
- 180Refined Risk Bounds for Unbounded Losses via Transductive PriorsWe revisit the sequential variants of linear regression with the squared loss, classification problems with hinge loss, and logistic regression, all characterized by unbounded losses in the setup where no assumptions are made on the magnitude of design vectors and the norm of the optimal vector of parameters. The key distinction from existing results lies in our assumption that the set of design vectors is known in advance (though their order is not), a setup sometimes referred to as transductivJian Qian, Alexander Rakhlin, Nikita Zhivotovskiy
- 181A Common Interface for Automatic DifferentiationFor scientific machine learning tasks with a lot of custom code, picking the right Automatic Differentiation (AD) system matters. Our Julia package DifferentiationInterface.jl provides a common frontend to a dozen AD backends, unlocking easy comparison and modular development. In particular, its built-in preparation mechanism leverages the strengths of each backend by amortizing one-time computations. This is key to enabling sophisticated features like sparsity handling without putting additionaGuillaume Dalle, Adrian Hill
- 182LazyDINO: Fast, Scalable, and Efficiently Amortized Bayesian Inversion via Structure-Exploiting and Surrogate-Driven Measure TransportWe present LazyDINO, a transport map variational inference method for fast, scalable, and efficiently amortized solutions of high-dimensional nonlinear Bayesian inverse problems with expensive parameter-to-observable (PtO) maps. Our method consists of an offline phase, in which we construct a derivative-informed neural surrogate of the PtO map using joint samples of the PtO map and its Jacobian as training data. During the online phase, when given observational data, we rapidly approximate the pLianghao Cao, Joshua Chen, Michael Brennan, Thomas O'Leary-Roseberry, Youssef Marzouk, Omar Ghattas
- 183The Distribution of Ridgeless Least Squares InterpolatorsThe Ridgeless minimum $\ell_2$-norm interpolator in overparametrized linear regression has attracted considerable attention in recent years in both machine learning and statistics communities. While it seems to defy conventional wisdom that overfitting leads to poor prediction, recent theoretical research on its $\ell_2$-type risks reveals that its norm minimizing property induces an `implicit regularization' that helps prediction in spite of interpolation. This paper takes a further step that aQiyang Han, Xiaocong Xu
- 184Nonparametric Estimation of a Factorizable Density using Diffusion ModelsIn recent years, diffusion models, and more generally score-based deep generative models, have achieved remarkable success in various applications, including image and audio generation. In this paper, we view diffusion models as an implicit approach to nonparametric density estimation and study them within a statistical framework to analyze their surprising performance. A key challenge in high-dimensional statistical inference is leveraging low-dimensional structures inherent in the data to mitiHyeok Kyu Kwon, Dongha Kim, Ilsang Ohn, Minwoo Chae
- 185Learning Bayesian Network Classifiers to Minimize Class Variable ParametersThis study proposes and evaluates a novel Bayesian network classifier which can asymptotically estimate the true probability distribution of the class variable with the fewest class variable parameters among all structures for which the class variable has no parent. Moreover, to search for an optimal structure of the proposed classifier, we propose (1) a depth-first search based method and (2) an integer programming based method. The proposed methods are guaranteed to obtain the true probabilityShouta Sugahara, Koya Kato, James Cussens, Maomi Ueno
- 186Simulation-based Calibration of Uncertainty Intervals under Approximate Bayesian EstimationThe mean field variational Bayes (VB) algorithm implemented in Stan is relatively fast and efficient, making it feasible to produce model-estimated official statistics on a rapid timeline. Yet, while consistent point estimates of parameters are achieved for continuous data models, the mean field approximation often produces inaccurate uncertainty quantification to the extent that parameters are correlated a posteriori. In this paper, we propose a simulation procedure that calibrates uncertaintyTerrance D. Savitsky, Julie Gershunskaya
- 187An Anytime Algorithm for Good Arm IdentificationIn good arm identification (GAI), the goal is to identify one arm whose average performance exceeds a given threshold, referred to as a good arm, if it exists. Few works have studied GAI in the fixed-budget setting when the sampling budget is fixed beforehand, or in the anytime setting, when a recommendation can be asked at any time. We propose APGAI, an anytime and parameter-free sampling rule for GAI in stochastic bandits. APGAI can be straightforwardly used in fixed-confidence and fixed-budgeMarc Jourdan, Andrée Delahaye-Duriez, Clémence Réda
- 188Extrapolated Markov Chain Oversampling Method for Imbalanced Text ClassificationText classification is the task of automatically assigning text documents correct labels from a predefined set of categories. In real-life (text) classification tasks, observations and misclassification costs are often unevenly distributed between the classes - known as the problem of imbalanced data. Synthetic oversampling is a popular approach to imbalanced classification. The idea is to generate synthetic observations in the minority class to balance the classes in the training set. Many geneAleksi Avela, Pauliina Ilmonen
- 189Neural Network Parameter-optimization of Gaussian Pre-marginalized Directed Acyclic GraphsFinding the parameters of a latent variable causal model is central to causal inference and causal identification. In this article, we show that existing graphical structures that are used in causal inference are not stable under marginalization of Gaussian Bayesian networks, and present a graphical structure that faithfully represents margins of Gaussian Bayesian networks. We present the first duality between parameter optimization of a latent variable model and training a feed-forward neural nMehrzad Saremi
- 190Flexible Functional Treatment Effect EstimationWe study treatment effect estimation with functional treatments where the average potential outcome functional is a function of functions, in contrast to continuous treatment effect estimation where the target is a function of real numbers. By considering a flexible scalar-on-function marginal structural model, a weight-modified kernel ridge regression (WMKRR) is adopted for estimation. The weights are constructed by directly minimizing the uniform balancing error resulting from a decompositionJiayi Wang, Raymond K. W. Wong, Xiaoke Zhang, Kwun Chuen Gary Chan
- 191Error Analysis for Deep ReLU Feedforward Density-Ratio Estimation with Bregman DivergenceWe consider the problem of density-ratio estimation using Bregman Divergence with Deep ReLU feedforward neural networks (BDD). We establish non-asymptotic error bounds for BDD density-ratio estimators, which are minimax optimal up to a logarithmic factor when the data distribution has finite support. As an application of our theoretical findings, we propose an estimator for the KL-divergence that is asymptotically normal, leveraging our convergence results for the deep density-ratio estimator anSiming Zheng, Guohao Shen, Yuanyuan Lin, Jian Huang
- 192A Reinforcement Learning Approach in Multi-Phase Second-Price Auction DesignWe study reserve price optimization in multi-phase second price auctions, where the seller's prior actions affect the bidders' later valuations through a Markov Decision Process (MDP). Compared to the bandit setting in existing works, the setting in ours involves three challenges. First, from the seller's perspective, we need to efficiently explore the environment in the presence of potentially untruthful bidders who aim to manipulate the seller's policy. Second, we want to minimize the seller'sRui Ai, Boxiang Lyu, Zhaoran Wang, Zhuoran Yang, Michael I. Jordan
- 193UQLM: A Python Package for Uncertainty Quantification in Large Language ModelsHallucinations, defined as instances where Large Language Models (LLMs) generate false or misleading content, pose a significant challenge that impacts the safety and trust of downstream applications. We introduce UQLM, a Python package for LLM hallucination detection using state-of-the-art uncertainty quantification (UQ) techniques. This toolkit offers a suite of UQ-based scorers that compute response-level confidence scores ranging from 0 to 1. This library provides an off-the-shelf solution fDylan Bouchard, Mohit Singh Chauhan, David Skarbrevik, Ho-Kyeong Ra, Viren Bajaj, Zeya Ahmad
- 194Nonlinear function-on-function regression by RKHSWe propose a nonlinear function-on-function regression model where both the covariate and the response are random functions. The nonlinear regression is carried out in two steps: we first construct Hilbert spaces to accommodate the functional covariate and the functional response, and then build a second-layer Hilbert space for the covariate to capture nonlinearity. The second-layer space is assumed to be a reproducing kernel Hilbert space, which is generated by a positive definite kernel determPeijun Sang, Bing Li
- 195Nonlocal Techniques for the Analysis of Deep ReLU Neural Network ApproximationsIn recent work concerned with the approximation and expressive powers of deep neural networks, Daubechies, DeVore, Foucart, Hanin, and Petrova introduced a system of piecewise linear functions, which can be easily reproduced by artificial neural networks with the ReLU activation function, and showed that it forms a Riesz basis of $L_2([0, 1])$. Their work was subsequently generalized to the multivariate setting by Schneider and Vybíral. In the work at hand, we show that this system serves as a RCornelia Schneider, Mario Ullrich, Jan Vybíral
- 196A Data-Augmented Contrastive Learning Approach to Nonparametric Density EstimationIn this paper, we introduce a data-augmented nonparametric noise contrastive estimation method to density estimation using deep neural networks. By leveraging the idea of contrastive learning, our density estimator exhibits efficiency with a one-step and simulation-free evaluation process, imposes no constraints on the neural network, and is shown to be consistent and asymptotically automatically normalized. A novel data augmentation procedure allows us to mitigate the influence of the choice ofChenghao Li, Yuanyuan Lin
- 197Guaranteed Nonconvex Low-Rank Tensor Estimation via Scaled Gradient DescentTensors, which give a faithful and effective representation to deliver the intrinsic structure of multi-dimensional data, play a crucial role in an increasing number of signal processing and machine learning problems. However, tensor data are often accompanied by arbitrary signal corruptions, including missing entries and sparse noise. A fundamental challenge is to reliably extract the meaningful information from corrupted tensor data in a statistically and computationally efficient manner. ThisTong Wu
- 198skwdro: a library for Wasserstein distributionally robust machine learningWe present skwdro, a Python library for training robust machine learning models. The library is based on distributionally robust optimization using Wasserstein distances, popular in optimal transport and machine learnings. The goal of the library is to make the training of robust models easier for a wide audience by proposing a wrapper for PyTorch modules, enabling model loss' robustification with minimal code changes. It comes along with scikit-learn compatible estimators for some popular objecFlorian Vincent, Waïss Azizian, Franck Iutzeler, Jérôme Malick
- 199Extending Mean-Field Variational Inference via Entropic Regularization: Theory and ComputationVariational inference (VI) has emerged as a popular method for approximate inference for high-dimensional Bayesian models. In this paper, we propose a novel VI method that extends the naive mean field via entropic regularization, referred to as $\Xi$-variational inference ($\Xi$-VI). $\Xi$-VI has a close connection to the entropic optimal transport problem and benefits from the computationally efficient Sinkhorn algorithm. We show that $\Xi$-variational posteriors effectively recover the true poBohan Wu, David M. Blei
- 200Stochastic Gradient Methods: Bias, Stability and GeneralizationRecent developments of stochastic optimization often suggest biased gradient estimators to improve either the robustness, communication efficiency or computational speed. Representative biased stochastic gradient methods (BSGMs) include Zeroth-order stochastic gradient descent (SGD), Clipped-SGD and SGD with delayed gradients. The practical success of BSGMs motivates a lot of convergence analysis to explain their impressive training behaviour. As a comparison, there is far less work on their genShuang Zeng, Yunwen Lei
- 201Classification Under Local Differential Privacy with Model Reversal and Model AveragingLocal differential privacy has become a central topic in data privacy research, offering strong privacy guarantees by perturbing user data at the source and removing the need for a trusted curator. However, the noise introduced by local differential privacy often significantly reduces data utility. To address this issue, we reinterpret private learning under local differential privacy as a transfer learning problem, where the noisy data serve as the source domain and the unobserved clean data asCaihong Qin, Yang Bai
- 202Identifying Weight-Variant Latent Causal ModelsThe task of causal representation learning aims to uncover latent higher-level causal variables that affect lower-level observations. Identifying the true latent causal variables from observed data, while allowing instantaneous causal relations among latent variables, remains a challenge, however. To this end, we start with the analysis of three intrinsic indeterminacies in identifying latent variables from observations: transitivity, permutation indeterminacy, and scaling indeterminacy. We findYuhang Liu, Zhen Zhang, Dong Gong, Mingming Gong, Biwei Huang, Anton van den Hengel, Kun Zhang, Javen Qinfeng Shi
- 203Efficient frequent directions algorithms for approximate decomposition of matrices and higher-order tensorsIn the framework of the FD (frequent directions) algorithm, we first develop two efficient algorithms for low-rank matrix approximations under the embedding matrices composed of the product of any SpEmb (sparse embedding) matrix and any standard Gaussian matrix, or any SpEmb matrix and any SRHT (subsampled randomized Hadamard transform) matrix. The theoretical results are also achieved based on the bounds of singular values of standard Gaussian matrices and the theoretical results for SpEmb andMaolin Che, Yimin Wei, Hong Yan
- 204Online Detection of Changes in Moment--Based Projections: When to Retrain Deep Learners or Update Portfolios?Training deep learning neural networks often requires massive amounts of computational ressources. We propose to sequentially monitor network predictions to trigger retraining only if the predictions are no longer valid. This can reduce drastically computational costs and opens a door to green deep learning. Our approach is based on the relationship to projected second moments monitoring, a problem also arising in other areas such as computational finance. Various open-end as well as closed-endAnsgar Steland
- 205The surrogate Gibbs-posterior of a corrected stochastic MALA: Towards uncertainty quantification for neural networksMALA is a popular gradient-based Markov chain Monte Carlo method to access the Gibbs-posterior distribution. Stochastic MALA (sMALA) scales to large data sets, but changes the target distribution from the Gibbs-posterior to a surrogate posterior which only exploits a reduced sample size. We introduce a corrected stochastic MALA (csMALA) with a simple correction term for which distance between the resulting surrogate posterior and the original Gibbs-posterior decreases in the full sample size whiSebastian Bieringer, Gregor Kasieczka, Maximilian F. Steffen, Mathias Trabs