The paper argues that workload traces observed in industrial settings represent a negotiation between the data platform and the platform’s users. Users mold their queries to run well on the platform, and engineers tune the platform to better meet user demands. This cycle, while great for both users and data platforms, creates a survivorship bias in the observed workloads: the most frequently processed queries are precisely the queries that are already working well.

The paper’s authors include Ryan Marcus (assistant professor), Jeffrey Tao (PhD student), Peizhi Wu (PhD student alumnus), and Zijie Zhao (PhD student).

Penn students presented five posters:

• Zhen Ping Khor presented “Adaptive load balancing in DAG-based consensus protocols” (link)
• Zijie Zhao presented “Low-Overhead Adaptive Execution via Reinforcement Learning with Bootstrapped Counterfactuals”
• Jeff Tao presented “Adversarial Benchmark Generation” (link)
• Phillip Hilliard presented “Accurate ML Processing under Real-Time Constraints” (link)
• Varun Jana presented “Semantic Indexing for Unstructured Data Management at Scale”

In addition to student posters, Ryan presented one of the three keynotes, “Next Steps for Learned Query Optimization” (slides).

Congratulations to Professor Ryan Marcus, for receiving a 2025 Google ML and Systems Junior Faculty Award. This is a recognition of both the promise and impact of Ryan’s work on using machine learning techniques to accelerate tasks in data management!

CausalMesh: A Causal Cache for Stateful Serverless Computing: Haoran Zhang (University of Pennsylvania); Shuai Mu (Stony Brook University); Sebastian Angel (University of Pennsylvania); Vincent Liu (University of Pennsylvania). In stateful serverless computing, workflows are broken into functions that may run on different physical machines, each with its own local cache. This distribution can lead to consistency errors, where one function reads stale data from its cache because a previous function in the same workflow wrote an update to a different machine’s cache. To solve this, researchers at Penn and Stony Brook developed CausalMesh, a novel caching system that guarantees “causal consistency,” ensuring operations are seen in a logical, cause-and-effect order across all machines. A key innovation of CausalMesh is that it provides this guarantee for most read/write operations without requiring costly coordination between servers or aborting transactions. As a result, CausalMesh delivers lower latency and higher throughput than existing approaches, enabling faster and more reliable state management in serverless applications.

A Practical Theory of Generalization in Selectivity Learning: Peizhi Wu (University of Pennsylvania), Haoshu Xu (University of Pennsylvania), Ryan Marcus (University of Pennsylvania), Zack Ives (University of Pennsylvania). This research provides a theoretical understanding of machine learning models used for query optimization in databases. While these models perform well in practice, there has been a significant gap in explaining why they work, especially when they encounter new or different queries (“out-of-distribution” or OOD) than those they were trained on. The paper bridges this gap by establishing the first theoretical guarantees for how these models generalize to OOD queries. Based on these new insights, the authors developed practical strategies that significantly improve the accuracy and real-world performance of existing models on unseen query types, making them more robust and reliable without sacrificing their original performance.

Holistic query Approximation via RL Modeling. Susan Davidson (University of Pennsylvania), Tova Milo (Tel Aviv University), Kathy Razmadze (Tel Aviv University), Gal Zeevi (Tel Aviv University). To accelerate slow queries during data exploration on large databases, researchers at Tel Aviv University and Penn have developed HARLM, a novel system for approximate query processing. While existing methods speed up aggregate queries (like COUNT or AVG) by using data samples, they fail to support non-aggregate queries that retrieve specific rows. HARLM presents a holistic solution by using Reinforcement Learning to identify an optimized, smaller subset of the data that works for both query types. This approach effectively learns to create a representative data sample that maximizes query accuracy while dramatically reducing execution time. Experiments show that HARLM significantly outperforms baseline methods, improving result accuracy by 30% and providing a 10-35x speedup.

SHARQ: Explainability Framework for Association Rules on Relational Data: Hadar Ben‑Efraim (Bar-Ilan University), Susan B. Davidson (University of Pennsylvania), Amit Somech (Bar-Ilan University). Association rule mining is a widely used technique for discovering patterns (e.g., “customers who buy X also buy Y”) in large datasets. However, a major challenge has been to quantify the actual importance of an individual data element, like “X,” to the entire set of rules it participates in. This paper introduces SHARQ, a novel method that uses Shapley values, a concept from cooperative game theory, to fairly and accurately measure the contribution of each element. While a naive calculation would be exponentially slow, the researchers developed highly efficient algorithms that compute this score in near-linear time. This breakthrough makes it practical to rank data elements, entire rules, and even attributes by their influence, providing a powerful new tool for explaining and gaining deeper insights from mined data.

Data-Agnostic Cardinality Learning from Imperfect Workloads: Peizhi Wu (University of Pennsylvania), Rong Kang (ByteDance);Tieying Zhang (Bytedance), Jianjun Chen (Bytedance), Ryan Marcus (University of Pennsylvania), Zack Ives (University of Pennsylvania). The authors, at Bytedance and Penn, have developed a new system called GRASP for cardinality estimation, a crucial task in database query optimization. Traditional methods need direct access to data, which is often restricted, while existing learning-based approaches struggle with the incomplete and imbalanced query workloads found in real-world scenarios. GRASP is a data-agnostic system specifically designed for these imperfect conditions. It uses a novel compositional design that allows it to generalize to new queries and is robust to skewed training data. By effectively modeling data distributions and join correlations without seeing the underlying data, GRASP consistently outperforms other query-driven models and, remarkably, can even match or exceed the accuracy of traditional methods that have full data access.

(AIDB Workshop) Exploring Wavelet Trees as Space-Efficient Physical-to-Sorted Mapping for Learned Indexes. Anwesha Saha (Boston University), Aneesh Raman (Boston University), Ryan Marcus (University of Pennsylvania), Manos Athanassoulis (Boston University). This paper explores Wavelet Trees as a compact way to map data between its physical and sorted order for learned indexes, which use machine learning models to replace traditional B+-tree nodes. The authors introduce Integer Wavelet Trees (IWTs), which significantly reduce memory usage—up to 84% less than B+-trees—but initially suffer from slow lookups due to cache inefficiencies. To address this, they propose T-way IWTs, which improve lookup speed while maintaining space efficiency, achieving 46% smaller memory footprints and 12% faster lookups compared to B+-trees. This study lays the groundwork for future designs, including their new idea of constellation maps, aimed at balancing speed and memory for learned index mappings. This paper was a best paper honorable mention at the workshop!

At the aiDM workshop, co-chaired by our own Ryan Marcus, there are two papers:

• SERAG: Self-Evolving RAG System for Query Optimization, Hanwen Liu, Qihan Zhang, University of Southern California, Ryan Marcus, University of Pennsylvania, Ibrahim Sabek, University of Southern California.
• Data-driven Adaptive Processing of Streaming ML Queries, by Phillip Hilliard, Rajeev Alur, Zachary Ives, University of Pennsylvania. This paper describes an adaptive query processing technique targeted at stream systems that incorporate machine learning components. When given a set of alternative machine learning models with different cost-accuracy trade-offs, it dynamically chooses the model that maximizes accuracy while satisfying a budgetary or quality-of-service constraint.

At the main SIGMOD conference, the following papers will be presented.

Low Rank Learning for Offline Query Optimization
Zixuan Yi (University of Pennsylvania)*; Yao Tian (The Hong Kong University of Science and Technology); Zack Ives (University of Pennsylvania); Ryan Marcus (University of Pennsylvania).

This paper develops a novel technique based on low-rank matrix factorization, which allows a query optimizer to predict which query processing strategies will be useful for one query, based on performance of other queries.

Learned Offline Query Planning via Bayesian Optimization
Jeffrey Tao; Natalie Maus; Haydn Jones; Yimeng Zeng; Jacob Gardner; Ryan Marcus.

Targeting queries that are going to be executed thousands of times, we propose an offline query optimizer that searches a wide variety of plans and incorporates query execution as a primitive. Our offline query optimizer combines variational auto-encoders with Bayesian optimization to find optimized plans for a given query.

• SHARQ: Explainability Framework for Association Rules on Relational Data
  Hadar Ben Efraim (Bar-Ilan University); Susan B Davidson (University of Pennsylvania); Amit Somech (Bar-Ilan University)*. Association rules are an important technique for gaining insights over large relational datasets. However, it is difficult to explain the relative importance of data elements with respect to the rules in which they appear. This paper develops a measure of an element’s contribution to a set of association rules based on Shapley values, denoted SHARQ (ShApley Rules Quantification).
• Physical Visualization Design: Decoupling Interface and System Design
  Yiru Chen (Columbia University)*; Xupeng Li (Columbia University); Jeffrey Tao (University of Pennsylvania); Alana Ramjit (Cornell Tech); Ravi Netravali (Princeton University); Subrata Mitra (Adobe Research); Aditya Parameswaran (University of California, Berkeley); Javad Ghaderi (Columbia University); Dan Rubenstein (Columbia University); Eugene Wu (Columbia University)
• CARINA: An Efficient CXL-Oriented Embedding Serving System for Recommendation Models
  Peiqi Yin (The Chinese University of Hong Kong)*; Qihui Zhou (CUHK); Xiao Yan (Centre for Perceptual and Interactive Intelligence (CPII) ); Chao Wang (The Chinese University of Hong Kong); Eric Lo (Chinese University of Hong Kong); Changji Li (CUHK); Lan Lu (University of Pennsylvania ); Hua Fan (Alibaba Cloud); Wenchao Zhou (Alibaba Group); Ming-Chang YANG (The Chinese University of Hong Kong); James Cheng (CUHK)

At the demo sessions:

ScaleLLM: A technique for scalable LLM-augmented data systems.

Paul Loh (University of Pennsylvania); Ashwin Alaparthi (University of Pennsylvania); Ryan Marcus (University of Pennsylvania);

• PY-SHARQ: A Holistic Python Library for Explaining Association Rules on Relational Data
  Hadar Ben-Efraim (Bar-Ilan University), Susan Davidson (University of Pennsylvania), Amit Somech (Bar-Ilan University)

We hope to see you in Berlin!

The AIRFoundry team is led by Professors Daeyeon Lee (Penn CBE), David Issadore (Penn BE), Zack Ives (Penn CIS), Drew Weissman (Penn Medicine), and Claribel Acevedo-Velez (U Puerto Rico); in collaboration with many investigators spanning Penn, Drexel, and U Puerto Rico.