The paper argues that workload traces observed in industrial settings represent a negotiation between the data platform and the platform’s users. Users mold their queries to run well on the platform, and engineers tune the platform to better meet user demands. This cycle, while great for both users and data platforms, creates a survivorship bias in the observed workloads: the most frequently processed queries are precisely the queries that are already working well.

The paper’s authors include Ryan Marcus (assistant professor), Jeffrey Tao (PhD student), Peizhi Wu (PhD student alumnus), and Zijie Zhao (PhD student).

CausalMesh: A Causal Cache for Stateful Serverless Computing: Haoran Zhang (University of Pennsylvania); Shuai Mu (Stony Brook University); Sebastian Angel (University of Pennsylvania); Vincent Liu (University of Pennsylvania). In stateful serverless computing, workflows are broken into functions that may run on different physical machines, each with its own local cache. This distribution can lead to consistency errors, where one function reads stale data from its cache because a previous function in the same workflow wrote an update to a different machine’s cache. To solve this, researchers at Penn and Stony Brook developed CausalMesh, a novel caching system that guarantees “causal consistency,” ensuring operations are seen in a logical, cause-and-effect order across all machines. A key innovation of CausalMesh is that it provides this guarantee for most read/write operations without requiring costly coordination between servers or aborting transactions. As a result, CausalMesh delivers lower latency and higher throughput than existing approaches, enabling faster and more reliable state management in serverless applications.

A Practical Theory of Generalization in Selectivity Learning: Peizhi Wu (University of Pennsylvania), Haoshu Xu (University of Pennsylvania), Ryan Marcus (University of Pennsylvania), Zack Ives (University of Pennsylvania). This research provides a theoretical understanding of machine learning models used for query optimization in databases. While these models perform well in practice, there has been a significant gap in explaining why they work, especially when they encounter new or different queries (“out-of-distribution” or OOD) than those they were trained on. The paper bridges this gap by establishing the first theoretical guarantees for how these models generalize to OOD queries. Based on these new insights, the authors developed practical strategies that significantly improve the accuracy and real-world performance of existing models on unseen query types, making them more robust and reliable without sacrificing their original performance.

Holistic query Approximation via RL Modeling. Susan Davidson (University of Pennsylvania), Tova Milo (Tel Aviv University), Kathy Razmadze (Tel Aviv University), Gal Zeevi (Tel Aviv University). To accelerate slow queries during data exploration on large databases, researchers at Tel Aviv University and Penn have developed HARLM, a novel system for approximate query processing. While existing methods speed up aggregate queries (like COUNT or AVG) by using data samples, they fail to support non-aggregate queries that retrieve specific rows. HARLM presents a holistic solution by using Reinforcement Learning to identify an optimized, smaller subset of the data that works for both query types. This approach effectively learns to create a representative data sample that maximizes query accuracy while dramatically reducing execution time. Experiments show that HARLM significantly outperforms baseline methods, improving result accuracy by 30% and providing a 10-35x speedup.

SHARQ: Explainability Framework for Association Rules on Relational Data: Hadar Ben‑Efraim (Bar-Ilan University), Susan B. Davidson (University of Pennsylvania), Amit Somech (Bar-Ilan University). Association rule mining is a widely used technique for discovering patterns (e.g., “customers who buy X also buy Y”) in large datasets. However, a major challenge has been to quantify the actual importance of an individual data element, like “X,” to the entire set of rules it participates in. This paper introduces SHARQ, a novel method that uses Shapley values, a concept from cooperative game theory, to fairly and accurately measure the contribution of each element. While a naive calculation would be exponentially slow, the researchers developed highly efficient algorithms that compute this score in near-linear time. This breakthrough makes it practical to rank data elements, entire rules, and even attributes by their influence, providing a powerful new tool for explaining and gaining deeper insights from mined data.

Data-Agnostic Cardinality Learning from Imperfect Workloads: Peizhi Wu (University of Pennsylvania), Rong Kang (ByteDance);Tieying Zhang (Bytedance), Jianjun Chen (Bytedance), Ryan Marcus (University of Pennsylvania), Zack Ives (University of Pennsylvania). The authors, at Bytedance and Penn, have developed a new system called GRASP for cardinality estimation, a crucial task in database query optimization. Traditional methods need direct access to data, which is often restricted, while existing learning-based approaches struggle with the incomplete and imbalanced query workloads found in real-world scenarios. GRASP is a data-agnostic system specifically designed for these imperfect conditions. It uses a novel compositional design that allows it to generalize to new queries and is robust to skewed training data. By effectively modeling data distributions and join correlations without seeing the underlying data, GRASP consistently outperforms other query-driven models and, remarkably, can even match or exceed the accuracy of traditional methods that have full data access.

(AIDB Workshop) Exploring Wavelet Trees as Space-Efficient Physical-to-Sorted Mapping for Learned Indexes. Anwesha Saha (Boston University), Aneesh Raman (Boston University), Ryan Marcus (University of Pennsylvania), Manos Athanassoulis (Boston University). This paper explores Wavelet Trees as a compact way to map data between its physical and sorted order for learned indexes, which use machine learning models to replace traditional B+-tree nodes. The authors introduce Integer Wavelet Trees (IWTs), which significantly reduce memory usage—up to 84% less than B+-trees—but initially suffer from slow lookups due to cache inefficiencies. To address this, they propose T-way IWTs, which improve lookup speed while maintaining space efficiency, achieving 46% smaller memory footprints and 12% faster lookups compared to B+-trees. This study lays the groundwork for future designs, including their new idea of constellation maps, aimed at balancing speed and memory for learned index mappings. This paper was a best paper honorable mention at the workshop!

At the aiDM workshop, co-chaired by our own Ryan Marcus, there are two papers:

• SERAG: Self-Evolving RAG System for Query Optimization, Hanwen Liu, Qihan Zhang, University of Southern California, Ryan Marcus, University of Pennsylvania, Ibrahim Sabek, University of Southern California.
• Data-driven Adaptive Processing of Streaming ML Queries, by Phillip Hilliard, Rajeev Alur, Zachary Ives, University of Pennsylvania. This paper describes an adaptive query processing technique targeted at stream systems that incorporate machine learning components. When given a set of alternative machine learning models with different cost-accuracy trade-offs, it dynamically chooses the model that maximizes accuracy while satisfying a budgetary or quality-of-service constraint.

At the main SIGMOD conference, the following papers will be presented.

Low Rank Learning for Offline Query Optimization
Zixuan Yi (University of Pennsylvania)*; Yao Tian (The Hong Kong University of Science and Technology); Zack Ives (University of Pennsylvania); Ryan Marcus (University of Pennsylvania).

This paper develops a novel technique based on low-rank matrix factorization, which allows a query optimizer to predict which query processing strategies will be useful for one query, based on performance of other queries.

Learned Offline Query Planning via Bayesian Optimization
Jeffrey Tao; Natalie Maus; Haydn Jones; Yimeng Zeng; Jacob Gardner; Ryan Marcus.

Targeting queries that are going to be executed thousands of times, we propose an offline query optimizer that searches a wide variety of plans and incorporates query execution as a primitive. Our offline query optimizer combines variational auto-encoders with Bayesian optimization to find optimized plans for a given query.

• SHARQ: Explainability Framework for Association Rules on Relational Data
  Hadar Ben Efraim (Bar-Ilan University); Susan B Davidson (University of Pennsylvania); Amit Somech (Bar-Ilan University)*. Association rules are an important technique for gaining insights over large relational datasets. However, it is difficult to explain the relative importance of data elements with respect to the rules in which they appear. This paper develops a measure of an element’s contribution to a set of association rules based on Shapley values, denoted SHARQ (ShApley Rules Quantification).
• Physical Visualization Design: Decoupling Interface and System Design
  Yiru Chen (Columbia University)*; Xupeng Li (Columbia University); Jeffrey Tao (University of Pennsylvania); Alana Ramjit (Cornell Tech); Ravi Netravali (Princeton University); Subrata Mitra (Adobe Research); Aditya Parameswaran (University of California, Berkeley); Javad Ghaderi (Columbia University); Dan Rubenstein (Columbia University); Eugene Wu (Columbia University)
• CARINA: An Efficient CXL-Oriented Embedding Serving System for Recommendation Models
  Peiqi Yin (The Chinese University of Hong Kong)*; Qihui Zhou (CUHK); Xiao Yan (Centre for Perceptual and Interactive Intelligence (CPII) ); Chao Wang (The Chinese University of Hong Kong); Eric Lo (Chinese University of Hong Kong); Changji Li (CUHK); Lan Lu (University of Pennsylvania ); Hua Fan (Alibaba Cloud); Wenchao Zhou (Alibaba Group); Ming-Chang YANG (The Chinese University of Hong Kong); James Cheng (CUHK)

At the demo sessions:

ScaleLLM: A technique for scalable LLM-augmented data systems.

Paul Loh (University of Pennsylvania); Ashwin Alaparthi (University of Pennsylvania); Ryan Marcus (University of Pennsylvania);

• PY-SHARQ: A Holistic Python Library for Explaining Association Rules on Relational Data
  Hadar Ben-Efraim (Bar-Ilan University), Susan Davidson (University of Pennsylvania), Amit Somech (Bar-Ilan University)

We hope to see you in Berlin!

Penn will again be well-represented, with 3 papers in the Research Track:

In Towards Full Stack Adaptivity in Permissioned Blockchains, PhD student Chenyuan Wu, postdoc alumnus Mohammad Javad Amiri (Stony Brook University), undergrad student Haoyun Qin (Class of 2025), PhD student Bhavana Mehta, and Profs. Ryan Marcus and Boon Thau Loo study the problem of supporting a (virtual) distributed database with untrusted components — using a learning-based approach. This paper articulates a vision for a learning-based untrustworthy distributed database. We focus on permissioned blockchain systems as an emerging instance of untrustworthy distributed databases and argue that as novel smart contracts, modern hardware, and new cloud platforms arise, future-proof permissioned blockchain systems need to be designed with full-stack adaptivity in mind. At the application level, a future-proof system must adaptively learn the best-performing transaction processing paradigm and quickly adapt to new hardware and unanticipated workload changes on the fly. Likewise, the Byzantine consensus layer must dynamically adjust itself to the workloads, faulty conditions, and network configuration while maintaining compatibility with the transaction processing paradigm. At the infrastructure level, cloud providers must enable cross-layer adaptation, which identifies performance bottlenecks and possible attacks, and determines at runtime the degree of resource disaggregation that best meets application requirements. Within this vision of the future, the paper outlines several research challenges together with some preliminary approaches.

In Relational Query Synthesis ⋈︁ Decision Tree Learning, PhD student Aaditya Naik, PhD alumnus Aalok Thakkar (Ashoka University), PhD student Adam Stein, and Profs. Rajeev Alur and Mayur Naik address the problem of supporting synthesis of SQL queries and consider its interaction with machine learning. They study the problem of synthesizing select-project-join (SPJ) queries from input-output examples. Search-based synthesis techniques are suited to synthesizing projections and joins by navigating the network of relational tables but require additional supervision for synthesizing comparison predicates. On the other hand, decision tree learning techniques are suited to synthesizing comparison predicates when the input database can be summarized as a single labelled relational table. In this paper, they adapt and interleave methods from the domains of relational query synthesis and decision tree learning, and present an end-to-end framework for synthesizing relational queries with categorical and numerical comparison predicates. Their technique guarantees the completeness of the synthesis procedure and strongly encourages minimality of the synthesized program. They present Libra, an implementation of this technique and evaluate it on a benchmark suite of 1,475 instances of queries over 159 databases with multiple tables. Libra solves 1,361 of these instances in an average of 59 seconds per instance. It outperforms state-of-the-art program synthesis tools Scythe and PatSQL in terms of both the running time and the quality of the synthesized programs.

In Searching Data Lakes for Nested and Joined Data, PhD alumnus Yi Zhang (AWS) and undergrad alumnus Peter Chen (PhD student at MIT) and Prof. Zachary Ives consider how to perform search for hierarchical (JSON, Pandas dataframe) or joined data, within a data lake of data that has been indexed in first-normal-form. Exploratory data science is driving new data management platforms that assist data scientists with common tasks, such as the integration and wrangling steps required to assemble training datasets. Such tools take the data scientists’ work-in-progress data as a search object (table or JSON), and find relevant supplementary data from an organizational data lake, which can be unioned or joined with the current data – adding instances or features. Existing data lake search tools seek to find single, relational tables at a time — to match or join with a search table. Yet many data science applications revolve around finding matches to hierarchical data, which can only be matched by creating views simultaneously joining and transforming several tables in the data lake. In this paper, they extend the Juneau data lake search system to search for this broader class of matches at scale. Their contribution is a general framework for efficiently merging ranked results, leveraging novel techniques for indexing and sketching, and incorporating existing single-table search techniques and ranking functions. They experimentally validate the benefits of their methods and their broad applicability using real data from data science computational notebooks. Their results indicate that, with respect to different ranking functions, their approach can return the optimal set of views up to 4.81x faster and 43% more related compared to heuristics baselines, and increase the data domain coverage by up to 28%. As a case study to show the usability of their augmentation to data science downstream tasks, their methods can reduce the regression error by up to 6.63%, and improve the classification accuracy by up to 19.5% for ML models.

Ph.D. student Soonbo Han (advisor: Zachary Ives) presented his work titled “Implementation Strategies for Views over Property Graphs,” which won the best paper award! Soonbo’s work shows query rewriting techniques can take advantage of semantic views over graph data, including how to index and maintain such views dynamically.

Ph.D. student Jiaming Liang (advisor: Zachary Ives) presented his work titled “RITA: Group Attention is All You Need for Timeseries Analytics.” Jiaming’s work shows how careful grouping and caching of semantically-similar inputs can accelerate neural attention mechanisms, allowing attention networks to scale up to previously-impossible tasks.

Due to visa issues, Zack presented a paper from Ph.D. student Peizhi Wu, titled “Modeling Shifting Workloads for Learned Database Systems.” Peizhi’s work shows how to keep learned database components up to date with data drift using a carefully-tuned replay buffer.

Visiting Ph.D. student Yao Tian (advisor: Xiaofang Zhou, Penn supervisors: Zachary Ives and Ryan Marcus) presented her work titled “A Learned Cuckoo Filter for Approximate Membership Queries over Variable-sized Sliding Windows on Data Streams.” Yao’s work combines traditional Cuckoo filters with deep learning models to enable approximate membership queries over dynamically-sized windows, achieving significantly higher accuracy than previous results.

At the aiDM workshop, Ph.D. student Zixuan Yi (advisor: Ryan Marcus and Zachary Ives) presented her work titled “Low Rank Approximation for Learned Query Optimization.” Zixuan’s work shows how linear methods for approximating low rank matrices can be used to learn to steer an entire query workload at once.

Later this summer, Penn will present several papers at VLDB in Guangzhou, China!

Ph.D. student Chenyuan Wu (supervisor: Boon Thau Loo) presented work on FlexChain and AdaChain. FlexChain (PDF) is the first work to present a blockchain that disaggregates compute, storage, RAM, and cold storage to better serve diverse workloads. AdaChain (PDF) shows how reinforcement learning can be used to adaptively switch a blockchain system between different architectures, keeping up with workload or hardware changes.

AutoSteer (PDF), an extensible system for steering the query optimizer of any SQL database, was presented by collaborator and TUM Ph.D. student Christoph Anneser. AutoSteer is open source, and has been tested on Meta’s Presto cluster. The AutoSteer work was a collaboration between UPenn, Intel Labs, MIT, Meta, and TU Munich.

Ph.D. student Bhavana Mehta (supervisor: Boon Thau Loo) presented RLShard (PDF), a vision for a Byzantine fault tolerant, sharded, transactional database at the aiDB workshop. RLShard contains designs for both a fully decentralized, trustless system, as well as a semi-centralized version taking advantage of an administrative domain. Bhavana will continue developing both approaches as part of her dissertation.

Also at aiDB, assistant professor Ryan Marcus presented a vision for Learned Query Superoptimization (PDF), a set of new techniques to improve common repetitive queries in analytic database systems. Ryan plans to implement this vision in the coming years.

The Penn DB Group will see you at the next one!