Back to the Core-Memory Age: Running Operating Systems in NVRAM only - Rabenstein, Jonas; Nguyen, Dustin Tien; Giersch, Oliver; Eichler, Christian; Hönig, Timo; Nolte, Jörg; Schröder-Preikschat, Wolfgang (Lecture Notes in Computer Science, 2023)
The classic core memory was completely non-volatile and thus kept at least part of the operating system persistently in main memory, even over power cycles. Nowadays we can repeat this approach with NVRAM, but with terabytes of main memory on a completely different scale and with parts of the operating-system state stored in volatile CPU caches. In this paper, we discuss our experiences of running large modern operating systems including their applications entirely in NVRAM. We adapted stock Linux and FreeBSD kernels to work exclusively with NVRAM by hiding all DRAM from the kernels at boot time to establish a realistic performance baseline without changing anything else. Following this entirely NVRAM-agnostic approach, we could observe an effective performance penalty of a factor of about four, but only negligible increases in whole-system power draw. For our system with two CPU sockets and 56 cores total, we also observed a reduction in power draw in several scenarios. Due to prolonged execution times, the energy consumption increased as well for these measured workloads. While this might be discouraging at first sight, this result was achieved without any performance tuning as to the specific characteristics of today’s NVRAM technology. Therefore, we are also discussing means to mitigate the observed shortcomings by integrating NVRAM appropriately into the memory hierarchy of future robust persistent systems.<br
CRIS-Team Scopus Importer:2023-09-29
Keywords: NVM-only; NVRAM; FreeBSD; Linux; whole-system persistence

Cocoon: Custom-fitted kernel compiled on demand - Heinloth, Bernhard; Ammon, Marco; Nguyen, Dustin Tien; Hönig, Timo; Sieh, Volkmar; Schröder-Preikschat, Wolfgang (PLOS 2019 - Proceedings of the 10th Workshop on Programming Languages and Operating Systems, Part of SOSP 2019, 2019)
As computer processors and their hardware designs continuously evolve, operating systems provide many different assembly-level implementations for the same functionality. This enables support for new platforms and ensures backward compatibility for older ones at the same time. However, the source code of operating systems grows more complex and becomes much harder to maintain. In this paper we explore ways to build made-to-measure system software by relegating work to the compiler which has necessary knowledge about the system at hand. We propose Cocoon, an approach for compiling a system-tailored and -optimized kernel at boot time. For two operating systems (i.e., Linux and FreeBSD) we demonstrate the soundness of the approach by hands of a prototypical implementation. The implementation shows various aspects of Cocoon, such as the ability to remove hard-to-maintain code while preserving and even increasing the system performance.
CRIS-Team Scopus Importer:2019-12-10

Honey, I Shrunk the Guests Page Access Tracking using a Minimal Virtualisation Layer - Nguyen, Dustin Tien; Rußer, Sebastian; Ott, Maximilian; Kapitza, Rüdiger; Schröder-Preikschat, Wolfgang; Nolte, Jörg (4th Workshop on Heterogeneous Composable and Disaggregated Systems, 2025)
Advances in memory-intensive areas such as machine learning and graph processing drive demand for suited memory technologies. Part of the answer lies in multi-memory architectures with fast and high-capacity memory tiers. Heterogeneous systems can be very diverse, ranging from NUMA to persistent memory and CXL-attached memory. Efficient usage requires support from the operating system (OS) with knowledge of memory performance characteristics and detailed insight into the page access behaviour of processes. Gathering statistics on the read/write-ratio of accesses and (huge) page utilisation is expensive with common methods. In this paper, we propose vmmload, a minimal virtualisation layer to monitor user space programs using a virtualisation-based page modification log (PML) for efficiently gathering runtime statistics. While PML is a hardware extension that was originally designed to track virtual machine memory access, vmmload enables observation of individual processes. vmmload monitored user space processes run with PML-enabled virtualisation, but are still able to interface with the host OS. We explore the viability of using PML without a conventional virtual machine and providing the host OS with the ability to use these extensions directl

NVall: A Crash-Resistant and Kernel-Compatible Memory Allocator for NVRAM - Nguyen, Dustin Tien; Wiedemann, Ole; Nolte, Jörg; Schröder-Preikschat, Wolfgang (FG-BS Herbsttreffens 2023, 2023)
Byte-addressable non-volatile memory is essentially persistent, but slower main memory that needs to be managed accordingly. Typical memory allocators for volatile memory are highly efficient today, but usually never had to be designed to keep their state in main memory consistent at all times against the background of system crashes. In this paper we present NVall, a crash-resistant kernel-level memory allocator for non-volatile RAM (NVRAM). The allocator works in a transactional manner, uses existing volatile memory to improve the performance of normal operation and is able to recover its volatile state from persistent data after a system crash. We implemented the allocator for the FreeBSD kernel and compare its performance against the standard (non-crash-resistant) in-kernel allocator of FreeBS
Keywords: Operating System; NVRAM; Memory; Storage; FreeBSD

On the Performance of NVRAM-based Operating Systems: A Case Study with Linux and FreeBSD - Rabenstein, Jonas; Nguyen, Dustin Tien; Giersch, Oliver; Eichler, Christian; Hönig, Timo; Nolte, Jörg; Schröder-Preikschat, Wolfgang (2023)
Classical core memory was entirely non- volatile and could keep at least part of the operating system (OS) in main memory even across power cycles. These days we can have terabytes of NVRAM to repeat this approach, albeit on an entirely different scale and with large parts of the OS state still kept in the volatile CPU caches. In this paper, we discuss our experiences of running large modern operating systems including their applications entirely in NVRAM. We adapted stock Linux and FreeBSD kernels to work exclusively with NVRAM by hiding all DRAM from the kernels at boot time to establish a realistic performance baseline without changing anything else. Following this entirely NVRAM-agnostic approach, we could observe an effective performance penalty of a factor of about four, but only negligible increases in whole-system power draw. For our system with two CPU sockets and 56 cores total, we also observed a reduction in power draw in several scenarios. Due to prolonged execution times, the energy consumption increased as well for these measured workloads. While this might be discouraging at first sight, this result was achieved without any performance tuning as to the specific characteristics of today’s NVRAM technology. Therefore, we are also discussing means to mitigate the observed shortcomings by integrating NVRAM appropriately into the memory hierarchy of future robust persistent systems.<br
Keywords: NVRAM; operating systems; FreeBSD; Linux

Virtual Memory Revisited for Tiered Memory - Giersch, Oliver; Nguyen, Dustin Tien; Nolte, Jörg; Schröder-Preikschat, Wolfgang (APSys 2024 - Proceedings of the 15th ACM SIGOPS Asia-Pacific Workshop on Systems, 2024)
The landscape of memory technologies has grown substantially more diverse and heterogeneous, with a number of special-purpose memory technologies reaching market maturity in recent years, such as NVRAM (Non-Volatile Memory), HBM (High-Bandwidth Memory) or the upcoming CXL.mem. Future computing systems will have to contend with byte-addressable and cache-coherent main memory that is highly diverse, either in terms of the fundamental performance characteristics of its underlying memory technology, or due to the fact that CXL.mem will allow accessing homogenous but remote memory. The VM (Virtual Memory) subsystems of today have been empirically optimized over decades, but were designed for a single tier of main memory only. While recent approaches that integrated tiered memory have focused on better memory access metrics to improve page placement strategies, we seek to leverage the proven heuristics that are already in place. In times of high load we assign pages from slower memory, demote inactive pages according to standard page replacement strategies and use lazy page promotions when sufficient resources are available again to retain the inherently high reactivity of demand paging with regard to working set changes. We integrate these mechanisms into the VM subsystem of a full-featured mainstream operating system (FreeBSD) with minimal adaptations. The resulting multi-tier VM system runs in a stable and efficient manner under high stress even with highly unfavorable work loads.
CRIS-Team Scopus Importer:2024-10-04
Keywords: Operating Systems; Resource disaggregation

Where are the Joules? Energy Demand Analysis of Heterogeneous Memory Technologies - Preisner, Thomas; Nguyen, Dustin Tien; Vögele, Manuel; Szymanski, Matthias; Hönig, Timo; Kapitza, Rüdiger; Schröder-Preikschat, Wolfgang (4th Workshop on Heterogeneous Composable and Disaggregated Systems, 2025)
New heterogeneous memory architectures are emerging with the increasing availability of different memory technologies. The growing number of byte-addressable memory technologies (e.g., NVRAM and CXL) make the selection of components for concrete systems increasingly difficult, as each memory technology has specific advantages and disadvantages. Currently, decisions are made based on a variety of performance measurements and recommendations with respective implementation guidelines. However, traditional performance metrics often overlook the unique energy-demand dynamics of each specific memory technology, especially with diverse memory access patterns and workloads. This paper highlights the importance of memory utilization with different technologies and their environmental impact (i.e., energy demand). To this end, we present meBench – a benchmark generator that systematically quantifies the (non-uniform) energy demand of different memory types. We reveal the relevance of the generated results by contrasting them with the assumptions made in previous research on carbon-aware memory allocatio
Keywords: Memory Technologies; Energy Consumption; Benchmarking; Heterogeneous Systems; DRAM; Non-Volatile Memory; Optane; CXL