Dependable Non-Volatile Memory

Non-volatile memory or, more precisely, persistent memory, is a new type of byte-addressable memory that maintains its state without a constant power supply. In the first instance DRAM based solutions that are backed up by a super-cap battery like the Viking DDR4 NVDIMM-N were deployed in data centers. Recently, however, products based on novel technologies like phase change memory (PCM) also hit the market. Scientific results indicate that PCM can achieve a 25x to 100x higher access rate compared to NAND FLASH, paired with a 2x-4x higher density than DRAM. With this characteristics, persistent memory solutions not only raise the model of persistence but also bridge the gap between DRAM and FLASH memory. This enables manifold solutions like libraries for direct access, file-systems or databases that outperform existing solutions by orders of magnitude.

With the currently available persistent memory solutions, only a portion of the main memory can be persistent, leaving the remaining memory, the CPU and all buffers on the main board still volatile. A hybrid system like this, has its state distributed across volatile and non-volatile parts. In the case of a power failure the volatile part of the state is lost and the remaining persistent part may be left inconsistent and consequently not useful anymore. To prevent inconsistencies in the persistent state, current solutions maintain logs and frequently enforce durability on the volatile state parts. This, however, not only decreases performance but also increases wear-out and the probability of faults on PCM-based persistent memory.

Additionally, data neither in DRAM nor PCM is safe from transient faults. For DRAM the environmental radiation is the main cause of errors. Early studies indicate that approximately 8% of all DIMMS are affected by errors per year. Recent results even show that common hardware protected DRAM is poorly suited for many enterprise data centers. Although for PCM there are no large scale studies yet, early reports indicate that, due to the short term resistance drift, data written to PCM cells will not always be stored correctly. To achieve a dependability similar to good DRAM devices, an 8-bit error correction combined with scrubbing is needed if no other dependability measures are applied.

Our framework for Dependable Non-Volatile Memory (DNV Memory) addresses the lack of efficiency while tolerating power failures and provides transparent fault tolerance for persistent data.

Concepts

In order to create persistent data as easy as volatile data, DNV Memory provides direct access to persistent memory through a C/C++ interface that is similar to malloc and free. Static persistent variables can also be created by using annotations. Finally, transaction semantics are provided for preserving consistency of persistent data. The code snippet below shows a function that uses our API to attach new elements to the head of a persistent linked list:

    DNV_OBJ node_t* head_;
    push_front (widged_t widged) { 
        __transaction_atomic { 
            node_t* node = dnv_malloc(sizeof(node_t));
            node->payload = widged;
            node->next = head_;
            head_ = node;
        }
    }
    

Based on our API, DNV Memory maintains the efficiency of persistent memory by enforcing Durability on Demand, that means only when a power outage is imminent. Furthermore, we introduce Reliable Transactions which transparently protect all persistent data in the user-space from transient faults. Finally our architecture is lightweight and does neither introduces extensive modifications to the OS kernel nor requires hardware changes. Our Dependable Driver is the only component inside the kernel space that uses persistent data, therefore the reliable computing base is kept at a minimum. To be dependable, our driver supports atomic state transitions and fault tolerance.

Durability on Demand

To keep persistent data consistent in the presence of a power failure, our framework provides transactional semantics (see __transaction_atomic {...} in the example above). However, other than previous work, our software transactions are using volatile logs and do not enforce durability on every commit. Only when a power failure is imminent, the transaction logs are copied to persistent memory followed by enforcing durability.

This approach requires the ability to detect power outages in advance, which we do with the help of a hardware power failure detector (PFD). This simple device is plugged into the same power source as the machine. In the case of a voltage drop at the power source, the PFD emits an interrupt to the affected machine. Exploiting the energy that is stored within the power supply, the machine can continue running for a short time to copy the log to persistent memory and enforce durability of persistent data.

Albeit our PFD is an external hardware device, its functionality can be easily integrated into a power supply, making the deployment of Durability on Demand seamless for the end-user.

In summary, Durability on Demand improves performance up to a factor of 3.5X. Additionally, due to reduction of writes, systems with phase change memory benefit from reduced wear-out and transient fault probability.

Reliable Transactions

Since persistent data typically stores valuable information, it can be seen as critical. However, neither DRAM nor PCM-based persistent memory is safe from transient faults. The absence of a trivial solution for recovering from errors via restarts makes a dedicated fault tolerance solution all the more important. With Reliable Transactions, DNV Memory provides a transparent and flexible approach for tolerating data corruptions in persistent data. Within a transaction, our write operations store an additional error correcting code (ECC) word in persistent memory while read operations are augmented with a data verification step. In order to provide a strong multi-bit error correction, paired with a low error detection overhead, we use an engineered approach. For a fast detection of errors we utilize CRC32 (D) since it is commonly implemented in hardware on many CPU architectures. Although CRC32 provides a respectable hamming distance of 8, for data up to 177 bit, it cannot correct errors on its own. We augment, therefore, the error detection with an error correction hint (C). This provides information about the location and the number of errors. However, the error correction hint, is imprecise, thus, error correction is eventually executed by trial and error, using C and D to reduce the amount of trials.

Efficiency and Programming Support

We store the original data and its ECC-Words (E) in an interleaved data layout to optimize for cache efficiency. However, with this layout, any access without transaction may corrupt persistent data. Since accessing persistent data without transactions is inherently unsafe in case of power failures. We add programming support, that prevents unintentional access of persistent data outside a transaction. For this reason two virtual memory sections are introduced. The actual persistent data, interleaved with its ECC, is store in the TxAccess section. Access to this section is obscured from the user because our API never externalizes pointers to this section. Instead, all pointers externalized by DNV Memory are pointing to the TxStaging section, which is mapped without any access rights. That way, accessing persistent memory without using transactions leads to segmentation faults that are easy to debug.

Architecture

DNV Memory is build for systems research on persistent memory. Therefore we follow a lightweight, modular design consisting of a user-space library and a driver in the kernel space. No modifications to the compiler, the OS kernel or the hardware is needed. That way DNV Memory can be deployed easily on any Linux system with direct access to persistent memory. DNV Memory can even be used without persistent memory at all. However, the persistent data will not survive power failures in that case.

Another goal of our architecture is the holistic protection of persistent data. In the user space protection is applied through our Reliable Transactions. For protecting the persistent data that is only accessible by the Dependable Driver itself, we follow a tailored approach. This combines the idea of versioning with redundant data layout for optimized performance and minimal dependencies.

Project members at IBR

Prof. Dr. Rüdiger Kapitza
Abteilungsleiter
kapitza[[at]]ibr.cs.tu-bs.de
+49 531 3913294
Room 135

Marc Kastner
Hiwi
Projekt DanceOS

Phil Lindow
Hiwi
Projekt danceOS

Rouven Scholz
Hiwi
Cloud Computing

Arthur Martens
Ehemaliger Wissenschaftlicher Mitarbeiter
martens[[at]]ibr.cs.tu-bs.de

Publications

• Arthur Martens, Rouven Scholz, Phil Lindow, Niklas Lehnfeld, Marc A. Kastner and Rüdiger Kapitza: Dependable Non-Volatile Memory, in Proceedings of the 11th ACM International Systems and Storage Conference, SYSTOR '18, Haifa, Israel, ACM, 2018 (martens2018systor, DOI, BibTeX)
• Arthur Martens, Rouven Scholz, Phil Lindow, Marc A. Kastner and Rüdiger Kapitza: DeNoVo malloc: Validating Data in Persistent Memory, in 12th European Conference on Computer Systems (EuroSys '17), Poster Session, Belgrade, Serbia, April 2017 (martens2017denovomalloc, BibTeX)

DNV Memory is part of danceOS and partially supported under the priority program SPP1500 grant no. KA 3171/2-3

Student Theses

If you are interested in writing a thesis regarding this project, please feel free to contact us.