History

The Human Mortality Database (HMD) Project began in autumn 2000 at the initiative of John R. Wilmoth, at the time Professor at the University of California at Berkeley (UCB), and James W. Vaupel, at the time Director of the Max Planck Institute for Demographic Research (MPIDR) in Rostock (Germany). It was established as a bilateral scientific project to be jointly developed by researchers at the University of California, Berkeley (Department of Demography) and the Max-Planck Institute for Demographic Research (MPIDR).

Original motivation for the HMD

From the beginning, the HMD was thought of as a major data resource to foster research on the longevity revolution that resulted in a greater increase in life expectancy during the last 100 years than during all preceding human history. It was especially important to make sure that the HMD enabled demographers to investigate the progress in longevity that began in the late 1970s by an unexpected and robust mortality decline at old and very old ages. This, in turn, led to questioning former speculations about the limits to human longevity and population aging.

Demographers address this fundamental question by an in-depth analysis of mortality including survival to advanced ages. Thus, the mortality and population data for such investigation had to include ages beyond 100 for as many countries as possible, be reliable and precise even at advanced ages, be consistent across time and space, cover long periods without gaps and ruptures, and provide fine details in terms of age, time, and birth cohort. In 2000, none of the existing international databases met all these criteria. HMD successfully filled this challenging gap.

Methodological background

Fortunately, the HMD project benefited from the important scientific and practical groundwork carried out in the 1990s. In 1994-96, Väinö Kannisto produced two seminal books documenting the mortality decline at ages 80 and older in developed countries. Together with Roger Thatcher and James W. Vaupel, he estimated survival, proposed data quality tests, and developed methods for the reliable re-estimation of populations at ages 80+. This methodology was pioneered by the Kannisto-Thatcher Database on Old Age Mortality (KTD) that was founded in 1993 at Odense University Medical School in Denmark by K.Andreev, V.Kannisto, R.Thatcher under the supervision of J.W.Vaupel. The KTD has been maintained at MPIDR since 1996. The KTD focused on survival ages 80 and older.

Precursors of the HMD

In 1997, John R. Wilmoth at the University of California, Berkeley, launched the Berkeley Mortality Database (BMD) that provided mortality estimates by single-year of age from birth up to age 110 for France, Japan, Sweden, and the United States. The HMD adopted data presentation formats and several calculation methods from the BMD. At the same time, HMD methods for ages 80+ were strongly influenced by the KTD. This way, HMD combined the best of these earlier efforts into a single project. It was anticipated that the database would eventually include around 40 to 45 countries with reliable vital registration and censuses systems.

Organizing principles

During the preparation phase that lasted from October 2000 to May 2002, the HMD Team developed the first version of the HMD Methods Protocol that involved numerous calculations and data checks and established data formats, data processing, and data checking procedures.

During this initial phase, the workflow and precise responsibilities of each team member were carefully established. It was decided that a member of the HMD team would be designated as the country specialist for each HMD country. Country specialists establish contacts with in-country experts (typically at the national statistical office or in the academic community), carry out all systematic steps involved in assessing the reliability of the input data, verify the results of calculations, and update existing country series. For all formal checks and computations, a set of uniform methods and visuals and corresponding MATLAB and SAS or R codes are applied. All the methods were described in detail in the HMD Methods Protocol.

By mid-Spring 2002, the newly developed HMD “machinery” had been applied to 17 country series. With this initial set of countries and the newly designed HMD website, the HMD was launched online in May 2002 and presented at the PAA meeting.

A continually improving project

In addition to the regular updates of country-specific data series, we also continue to work on improving the HMD methods. There have been six successive versions of the HMD Methods Protocol and numerous modifications to the data processing routines. However, these adjustments and enhancements have never fundamentally altered the initial data structures and methods.

At present, the HMD is one of the most used and cited data resources in demography. It has become a common reference in the social sciences, statistics, public health, econometrics, biometrics, gerontology, epidemiology, actuarial science, and other disciplines. Since the 2010s, HMD has been extensively used by the UN Population Division as both a direct data source for updating statistical series in the HMD countries and also an analog for the model life tables.

Due to very strict data quality requirements, the HMD is limited to countries with good quality population data. Therefore, countries and areas in the HMD are relatively wealthy industrialized countries. To (partly) compensate for this shortcoming, the HMD Team has built a companion database named the Human Life-Table Database (HLD). The HLD includes life tables for many national and sub-national populations from both developed and developing countries. The HLD life tables originate from official and non-official publications and are very diverse in the methods implemented and the data quality. The HLD was launched in June 2002.

Adding the weekly mortality data series to the HMD

In response to the COVID-19 pandemic, the HMD team decided to establish a new data resource: Short-term Mortality Fluctuations (STMF@HMD) data series. Objective and internationally comparable data are crucial to determine the effectiveness of different strategies used to address epidemics. Weekly death counts provide the most objective and comparable way of assessing the scale of short-term mortality elevations across countries and time. More details about this data project can be found in our paper

Adding cause-of-death information to the HMD

In the latest development, it was decided to integrate Human Cause-of-Death Database (HCD) in HMD, to allow users immediate access to all mortality data, including medical causes of death. The original HCD was built on the foundation provided by long-term research conducted at INED by Jacques Vallin and France Meslé. These two researchers were the first to ever attempt to reconstruct coherent historical time series of deaths by cause. Until then, it had been impossible to follow long term changes in the detailed cause-of-death structure of mortality for any country, as successive changes in the International Classification of Diseases (ICD) in use by most of the HMD countries resulted in sometimes dramatic disruptions in the statistical series.

A number of projects drew from the work initially conducted at INED and demonstrated the applicability of the approach developed by Jacques Vallin and France Meslé to other countries with high quality cause-of-death data, even among those not using the ICD (e.g., those using the USSR classification). Eventually, this collective effort led to an international project funded first by a joint French and German initiative (ANR-DFG, 2007), then by the AXA Research Fund, and then again by the ANR-DFG program. The project resulted in the production of coherent series with a classification of all deaths into the ICD-10 scheme spanning from the late 1970s or early 1980s (depending on the country) to the most recently available data for 16 countries. The reconstructed series were were published in the Human Cause-of-Death Database in March 2016. The HCD data series were integrated into the HMD in 2024.

The conception of the HMD was greatly inspired and supported by James W. Vaupel (the MPIDR Director at the time). In 2000-2014, the HMD project was jointly led by John R. Wilmoth (at UCB at the time), the founding Director of the HMD, and Vladimir M. Shkolnikov, the HMD Co-Director. In 2014-2021, Vladimir M. Shkolnikov served as the HMD Director with Magali Barbieri (UCB and INED) as the HMD Associate Director. Since 2022, the HMD project has been headed by two Co-Directors - Dmitry A. Jdanov (MPIDR) and Magali Barbieri (UCB).