Abstract

Between 1914 and 1919 hundreds of thousands of New Zealand men (and a small number of women) were enlisted to serve in the New Zealand Expeditionary Force to aid the allied forces in their fight against Germany. By analysing biographical information available from the Auckland War Memorial Museum Cenotaph database I will find out about those who served, especially the 18,000 casualties, including where, how, and when did they die, what patterns can be found in their deaths and what generalities can be made about who was more likely to live or die through the war.

Introduction

The First World War was one of the major events of the twentieth century. More than 120,000 New Zealanders enlisted and 18,000 died. My grandfather and his two brothers served in the NZEF and left behind a stack of letters written home during the war. These letters led to my interest in the experiences of those who served. My grandfather was wounded in the chest by shell fragments three days after entering the trenches rendering him unfit for further service (and perhaps saving him from a grim death in New Zealand’s deadliest battle). He lived to marry the widow of his brother, who had survived months as a water carrier near the front lines, only to return to New Zealand and die of influenza 5 years after the war ended. Their younger brother was a brilliant scholar whose recurring illnesses kept him away from the font lines leading to a job offer as an Army instructor, then a scholarship at Cambridge university. Through sheer good luck these three brothers lived through the war, but two of their cousins were not so lucky. One died of disease contracted at the Trentham training camp and was buried before leaving New Zealand. His younger brother was caught up in the battle of Passchendale and died from the effects of gas poisoning. Learning about the diverse stories of these men showed me that there was no “typical” war story. The reasons men went to war, the places they served and the experiences they had that led to their survival – or their death – was as varied as the men themselves.

The Data Source

The Auckland War Memorial Museum Cenotaph Database pulls together information about New Zealand’s service personnel from many different sources. The site was originally conceived to memorialise those who died at war, but has since expanded to include all service personnel who have since died. It purports to include “almost all of those who served in WWI”. The data is searchable through a front-end on the museum website (http://www.aucklandmuseum.com/war-memorial/online-cenotaph/search) where it is assembled into an individual web page for each person who served. It is also available in raw format through an API (http://api.aucklandmuseum.com/).

Workflow

Accessing the Data

The cenotaph data is accessible through an API using either a simple string search or a SPARQL query. As I am unfamiliar with the SPARQL query language (and since queries of this type have greater restrictions on use) I decided to use a simple search for the string “World War I, 1914-1918”. In theory this should be the value of the “war” field for all of those who served in World War One. In fact the search returned 96,203 results – short of the 103,188 results obtained using the web site’s front-end. I’m not sure why there is a discrepancy in this number (if anything I would expect a general string search to return more results than a search on a single field). The front-end doesn’t offer the full results of a search in a downloadable format, so I had to make do with the 96,203 results from the API. These were returned in JSON format. I fetched them in page sizes of 1000 and saved them as local JSON files (the code for this is contained in the notebook “Download JSON”).

Converting to CSV

The JSON structure returned was hierarchical, so the next step was to flatten these files so they could be loaded into a dataframe (this was carried out in the notebook “Import from JSON”). I used a convenient library called FlattenJSON to convert the lists inside the JSON file into individual fields. This created an extremely unwieldy number of columns (one for every potential value of a list), so at this stage I also filtered the number of columns down to a subset of that looked like they could be useful or interesting. I then saved these flattened tables as individual CSV files.

Data wrangling and cleaning

This was carried out and documented in the notebook “Data Wrangling”. I first assembled the individual CSV files into one dataframe, then went though each set of fields and determined the volume and quality of data, whether any cleaning or converting needed to take place and whether or not the data would potentially be useful for analysis. A number of field sets that had multiple values were separated out into “normalized” dataframes, reducing the number of redundant columns (which were often mostly filled with NaN values). This drastically reduced the overall memory requirement. This set of dataframes was then re-saved as CSV files, ready for analysis in this notebook.

Quality of the Data and Assumptions

Considering the dataset has been assembled from many varied sources, it is inevitable that the quality is extremely variable. Transcription errors are evident throughout the data and I have picked these up as I have gone as best as I can.

One big question I had about the data is why a reasonable chunk of records is completely missing. Out of around 120,000 who enlisted, only 73,000 or so records remain after the data wrangling/cleaning phase. While this is still more than enough to proceed with analysis, there is a question of which records are missing and why. If they represent a particular type of record this could easily invalidate some of the conclusions. However since I have no knowledge of why these records are missing I will proceed with the assumption that they are randomly distributed through the data. This assumption applies to any field I have analysed where there is a large amount of missing data. Often it is unclear whether missing data implies there is semantically no value for that field or whether the value is unknown.

The biggest disappointment is in the date of birth field, which has values for only 5,700 or so records. The age of soldiers would seem to be a defining characteristic of their experience at war and it is a pity that this is accessible for such a small proportion of the total.

The most complete part of the data are the death fields. Out of 18,000 who died because of the war 15,000 of these deaths are recorded in the data, with well used fields for place, age and cause of death. For this reason I chose to mostly focus on this aspect.

Main Analysis

Analysing ages

What is the distribution of the ages of the soldiers who died?

The age at death data is right skewed with a median of 26 and a mean of 27.4. Most of the the soldiers who died during the war were young men, the majority under thirty. A handful of outliers were older than their mid-forties.

Why were the men who died mostly young? It could be the case than younger are more reckless than older men – ready to face the action head-on. Older men could be more cautious and concerned with self-preservation.

However these ages may just reflect the ages of the soldiers as a whole. It makes sense that the majority of men who volunteered were quite young as they would have had more of a thirst for adventure and less responsibilities to tie them to home. Although conscription applied to men aged between 20 and 45, single men were the first to be conscripted, married men were not called up until October 1917 and men with children not until April 1818. (https://nzhistory.govt.nz/war/recruiting-and-conscription)

The oldest person to die in the war was in his sixties. Who was he and how did he die?

Ferdinand Campion Batchelor was a respected medical practitioner who served in Egypt and died on his return to New Zealand.

https://www.teara.govt.nz/en/1966/batchelor-ferdinand-campion

Is the distribution of the ages of men who died different from those who didn’t?

Since the date of death field can’t be compared between men who died and didn’t die during the war (obviously those who didn’t die during the war have no value for this field) I’ll instead compare the enlistment ages between the two groups.

Unfortunately there is much less data of ages of the men who didn’t die during the war so I’d first like make sure there is enough data for both groups to compare.

This is a small proportion of the total, but enough to be compare the two groups. I am assuming that records with this field are randomly distributed through the total population and not representative of a particular subset (though I am aware this assumption could be incorrect).

The summary statistics seem to indicate that men who died were indeed younger than those didn’t have a lower mean, median and quartile values. A boxplot and histogram will give a more visual representation of this information.

These plots show that in general men who died during the war were younger when they enlisted than those who lived through the war. The distribution of the ages of men who lived through the war has a longer tail and more outliers implying that older men were more likely to live through the war.

Is this because younger men are more reckless and have less regard for their own lives? Before coming to that conclusion, it is worth considering this is more depth. There could be other influencing factors on age at enlistment that lead to a higher death rate.

Age and Rank

A third variable that might have an effect on both age and death rate is rank. Presumably older men would hold higher ranks, and higher ranked men may be less likely to die (higher ranked officers may be less likely to be directly involved in battle).

To compare rank I will use the embarkation data, as this gives the most complete information on rank. I’ll compare the ages using the age at embarkation field. Each soldier may have multiple embarkations as some would have returned to New Zealand for a leave then embarked a second time (or more). I’ll look at the age of the soldier’s first embarkation, which generally occurred several months after their enlistment.

Not every soldier who died has embarkation data – some died during their training in New Zealand. We need to be aware that by using the embarkation fields we are actually excluding those who died before they left New Zealand.

The embarkations have already been ordered by date, so by removing the duplicates of the index (which is an index into the dataframe so represents individual records) we are left with each person’s first embarkation.

Looking at the rank field – how many separate ranks are there?

The ten most common ranks:

The rank entries look fairly consistent. There are 101 different ranks beings used but these are mostly related to different rank names in the different divisions. Since I want to be able to compare the ranks by their level I’ll only use New Zealand Army ranks (these are by far the most common), and assign each a rank order, grouping the four bottom ranks together then using ascending numbers for ascending ranks.

I’ll use an inner join to merge the first_embarkations with rank_levels – this will add rank order to first embarkations data frame and remove the extraneous ranks.

Is there a difference in age at embarkation between the different ranks?

From rank 1 up to rank 6 the ages are very similar, though there is a general trend of the minimum ages increasing and a slight increase in the medians. Rank 7 though 10 show a definite increase in age with the rank level with an increase in spread. The middle halves of the data significantly increase with each higher rank.

There is an unfortunate lack of data for the very highest ranks (only 1 data point for rank 11 – Major General, and none for 12 – Lieutenant General). My conclusion would be that age does generally increase with increase in rank – not as much in the lower ranks, but significantly from Captain upwards.

Is there a difference in age at death between the ranks?

This does in fact show a very similar pattern of increasing age with rank (excepting the first three ranks). This lends evidence to the theory that the rank of soldiers may be an important influence in the overall trend of age of death.

Does the death rate vary between the ranks?

A stereotype of war is of highly ranked men sending the lower ranked recruits into the fray while they wait the battle out in safety. If this is true we should see a reduction in death rate for higher ranked soldiers.

I’ll add another field called simply “died”, which is an integer casted from the diedDuringWar boolean field. This is an easy way of calculating death rates (the mean value is equivalent to death rate).

First I’ll examine the total numbers embarked of each rank compared to the numbers who died.

It is interesting to visualize the distributions of the different ranks, but because of the wildly different total counts this plot isn’t very useful in determining whether the death rates vary between the ranks.

Next I’ll plot death rates for each rank by averaging the “died” field.

There is no obvious trend in this plot. The death rates are very similar up to rank 4 (Sergeant). Surprisingly rank 5 and 6 (Second Lieutenant and Lieutenant) have higher death rates, then there is a marked decrease for rank 7 (Captain). Rank 8, 9 and 10 (Major, Lieutenant Colonel, Colonel) have very similar death rates though slightly lower than ranks 1-4.

It is worth noting that ranks up to 4 (Sergeant) are non-commissioned ranks, while 5 (Second Lieutenant) and above are commissioned officers. My understanding is that commissioned officers were professionals recruited into the Army while the non-commissioned officers were mostly volunteers or conscripted men who had been promoted to a higher rank through good conduct. This may explain why ranks 1 through 4 show a different pattern to 5 and above.

Second Lieutenant and Lieutenant (rank 5 and 6) are the lowest of the commissioned ranks – these men may have been responsible for leading companies into battle. It is interesting that their death rates are markedly higher than even the lowest ranked soldiers. This could indicate that these men took their leadership roles seriously and were willing to risk themselves for the men they were leading.

The higher ranked men probably had less of a hands-on role in battle, but these rates show that they were by no means completely removed from the dangers of war. These men probably would have also served for much longer than the volunteers of ranks 1-4 (many of who joined up mid-way through the war), so may have been more likely to be involved in a higher number of deadly battles and hence have more chance of being killed.

Is there any interaction between age, rank and probability of dying during the war?

First I’ll look at the three age fields together – age at enlistment, embarkation and death, dropping the records that don’t have both an age of enlistment and embarkation.

This clearly shows a pattern of increasing age to rank. The ages are fairly even up to rank 4 (Lance Corporal), then increase at rank 5 (Second Lieutenant) and decrease for rank 6 (Lieutenant). This decrease is unexpected and may indicate that there is something unusual about this rank that I don’t understand (one possibility is that outstanding leaders could have been promoted straight to this rank, short-cutting the lower ranks). As expected, the ages then climb steeply as the rank gets higher.

The dotted lines on this graph indicate the median age of men who died during the war. These are generally higher (with the exception of rank level 3 – Corporal) than for the men who didn’t die. This is a strong indication that rank does have an influence on the ages of men who died.

Looking at just the age at embarkation in more detail:

This pattern is slightly different to the plot above, probably reflecting the increase in amount of data (this data includes records that don’t have an enlistment age).

The main trend for men who died during the war is that they were generally older the higher their rank. However in ranks 5 and 6 (Second Lieutenant and Lieutenant) the men who lived were actually younger than those who died. These are the lowest of the commissioned ranks so many of these younger men may have been very inexperienced and given less responsibility, which kept them out of the main action.

We have seen that rank is strong influence on age and that men in higher ranks were slightly less likely to die. To answer the question of whether younger men were more likely to die because they were more reckless, my overall judgment is that that rank does explain some, but not all of the effect of age on death rate. It might be somewhat true that younger men were more reckless, but it is also somewhat true that they were generally lower ranked and hence more likely to be involved in conflict.

Date of embarkation and survival rate

What affect did the date of embarkation have on chances of surviving the war?

Another reason for the difference between the ages of those who died and didn’t die might be that the the men who enlisted earlier in the war were more likely to have died (simply because they were at war longer). I can test this by looking at boxplots of the years that the men were born (rather than their age).

These distributions are almost identical apart from a few outliers (representing much older men who didn’t die).

We can get an idea of the affect of enlisting early on the death rate by comparing the distributions of numbers of embarkations by date for the men who died and didn’t die during the war.

The distribution for the men who didn’t die reflects the embarkation distribution as a whole. Many men enlisted almost straight away and were sent away very early on in the war. This dropped off, then geared up again 1000 days into the war – this corresponds to late 1916, just after conscription was introduced.

The distribution for men who did die is much more right skewed. More of the men who died enlisted in the first half than the second half of the war.

Next I’ll look at death rates by date of embarkation by grouping the embarkation date and taking the mean of the “died” field.

Those first two dates are way before the war started, so I’ll delete them.

Resample into quarters to create a time series.

Death rates for those who embarked up until December 1916 were high, though variable. The middle of 1915 was an especially bad time to embark (astoundingly, over half died), but those who embaked towards the end of that year had a much healthier survival rate (20%). From December 1916 onwards the death rates started lowering. Less than 10% of those who embarked mid 1917 until the start of 1918 died. The peak in mid 1918 is surprising. By this time the war was coming to an end (the war ended 11 November 1918). Maybe these “fresh” troops were used extensively in the final push through France, replacing the troops who had already spent many months in the trenches.

So men who enlisted earlier were generally more likely to die than those who enlisted later, though the relationship between time served and date rate is not as clear cut as I would have anticipated. It seems that more factors were involved than just an increasing probability of being killed with each passing day.

Were younger men more reckless than older men?

If younger men were more reckless than older men we should see a pattern of younger men dying sooner after joining up.

To test this if this is the case I’ll create a field that records the number of days between first embarkation and date of death.

Examining the field to test for the validity of the data:

The minimum value of -300 indicates there are some data errors here. How many negative values are there?

Only 3 records have a negative number of days until death so proceed, but remove those values.

First I’ll look at the overall distribution of days from embarkation to death.

This is a right skewed distribution with a long right tail. The median of 333 days indicates that half of the troops in this data died within about 10 months of leaving New Zealand. However a quarter of those who died survied longer than 19 months (600 days). A handful survived longer than 4 years. The shape of this plot suggests this data could be a good fit for a gamma distribution.

Next I’ll plot the relationship between age at death and number of days from embarkation until death by using a scatter plot.

Visually, there does look to be general increasing trend of number of days to death and age, but this is probably related to the increasing minimum ages, which is actually just a consequence of the troops getting older the longer they have been at war. Plotting days to death against age at enlistment would show a clearer result.

In fact I’ll plot it against age at embarkation since there are more results for this field and the first embarkation date would be quite closely related to enlistment date. This would exclude deaths that happened before embarkation, but those deaths would have have been related to battle so I feel there is justification in excluding them.

First I’ll plot age at enlistment against age at embarkation to test my assumption that there is a direct relationship between these two fields.

As expected there is a very strong relationship here, with only a few outliers (some of which are obviously bad data since it is unlikely embarkation age would be younger than enlistment age).

There doesn’t seem to be any strong relationship between these two fields. There is certainly a high concentration of younger soldiers who were killed not long after their embarkation date, but this reflects the right skewness of both the ages and days to death data.

An improvement to this would be to only include deaths that occurred as a result of the fighting (excluding disease and sickness related deaths for example). I’ll come back to this later when I examine cause of death.

Place of death

What patterns are there in the places that casualties occured?

First I need to organise the place of death field into manageable categories.

The field is generally presented as a series of more specific localities like town/country or country/region. Unfortunately there looks to be some inconsistencies and generalisations (for instance just “France” with no more specific information for a large number of records).

I’ll split the field into the second most specific (“place”) and most specific (“region”) fields.

Set an “Unspecificed” value to the place fields where the placeOfDeath only gives a general region.

A categorical column will allow the places to be sorted by country order instead of name.

Find places that have a death count of more than 100, display in a table and plot.

Overall France had by far the highest number of deaths, followed by Belgium and Turkey. The three specific places that stand out as having a high number deaths are Ypres, Northern France and Gallipoli. These correspond with the three major campaigns of the war – Gallipoli, Flanders Fields and The Somme. It is perhaps surprising that a large number of deaths occurred in New Zealand (and in particular Wellington). The main training camp of Trentham was situated in Upper Hutt, so Wellington has probably been given as the location of the deaths that occurred here. I will explore this further later.

When and where did the most deaths occur?

I’ll use a strip plot to plot death by date in places where 100 or more deaths occurred.

This plot clearly shows the pattern of deaths according to the place they occurred, darker regions indicating times of higher concentrations of deaths. Long and deadly campaigns like Gallipoli (April 1915 – January 1916) and the Battle of the Somme (July to November 1916) can be clearly identified. The darkest splotches highlight the most intense conflicts and their length – the Battle of Messines (Ypres and Belguim, mid 1917), Battle of Passchendale (October 1917), Battle of Bapaume (August 1918) and the short but intense battles that liberated the French towns of Le Cateau, Le Quesnoy and Havrincourt. The pattern for France (with no specific place) shows almost continuous deaths from early in 1916, gradually dropping off, then heightening as the push was made through France in 1918. The deaths in New Zealand and Wellington show a gradual trickle of deaths becoming slightly more concentrated from the start to the finish of the war with a short burst late in 1918, which may relate to a disease outbreak. The deaths in the UK show several more concentrated patches that may also be disease outbreaks.

Cause of death

What is the distribution of cause of death?

A quick look at the Cause of Death field:

This data in this field very consistently entered.

I’m not sure about some of these categorisations though. Why is there a distinction between disease and sickness? What does Killed on Active Service mean if it’s not being killed in action or accidentally? What are natural causes if it’s not sickness or disease?

Let’s look at plot of the counts of the different causes of deaths.

Unsurprisingly Killed in Action was by far the most common way to die, followed by Died of wounds. However a surprising large number also died from sickness and disease as well as accidental deaths.

I’ll filter by the most common cause of death values in order to produce a cleaner plot against place of death.