A new OWL ontology has been released for the RDF markup of graves, human remains and commemorative monuments. It supports complex relationships between the remains, the grave and the different markers in use; recording the movement of remains is also possible as well as the linkage to the identity of the deceased.

The documentation is available here and the ontology in OWL format here. While the ontology is meant to support Muninn’s data, it can be used on a stand alone basis for other projects.

We looked at the messages being passed between the officers in charge of various companies as they battled at the front and the commanders some distance behind the lines. In total there are about 200 dispatches extracted from the war diary from about a dozen officers. We wanted to know: When crises intensify at the Front, do the officers in charge write less or not at all until the fighting slows? We speculated that this would be the case owing to the confusion and the increased need for hands-on leadership at such times.

We found that on the contrary, at the times of greatest activity, messages from the officers in charge at the front not only increased in number but in length at the times of greatest activity, when the troops and their officers were under fire or in other critical situations. These urgent messages tend to be an average of two sentences longer than other non-urgent messages.

The histogram shown here gives an idea of what is going on: messages get longer the more adverse front line conditions become. Using close reading techniques, we analyzed the long messages that our data-mining search had identified and drew the following conclusions about why these were written at the very times that we would have thought officers were far too busy to write them:

Front-line officers could not always follow the orders issued by commanders who were frequently working without up-to-date information about the situation on the ground. When HQ commands were not carried out at the Front, the officer in charge would write out a fuller explanation of his company’s actions than he would ordinarily make so as to explain decisions made by junior officers at the front. These officers must ask for permission after the fact: “I trust this meets with your approval.”

Furthermore, since the commanders lack crucial information, officers-in-charge need to convey that information thoroughly and accurately, which explains the details and time taken by company officers to write comprehensive dispatches to their superiors.

These longer memorandum also offer evidence of group decision making when confusion at the front is particularly acute: front line officers will make contact with other officers in charge of adjacent companies in order to establish a clearer picture of what is occurring at the front and to form a plan that they then forward back to their commanders: “I talked the matter over with Major Hudson of the RCR and he and his junior officers agreed with me that the trench was too crowded and that I should withdraw.”

The data used in this specific research is derived from the war diaries transcribed by volunteers at the Canadian Expeditionary Force Study Group and the analysis is an ongoing collaboration between Shelley Hulan and Rob Warren.

Jonathan Rochkind’ flippant remark ‘Sorry Linked Open Data people’ was made in jest and drew a lot of responses. But it in the end you can’t say that text-mining is better than linked open data anymore than apples are better than submarines as they don’t perform the same function. Similarly, the problem of interpreting archival contents is central to Muninn’s role as well as the question of what is an authoritative source (at least in the knowledge sense).

There is no question that “applying statistical analysis text-mining ‘best guess’ type techniques, provides more relationships than dbpedia alone does“. Generating relationships is easy, since anything can be a relationship, the hard part is figuring out which class of relationship actually means something even before the accuracy of its instances are judged.

Dbpedia on the other hand creates its relationships from very specific mark ups within wikipedia. Both the creation of the wikipedia rdf and the wikipedia infoboxes are human-driven and the number of classes is limited by the manpower available. There is still a possibility of errors, someone might have written in Bern as the dbpedia:capital of (dbpedia:populated_place) Germany, but there is no question this is the class that was intended to hold that information.

Muninn uses linked open data to publish its processed, cleaned up, results. Behind the curtain, many domain-specific statistical and logical models (including statistical mining)look for particular patterns. However, as opposed to free-running relationship mining these are constrained to the specific linked open data tags that they actually fill. For example, Muninn automatically fills in foaf:knows tags if its calculates that two people know each other within a .95 confidence using a statistical ‘best guess’. But the rel:friendOf tag require specific evidence before it is instantiated, such as a record of a communication or an entry in a diary.

Most statistical data-mining algorithms won’t understand the relationships between the Dominion of Canada (oddly it currently redirects to Canada instead of Canada under Imperial control) as a location and the Dominion of Newfoundland as a political entity. They will happily ignore the untidy ‘Dominion’, ‘location’ and ‘political’ tokens and simply file one as a owl:partOf the other since that is the strongest signal on the graph. It’s a simple, easy to understand, wrong answer that causes all sorts of confusion down the line because it ‘looks right’ but isn’t.

Should we should consider wikipedia as an ‘authority’? It is an outstanding resource for general background data and as a statistical resource to design things like tag clouds, auto tagging html text and creating specialized dictionaries. It does well by the law of averages since the crowd eventually fixes things. This also means that at any given time something will be broken at the page level and that is a problem for an authority.
Do you really want your project to catalogue Japan as a rogue legislative element of the European Union because someone messed up a page edit last night? Or watch your reasoner spin madly because there is an edit war on wheter Henry Kissinger is_a war criminal? The United States is part of the British Empire right? This is like some of the problems that Open Street Maps is having with bits of countries getting flooded after bad edits.

Some data lends itself to statistical or social consensus making for a variety of reasons such as ease of observation. It’s easy to fix a bad restaurant location and it only causes limited nuisances for downstream users. Fixing a bad coastline is a bit harder since few people are willing to go out with a transit to survey it.

When possible, Muninn links to the appropriate dbpedia triple using rdf:seeAlso for background documents. It’s not clear that it is a good idea to link Muninn triples with the dbpedia ones using owl:sameAs at this point since we don’t know how authority and stability of triples works.

To create a database from the documents requires some interpretation of the information beyond normal indexing or finding aids. Interpretation can range from codifying referencing standards to hard-core detective work. Right now, Muninn focuses on extracting basic facts and trying to link them across documents and entities, cleaning up ranks and names to a point where there is enough of a basic database to add more information. Eventually, there will be enough room for very high level analysis of the data and some interpretation will be done automatically.

The nice thing about the Great War is the large number of forms in use, even if not all these are typed since this pre-classifies some of the information for us. Yet, there are many cases where the forms are not responsive to the users needs, so the clerk crosses out fields and replaces them by his own. It will be interesting in the long term to track when and why people do so. Text analysis is a little bit more demanding in that it gives good results when customized to the specific document or text types but that requires some hand holding and hand-coding by a human that needs to be doing too many things at once. There is an experimental text search interface that searches the human readable parts of the linked open data as well as other texts in the collection along with linguistic mood and style. The interface and presentation is still something that is being worked on and comments are welcome.

A interesting occurrence is when faulty data is used within a document and is propagated within the organisations’ other document, such as impossible dates of birth. The date is obviously wrong, but is still useful as a means of referencing other documents within the archive. It is impossible to trick most databases into storing this information since it fails basic consistency checks, through we are able to force rdf into supporting it through create use of mark-ups. Thus we need to be able to provide information which is clearly wrong from a temporal aspect because it is still useful for linkages purposes.

If you are a fan of linked open data, there is an experimental SPARQL interface to some of the database. It is answering queries at host www.muninn-project.org, port 8081 as a sparql endpoint.

The RDF dataset is a small subset of the Muninn database and uses the friend of a friend vocabulary. Not all relationships are available through the end-point, but more will be added over time. Comments welcome!

Poppy on the grave of Sargeant A. J. Winslow, CEF at Vimy Ridge

On the occasion of Rememberance Day Muninn is launching its web interface to its catalog. The data so far is what has been processed from the Library and Archives Canada data and the Red Cross Wounded and Missing files from the Australian War Memorial.

Most of the initial work we have been doing is identifying what is in the actual image files. This sounds odd but what actually happens is that the documents are scanned to computer files without noting what the document was. So while we might know that a series of 10 images concern a perticular soldier, we don’t know what peice of paper is recorded in those files. We need to know this before we try and extract the information from those documents and that’s what have been taking our time.

Muninn gets its computer time on the SHARCNET computer cluster (a division of Compute Canada). Instead of using people to review the archives we use computer programs that extract the data from scanned copies. This is a little bit more complicated than run of the mill Optical Character Recognition since the quality of some documents is very poor, the text is sometimes hand-written and forms have check-boxes, rubber stamps and other oddities that require different approaches. Please bare with us while we iron out the bugs from the interface and make it a little bit more pleasing than its current basic look.

The first image is of Lt.-Col. John McCrae who wrote the poem “In Flanders Fields“. The image was taken from the Library and Archives Canada photostream on flickr, and is cropped to remove his dog from the picture and scaled to enhance his facial features. This is followed by an image of General Currie, the commander of Canadian Troops in France, with a pixel size of 25. The image was sourced from the Library and Archives Canada photostream on flickr and did not need to be manipulated.

Major Georges P. Vanier follows using an image from the Library and Archives Canada photostream on flickr, the image was cropped and resized to focus on the profile and enhaced the contrast.

The image of General Sir Samuel Hughes was not changed. The image of  Field Marshal Douglas Haig is curtosy of the Wikipedia Commons and was enlarged and re-centered. The landstrum infantryman with the spiked (the spike was meant to catch cavalry swords) helmet is cropped from a postcard scan on flickr from the Great War Flickr Pool, the württembergisch infantryman is also from another postcard scan in the same flickr pool.

Loading data from the drive into the Sharcnet storage racks.

The first data dump from Library and Archives Canada has been shipped to the Sharcnet data-centre and loaded onto the cluster for processing. The data contains scanned images of the enlistment papers of Canadian Expeditionary Force soldiers (about a million images) and the full personnel file of about 200 soldiers (about twenty thousand images).

The hard drive was first picked up in Ottawa and then travelled with a Muninn staffer to Waterloo, Ontario to one of the Sharcnet machine rooms. The contents were copied directly to the disk array of one of the computer cluster to be worked on. The first step will be to catalog every image and link it to its subject. Since the contents of the image is not allways known, we have to identify the form that was scanned and the information contained in it before we are able to extract the information.

Never underestimate the bandwidth of a passager train on its way to the data centre.

It has been asked why we use hard-drives to move the data from a donor institution to Sharcnet instead of just sending it over the Internet? This has mostly to do with the practical considerations of moving and managing large amounts of data amongst different organisations and systems. Donor institutions do not allways have the facilities available to transfer large amounts of data over the wire, nor may they be comfortable doing it for IT security reasons. Another has to do with backing up what is essentially primary source data in the distributed computing system that has become the Muninn back-end: a hard-drive on a shelf is an insurance policy against computing mishaps.

The data is being worked on now and should be visible on the online catalog system shortly. We will announce results and extracted data-sets on the blog.

Besides the pre-wrapped ‘simple’ data sets that can be downloaded from the website, the database can be queried to generate very detailed dumps of the information Muninn has collected, including ambiguous data that we might be resolved but still need more work. Our first hard drive went out by courier yesterday and we look forward to working with the files.