The Muninn Project - RDF

Transcribed CEF Medical Files as Linked Open Data on the Canada Open Data Portal

warren — Mon, 16 Feb 2015 14:10:26 +0000

One of the collections that Library and Archives Canada has been digitizing and putting for access online has been the personnel records of the soldiers of the Canadian Expeditionary Force sent to Europe during the Great War. A typical personnel file is a folder containing about 100 pages of documentation about the soldier himself and sometimes includes his medical records in the form of temperature charts, dental records and medical case sheets. In this project it was decided to focus on the contents of the "Medical Case Sheet" that is a lined form used by hospital staff to record information about their patient.

Late last year, a partnership was struck between Library and Archives Canada and the Muninn Project to explore the possibilities of crowd-sourcing to extract the content out of digitized historical documents and create Linked Open Data with them. This project had for goals to foster a relational network of resources on information retrieval research, making documentary heritage information discoverable and searchable and to contribute to the pool of Great War resources freely available to the public during the commemoration period of the Great War. The results of this trial project was a partial transcription of the Medical Case Sheets from a sample of the personnel files of the CEF that has been released by Library and Archives Canada to the Canada Open Data Portal.

The dataset is released in RDF/XML format with links to Muninn, LOC Subject Headings and DBpedia. As an additional bonus, this dataset is the first data set on the Canada Open Data Portal that ranks five stars (✭✭✭✭✭) on the Tim Berners-Lee data deployment scheme! A void description of the data is here.

Making sense of scanned historical documents

One of the problems with imaged historic documents is that searching them in a large collection is difficult when the meta-data identifying the image contents is missing. Creating that meta-data manually is time consuming: a person has to physically look at each image and type in some meaningful descriptors for the document. Digital image processing is a lower cost alternative can help in generating this meta-data [1,2] by extracting information from the raw images themselves.

In the case of a CEF Soldier's file, the individual pages in the folder are unordered and directly scanned to a PDF file. The breath of information that is available within the scanned personnel files of the Canadian Expeditionary Forces (CEF) collection includes everything from handwritten notes to hospital temperature charts.Finding these specific imaged documents within the file was the first problem: it is not known which pages within the PDF file contains the form that interest us. Using an image analysis script, a sample of 1,000 CEF files was analyzed to locate these forms within the service files for them to be transcribed. Out of the 1,000 sampled soldiers, and the resulting 10,000 images, about 500 Medical Case Sheet forms were found and submitted for transcription.

The data is actually quite noisy - at times the hand-writing (as shown above) has bled into the paper and is no longer legible. The handwriting was not particularly neat when it was first written and time has made it harder still to transcribe. In keeping with the archival direction of the project, the text is not edited nor is the short-hand used within the medical files expanded or translated. As an example, the handwriting might be transcribed from the document as "TXH GSW of Leg rt". What this actually means is "TXH(?) Gun Shot Wound Of Leg Right". Keep in mind that the contents of the files are not a compelling narrative that tells a good story, but the record keeping tool used by medical professionals trying to get a soldier back to health.

Of course, this is a Linked Open Data set which makes it easy to annotate and link different interpretation of the same transcribed material. The dataset uses the NLP Interchange Format ontology, the W3 Provenance ontology, the Void dataset vocabulary and the FOAF ontology to markup the contents of the transcriptions and serialized in RDF/XML. The provenance information in the dataset records not only what page of which document the specific image was taken from, but also the individual transcriptions that were submitted and the ones that were recognized as begin valid. The sub-images that show the specific phrases that are being transcribed are not located in this dataset but will eventually be distributed through another mechanism. This was done to reduce the dataset size to a manageable amount; that sub dataset will likely use Open Annotation to record the locations that the image was cut out from.

What were the most common complaints and/or injuries?

The contents of the medical case sheets represents all of the incidents that one would expect to occur in a large body of men involved in the conduct of war. Not all injuries were directly related to combat, besides the usual problems of every day life (such as measles), venereal diseases occurred often as well as respiratory ailment that would affect people in cold wet trenches.

The figure above is a count of the number of incidences that occurred in all of the transcribed medical case sheets. As the project only dealt with 1,000 service records, this represents only a small sample of the experiences of the more than 500,000 CEF soldiers. Take note, that some afflictions occur multiple times, such as firearms-related injuries, to the same soldier over the course of their service and are counted multiple times. Hence, this chart is not a true statistical representation of the injuries sustained by CEF troops.

Not especially surprising is the high instances of coughs, pneumonia and influenza that were pervasive in a trench warfare environment and could spread quickly to large groups of men in dugouts or barracks in less than hygienic situations. Influenza, or the Spanish Flu, would eventually kill more people than the Great War did. Venereal diseases were also common, 1 in 9 soldiers were affected according to some sources, which at that time was listed as a self-inflicted injury and would result in a soldier's pay being docked while he got treatment.

The data is available on the Canadian Open Data Portal in Linked Open Data and plain text format.

Language English

Tags:

CEF

RDF

Medical Files

LAC

The 100 year old April Fool's joke

warren — Mon, 01 Apr 2013 17:15:58 +0000

Perhaps you think you're funny?

I'd like to see you pull a prank that still causes problems 100 years after the fact.

Take a look at this group of soldiers from the CEF. They pulled a prank on their enlistment officer about their date of birth. A good one.

For example, examine the records of Scott Mcconnell, Born Feb 31, 1884, Joseph Carriere, Born Feb 31, 1895, Harold Bennington, Born Feb 31, 1890 and Frederick Handley, Born Feb 31, 1874 among many others. Notice anything in common? They all wrote down impossible, but valid looking, birthdates on their enlistement form as a joke that keeps on creating chaos to this day.

Harry Baird was one of the soldiers of the Canadian Expeditionary Force with a sense of humour and he likely enjoyed pulling a fast one on an officer at enlistment. Some of his letters are available online at Canadian Letters where some of his more colourful descriptions of the war are recorded. Other reasons for recording impossible dates might have included hiding ones true identity while re-enlisting or just as a means of showing dissent at being drafted.

And so, a 100 years after the fact, every time someone tries to create a database about the Great War, what happens?

muninn=> select date('1989-02-31');
ERROR:  date/time field value out of range: "1989-02-31"
LINE 1: select date('1989-02-31');
muninn=>

Which likely has driven more than one database to distration. When the LAC originally indexed the CEF papers online, they indexed the birthdates as strings to get around this problem as well as the other data quality problems that exist in working with historical dates. SQL databases will not allow impossible dates to be entered and dealing with imprecise dates requires the use of time intervals or really creative and specialized schemas. Also, most date implementations in databases or operating systems are limited in their ranges to post-1901 or post-1970 dates which makes the process especially painful. (Note: Date::Calc is a good tool to get around this problem in perl.)

This is actually an interesting problem in attempting to balance archival integrity and data quality: the date is impossible but it has perpetuated itself into the bureaucracy and behaves a little bit like a primary key that can be used for additional document retrieval. As show above, a normal SQL database has no hope of working with this type of data. One of the great things about using linked open data is that we can use ontologies to record information that is wrong without creating logical or syntactic errors. Using the W3 Time Ontology for example, we can record an invalid date without necessarily committing to a date value without loosing the date string associated:

<owl:time rdf:about="Birth">
 <dc:source rdf:resource="LAC Entry"/>
 <rdfs:label>Birth of Harry Baird on 31/02/1893.</rdfs:label>
 <rdf:value>1893-02-31</rdf:value>
 <foaf:name>1893-02-31</foaf:name>
 <mil:hasPrincipal rdf:resource="Harry Baird"/>
</owl:time>

However, if we believe that part of the date information is true, and that Harry was actually born somewhere in 1893, we can use only part of the date information by adding a partial Date Time description:

<owl:time rdf:about="Birth">
 <dc:source rdf:resource="LAC Entry"/>
 <time:hasDateTimeDescription>
  <time:DateTimeDescription rdf:about="...">
   <time:year rdf:datatype="&xsd:GYear">1893</time:year>
  <time:DateTimeDescription>
 </time:hasDateTimeDescription>
 <rdfs:label>Birth of Harry Baird on 31/02/1893.</rdfs:label>
 <rdf:value>1893-02-31</rdf:value>
 <foaf:name>1893-02-31</foaf:name>
 <mil:hasPrincipal rdf:resource="Harry Baird"/>
</owl:time>

By doing this, we can leverage any partial information that is available even through its accuracy leaves to be desired.

But remember, the best of jokes are those that keep on giving for years to come.

Language English

Tags:

US Department of Defense Dictionary of Military and Associated Terms as SKOS RDF

warren — Tue, 18 Sep 2012 04:18:13 +0000

I have converted the US Department of Defense Dictionary of Military and Associated Terms into an RDF SKOS dictionary that can be used to link RDF documents into an authoritive term list or to disambiguate terms. The dictionary OWL file is rather large with about 3,400 terms taking about 2.5MB of storage. Documentation for the dictionary is sparse as the RDF contains most of the necessary definitions already.

Language English

Tags:

RDF

SKOS

Dictionary

Military Ontology Available

warren — Fri, 29 Jun 2012 00:00:00 +0000

The Muninn Military and Organization ontologies have been released in OWL format on the Muninn RDF server and comments are welcome. These provide ontological terms for the markup of information about historical data and they provide more depth in terms of supporting changes to and the lineage of entities.

One of the ongoing problems with RDF markup and ontologies is that most of them are meant to support the expression of facts about the world as it currently is and not as it was or will be. This can create some issues in that most data is meant to be true at only one point in time.

The ontology includes support for military ranks, roles, organization types and service-specific chains of commands. Limited NATO rank equivalencies are provided and the country specific rank instances are being worked on. Any comments and suggestions are welcome.

Language English

Tags:

Authority, Statistics and RDF

warren — Tue, 31 May 2011 23:30:00 +0000

The recent discussion on the LOD-LAM website have prompted me to begin writhing about some of the data available on Muninn and how it is generated. Creating a database from the contents of war archives nearly a century old presents some special challenges, some old some new. Jonathan Rochkind' flippant remark 'Sorry Linked Open Data people' was made in jest and drew a lot of responses. But it in the end you can't say that text-mining is better than linked open data anymore than apples are better than submarines as they don't perform the same function. Similarly, the problem of interpreting archival contents is central to Muninn's role as well as the question of what is an authoritative source (at least in the knowledge sense). There is no question that "applying statistical analysis text-mining ‘best guess’ type techniques, provides more relationships than dbpedia alone does".

Generating relationships is easy, since anything can be a relationship, the hard part is figuring out which class of relationship actually means something even before the accuracy of its instances are judged. Dbpedia on the other hand creates its relationships from very specific mark ups within wikipedia. Both the creation of the wikipedia rdf and the wikipedia infoboxes are human-driven and the number of classes is limited by the manpower available. There is still a possibility of errors, someone might have written in Bern as the dbpedia:capital of (dbpedia:populated_place) Germany, but there is no question this is the class that was intended to hold that information. Muninn uses linked open data to publish its processed, cleaned up, results. Behind the curtain, many domain-specific statistical and logical models (including statistical mining)look for particular patterns. However, as opposed to free-running relationship mining these are constrained to the specific linked open data tags that they actually fill. For example, Muninn automatically fills in foaf:knows tags if its calculates that two people know each other within a .95 confidence using a statistical 'best guess'. But the rel:friendOf tag require specific evidence before it is instantiated, such as a record of a communication or an entry in a diary. Most statistical data-mining algorithms won't understand the relationships between the Dominion of Canada (oddly it currently redirects to Canada instead of Canada under Imperial control) as a location and the Dominion of Newfoundland as a political entity. They will happily ignore the untidy 'Dominion', 'location' and 'political' tokens and simply file one as a owl:partOf the other since that is the strongest signal on the graph. It's a simple, easy to understand, wrong answer that causes all sorts of confusion down the line because it 'looks right' but isn't. Should we should consider wikipedia as an 'authority'? It is an outstanding resource for general background data and as a statistical resource to design things like tag clouds, auto tagging html text and creating specialized dictionaries. It does well by the law of averages since the crowd eventually fixes things. This also means that at any given time something will be broken at the page level and that is a problem for an authority. Do you really want your project to catalogue Japan as a rogue legislative element of the European Union because someone messed up a page edit last night? Or watch your reasoner spin madly because there is an edit war on whether Henry Kissinger is_a war criminal? The United States is part of the British Empire right? This is like some of the problems that Open Street Maps is having with bits of countries getting flooded after bad edits. Some data lends itself to statistical or social consensus making for a variety of reasons such as ease of observation. It's easy to fix a bad restaurant location and it only causes limited nuisances for downstream users. Fixing a bad coastline is a bit harder since few people are willing to go out with a transit to survey it. When possible, Muninn links to the appropriate dbpedia triple using rdf:seeAlso for background documents. It's not clear that it is a good idea to link Muninn triples with the dbpedia ones using owl:sameAs at this point since we don't know how authority and stability of triples works. To create a database from the documents requires some interpretation of the information beyond normal indexing or finding aids. Interpretation can range from codifying referencing standards to hard-core detective work. Right now, Muninn focuses on extracting basic facts and trying to link them across documents and entities, cleaning up ranks and names to a point where there is enough of a basic database to add more information. Eventually, there will be enough room for very high level analysis of the data and some interpretation will be done automatically. The nice thing about the Great War is the large number of forms in use, even if not all these are typed since this pre-classifies some of the information for us. Yet, there are many cases where the forms are not responsive to the users needs, so the clerk crosses out fields and replaces them by his own. It will be interesting in the long term to track when and why people do so. Text analysis is a little bit more demanding in that it gives good results when customized to the specific document or text types but that requires some hand holding and hand-coding by a human that needs to be doing too many things at once. There is an experimental text search interface that searches the human readable parts of the linked open data as well as other texts in the collection along with linguistic mood and style. The interface and presentation is still something that is being worked on and comments are welcome. A interesting occurrence is when faulty data is used within a document and is propagated within the organizations' other document, such as impossible dates of birth. The date is obviously wrong, but is still useful as a means of referencing other documents within the archive. It is impossible to trick most databases into storing this information since it fails basic consistency checks, through we are able to force rdf into supporting it through create use of mark-ups. Thus we need to be able to provide information which is clearly wrong from a temporal aspect because it is still useful for linkages purposes.

Language English

Tags:

RDF

Relationships

DBpedia

SPARQL and Linked Open Data

warren — Wed, 25 May 2011 05:23:42 +0000

After a few hiccups with the SPARQL database and the web front end, the Muninn website will be undergoing some major re-work. I'll update this blog post as the new interface features go online. Update: Feb 23, 2012 - The SPARQL server at http://rdf.muninn-project.org/sparql is answering queries.

Language English

Tags: