Catholic Research Resources Alliance (CRRA) Blog

This posting outlines how the names & addresses of the “Catholic Portal” are made available. The purpose of this posting is mostly documentation. Documentation for myself, since I always forget. And documentation so somebody else can do the work after I win the lottery and move to the beach to drink cocktails with umbrellas in them.

Here goes:

1. Extract data – Open the spreadsheet. Activate the ACCU tab. Copy all of the data sans the “cool” data entry macros. Create a new spreadsheet. Paste all of the previously cut data into the new spreadsheet. Save the new spreadsheet for future reference with the name catholic_libraries.xls.
2. Extract more data – Repeat Step #1 for the tab labeled Tab 2, but save the newly created spreadsheet with the name atla.xls.
3. Clean – Open catholic_libraries.xls and delete columns so the only ones remaining are: last name, first name, school, address, city, state, zip code, and email address. Make sure the remaining columns are in the the order listed above. During this process the data may need further cleaning. For example, curly quotes need to be straightened. Carriage returns inside cells need to be removed. Make sure city and state values contain only… city and state values. No countries.
4. Sort – Sort catholic_libraries.xls in ascending order by school.
5. Save – Save the cleaned and sorted data as a tab-delimited text file with the name catholic_libraries.db. Make sure the resulting text file is Unix-based and not DOS- or Macintosh-based. Additionally, Excel often tries to do you a favor by surrounding fields containing commas with quotes. Remove the quotes.
6. Go to Step #3 – Repeat the process for the file named atla.xls, but include only the last name, first name, school, city, state, and email address in the saved data, and call the result atla.db.
7. Mount – Mount the saved database files (catholic_libraries.db and atla.db) by saving them to the Portal. They are expected to live in Y:\data\vufind\web\etc.

The result of this work should then be visible under the Directory tab.

This process is a bit tedious, but since the directory does not change very often, and now that I have documented the process, the next time the directory needs updating things will be easier. On the other hand, as the Portal grows, there will be a need for a real database, and it will be able to support additional functions, such as document delivery. Stay tuned.

Today we had a CRRA Digital Access Committee (DAC) meeting via the telephone. Attendees included:

• Ann Hanlon
• Demian Katz
• Eric Frierson
• Eric Morgan
• Kevin Cawley
• Pat Lawton
• Susan Leister
• Thomas Leonhardt

I did a bit of “Portal” show & tell demonstrating the work done to date on indexing EAD files. (See the previous blog posting.) We then discussed ways the indexing/display could be improved. Suggestions included:

• putting the words “Archival material” into the format field of the Solr index thus allowing better faceting
• reading the value of langmaterials and using it as the value for Solr’s language fields, again allowing for better faceting
• reading all of the fields associated with a given container-level element and putting them into Solr’s allfields field to improve indexing
• extracting the last value of our current “title”, using it as our title, and using the remaining values as some sort of supplemental description or alternatively, simply reversing the “title” string

We then brainstormed ways to resolve character encoding issues, the feasibility of making our metadata available via Web servers, and the status of the metadata guidelines.

We felt we had discussed it all, so the meeting was over.

This posting outlines how I am currently indexing MARC and EAD files in VUFind with Solr for the CRRA. (Boy, there are a lot of acronyms in that sentence!)

Background

The Catholic Research Resources Alliance (CRRA) is a member-driven organization with the purpose of making available “rare, unique, and uncommon” research materials for Catholic scholarship. Presently the membership is primarily made up of libraries and archives who pool together their metadata records, have them indexed, and provide access to the index. My responsibility is to build and maintain the technical infrastructure supporting this endeavor.

A couple of years ago much of the CRRA metadata was manifested as MARC, and at that time VUFind was selected as the tool we would use to index, search, and display this content. About six months ago the Alliance realized the growing necessity of including EAD files as well. At the same time, the ability of accomodate non-MARC metadata was increasingly becoming a VUFind reality. New ground still had to be broken; processes needed to be implemented allowing VUFind (and the underlying Solr indexer) to understand how to work with materials which were not book-like.

The balance of this posting describes in greater detail how I am beginning to accomodate MARC as well as EAD metadata into VUFind’s interface with Solr.

Assumptions

The system runs on a number of assumptions. First, it is assumed it is the members’ responsibility to create and maintain their metadata. Second, it is my responsibility to index it and make it available for display. Moreover, it is assumed each metadata record incudes at least three values: 1) a unique identifier, 2) a human-readable description of an item, and 3) an address pointing to the location of the item. For MARC records, these things reside in the 001, 245, and 099 fields. For EAD files, they have been designated as the id attribute of unitid elements, the content of unititle elements, and the url attribute of the eadid element and from there the location of the item.

Additionally, it is assumed all metadata records, whether MARC or EAD, are available for harvesting from a Web server. In other words, each member who wants to have their MARC records available in the CRRA needs to export their records to a single file and make them accessible via a URL. Similarly, all EAD files which are intended to be indexed need to be in a single Web-accessible directory and the URL of the directory needs to be known. Making member metadata accessible via a Web server has three benefits: 1) it facilitates automation, 2) it distributes the responsibility of archiving metadata across the membership, 3) it enables the metadata to be harvested by other applications and used for other things. “Can you say ‘linked data?’”

Files and Perl scripts

Given these assumptions, the following sets of files and Perl scripts are used to do the work. The first set is core the both of the other two:

• libraries.db – A “database” of CRRA participants consisting of their names, libraries, and URLs where their metadata records can be found. This file is used by just about every other script in the system.
• subroutines.pl – A tiny library of Perl subroutines, mostly to read the contents of libraries.db.

This second set is used to index MARC metadata:

• marc-harvest.pl – Copies (mirrors) remote MARC files locally
• marc-add-code.pl – Validates and updates the values of MARC 001 fields making sure they exist and are unique
• marc-index.pl – Slurps up a Solr marc.properties template (template.txt), makes the appropriate substitutions, and indexes the MARC records associated with a given library
• marc-build.sh – A shell script used to run all of the MARC-based scripts. One ring to rule them all.

The third is used to index EAD files:

• ead-harvest.pl – Copies (mirrors) remote XML files locally
• ead-validate.pl – Makes sure the mirrored XML files are well-formed, conform to the EAD DTD, and include an eadid url attribute (done with a stupid stylesheet called geturl.xsl)
• ead-transform.pl – Makes sure each EAD container-level element includes a unitid with a unique id attribute, saves the result to a local cache, and transforms these same files into HTML. The first process is done with a stylesheet called addunitid.xsl. The second process is done with another stylesheet called ead2html.xsl.
• ead-index.pl – Indexes all the cached/transformed EAD files by parsing out container-level elements, creating an XML stream of records of my own design, parsing the result, and passing each record on to Solr. The heart of this script is a fourth stylesheet — ead2solr.xsl
• ead-build.sh – A shell script used to run all of the EAD-based scripts. Another ring to rule them all.

The “secret” to indexing EAD files is really no secret. I simply followed Demian Katz’s instructions. In a nutshell, to index non-MARC content the developer needs to:

• Parse the given metadata into records. I do this with ead2solr.xsl.
• Map each of the record’s values to as many of the underlying Solr fields as possible. Presently I only have titles and I do this through ead2solr.xsl as well.
• Create an XML snippet representing each record and map it to the Solr fullrecord field, described below.
• Denote a record type. I call mine ead.
• Save the whole thing to Solr, done with ead-index.pl.

Currently, my XML snippet (Item #3) looks like this:

  <record>
    <id>unaead_id2635150</id>
    <title>Catholic Church. Archdiocese of Detroit (Mich.)
      Collection -- Catholic Church. Archdiocese of
      Detroit (Mich.): Manuscripts -- Letters -- Bp.
      Baraga to his sister Amalia
    </title>
    <date>1836/1203</date>
    <url description='View remote, canonical version of EAD'>

http://archives.nd.edu/findaids/ead/xml/det.xml

    </url>
    <url description='View local version of EAD file'>

http://zoia.library.nd.edu/sandbox/crra-data/ead/una-det.html#id2635150

    </url>
  </record>

The VUFind application provides seamless access to the index through its search box, but a bit of work needs to be done to display search results. Specifically a “record driver” needs to be written to accomodate new record types (Item #4, above). This driver inherits methods from a parent driver, IndexRecord.php, and the developer needs to override some of the methods found there with methods considering the content of the fullrecord field. Presently, the only thing I have in my record driver (EadRecord.php) is a method to extract URLs. In the future I will need to include methods to extract names of CRRA members, names of their libraries, and additional descriptive metadata.

You can see the fruits of these efforts in the CRRA “sandbox” — something we are affectionately calling “The Green Interface”.

Issues

The whole process functions and could be run automatically from cron on a daily basis, but there is plenty of room for improvement. Issues include:

• speed – The indexing process is slower than I’d like. I think throwing more hardware thrown at the problem will make things faster.
• invalid data and stale URLs – A small percentage of the MARC and EAD files do not include the required metadata values. No unique identifiers. Malformed MARC leaders. Non-validating EAD files and/or eadid url attributes pointing to broken locations. This is where metadata maintenance comes in.
• character encoding – This is one of the bigger problems. Trying to figure out whether or not a MARC record has been exported as UTF-8 is difficult. Solr assumes UTF-8 and I don’t think it even knows about MARC-8. When MARC data is not encoded as UTF-8, search results look really ugly. Similarly, some of the EAD files, because of similar issues, really display poorly after they have been transformed, indexed, searched, and displayed.

None of these things are insurmountable. They will be addressed.

Next steps

My immediate next steps focus on richer search results. I need to extract additional information from the EAD files to supplement the content of my fullrecord field. After that I will explore the creation of “collection-level” records by indexing the headers of EAD files. These records will be fuller because they will have things like controlled vocabularies, scope notes/abstracts, and biographies from which to draw. Once the fullrecord fields are enhanced, I will need to go back to EadRecord.php and enhance its functionality. After that I will see about creating reports listing errors in metadata files. These reports will be designed to share with members making it easier for them to maintain their content.

All of that sounds like plenty to me. Wish me luck.

I have made significant progress in the process of harvesting EAD files and preparing them for ingestion into the “Catholic Portal”. This posting outlines the successes.

Assuming a Catholic Research Resources Alliance members place their EAD files in a HTTP-accessible directory, and those files have a .xml extension, then the following Perl scripts enable me to harvest and prepare them for indexing:

• harvest-ead.pl – reads remote HTTP-accessible directories and copies all of the .xml files found there to a local cache
• validate.pl – makes sure the cached XML files are well-formed and conform to the EAD DTD, and if not, then move the files to a different directory
• transform.pl – reads the validated XML files, adds id attributes to all unitid elements through the use of a stylesheet (addunitid.xsl), transforms the resulting XML into HTML using another stylesheet (ead2html.xsl), and saves the result to an HTTP-accessible directory

What was really cool and a huge time-saver was the use of ead2html.xsl. Originally named AAAv2002-HTML.xsl, found on a page called User Contributed Stylesheets, and submitted by Stephanie Ashley, this stylesheet took my id attributes and automatically made named anchors for me. Boy, did I get lucky. “Thank you, Stephanie!”

My next step is to revisit my indexing routines.

This is the briefest of travelogues reporting on a meeting about EAD files at Marquette University for the Catholic Research Resources Alliance on September 20, 2010.

A few members of the Alliance were previously awarded a CLIR grant to catalog previously uncataloged special collections items. These members are now doing the work using EAD (Encoded Archive Description) with the intent of sharing the resulting metadata with the “Catholic Portal”. The purposes of the meeting were to build relationships between these particular Alliance members and to discuss progress on the grant. In attendance where people from St. Catherine University (Deborah Kloiber and Emily Asch), Marquette University (Matt Blessing, Ann Hanlon, Bill Fliss, and Jean Zanoni), and the University of Notre Dame (Pat Lawton, Kevin Cawley, and Eric Lease Morgan).

Of primary concern was the particular way people were using EAD and whether or not it would lend itself to indexing by the “Portal” software. Consequently, I spent a lot of the time describing the technical infrastructure of VUFind and how it interfaced with Solr, the underlying indexer/search engine. In short, the absolute need for unique identifiers, human-readable descriptions of items, and location codes were enumerated. The former two can be garnered from the unitid and unittitle elements of a EAD did elements. The later can be gotten from the url attribute of the eadid element. Everybody was confident their EAD files would contain these values.

We then went around the table doing a bit of show & tell against our EAD. The folks of St. Catherine’s were using the Archivist’s Tool kit to “catalog” their Ade Bethune collection. Marquette University was using a Microsoft Access database to “catalog” Dorothy Day content.

Time tables where then outlined. The whole CLIR project is expected to be finished by December of 2011. Participants in attendance thought their work would be done by the end of Spring 2011, and the remaining time would be spent on putting the content onto the “Portal” as well as doing various types of publicity (conference presentations, etc.).

The meeting was over around noon, and we all retired to the faculty club for lunch. (“Thank you, Marquette!”)

In retrospect, there may be two additional issues needing to be addressed. First, I originally planned to assign or replace unitid values with locally generated, “Catholic Portal” specific values, but I have since learned that unitid information is often times used as a sort of call number and therefore necessary for location. Replacing (removing) such values from the EAD files may make work down the line more difficult. Maybe I should be getting the unique values from an id attribute of the unitid element instead?

Second, as a group we may need to decide how to encode dates. Dates can be nested within unittitle elements as well as free-standing elements in the did. Just as importantly, they can take all sort of forms. In order to make sorting and faceting feasible, the Alliance may need to figure out ways to standardize and normalize dates.

Today I indexed some of the metadata I extracted yesterday using a script called index-ead.pl. Of all the scripts I’ve written so far, this one is the most straight-forward. Read locally-developed XML file. Extract the unique identifier, title, and date. Associate each with VUFind/Solr fields. Commit.

You can (temporarily) see the fruits of these labors because all of the records have been associated with the Eric Lease Morgan Foo Bar Library. The result is a list of container-level records with very little additional information.

By the way, as of today I am running a version of VUFind as retrieved from the development trunk, specifically, revision 3029. When upgrading from revision to revision, it is important to retain one’s config.ini file and reindex. The process is not painful, if done infrequently. As time goes on I will also need to retain locally developed hacks, such as the ones I need to write below.

The next steps are to write the MARC record driver so it does not attempt to do automatic look-ups for call numbers, but rather extracts such information from of the local index. A second next step is to write an EAD record driver to accomodate the special cases of… EAD records.