Introduction

As you may or may not know, OAI-PMH is a “standard” protocol designed for harvesting metadata. It only understands six commands (or in OAI-PMH parlance, “verbs”). These commands are sent to remote computers in the form of URLs, and the remote computer is expected to respond in the form of specifically shaped XML streams. These commands include:

• Identify – Lists who manages the repository and what type of content it contains.
• ListMetadataFormats – Lists the various metadata schemes used to describe the repository’s content. At least one of these schemes must be Dublin Core.
• ListSets – Specifies how the repository’s content is subdivided. There can be zero or more of these subdivisions.
• ListIdentifiers – Returns a list of keys pointing to specific records in the repository.
• ListRecords – An enhanced version of ListIdentifiers, this verb downloads whole records, not just identifiers.
• GetRecord – Given a specific identifier, this verb retrieves a single record.

Through a conversation of these verbs and the returned XML streams, metadata between computers can be exchanged. It is then up to the computer doing the harvesting to implement some sort of cool and interesting service with the harvested content. Here at Catholic Portal Central we want to index the metadata and provide immediate access to remote digitized content.

Investigations

At least three Catholic Research Resources Alliance (CRRA) members have OAI-PMH repositories: Duquesne University, Boston College, and Loyola University Chicago. Using a little Perl script, I most recently investigated the content of the repositories of Boston College and Loyola University Chicago. Through this process I learned what metadata formats they supported, what sets were used to subdivided their collections, and output Dublin Core metadata from a few selected sets.

The harvested Dublin Core metadata was typical of OAI-PMH repositories: thin, a bit ambiguous, and somewhat inconsistant across repositories. It was thin because many of the Dublin Core elements are left unpopulated. It is ambiguous because many of the fields are repeated, and the values of repeated elements are of different types. For example, a description field may be empty, contain an abstract of the work, the full text of the work, or the process used to digitize the material. It is inconsistant because things like dates, names, and subject entries are formatted differently. In some places names are listed in first name/last name order. Other times it is last name/first name order. Dates can be anything from “February 12, 2012″ to “2012-02-12″ to “Twelfth Century”. None of this is new the world of OAI-PMH. It is typical.

All is not lost. There are patterns to this apparent randomness. Using my script I can sometimes output titles, descriptions, subject headings, and URLs of digitized objects. For example, here is such a list from the Loyola University Chicago repository:

item: 46

key: oai:content.library.luc.edu:coll6/45

title(s): Letter to the Secretary of the Literary Agency of London, 1908
title(s): Catholic Women Poets

identifier(s): cudahy219e3

identifier(s): 003_kayesmith_1908;pg3.jpg

identifier(s): http://content.library.luc.edu/u?/coll6,45

subject(s): Shelia Kaye-Smith; poets; women poets; Catholic poets

subject(s): Local

description(s): third page of letter requesting appointment

description(s): does not suit you any other time up to 4 15 will do Would you kindly send a reply to me c o Miss F E Walters Girton College Cambridge With apologies for troubling you believe me Yours faithfully Sheila Kaye Smith

description(s): Master file scanned at 600 dpi RGB in reflective mode from original document using MicroTek ScanMaker 1000XL

description(s): http://www.luc.edu.archives

type: image

From this output it becomes apparent that the first title is the title of the artifact, the third identifier is the URL of the digitized object, the first subject field is a delimited list of keywords, the first description is a sort of abstract, and the type field contains a value denoting what kind of digitized thing is in question. Thus, the output follows a pattern, and computers are very good at patterns, therefore a computer program could easily be written to read this particular OAI-PMH output and stored in the Portal’s index.

Next steps

My next steps are two-fold. First, I will harvest and index some of the metadata from selected Loyola University Chicago OAI-PMH sets. Second, I will let colleagues from various CRRA committees (specifically the Digital Access Committee as well as the Collection Committee) peruse the results. In the end I hope to get feedback on how to proceed. Should I index more content? Less? None? If more, then how should records be displayed, and exactly how ought the Dublin Core metadata be mapped to VuFind’s underlying Solr index fields?

All of this work is entirely feasible. At the same time it is not enormously scalable. Hand-crafting the parsing of OAI-PMH output, and handcrafting how it all gets mapped to Solr’s index is time consuming and fragile. The Portal Home Planet can easily do this work for no more than a dozen different repositories, but after that some other means of production will need to be examined.

April 2012

This month’s update includes:

• A Focus on Members, from Janice Welburn, Chair, CRRA Board of Directors
  To guide us in developing effective strategies for successful member engagement, the Board has set up a Membership committee and I’m delighted to welcome a current Board member, Evelyn Minick, University Librarian, Saint Joseph’s University, as the chair … The Committee’s major objectives are to grow the membership and ensure retention of current members …
• CRRA Collections Spotlight: The Philadelphia Archdiocesan Historical Research Center Catholic Newspaper Collection, by Shawn Weldon
  The Philadelphia Archdiocesan Historical Research Center (PAHRC) holds one of the largest collections of Catholic newspapers in the United States …
• Update on the Digital Access Committee (DAC), from Demian Katz, DAC Chair
  In spite of changes, DAC has pressed forward with several initiatives.  The Catholic Portal, still the centerpiece of CRRA’s website, is under continuous improvement, both in response to member feedback gathered during usability testing and due to new features in the underlying VuFind software …
• Mark Your Calendars: All-Members Meeting, Anaheim, CA, June 25-26, 2012, all are invited;
  Archival Networks and EAD Consortia at SAA in August (San Diego); Fall Symposium at DePaul University, Oct. 15-16, 2012
• Position Announcement:Duquesne University

A Focus on Members
from Janice Welburn Chair, CRRA Board of Directors

 The new strategic plan affirms the importance of a strong value proposition for members.  To guide us in developing effective strategies for successful member engagement, the Board has set up a Membership committee and I’m delighted to welcome a current Board member, Evelyn Minick, University Librarian, Saint Joseph’s University, as the chair.  Evelyn’s deep commitment to our mission, keen insights into member expectations and effective leadership of the task force that developed a multi-tiered dues schedule, make her an excellent choice to guide our membership development and support. While we may add other members over time, I am pleased to announce the initial membership:

• Kris Brancolini, Dean of the Library, Loyola Marymount University, Los Angeles
• Theresa Byrd, University Librarian, University of San Diego; also a Board member
• Melody McMahon, Director of the Paul Bechtold Library, Catholic Theological Union, Chicago
• Tom Messner, Library Director, Barry University, Miami Shores, FL
• Laverna Saunders, Library Director, Duquesne University, Pittsburgh
• Bob Seal, Dean of Libraries, Loyola University Chicago
• Kathy Webb, Dean of University Libraries, University of Dayton
• Jennifer Younger, ex officio, Executive Director, CRRA

The Committee’s major objectives are to grow the membership and ensure retention of current members.  It is advisory to the Board.  Although the Committee plays a central role, it is important to emphasize that the Committee will consult broadly with members on needs and expectations of membership, as well as actively seek suggestions from individuals and committees on prospective members.  We want to continue our participative tradition of reaching out to potential members as noted in our protocol for inviting new members.  The charge to the Membership Committee will be accessible shortly along with the full roster on our website.

 THE PHILADELPHIA ARCHDIOCESAN HISTORICAL RESEARCH CENTER

CATHOLIC NEWSPAPER COLLECTION

 The Philadelphia Archdiocesan Historical Research Center (PAHRC) holds one of the largest collections of Catholic newspapers in the United States. These newspapers were collected by the American Catholic Historical Society of Philadelphia which was founded in 1886 to collect material documenting the history of Catholicism in the United States. The ACHS collections, including manuscripts, newspapers, periodicals, pamphlets, books, artifacts and graphic material, were given to the Archdiocese of Philadelphia in the 1930’s. In 1989, the ACHS Collection was merged with the Archives of the Archdiocese of Philadelphia to form PAHRC.

The newspaper collection contains Catholic newspapers from throughout the United States as well as some Catholic newspapers from Canada, England, Ireland, France and Italy.  The collection contains over 300 titles, representing 35 states and the District of Columbia, and covers the period primarily from the 1820’s through the 1940’s. The bulk of the collection dates from the 1840’s through the 1920’s.

Included are early and prominent Catholic newspapers such as The Catholic Press/The United States Catholic Press (Hartford), The Catholic Miscellany (Charleston), The Catholic Herald (Philadelphia), The Catholic Mirror (Baltimore), The Catholic Advocate (Louisville), The Pilot (Boston), The Catholic Telegraph (Cincinnati) and The Freeman’s Journal (New York City). The collection also contains many ethnic newspapers, including Irish-American, German-American and Polish-American newspapers, as well as newspapers published for a juvenile audience, society newspapers and papers published for the support of Catholic institutions.

Notable are some of the first black Catholic newspapers published in the United States. There is a good run of the American Catholic Tribune, originally published in Cincinnati and later in Detroit, for the years 1887-1894. There are some issues of The Journal, a Philadelphia black Catholic newspaper that was published for a few months in 1892. The collection also includes Volume I, Number 1 (February 18, 1905) of The Catholic Herald, a black Catholic newspaper in Washington, D.C. which may be the only issue published. For more information on black Catholic newspapers and periodicals in the PAHRC collection see the following: http://www.pahrc.net/index.php/black-catholic-periodicals/

The collection also contains other rare titles such as Sina Sapa Wocekiye Taeyanpaha, a North Dakota newspaper published in the Sioux language, The Catholic Visitor (Richmond, Virginia), The New Jersey Catholic Journal (Trenton, New Jersey) and perhaps the only significant collection of Redpath’s Illustrated Weekly, a primarily Irish national newspaper published in New York City by the journalist and social activist James Redpath.

Despite the size and research value of the collection, there are some issues that impact its usefulness. Although there are very large runs of issues for the major newspapers, none of the titles is complete and there are gaps and issues missing. Most of the rarer newspapers may contain only a few issues or a few years of the paper. The most pressing issue is that the collection is maintained primarily in hard copy and a significant number of the newspapers are in very fragile condition and in need of immediate conservation and preservation. One of the advantages of membership in the CRRA is the opportunity to cooperate with other repositories facing these same issues to create a comprehensive online inventory and directory of North American Catholic newspapers and to facilitate the eventual digitization of the various collections. One of my goals as a member of the CRRA Newspapers Taskforce is to assist in the realization of these projects.

To view the contents of the newspaper collection at PAHRC see the following:  http://pahrc.pastperfect-online.com/30664cgi/mweb.exe?request=clicksearch;dtype=d;subset=0;_t1101=newspapers

–Shawn Weldon, PAHRC

Member, Collections Committee

Member, Catholic Newspapers Task Force

Update on the Digital Access Committee (DAC), from Demian Katz, DAC Chair

The Digital Access Committee has had some recent membership changes, bidding farewell to Ann Hanlon (Marquette) and welcoming new member Megan Bernal (DePaul).

In spite of changes, DAC has pressed forward with several initiatives.  The Catholic Portal, still the centerpiece of CRRA’s website, is under continuous improvement, both in response to member feedback gathered during usability testing and due to new features in the underlying VuFind software used to run it.  Additionally, DAC has begun looking at some new software that can be used to expand and improve CRRA’s online presence.  The Concrete5 Content Management System is an open source tool for building websites, and DAC hopes to use it to improve the quality and simplify the maintenance of the informational pages that accompany the Catholic Portal on catholicresearch.net.

Archon and Archivists’ Toolkit are both packages for building EAD files for archival description, and DAC has been weighing the benefits of installing one of these packages to help members build finding aids.  Finally, DAC is also preparing to support the Newspapers Task Force as needed as efforts progress.

Mark Your Calendars!  Upcoming Events

 All-Members Meeting

Anaheim, CA

June 25-26, 2012

CRRA colleagues,
As you make plans for the ALA conference and/or the following ATLA conference, we hope you will also make time for the CRRA All-Members meeting.  This announcement also appeared in the CRRA March Update but we thought it might be good to send it out again after the Easter holiday. –Jennifer

You are invited to the annual All-Members meeting.  While we don’t know specific locations at this time, we will hold our events in easily accessible locations. The Anaheim Resort Transit Trolley has numerous routes connecting hotels, restaurants, shops, convention center and the Crystal Cathedral, and we will provide directions for getting to CRRA events.  Later we will ask for RSVP’s from those attending Monday’s dinner and/or Tuesday’s meeting and/or lunch so as to provide appropriately for a dinner reservation, and for breaks and lunch on Tuesday. On Monday evening, June 25, we will meet for dinner at a casual restaurant. We meet about 6:30.  We will make a group reservation.

We meet on Tuesday, June 26, from 9:00 a.m. through 12:30 p.m. followed by lunch (optional).  Our agenda is focused on mission-support for the next year: identifying top priorities, ideas for forming local teams and expanding our understanding of Catholic Studies.  With the announcement that the Board has adopted a five year strategic plan, we will be asking committees to develop their annual goals in this context and will be inviting all members to participate in identifying high priorities for the coming year.

Agenda

·         Welcome, Janice Welburn, chair, Board of Directors

·         Annual goals, objectives and priorities – Moderator, Pat Lawton

·         Forming institutional teams – Panel discussion TBA

·         Catholic Studies and challenges facing Catholic educators – Rev. James Heft, S.M. President, Institute for Advanced Catholic Studies at the University of Southern California and Member, CRRA Leadership Council

We look forward to meeting with as many of you as can be there. Please share this invitation with any others at your institution who may also be in Anaheim.  Traditionally, our meetings are open to others interested in our mission and activities. If you know of others who might like to attend, you can share this information or request that Pat or Jennifer do so.  See you there.

Jennifer Younger
CRRA Executive Director

Attending SAA in San Diego in August?  This session on Networks and EAD Consortia may be of interest:

Archival Networks and EAD Consortia
EAD consortia and aggregators of archival resources share broad interests in the ongoing exchange of information about each others’ projects and programs.  Why reinvent the wheel?

Where: SAA 76th Annual Meeting, San Diego Hilton Bayfront — room to be determined.  Please consult conference program for location details, once available.

When: Thursday, August 9, 2012, 12:00-1:15 pm

Goal: to increase communication across consortia, in order to share expertise and develop a common vision for broader archival description and discovery networks.

Agenda: brief regional/statewide/national program updates, followed by structured discussion.  Additional agenda details forthcoming.

Anyone interested is welcome to attend.

Jodi Allison-Bunnell, Orbis Cascade and NWDA
Jennifer Schaffner, OCLC Research
Adrian Turner, Online Archive of California and the California Digital Library

Fall Symposium at DePaul University, Oct. 15-16, 2012
Continuing on the success of the November 2011 Duquesne Symposium, plans are underway for a Fall Symposium to be held at DePaul University Oct. 15-16.  Please hold this date, and watch the CRRA Update for further details.

Position Available: Reference & Instruction Librarian, Duquesne University

 Duquesne University

Gumberg Library

Reference & Instruction Librarian
NATURE OF WORK:
This non-tenured library faculty position reports to the Director of Information Services.  This is primarily a public service position with significant instructional and liaison duties.  Knowledge of information sources, interpersonal skills, instructional skills, and technology skills are of highest importance for this position.  Provides reference service and instruction to enable members of the Duquesne University community and guest users to find and effectively make use of library resources and other information sources.

For the full posting, please see: http://www.duq.edu/hr/faculty/faculty-jobs-openings/gumberg.cfm

CRRA Update is an electronic newsletter distributed via email to provide members with an update of CRRA activities.  Please contact Pat Lawton at 574.631.1324 or email plawton@nd.edu with your questions, comments, or news to share. We welcome your news items!

———
CRRA Calendar: http://tiny.cc/Calendar798
CRRA Contact page: http://www.catholicresearch.net/About/Contact
CRRA blog: http://www.catholicresearch.net/blog/

More specifically, the Portal’s Web server log files are read on a daily basis, normalized, and saved to an underlying database. A number of queries are then applied to the database to create rudimentarily lists of tabulations. Each one of the reports are described below:

• Hosts – This report lists the Internet address or name of the top 100 computers using the Portal. To the best of our ability, the list excludes Internet robots and spiders, but the list needs to be updated. As of this writing, it is quite likely that many of the top computers are still robots, and the host named university.archives.nd.edu is probably the most frequent user of the Portal with shunat236-189.shu.edu coming in at a close second.
• Page count – This is a list of the number of hits the Portal received on any given day. Obviously the script creating this report needs to be updated in order to output data for the current year.
• Query strings – This is a tabulation of the most frequently used search terms applied against the Portal. The “null” query is probably a simple hit against the “browse” link at the bottom of the Portal’s home page and/or simply clicking the search box’s Find button. The queries in quotes are probably from clicks on hot linked search results.
• Referrers – This is a list of the websites where people came from before they visited the Portal. A whole lot of these websites are places where blog postings about the Portal appear. Many are spam. Some are HTML versions of the EAD finding aids. Further down the list one can begin to see Google searches.
• Referrers engines – This report is just exactly like the Referrers report except it only includes search engines (Google, Yahoo, and Bing).
• Tabs – This is a list of the most frequently used links used across the top of the Portal’s home page.
• Top records – This is a tabulation of the most frequently viewed records in the Portal. The first item on the list is an error, but as of this writing the most frequently viewed record is something from Catholic University of America.
• Types of searches – From this report is all but obvious that the overwhelming majority of the searches applied against the Portal are free text searches. Nobody uses the advanced search form.
• Whose records – This is a list of the names of the libraries/institutions whose records are viewed most frequently.

For a more technical description of how these reports are generated, see the blog posting entitled “Data warehousing Web server log files” as well as a follow-up posting called “Progress with statistics reporting“.

These reports can be improved in any number of ways. First, they could be represented graphically — pie charts, histograms, etc. Second, they could be re-generated on a month-by-month basis to look for trends over time. Luckily just about all the necessary data has been preserved. Alternatively, a peek at the Portal’s Google Analystics site may illuminate additional trends.

For the longest time the EAD files harvested for the Portal were validated against the EAD DTD. These files have no namespace declarations, and transformations were relatively easy. It was almost trivial for me to add unitid attributes to did-level elements. It was almost trivial for me to loop through the input files to extract did-level elements for indexing. Using a stylesheet I found through the Library Of Congress, it was easy for me to convert the EAD into an HTML file for online reading.

When I started getting EAD files generated from the venerable Archivist’s Toolkit my processes broke because these new files were validated against EAD schema which is full of two or three namespaces. None of my XPath statements worked. A number of people offered a number of suggestions. Some of them required the use of XSLT 2.0, which is not an option for me. Others thought I should update my existing stylesheets to accomodate the namespaces, but that would have been too complicated and not scalable.

In the end, I chose a different solution which was alluded to by a number of other people — remove the namespaces. Each person offered a slightly different take on the problem, but in the end I went for a brute force method I found in the TEI community Web space:

<xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
  <xsl:output method="xml" indent="no" />
  <xsl:template match="/|comment()|processing-instruction()">
    <xsl:copy>
      <xsl:apply-templates />
    </xsl:copy>
  </xsl:template>
  <xsl:template match="*">
    <xsl:element name="{local-name()}">
      <xsl:apply-templates select="@*|node()" />
    </xsl:element>
  </xsl:template>
  <xsl:template match="@*">
    <xsl:attribute name="{local-name()}">
      <xsl:value-of select="." />
    </xsl:attribute>
  </xsl:template>
</xsl:stylesheet>

Consequently, my XML processing pipeline now looks this:

1. harvest EAD files
2. validated them
3. strip namespaces
4. add unitids
5. transform them into HTML
6. index them
7. done

The next thing to do is improve Step #5 since the generic EAD to HTML transformation is just that — too generic.

The Problem

The vast majority of the content in the “Portal” is not digitized. Consequently, when items of interest are identified, the reader is left hanging because the Portal does not support document delivery. “Now that I have found this item, how do I get it?”

The Solution

The solution is not perfect, but rather a step in the right direction. Instead of delivering the item, the solution is to provide a means for the reader (I don’t use the word “user” anymore) to easily connect with the member institution libraries through a directory. Specifically, create a directory of member institution libraries/archives complete with names, addresses, and other pieces of contact information. Hyperlink each and every search result to specific entires in the directory and thus enable readers get in touch with member institutions.

I have implemented this in the Portal’s “sandbox”. Search for any item, and from both the search result page as well as detail holdings pages, the reader can click on the institutions’ library and be shown a (bogus) directory.

The implementation was much easier than I anticipated, and the key was found in the identifiers of each indexed record. (All puns intended.) Each indexed record in the Portal is prefixed with a code denoting the library holding the item. For example, Boston College’s code is bcu, and Loyola Marymount University’s code is lmu. When search items are returned VuFind’s IndexRecord record driver is called. In that code I am able to extract each record’s identifier, and parse out is first three characters — the code. I then pass this identifier and the library’s name on to my template for display:

$interface->assign('CRRALibrary', $this->fields['building'][0]);
$interface->assign('CRRAKey', substr ($this->fields['id'], 0, 3 ));

In the template I hyperlink the holding library’s name with the directory’s URL, and specifically, a named anchor for the library:

<a href='http://zoia.library.nd.edu/tmp/directory.html#{$CRRAKey}'>{$CRRALibrary}</a>

The directory I created was rudimentary at best, and it will be up to people other than me and including myself to determine how the directory gets created and what it looks like.

Hooray for open source software and object oriented programming techniques!

• add a a note to the email dialog box denoting how the from field is mandatory and requires an email address
• create a directory of institutions, and from search results hyperlink institutions’ names to the directory
• update the “Portal” look & feel (theme) so it is based on the “blueprint” theme
• turn off the “Suggested Topics” feature
• fix the author searches so when author names are clicked the content displays correctly
• make the login links float to the right instead of the left
• change the red text — such as the text in the search box — to black
• change the login label to read “Login / Create account”

On my mark. Get set. Go.

This is a set of (draft) prescriptive instructions describing how to make MARC and EAD metadata available in the “Catholic Portal“.

Introduction

At its core, the “Portal” is an index — a list of pointers to content items. Access to this index is implemented through a form-based interface. Readers enter queries into the form, and items are returned. Readers are then expected to select items of interest from the returned list, and use them for the purposes of research and scholarship. In order to implement this functionality, each content item in the index requires, at the very least, three elements: 1) a unique identifier, 2) a human-readable description of the item, and 3) a location code where the item can be acquired.

The MARC and EAD metadata schemes are well-suited for indexing. After making sets of MARC records and/or EAD files transparently accessible on a Web server, it is easy to harvest the metadata, integrate it into the Portal’s index, and provide access to the content items.

The balance of this posting describes how to make MARC and EAD files available for harvesting.

MARC

Here’s the short version. Export all the MARC records from your integrated library system you think are apropos to the “Catholic Portal” making sure they are encoded using the UTF-8 character set. Save the resulting file on a Web server, and tell Eric Morgan the URL of the resulting file. Eric will do the rest.

Here’s the long version. Remember, every record in the Portal needs a unique identifier, a human-readable description, and a location code. For MARC records, this means every record first needs a value in the 001 field. Any value will do as long as it is unique to your set of records. Second, each MARC record needs something in the 245 field. At the very least this will be the human-readable description. All the other descriptive and analytic fields will supplement this description. Third, each MARC record needs to have a location code, and this is the item’s call number. This value will most likely be extracted from the 090 field.

Helping you decide which MARC records to extract from your integrated library system is beyond the scope of this document. But once you have figured that out it is recommended you denote which items are to be extracted by updating them with a local note. Here at the University of Notre Dame, we put the letters CRRA in field 590 subfield a. Once this is done it is relatively easy for the systems librarian to do a search for CRRA in field 590 subfield a, and dump the resulting records to a file. Alternatively, the systems librarian might search for all items whose call numbers begin with BX and dump the resulting set. The process you use to denote and export your MARC records depends on your local environment.

When exporting your MARC records from your integrated library system, it is imperative the records be encoded using the UTF-8 character set and not something else. The Portal’s underlying indexer does not deal very well with encodings of another kind. If your system does not export records as UTF-8, and it exports things in MARC-8 instead, then use an open source application called yaz-marcdump from Index Data to transform your records from one encoding into another. Once yaz-marcdump is installed you can execute a command like the following to do the transformation:

yaz-marcdump -f MARC-8 -t UTF-8 -o marc -l 9=97 input.mrc > output.mrc

The command translates MARC records from (-f) MARC-8 encoding to (-t) UTF-8 encoding. It outputs (-o) the result as MARC records, and inserts the letter a (ASCII character 97) into the leader (-l) at position 9. It uses the file named input.mrc as input, and it outputs the result to a file named output.mrc.

Every time you export your records, you should export everything that you feel is relevant to the portal. Do not worry about additions, changes, nor deletions. We here at Portal Central handle this issue by deleting all of your records locally and re-indexing the whole lot.

After the records have been exported, save them on a Web server, and finally, tell Eric Morgan the URL of the resulting file. Please don’t change the name of the URL. Eric will harvest the records and incorporate them into the index. As of this writing it is a good idea to tell Eric when new records are available, but at some point in time this won’t be necessary.

EAD

Here’s the short version. Use validated EAD files to encode the content you deem apropos to the Portal. Save all the EAD files in a single directory on a Web server making sure each file is given a .xml extension. Tell Eric Morgan the URL of the directory, and he will take care of the rest.

Here’s the longer version. Use whatever tool you desire to create EAD files describing the archival content you deem appropriate for the Portal. There are any number of available editors and applications facilitating this process. Make sure the resulting EAD files validate against the EAD DTD or schema. It doesn’t really matter which one, but right now validation against the DTD is easier to handle here at Portal Central.

Each did-level element in your EAD files will eventually become a record in the Portal’s index. During pre-processing here at Portal Central, unique unitid attributes will be added to each did-level element, if no unitid attributes exist in the first place. This pre-processing satisfies the need for unique identifiers. You need to do nothing in regards to unique identifiers.

Each did-level unittitle element will recursively be combined with its parent did/unittitle element to form a human-readable description of each content item. Consequently, there is nothing you need to do in regards to human-readable descriptions.

The location of items found in EAD files is facilitated in three ways. First, the name of your hosting institution and library/archive will be associated with each search result, thus the need for location information will be satisfied but only in a rudimentary way. Second, through the use of the url attribute of the eadid element, location information is re-enforced. Specifically, you are expected to include a value in the url attribute of the eadid element. This value is expected to point to a human-readable version of your EAD file on your Web server. Portal search results include hot links with a label similar to “View finding aid at owning institution”. The hot links will be the same as the value in the url attribute. Your human-readable version of the EAD file is then expected to include instructions and contact information describing how to acquire items of interest. Finally, search results will include a second hot link labeled similar to “View finding aid in Portal display”. These hot links will equal to a URL pointing to a local HTML file transformed from the original EAD. Again, location and contact information should be a part of the HTML because it was a part of the original EAD.

In summary, create complete and valid EAD files making sure you include values in the url attributes of the eadid elements.

Once you have created your EAD files, save them in a single directory on a Web server, and tell Eric Morgan the URL of the directory. Make sure each EAD file ends with a .xml extension. Eric will then regularly harvest all the .xml files from your directory, re-validate them, make sure they include url attributes, add unique identifiers to each did-level element, and index each did-level element.

Starting somewhere near version 1.2, VuFind supports the indexing of arbitrary metadata types. Content from OAI repositories was the original example. Later, I figured out how to index EAD files. This was a break through for the “Portal”. Give credit to open source software.

With the addition of the Philadelphia Archdiocesan Historical Research Center (PAHRC) into the Catholic Research Resources Alliance, a new metdata format needed to be accepted — metadata other than EAD or MARC. PAHRC uses “cataloging” software called PastPerfect. From what I can tell, it is a sophisticated FoxPro/Microsoft Access database application. It provides the means for institutions to do data entry, and have their holdings searched, and ultimately displayed on the Web.

PastPerfect can export its metadata in a form of Dublin Core. After working closely with Shawn Weldon, Faith Charlton (both of PAHRC), and Brian Gomez (Past Perfect, Inc), the metadata exported by PAHRC was tweaked to be less ambiguous and more accurate. Once this was done I was able to harvest the metadata, parse it into something usable by VuFind’s Solr indexer, and make it available through the Portal. I did this with a script called pastperfect-index.pl. The result is a set of searchable records from PAHRC.

My current implementation is specific to PAHRC, and when other PastPerfect libraries/archives come on board, it will not be too difficult to abstract my implementation to support other institutions. That work is left to the future, when and if it occurs.

Fun with open source software!

The “Catholic Portal” is intended to contain links to and content of a rare and infrequently held nature. Every once in a while search results return duplicate records. For example, yesterday, it was brought to our attention that there are five records with the title Life Of Mrs. Eliza A. Seton. On one hand, few if any of these records are duplicates because between the five of them they are held by two different institutions. And each institution owns multiple editions. In the sense of a “catalog”, this is perfectly acceptable, if not expected. On the other hand, the Portal is not a catalog but rather an index, and each of the five items are really a variation on a theme. Should these records be merged?

Demian Katz shared with me and the Portal’s Digital Access Committee a query that can be applied the Portal’s underlying Solr index, here, with carriage returns added for readability:

http://localhost:8080/solr/biblio/select/?
q=*%3A*&rows=0&start=0&facet=true&facet.mincount=2&
facet.limit=-1&facet.field=oclc_num&facet.field=isbn

The result of this query is a list of OCLC and ISBN numbers which occur in the index at least two times. According to the result, which only matches on the OCLC or ISBN keys, there are no records in the index appearing more than three times. Furthermore, there are about 1,100 duplicated OCLC numbers and about 300 duplicated ISBN numbers. Considering the total number of records (93,000) in the index, this represents a total duplication rate of approximately 1.5%. Is this value too high?

In an ideal world, there would be no duplicate records and/or duplicates would be merged into a single record. Unfortunately, the definition of “duplicate” is ambiguous, and a process for eliminating duplicates has not been implemented. To a Walt Witman scholar, the difference between various editions of The Leaves Of Grass is definitely significant. Thus, sometimes the differences in editions is very important. Other times and for other people, this is not always so important. In an ideal world, there would be no duplicates and a single record would warrant a de-duplication process, but the expense of de-duplicating that single record may be very high, especially if there is no de-duplication process in place. How many records — or what percentage of records — warrants a de-duplication process, especially considering the other things that have been set as priorities for the Portal? Honestly, I don’t know the answer.

Marta Deyrup and Martha Loesch, catalogers at (CRRA instution) Seton Hall University, and Pat Lawton, digital projects librarian for the CRRA, have released the results of their Survey of Digitized Rare Catholica held by Catholic universities, colleges, seminaries and archives in the U.S. and Canada. You may view the Summary Report of Results and the results data.

Specifically, I needed to delete all of the records in the Portal from the University of Notre Dame Archives because the Archives wanted to totally replace what finding aids were available. This meant deleting more than a 100,000 records from the underlying index. After a bit of investigation, I learned that at the following one-liner from the command line would do the trick:

curl http://localhost:8080/solr/biblio/update?commit=true -H "Content-Type: text/xml" --data-binary '<delete><query>id:unaead_*</query></delete>'

In short, curl is a command-line HTTP client. It is being told to first connect to the local host on port 8080. It is then told to find all the records matching the query “id:unaead_*” and delete them from the index named biblio. Once that is done, the underlying index is expected to commit the changes. Deleting these records took about ten minutes. I was then able to use my previously created scripts to harvest, validate, transform, and index the Archives’ content painlessly.

It is a pleasure when things work in the way they were designed! Now if I could only get my local indexing process to work faster.

Liberty Bell

cheese steak sandwich

Joe Lucia (Villanova University) initialized the meeting and set the stage by recommending a book called The Googlization of Everything. It advocates the creation of an open knowledge commons similar to the ones at the root of the fledgling Digital Public Library of America. To paraphrase his remarks, “Everything we do in our shop here embrases the open knowledge commons concept… Libraries are not just purveyors of content, but also creators of content — The New Resource Sharing. We [librarians] can become agents of information creation.”

The first presentation was given by Amy Baker Williams (University of Pittsburgh), and she described her process for conserving the maps of local coal mines. In the Pittsburgh (Pennsylvania) area there are many coal mines dating back as far as 1750. Some of the oldest maps of the mines date from 1850. A few years ago some miners were trapped in a mine, and if maps of the mines had been easily accessible, then rescue efforts would have been simplified. Since then concerted efforts have been made to preserve, digitize, and make accessible as many of these coal mining maps as possible in order to prevent similar accidents from happening in the future. I found the process used to flatten the maps to be the most interesting. Basically they are re-hydrated and unrolled. Moving the maps from the conservation lab to the scanning location was also interesting because, ironically, the maps are rolled up again for transportation as well as long-term storage. For more detail, see the website.

My presentation was next, and I shared with the audience how the Catholic Research Resources Alliance (CRRA) is using VuFind to implement the “Catholic Portal”. I first described the mission and history of the CRRA. I then outlined the Portal’s technical architecture as well as the process I used to index EAD files. Finally, I described how text mining functions have been integrated into the Portal’s interface emphasizing the possibilities for libraries in general.

Falvey Library

mural

During lunch we broke up into groups, and I sat with the folks interested in the digital humanities. For the most part we went around the table sharing common war stories. Most of our initiatives where fledgling, but there was plenty of enthusiasm.

After lunch a sort of “unconference” session was facilitated by David Upsal (Villanova University). The discussion topic that made itself apparent was the challenge of the profession to serve both traditional librarianship as well as librarianship in the current environment. If my memory serves me correctly, some of the suggested solutions included more resources (people and money), permission to “play” with new technology, a redefinition of library purpose, and greater collaboration between different types of libraries (public, academic, etc.)

The next presentation was given by Eric Zino (LYRASIS) who described how LYRASIS has been working with the Sloan Foundation and the Internet Archive to facilitate the digitization of 20,000,000 pages of library content. Approximately 160 libraries have been participating in the project with LYRASIS. Subsidized by the Foundation, partipants package up their content and ship it to the Internet Archive. The content gets digitized, returned to the owning library, and the digital versions are made accessible at the Archive. From my perspective, this is exactly how any other library works with the Archive, except in this case LYRASIS does a bit of hand-holding during the process. Not all media is digitized by the Archive though. Some things, such as microfilm, are scanned by a different vendor — Creekside Digital.

The last presentation of the day was given by Bob Behary (Duquesne University), and he shared with the audience how Duquesne is digitizing a newspaper called the Pittsburgh Catholic. The project was initiated by a Catholic order called the Spiritans (the founding order of Duquesne University) with evangelism at its root. At first digitized versions of the newspaper were put on CDs and distributed. This has evolved over time, and now the content is housed in a ContentDM system. The collection has proven useful in a number of ways, including: local & regional church histories, literature allusions (such as Emily Dickinson), and United States history. Behary listed a number of key considerations for any digitization effort: 1) get administrative support, 2) make sure the project fits within the mission of the institution, 3) make sure to use sustainable technology, and 4) ensure knowledgable research advocates are a part of the process.

Vuee Award

Art Museum staircase

I believe the meeting was attended by fifty to seventy-five people. Most were from the immediate area, and it offered a easy opportunity for professional development. Kudos to the folks at Villanova for hosting the event. Just before the meeting concluded I was awarded the second annual “Vuee” for best presentation. It is a small shoebox-sized container in the shape of a book. I was very flattered. “Thank you very much!”

This posting describes how EAD files are indexed in the “Catholic Portal” with VUFind.

VUFind is a “next-generation library catalog” or “discovery system” application. Its primary purpose is to index bibliographic metadata and provide a reader-friendly interface to the result. The heart of this process is a Solr index made up of many bibliographic-like fields. These fields are the usual suspects including a host of variants on author, title, institution, building, collection, language, format, physical description, publisher, published date, edition, description (note), contents, URL, call number, ISSN, ISBN, OCLC number, series, topic, genre, geographic, era, illustration, full text, and record type. In order for EAD files to be searchable in the Portal, they need to have their metadata extracted, the metadata needs to be mapped to Solr fields, and the metadata needs to be added to the index. The balance of this posting describes this in more detail.

Pre-processing

Before any indexing can take place, bits of pre-processing are applied against the EAD files. In a nutshell, this pre-processing (and the Perl scripts doing the work) includes:

1. harvesting the EAD files from a remote HTTP server and caching them locally (ead-harvest.pl) – Done so the balance of the work can be done.
2. validating the EAD files against the DTD and/or schema (ead-validate.pl) – Done because we don’t want to practice GIGO (Garbage In, Garbage Out).
3. adding unique identifiers to each did-level element of the EAD files (ead-transform.pl) – The Solr indexer requires unique identifiers for each indexed item. This process provides the identifiers as well makes it easy to hyperlink directly to a place in the EAD through the use of HTML anchors.
4. transforming the EAD files into HTML and making the results Web accessible (ead-transform.pl) – Done because links to remote versions of the EAD files break, and humans do not read XML very well.

Indexing

The bulk of the indexing process centers around the acquisition of metadata, and it is completely handled by a Perl script named ead-index.pl:

1. The process begins by looking up the name of the institution and the name of the library from where the EAD file was created. These values are located in a rudimentary tab-delimited database.
2. Next, the value for record type is denoted. It is always “EAD”.
3. Third, a value for format is denoted. It is always “Archival material”.
4. Next, the language of the material is extracted from the /ead/archdesc/did/langmaterial/language element. If no language is specified, then language is denoted as “Unknown”.
5. Each did-level element from the EAD file is then examined pulling out its unique identifier (the id attribute of unitid element created in Step #3 of pre-processing), title (the unittitle element), and date (the unitdate element). The title metadata is a bit special since it is really a concatenation of all the parent title values of the given did element. This is done because each item in an EAD file is a part of the entire collection, and this enhanced title is intended to provide context.
6. At this point the metadata for each did-level element has been extracted and is mapped to a select number of Solr fields, namely:
   • id -> unique identifier;
   • title -> title
   • title_auth -> title
   • title_full -> title
   • title_fullStr -> title
   • title_full_unstemmed -> title
   • title_short -> title
   • title_sort -> title
   • publishDate -> date
   • format -> always “Archival material”, from Step #2
   • institution -> the name of the library’s hosting institution, from Step #1
   • building -> the name of the library, from Step #1
   • fullrecord -> An XML snippet containing the unique identifier, title, date, as well as two URLs pointing to HTML versions (local and remote) of the EAD file
   • recordtype -> always EAD, from Step #3
   • language -> language, from Step #4
7. Finally, the metadata is added to VuFind’s underlying Solr index.

Discussion

The indexing process is far from perfect. For example, in the current process, the entire head element of the EAD file is ignored. While it contains very rich metadata, such as controlled vocabulary terms and abstracts, these values describe the collection as a whole and do not necessarily apply to each individual did-level element.

Second, creating EAD files is laborious in the first place. There are not enough resources in most archival departments to describe did-level elements with much more detail than title and date. It would be nice to have a narrative summary describing of each did-level element, a more specific format, some key words or controlled vocabulary, a consistently formatted date, etc. But again, creating such metadata for each did-level element is expensive. Consequently, indexed items are not described as robustly as possible.

Third, while VuFind’s implementation of Solr is bibliographic in nature, it is heavily weighted towards bibliographic metadata describing books. OCLC number. Call number. ISBN & ISSN. Edition. Etc. There are no fields for EAD-specific things such as postal addresses, provenance, nor biographies.

Again, the process is not perfect, but it does enable the Catholic Research Resources Alliance to amalgamate the metadata of its member institutions and provide a searchable index to the result. Suggestions for improvement are welcome.

The registration deadline for this CRRA/Duquesne sponsored event is this Friday, October 15, 2011.  We are pleased that many of you have already registered for the event and if you have thought about registering, please do so now.  There is no fee to register.

The event features Catholic scholars, archivists, and librarians gathering together to consider the state of Catholic scholarship and how we can act together to advance and enhance freely available global access and discovery of important Catholic resources. The event will take place at Duquesne University (Pittsburgh) on Nov. 9-10.

We encourage librarians, scholars, and archivists interested in learning more about opportunities to make scholarly resources accessible to join in and meet new friends and colleagues.

A full roster of events and registration information is available at http://bit.ly/Duquesne_Symposium .   The registration deadline is this Friday, October 15, 2011.

We hope that you will join us in what promises to be a stimulating and productive conversation about Catholic scholarly research and the ways in which librarians and archivists support this research.

On behalf of Duquesne University and the Catholic Research Resources Alliance (CRRA),
•         Jennifer Younger, chair, Board of Directors at younger.1@nd.edu
•         Laverna Saunders, University Librarian, Gumberg Library, Duquesne University at lsaunders@duq.edu
•         Pat Lawton, CRRA Digital Projects Librarian at plawton@nd.edu

CRRA Update

July/August 2011

We begin a new semester and a month of many welcomes!  Join us in welcoming the Catholic Theological Union, board members, board and committee chairs, a new baby (Otto Ray Katz), the Five Year Strategic Plan Task Force members, and executive director, Jennifer Younger.

Welcome, Catholic Theological Union!

We are pleased to welcome Catholic Theological Union (CTU) The Paul Bechtold Library, under the leadership of Melody McMahon, to the CRRA.  Lisa Gonzalez, Electronic Resources Librarian, will join Melody in her work with CRRA, and will serve as a member of the Digital Access Committee.  Welcome, CTU, Melody and Lisa!  One of our goals for the coming year is to implement an instance of Archivists’ Toolkit, for which CTU has generously volunteered to act as a pilot user.

Catholic Theological Union is the largest Catholic graduate school of theology and ministry in the United States. Founded in 1968 in the spirit of Vatican II, there are currently 32 religious orders that send students to CTU, as well as lay students from the United States and around the world.  The Bechtold Library collection is particularly strong in materials pertaining to religious orders, Franciscan studies and catechetical materials. The library contains over 150,000 volumes, and has a varied art collection that was included in the Art and Architecture in Illinois Libraries project.

The library houses both the Weber-Killgallon Center collection of catechetical materials and the Stuhlmueller Room, which contains the personal library of Carroll Stuhlmueller. Stuhlmueller, a Passionist priest and Biblical scholar, served on the faculty of CTU until his death in 1994. In addition,collections housed in the CTU archives include the papers of catechists Gerard Weber and Irene H.

Murphy, the archives of the North American Academy of Liturgy, and the papers of the Women Religious Imprisoned Under Eastern European Communism project.

Melody Layton McMahon, Director of the Paul Bechtold Library

Melody Layton McMahonstarted her career as a music librarian (Juilliard, Cleveland Institute of Music) and then worked for twelve years at John Carroll University, a Jesuit liberal arts university. Always having an interest in theological librarianship, she came to CTU in 2008. She has been an active participant in the Catholic Library Association, the Ohio Theological Library Association, the Chicago Area Theological Library Association, and the American Theological Library Association, serving on a number of committees.

In recent years, Melody’s vocational interests have extended into writing, including numerous reviews,  an article on faculty-library collaboration in Theological Education (2005), co-editing an anthology of writings on theological librarianship (A Broadening Conversation: Classic Readings in Theological Librarianship, 2006), and serving as the Critical Reviews Editor for the recently-launched online journal Theological Librarianship: an Online Journal of the American Theological Library Association  (www.theolib.org). (M.S., School of Library Service, Columbia University; M.A., St. Mary Seminary and Graduate School of Theology)

Lisa Gonzalez, Electronic Resources Librarian, Paul Bechtold Library
Lisa Gonzalez has worked in libraries in Illinois and California, including Trinity Christian College and Azusa Pacific University. Lisa earned her M.L.I.S. from Dominican University and an M.A. in Theology from Fuller Theological Seminary. She has worked as the Electronic Resources Librarian at CTU since September 2008, and currently serves as the Communications Officer for CATLA, as well as serving as a member of the I-OPAC Team for the I-Share group catalog in Illinois.

Lisa serves as the newest member of the Digital Access Committee – welcome, Lisa!

Thank you to Departing Board members and Welcome to the 2011/12 Board

We owe thanks and a hearty “job well done” to three Board members who have served since the Board was set up in February 2008. Each of them has contributed mightily to defining and carrying out the CRRA mission, not only through their Board service but also in leading CRRA committees.  We have benefited from their unique blends of vision, commitment and proactive leadership and we thank you for giving so generously of your time and talents.

• Tom Leonhardt, St. Edward’s University and chair, Digital Access Committee
• Tim Meagher, The Catholic University of America and chair, Scholars Advisory Comm.
• Bob O’Neill, Boston College and chair, Collections Committee

In selecting new Board members, the Board considered factors relevant to overall Board composition.  We agreed it is desirable to have representation from small and large institutions as well as from sustaining members. We also wanted to bring in members from institutions not currently represented on the Board and to consider individuals who have expressed prior interest in Board service.  Although we did not specifically discuss geographic or gender distribution, the new Board is geographically diverse and gender-balanced. With the inclusion of the new position of executive director as an ex officio member, the Board increased from nine to ten members.

Roster of CRRA Board of Directors for 2011/12 (in alphabetical order)

• Theresa Byrd, University Librarian, University of San Diego
• Tyrone Cannon, Dean, University Libraries, University of San Francisco
• Steve Connaghan, Director of Libraries, The Catholic University of America
• Artemis Kirk, University Librarian, Georgetown University
• Joe Lucia, University Librarian and Director of Falvey Memorial Library, Villanova University
• Evelyn Minick, University Librarian, Saint Joseph’s University
• Susan Ohmer, Director, Office of Digital Asset Management, University of Notre Dame
• Tom Wall, University Librarian, Boston College
• Janice Welburn, Dean, University Libraries, Marquette University and Board chair
• Jennifer Younger, Executive Director, CRRA, ex officio 

CRRA Welcomes New Board and Committee Chairs
Stepping up to carry on the good work of our illustrious founding Board and committee chairs, Jennifer Younger, Tim Meagher, Bob O’Neill, and Tom Leonhardt are Janice Welburn (Board of Directors), Jean McManus (Scholars Advisory Committee), Matt Blessing (Collections Committee) and Demian Katz (Digital Access Committee).  We warmly welcome you and the opportunities your leadership will afford the Alliance. Please join us in welcoming our new board and committee chairs!

Janice Welburn, Chair, Board of Directors
Janice Welburn, Dean of University Libraries at Marquette University, continues Marquette’s role as a founding member of the CRRA.   Janice is the recipient of the prestigious 2011 Association of College and Research Libraries’ (ACRL) Academic/Research Librarian of the Year. The award, sponsored by YBP Library Services, recognizes an outstanding member of the library profession who has made a significant national or international contribution to academic/research librarianship and library development.

As Board member and co-chair of the Budget & Personnel Committee, Janice brings a thoughtful approach to understanding and advancing our mission. At Marquette, she actively engages a team that supports the CRRA through their activities. Activities include contribution of metadata records to the portal, committee participation and leadership of the successful collaborative CLIR grant for cataloging unique materials at Catholic, St. Catherine and Marquette; all of which have resulted in significant contributions to our growth and success.  This year, Janice anticipates the development of a five year plan for the CRRA that will inspire and guide our activities going forward.

Jean McManus, Chair, Scholars Advisory Committee
Jean McManus has an A.B. in English from Bryn Mawr College and an M.A. in Library Science from the University of Chicago. She has worked primarily in reference services and collection development, with formative stints in the worlds of serials and interlibrary loan. Since 1997, Jean has been at the University of Notre Dame (UND), and has been involved with CRRA and UND’s Team Catholic Portal since 2008.

Matt Blessing, Chair, Collections Committee
Matt Blessing is the head of the Department of Special Collections and University Archives at Marquette University. Prior to joining Marquette he served as director of the Mass Communications History Center at the Wisconsin Historical Society.  Matt also currently serves on the boards of Wisconsin Heritage Online and the Wisconsin Historical Records Advisory Board.  He is also active within the Midwest Archives Conference.

Marquette University serves as the archives for numerous Catholic groups and organizations, including the Catholic Worker movement, Religious Formation Conference, Catholic Library Association, Women’s Ordination Conference, National Catholic Rural Life Movement, in addition to several historic American Indian schools and missions .  Marquette also preserves the manuscripts of prominent Catholic writers,  including J.R.R. Tolkien, Dorothy Day, and Penny Lernoux.

Matt has served as a member of the Collections Committee since the 2007 meeting at Notre Dame.

Demian Katz, Chair, Digital Access Committee (DAC)
Demian Katz has a B.S. in Computer Science from West Chester University and an M.L.I.S. from the University of Pittsburgh.  Over the course of his career, he has worked both in and out of libraries as a computer programmer and as a provider of reference services.  He is currently employed by Villanova University, where he serves as the lead developer of the VuFind discovery software, and he is greatly enjoying the opportunity to apply both aspects of his background to a single job while working with an enthusiastic open source development community that includes the CRRA members behind the Catholic Portal.

Demian has served on the DAC committee since 2010 and – Demian is the proud father of the newly arrived Otto Ray Katz, born at 8:12 on 8/12!  Welcome to the world, Otto, and congratulations, Demian and family!

CRRA Welcomes Executive Director

Posted on behalf of Janice Welburn, chair, Board of Directors and co-chair, Budget & Personnel Committee and Artemis Kirk, co-chair, Budget & Personnel Committee

Jennifer Younger, CRRA Executive Director
In 2009, CRRA took decisive action to expand our capacity to carry out our mission by hiring a digital projects librarian, Pat Lawton, as our first staff member. As a result, we have in the last two years been able to set and achieve ambitious goals.  Working collaboratively across CRRA committees, the Board and individual members, we have grown from eight to twenty-seven members and the portal now provides access to over two hundred thousand items held by CRRA members. Scholars, librarians and archivists are noting the collections through their blogs and providing valuable input on highly valued resources to add to the portal.  Although many search queries come from individuals looking for the CRRA or the portal, we also see searches in which users are finding materials, such as the recent search query for “Monsignor Martin B. Hellriegel” whose papers are at Boston College with related materials held by Marquette University.

Just as our alliance has grown, so have our needs. In January 2011, the Board determined that an executive director would assist the CRRA in continuing its growth and impact.  On behalf of the Board, we are pleased to announce that Jennifer Younger began her appointment as the Executive Director on July 1, 2011. Together, she and Pat Lawton form an effective partnership that will lead and serve us well. Please join us in welcoming Jennifer. — Janice Welburn and Artemis Kirk

CRRA to Develop Five Year Strategic Plan/Task Force Members Named

From Janice Welburn, chair, Board of Directors and the Board of Directors

In June 2011, the CRRA Board of Directors approved the establishment of a task force for the development of a five year strategic plan (FY2012/13-2016/17) to identify key directions and goals for the CRRA and the Catholic portal. The plan will include a statement of core values, vision and mission, the benefits of CRRA in advancing Catholic scholarship, as well as directions and strategies to carry out the mission and deliver value to stakeholders.

We proposed a task force of five to seven members to include representation from CRRA committees, the Board, members at large, and Pat Lawton, Jennifer Younger and Terry Ehling, Strategic Consultant, as ex officio members (CRRA May/June Update). The responses to the call for volunteers came quickly and we are pleased to name the following individuals to the CRRA Five Year Strategic Planning Task Force.  After consultation with the members, we will appoint a chair. As noted in the announcement, the Task Force will solicit input from members as part of its work. We can achieve our mission only through the direction and participation of our members. We thank you for volunteering to serve.

Task Force Members include:

1. Jonathan Bengtson, University of St. Michael’s College and member, Scholars Advisory Committee
2. Stephanie Clark, Georgetown and member-at-large
3. Ann Hanlon, Marquette University, member of Digital Access Committee
4. Ingrid Hsieh-Yee, Catholic University and member-at-large
5. Joe Lucia, Villanova University and Board member
6. Lorraine Olley, University of Saint Mary at the Lake/Mundelein Seminary and member, Collections Committee
7. Diane Parr Walker, Notre Dame and member-at-large
8. Tom Wall, Boston College and Board member

Mark Your Calendars … for the Duquesne Symposium November 9-10, 2011

Advancing Catholic Scholarly Research: A Symposium at
Duquesne University
November 9-10, 2011

 We are pleased to invite you, members of your staff, students and faculty at your institution to a symposium to be held November 9-10, 2011 at Duquesne University. Invited Catholic scholars and librarians will discuss the “state of the art” of Catholic scholarship, directions that scholarship is headed, and how libraries, archives, and member organizations support and nurture future Catholic scholars and scholarship.

We are honored to have Leslie Tentler (keynote speaker), Professor of History at Catholic University, Paula M. Kane, John and Lucine O’Brien Marous Associate Professor of Contemporary Catholic Studies, University of Pittsburgh, Joseph P. Lucia, University Librarian and Director, Falvey Memorial Library, Villanova University, Dr. Kevin Mongrain, Ryan Chair for Newman Studies, Duquesne University and Executive Director, National Institute for Newman Studies and Dr. Michael Galligan-Stierle, President and CEO, ACCU speaking to the symposium theme, “Advancing Catholic Scholarship.”

Other events will include a Digital Projects Showcase, highlighting innovative technologies, best practices, future trends, and related scholarship related to our theme. CRRA members are especially encouraged to participate in the showcase by submitting poster proposals by September 26: <http://www.catholicresearch.net/info/events/Call_for_Posters.pdf>.

A full roster of events and registration information is available at http://bit.ly/Duquesne_Symposium. The registration deadline is October 15, 2011. Registration is open and space is limited.

We hope you will join us in what promises to be a stimulating and productive conversation about Catholic scholarly research and the ways in which librarians and archivists support this research.

On behalf of Duquesne University and the Catholic Research Resources Alliance (CRRA),

·         Jennifer Younger, chair, Board of Directors at younger.1@nd.edu

·         Laverna Saunders, University Librarian, Gumberg Library, Duquesne University at lsaunders@duq.edu

·         Pat Lawton, CRRA Digital Projects Librarian at plawton@nd.edu

Please mark your calendars …
… to attend a roundtable discussion on “Building a Catholic Archival Network” at the American Catholic Historical Association (ACHA) meeting in Chicago on Friday, January 6, 2012 from 9:30-11:30 a.m. at the Chicago Downtown Marriott, Northwestern Room. This topic is of vital importance to many of us and we invite you to join in as participants present their vantage points to realizing a network of Catholic archival resources.  Please join us.

Speakers will include:

Emilie Gagnet Lumis [Archdiocese of New Orleans, lleumas@arch-no.org]
Patricia A. Lawton [Catholic Research Resources Alliance, plawton@nd.edu]
Ellen D. Pierce [Maryknoll Mission Archives; epierce@maryknoll.org]
Chair: Robert E. Carbonneau [Passionist Historical Archives, NJ; RobCarb@cpprov.org]

Thank you, Malachy McCarthy, for taking the lead to assemble this discussion! We look forward to seeing you in Chicago.
CRRA Update is an electronic newsletter distributed via email to provide members with an update of CRRA activities.  Please contact Pat at 574.631.1324 or email plawton@nd.edu with your questions, comments, or news to share.

Announcing the Self-Subscribe Option to CRRA Update
We are testing a self-subscribe option for those interested in receiving the CRRA Update.  The self-subscribe option will make it easy for individuals from member institutions and beyond to sign up.

For current subscribers, there is nothing you need to do!  You will remain on our mailing list and will continue to receive Updates until you tell us to stop.  : )

We have tested the self-subscribe option over the last couple of months and it has worked well.  Please encourage colleagues interested in receiving our newsletter to sign up. We will monitor the list of subscribers and will continue to add names of all on our mailing list to our Contacts page under “Other Institutional Contacts.”

To self-subscribe to the CRRA Update:

1. Address a message to listserv@listserv.nd.edu
2. Enter “subscribe crra-updates-l” in the body of the message
3. Leave the subject line blank
4. Send

You will then need to confirm your subscription and you will then be subscribed.   List subscribers are then entered on the CRRA list of other institutional contacts: <http://www.catholicresearch.net/About/Contact>.

We appreciate any feedback you may have concerning this option.

All CRRA events and events of possible interest to members are posted to the CRRA calendar, please bookmark this link for future reference.

Check our progress and news on the CRRA blog: http://www.catholicresearch.net/blog/.

———
CRRA Calendar: http://tiny.cc/Calendar798
Duquesne/CRRA Symposium:
- Call for Posters http://www.catholicresearch.net/info/events/Call_for_Posters.pdf
- Symposium Details http://bit.ly/Duquesne_Symposium

Yesterday I broke down and re-wrote my log file import application. Instead of parsing the log, ingesting the results, and then post processing, I re-wrote the application so it does all of this in one pass. I also enhanced the program so it could take command line input. Specifically, if no arguments are supplied, then it will import yesterday’s log data. Otherwise it expects two inputs: 1) a beginning date, and 2) an ending date. If given these inputs, then I can drop the entire database and re-create it almost effortlessly. The script is called log-load.pl, and it is now running under cron so the database gets updated daily.

The next step is to create and automate reporting functions. I have already created a number of SQL queries. They are designed to be run from a shell script which outputs results to plain text files. These plain text files are presently put on the Web in a temporary location. This process is rather brain-dead. The next steps will include creating some sort of Web-based front-end allowing readers (increasingly I don’t use the word “users”) to complete some sort of form and get real-time results.

Wish me luck.

 

 Dear CRRA Colleagues,

We are pleased to invite you, members of your staff, students and faculty at your institution to a symposium to be held November 9-10, 2011 at Duquesne University. Invited Catholic scholars and librarians will discuss the “state of the art” of Catholic scholarship, directions that scholarship is headed, and how libraries, archives, and member organizations support and nurture future Catholic scholars and scholarship.

We are honored to have Leslie Tentler, Professor of History at Catholic University, Paula M. Kane, John and Lucine O’Brien Marous Associate Professor of Contemporary Catholic Studies, University of Pittsburgh, Joseph P. Lucia, University Librarian and Director, Falvey Memorial Library, Villanova University, Dr. Kevin Mongrain, Ryan Chair for Newman Studies, Duquesne University and Executive Director, National Institute for Newman Studies and Dr. Michael Galligan-Stierle, President and CEO, ACCU speaking to the symposium theme, “Advancing Catholic Scholarship.”

Other events will include a Digital Projects Showcase, highlighting innovative technologies, best practices, future trends, and related scholarship related to our theme. CRRA members are especially encouraged to participate in the showcase by submitting poster proposals by September 26: <http://www.catholicresearch.net/info/events/Call_for_Posters.pdf>.

A full roster of events and registration information is available at http://bit.ly/Duquesne_Symposium.   The registration deadline is October 15, 2011.

We hope that you will join us in what promises to be a stimulating and productive conversation about Catholic scholarly research and the ways in which librarians and archivists support this research.
On behalf of Duquesne University and the Catholic Research Resources Alliance (CRRA),

·         Jennifer Younger, chair, Board of Directors at younger.1@nd.edu

·         Laverna Saunders, University Librarian, Gumberg Library, Duquesne University at lsaunders@duq.edu

·         Pat Lawton, CRRA Digital Projects Librarian at plawton@nd.edu

In an effort to improve SEO (search engine optimization) I have done my best to implement sitemaps against the “Catholic Portal’s” VUFind implementation.

Sitemaps are XML files listing all the individual files/resources of a website. The intention and structure of these files is documented at Sitemaps.org. By exposing a site’s content in this way Internet robots/spiders can slurp up sitemap files’ URLs, go directly the resources without crawling, and index the content found there. In short, sitemaps make it easier for Internet indexers to do their job.

Implementing sitemaps in VUFind is relatively trivial. Edit a configuration file (web/conf/sitemap.ini), and run the sitemap file generator (php util/sitemap.php). See the VUFind documentation for more detail. Here at Portal Central I configured sitemap.ini with the following values:

• frequency = monthly
• countPerPage = 10000
• fileName = sitemap
• fileLocation = /shared/catholic_portal/data/data/sitemaps/
• baseSitemapUrl = http://www.catholicresearch.net/sitemaps
• baseSitemapFileName = baseSitemap

The only configuration which differs from the norm is the value of baseSitemapUrl. Instead of putting the sitemap files in the root of the VUFind filesystem I am having them saved in a directory called sitemaps. While such a thing is discouraged by the folks at Sitemaps.org, it keeps my filesystem clean, and more importantly, it makes it easier for me to migrate from one version of VUFind to another. Besides, the Google Webmaster tools ask for the specific location of one’s sitemap files and I don’t really need them to be discoverable by too many other indexers. All the other indexers pale in comparison.

Because I put all of the sitemap files in a separate directory, I needed to edit my httpd.conf file so VUFind does not try to interpret the directory as an action. My configuration follows:

  # sitemaps
  Alias /sitemaps /shared/catholic_portal/data/data/sitemaps/
  <Directory "/shared/catholic_portal/data/data/sitemaps">
    Options FollowSymLinks ExecCGI +Includes +Indexes
    AllowOverride FileInfo
    Order deny,allow
    Allow from all
  </Directory>

Is this an overly complicated solution? Maybe. It is cleaner? In my opinion, yes. But heck, all of this is only differences in configuration.

We here at “Catholic Portal” Central we generally run two simultaneous versions of VUFind. One is production, and the other is a test/sandbox implementation. Since the both run on the same hardware it is necessary to configure them differently. Here is a list of things I need to change when upgrading from the “sandbox” to production:

• the HTTP port from 88 to 80
• the path to the VUFind directory
• the Jetty port from 8088 to 8080
• the path to Solr
• the name of the MySQL database storing user accounts
• the name of the MySQL schema
• the location of the MySQL schema
• the environment variables defined on vufind.sh
• the name of the MySQL schema file, from vufind_test.ini to vufind.ini

I also need to copy content from the previous production environment to the new production environment, but I suppose I would not need to do this if I were to copy them when I initially create my sandbox in the first place. In any event, they include:

• interface/crra/css/common.css (for local styling)
• interface/crra/css/style-about.css (for local styling)
• layout-about.tpl (for local design)
• the contents of interface/crra/About (for local links)
• the contents of services/About (for local links)
• the contents of web/etc (for the directory, mostly)

All of this doesn’t count the local hacks/additions I’ve made to VUFind such as EADRecord.php, tweaks to MARCRecord.php, and the various Perl scripts used for indexing.

All in all though, much of the changes needed to upgrade from the sandbox to production have been listed here. ‘Off to buy that winning lottery ticket now…

1. wording in regards the linking of online finding aids
2. toggling the check box associated with filters

With this version there are also quite a number of additional records in the underlying index — around 280,000. This is because the finding aids (EAD files) have been indexed more completely.

Next steps include:

• turning on Google site maps to facilitate better SEO (search engine optimization)
• installing version 1.2 in the “sandbox” environment
• implementing the results of other CRRA member usability studies
• figuring out how to speed up the indexing process
• figuring out how to check for valid MARC records and thus paving the way for better automated updating.

Wish us luck.

In this issue you will find news items regarding:

• Tom Leonhardt Retires from CRRA Board of Directors and Digital Access Committee (DAC)
• Welcome to CRRA’s Six New Members
• From the Board: NEH Challenge Grant Proposal Deferred; CRAA to Develop Five Year Strategic Plan
• The Collections Committee Update: Developing Critical Mass Around Three Themes
• Member Institutions Continue Portal Usability Studies
• CRRA in ACCU News
• Mark your calendars to attend “Advancing Catholic Scholarly Research: A Symposium at Duquesne University.” November 9-10, 2011, Pittsburgh, PA.  Details at: http://bit.ly/lg0oCH

CRRA Welcomes Six New Members

Welcome to our newest members!

• Barry University (Miami Shores, FL);
• Catholic Theological Union (Chicago);
• The College of New Rochelle (New Rochelle, New York);
• Philadelphia Archdiocesan Historical Research Center (PAHRC) (Wynnewood, PA);
• St. Joseph’s College (Brooklyn, NY); and
• University of St. Mary at the Lake/Mundelein University (Mundelein, IL).

Look for more on our newest members in future updates.

Tom Leonhardt

After a distinguished career, Tom Leonhardt is retiring from St. Edward’s University, Austin, TX and returning to the Pacific Northwest.

From the beginning, Tom’s thoughtful voice has informed, encouraged and supported the development of the Alliance through his leadership on the Board and as Chair of the Digital Access Committee. Tom leaves with these words:

“I have been privileged to have been a part of CRRA from the time it was one person’s vision through an exciting period of growth led by another person with a shared vision and the organizational and leadership skills to make it all work. Working with other directors of Catholic university libraries brought me into a group that not only included collaborative achievements but the good fellowship that one finds over drinks and shared meals. There are other associations, too, that come with CRRA, the Digital Access Committee being the one that I know best. Thanks to you all for your fellowship and shared memories.”

Thank you, Tom, for your many contributions in shaping the CRRA.  We will miss you and wish you the very best.

NEH Challenge Grant Proposal Deferred: From the Board

One goal in the strategic plan for this year was to submit a grant proposal to the NEH Challenge Grant Program. However, as the Board and Terry Ehling (strategic consultant) reviewed the proposal, the Board decided we would benefit from doing a five year strategic plan before submitting the proposal. The proposal spoke clearly to the mission and positive impact on Catholic scholarship but it was not clear what the financial target of the grant should be.  The purpose of an NEH challenge grant is to assist the grantee in building capacity and infrastructure to carry out its mission. In looking ahead, we recognized that the annual goals for 2011/12 mark the third year of the current strategic plan and realized this is a good time to develop a five year strategic plan. Doing so we believe will allow us to create a bold and even stronger proposal of what we can do over the next five years in supporting the needs scholars on our own campuses have expressed in focus groups and usability studies.  In the context of the longer term directions and goals, we will also be able to determine an appropriate financial target to assist in building capacity and infrastructure.

Developing a Five Year Strategic Plan

The Board is setting up a Task Force to develop a five year (FY2012/13-2016/17) strategic plan <http://bit.ly/crra_5yr Plan > to identify key directions and goals for the CRRA and the Catholic portal. The plan will include a statement of core values, vision and mission; a statement on the benefits of CRRA in advancing Catholic scholarship, and directions and strategies to carry out the mission and deliver value to stakeholders. While the planning process is underway, we will continue to implement the goals for the coming year (2011-12) and especially the three priorities identified in the March membership meeting.

We propose a task force of five to seven members and with representation from each committee, the Board and members at large. Pat Lawton, Jennifer Younger and Terry Ehling will be ex officio members. If you are interested, please send an email to Pat or Jennifer who will create a list of interested members for the Board.

Once the task force is in place, more information will be forthcoming on the process, time line, key questions and most importantly, opportunities for member input. We can achieve our mission and aims only through the direction and participation of our members.

Portal Anecdote: User Connected with Parish History

A variety of interesting requests come through the “Contact us” link on our homepage and one of the most rewarding to date came recently from a woman “working on a history of the Church of the Magdalene and gathering all print sources that [she] can find.”  She referenced a record for a parish history listed in the portal.  In a short period of time, the item was located, scanned, and sent to the church via the requester.  She was pleased to receive the document, and we were pleased to know that the portal had enabled access to a valued resource to one very real user.

Users are finding us!  If you have portal success stories to share in future Updates, please pass them on.

CRRA in the News …

ACCU Summer 2011 Newsletter on the CRRA/Duquesne Symposium

The Summer 2011 ACCU Update includes an announcement of the upcoming November Symposium hosted by Duquesne University and the CRRA.  See the full article on page 12 of ACCU’s newsletter, available at: http://www.accunet.org/i4a/pages/index.cfm?pageid=3389.

For details about the CRRA/Duquesne Symposium, see http://bit.ly/lg0oCH.

The Collections Committee Update: Developing Critical Mass around Three Themes

The Collections Committee states goal 1.1. on the 2011/2012 strategic plan as: “Encourage development of critical mass around three themes: Women religious, Vatican 2, and Catholic social action.”   The Committee emphasizes that highlighting these themes is not prescriptive but rather suggestive. The idea is to encourage members to think about their own holdings in these areas and to add to the portal what they may have in these areas, thereby supporting the portal goal to build collections.

If you have rare or uncommon materials in these subject areas, these would be especially welcome additions to the portal. Of course, contributions relating to any of the twelve primary collecting themes are helpful and highly desirable. If you have any questions about priorities, please contact Pat Lawton or any member of the Collections Committee.

All CRRA events and events of possible interest to members are posted to the CRRA calendar, please bookmark this link for future reference.

Check our progress and news on the CRRA blog: http://www.catholicresearch.net/blog/.

CRRA Update is an electronic newsletter distributed via email to provide members with an update of CRRA activities.  Please contact Pat at 574.631.1324 or email plawton@nd.edu with your questions, comments, or news to share.

———
CRRA Calendar: http://tiny.cc/Calendar798
CRRA Goals for 2011/2012: http://bit.ly/lay6Za
Duquesne/CRRA Symposium: http://bit.ly/lg0oCH
Task Force to Develop a Five Year Strategic Plan: http://bit.ly/crra_5yrPlan

As membership in the Catholic Research Resources Alliance (CRRA) grows, so does the number of metadata formats the “Catholic Portal” is expected to support. When the CRRA was just beginning MARC was the predominate metadata format. After the content of university archives was recognized as significant, EAD became very important. Some institutions use neither MARC nor EAD to describe their special collections but instead use systems like ContentDM. These sorts of things are often accessible via OAI-PMH, and thus, at the very least, harvestable Dublin Core is available. In order to support discovery, all of these types of metadata need to be parsed, mapped to VuFind’s underlying Solr schema, and indexed.

It has come to my attention that some of the CRRA’s membership may be using an application called PastPerfect by PastPerfect Software, Inc. to describe their collections. After a bit of investigation, I learned that PastPerfect supports a number of exportable metadata formats. One of those formats is an XML file complete with Dublin Core elements. Here is a sample record:

<?xml version="1.0" encoding="windows-1252" standalone="yes"?>
<metadata>
  <dc-record>
    <type>text</type>
    <type>original</type>
    <type>cultural</type>
    <format>23 cm. 292. p. Includes index.</format>
    <title>Guide to the Use of Books and Libraries</title>
    <title>Book</title>
    <description>Guide to the Use of Books and Libraries....</description>
    <subject>Book</subject>
    <subject>1. Reference books.</subject>
    <subject>2. Libraries--Handbooks, manuals, etc.</subject>
    <subject>3. Library.</subject>
    <subject>Gates, Jean Key</subject>
    <creator>Gates, Jean Key</creator>
    <contributor>Wright, Richard R. and Susan Gamer, editors.</contributor>
    <publisher>McGraw-Hill Book Company</publisher>
    <date>1979</date>
    <identifier>2000.4.3</identifier>
    <language>English</language>
    <coverage>1979 - 1979</coverage>
    <coverage>New York, NY</coverage>
  </dc-record>
</metadata>

The XML is straight-forward enough and seems to be well-formed, but there does not seem to be any DTD nor schema for validation. The content in each of the elements comes straight from the data entry of the PastPerfect system so things like “1. ” or “2. ” in the subject elements are included apparently because that is what someone typed in. Similarly, subheadings delimited by “–” and the multiple values in the format element are reminiscent of MARC records and their ISBD (International Standard Bibliographic Description) codes. The repeating elements like coverage or type make things challenging, but not insurmountable. In short, the issues surrounding the mark-up are relatively minor. It is not ideal, but it is functional.

The bigger issue surrounds linking to the original item. While each metadata record includes a unique identifier, there is seemingly no way to enable the reader to either see a full-record display at the hosting institution or see the item being described; the PastPerfect records are not associated with an actionable URI (Universal Resource Identifier). This means each record, in order to be used and before it is indexed, will need to be associated with the postal address of the library/archive using PastPerfect, and readers will need to get in touch with the librarian/archivist if they want to use the item.

We don’t live in an ideal metadata world, and increasingly it seems the best we can hope for is well-formed and valid metadata. Whether our metadata is complete or accurate is completely in the hands of people, not computers.

A couple of years ago, when the “Portal” was just beginning, the modus operandi for ingesting MARC and EAD metadata was to send it to Notre Dame, save it on local hard disk, and index it. That process worked then, but as we grow it becomes less and less scalable.

Now-a-days the preferred method of getting your metadata to the Portal is through harvesting. Here is how it works:

1. Create metadata – Use whatever process you desire to create and edit your metadata. Much of what we suggest is outlined in a previous posting affectionately called “the recipe“.
2. Export metadata – If your metadata is in MARC format, then query your integrated library system for all things destined for the Portal, and save the result to a single file using the UTF-8 character set. If your metadata is in EAD format, then export it as individual files making sure they are well-formed and valid.
3. Expose metadata – In either case, MARC records or EAD files, the next step is to save the metadata on a Web server. Create or have created a directory on a Web server. Put the file of MARC records and/or the EAD files in the directory. There is no need to create a Web page. Just make sure the directory’s contents are listed automatically and by default. A good example is the work done by Marquette University.
4. Share the URL(s) – Once the files are on a Web server, they will have URLs. In the case of MARC records, send Notre Dame the URL of the MARC file. In the case of EAD files, send the URL of the directory.
5. Repeat – This is an never-ending process. Go to Step #1. As you create, edit, and export new or different metadata, save it in the Web-accessible directory. There is no need to send the updates to Notre Dame. They will be harvested on a regular basis. There is no need to denote which records are new, changed, or deleted. Previously indexed records will be discarded and the whole lot will be re-indexed.

There are many benefits to this process. First, the data gets duplicated. “Lot’s of copies keep stuff safe.” Second, Internet spiders and robots will find your data, index it, and make it accessible via their indexes. That is a good thing. Third, it gives you more control over the data and reduces the risk of Notre Dame loosing it.

Just like the previous “recipe”, what is described above is only an outline. Each institution will differ slightly in their implementation. If you have any questions, then please don’t hesitate to ask.

The Digital Access Committee had a conference call on Thursday, May 12. The purpose of the meeting was to discuss usability studies. The resources (time and money) required to do the studies was emphasized. Similarly, the need to have the studies done with the intended audience of the Portal — upper-class man, graduate students, faculty, and scholars — was also stressed.

Ideally each institutional member of the Committee will facilitate and complete a set of usability studies by Christmas. In that vein, the following tentative list of who will do studies by has been drafted:

• Seton Hall during June/July
• University of Toronto during July/August
• Marquette University during August/September
• Georgetown University during late August/early September
• Catholic Theological Union during September
• Villanova University during September/October

Individual committee members are expected to communicate with the committee as a whole by May 27 with more definite commitments.

For more information about the usability studies, see “Doing usability against the ‘Catholic Portal’“

The Catholic Research Resources Alliance (CRRA) or “Catholic Portal” brings together data and metadata for the purposes of Catholic research and scholarship. This process is facilitated through a number of groups dealing with administrtive issues, collection issues, metadata issues, etc. CRRA-Tech is a mailing list intended to support and discuss the computer technology issues of the CRRA such as but not limited to the harvesting of content and metadata, the validation of content and metadata, indexing technologies, library “discovery systems”, the programming languages (PHP, Java, Perl, and Javascript) used, log file analysis, casscading stylesheets, debugging tools, the role of open source software, etc. In short, CRRA-Tech provides a forum for discussing the computer infrastructure of the Portal.

If supporting research and scholarship through the use of computer technology is a part of your daily work and if your employer is as member of the CRRA, then consider subscribing to CRRA-Tech. To subscribe:

1. address a message to listserv@listserv.nd.edu
2. in the body of the message put “subscribe crra-tech”
3. send it away

You ought to get back a couple of confirmations, and you will be done.

To and from Berrien Springs (MI)

The Chicago Area Theological Library Association held its Spring Conference at Andrews University in Berrien Springs (Michigan). The conference was small, about 15 people attended. After the business meeting I gave a presentation on “next-generation library catalogs”, digital humanities, and the “Catholic Portal”. In a nutshell I compared & contrasted database applications (traditional library catalogs) with indexes (“discovery systems”). I then demonstrated a few text analysis tools and at the same time explained how these tools can be used to supplement the “close” reading process. Finally I described and demonstrated the “Catholic Portal”, and I showed how the ideas of “next-generation” library catalogs and text mining have been incorporated into it. I got lucky with the last part of the presentation because I had upgraded the Portal the previous day, and nothing went wrong during the demonstration.

After a vegetarian lunch in the University’s dining hall, we returned to the conference room for a set of lightning talks:

• Kate Ganski (University of Wisconsin, Milwaukee) described a needs-based marketing campaign which looked rather innovative and energetic
• Lisa Gonzalez (Catholic Theological Union) enumerated a number of cool (and “kewl”) tech tools to share with patrons
• Alan Krieger (University of Notre Dame) described the the Hesburgh Libraries’s newly created theological reading room and how it was being used
• Matt Ostercamp (North Park University) outlined ways to promote traditional reading in libraries, and of all the lightning talks, this one complemented my presentation the most
• Karl Stutzman (Associated Mennonite Biblical Seminary) reported on the process his library is going through to implement Primo

After the talks we were given a very nice tour of the University’s library and archive. Hosting the largest collection of Seventh Day Adventist materials in the world, the University archive was quite impressive. They actively digitize their materials and provide a home for a wide variety of materials. I was also impressed with the library’s service to the community. Specifically, they operated a charitable giving program where they received new (and used) books from a variety of sources and then shipped these books to fledgling libraries all over the world. They were putting their university’s values into practice.

I had a good time, and I appreciate the opportunity. “Thank you, Lisa G., for inviting me!”

Highlights in this issue include:

• CRRA welcomes new members: College of the Holy Cross and Creighton University!  Look for more on our newest members in next month’s Update.
• Welcome to Terry Ehling, Strategic Consultant to the CRRA!  
• News on membership dues
  The Membership Dues Task Force met with members of the Board and other member library directors to discuss its recommendation to adopt a multi-tiered dues structure and a proposal for dues for the coming year. 
• Highlights of the CRRA All-Members Meeting in Philadelphia. On Wed., March 30, 35 CRRA members and friends participated in the CRRA annual meeting. Agenda items included goals for fiscal year 2010/2011 (see proposed goals at http://bit.ly/DraftPlan2012), usability studies, and membership dues.

See http://bit.ly/Update_April2011 for more on these and other news items.

The Meeting began Tuesday afternoon, March 29, at Villanova University where attendees were treated to a number of show & tell presentations describing the digital library goings-on of the Falvey Library. Joseph Lucia began by listing a number of well-articulated reasons why open source software is akin to the values of librarianship. Most notably, he alluded to the Second Enclosure and the very real threats to the public commons. Other presentations outlined local digitization efforts using Tesseract, their institutional repository implementation, scholarly publishing with Open Journal System, and their newly released digital library software called VUDL. I am continually impressed with the work being done by the folks at Villanova. Administration has a vision, a plan, and puts the plan into practice. “We do things for the sake of scholarship… We collaborate and find partners.” This approach to digital librarianship seems to me to be the best long-term strategy and ensures sustainability. It is not so much about getting more money but instead about setting priorities and allocating resources accordingly.

The main event took place the following day at St. Joseph’s University. Attended by thirty people or so, this particular All-Members Meeting was the largest to date. This is not surprising since the Catholic Research Resources Alliance (CRRA) now has about twenty members. We’re growing! The morning’s session focused on two issues. The first was automated procedures for getting member metadata (MARC and EAD) into the “Portal”. This is where I elaborated on the “recipe”. Attendees seemed to think the procesure was feasible. The second discussion surrounded digitization efforts. While no plan was articulated many people believe it is necessary to include more full-text content in the Portal. This can be done through local digitizations projects, coordination with Villanova’s established program, or through the harvesting of content from the ‘Net. Some of this discussion bled into the interpretation of the Portal’s collection policy. It seems to me as if the policy may not be as prescriptive as necessary. Many people seem confused by it and desire clarification regarding their content selections. After lunch most of us participated in a discussion regarding usability studies. This is where I outlined how we did usability here at Notre Dame and the expectations for institutional CRRA members.

In summary, the Alliance feels like it is moving ahead at a measured pace.

Finally, I relished the location of this particular meeting. Let me explain. United States Highway Route 1 starts in Key West (Florida), goes through every major city along the East Coast, and terminates at the Canadian border in Maine. United States Highway Route 30 starts at the Atlantic Ocean in New Jersey, crosses the country as the Lincoln Highway (for the most part), and ends in Oregon at the Pacific Ocean. These two cross-country highways intersect at the western-most edge of Philadelphia. The following movie was taken at this intersection — a true crossroads of America.

Some day I would love to start at the end of either one of these roads, slowly drive to the other end, and take photographs all along the way.

On Thursday, February 24, the Hesburgh Libraries and the Catholic Research Resources Alliance (CRRA) sponsored the Digital Humanities Forum. The purpose of the event was to raise the awareness of the digital humanities across campus just a little bit. To that end we hosted two speakers and a couple of hands-on workshops.

The first speaker was Art Crivella (Crivella West). Crivella has recently become passionate about digital libraries and was instrumental in getting the digital access to Cardinal Newman content off the ground. “Books are very hard to let go of”, and “We are developing ways to read 90 million pages [of text]“, he said. He then described some of the work his company has been doing in this regard. Digitize the “best” text possible. Make it perfect. Divide the corpus into parts: published, oratory, tracts, and personal writings. Compile lists of words and phrases denoting broad subject areas, emotional connotations, and philosophic concepts. Implement a system — which looked a lot like a concordance — allowing scholars to search the corpus and identify relevant passages of text. Select paragraphs from the search results, click a button, and find similar paragraphs. He summarized by saying, “Algorithms of the 22nd Century are just as important as the works, sets, and commentary of collections. The sum of these things constitute the library of the near future.”

speakers

workshop

The second speaker was Ron Snyder, the Director of Advanced Technology at ITHAKA, and ITHAKA is the parent organization hosting JSTOR. The majority of the time, Snyder described the functionality of a JSTOR site called Data For Research (DFR). Built with open source software (most notably Solr), and accessible via a REST-ful application programmer interface (API), Data For Research provides a way to search the JSTOR content and apply data mining techniques against the result. The site outputs bibliographies, word frequencies, n-grams, keywords based on TFIDF, and references. It supports a bit of visualization, and data sets can be delivered to programmer’s desktops in the form of XML or CSV files. Snyder compared the DFR’s Web interface to a form of sculpture. “The DFR tool is like an ice sculpture where you whittle down your results.” This is true since it is entirely possible to access to the sum of the JSTOR content by first entering a couple of key words and use the resulting facets to reduce the set to a few items. The following day, Friday, Snyder facilitated two workshops. One akin to a traditional bibliographic instruction session, and the second a brief tutorial on how to use the API.

The first day’s session was attended by approximately 50 people. Just more than half were from the University, and the balance were members of the Catholic Research Resources Alliance. The workshops were attended by fewer but a similar mix of people. The feedback I received from the event was more or less positive. Words used to describe it included “interesting”, “intense”, and “thought provoking”. I believe the Digital Humanities Forum accomplished its goal.

• why do usability
• expectations of institutional Committee members
• locally required resources
• doing the work
• work with the Committee to refine the questions
• practice with the technology
• schedule testers
• do studies
• evaluate results
• return results to Committee
• summary and conclusion

This document is also available as a PDF document for printing, a second PDF document designed as a set of slides, and just for fun, an EPUB file for your mobile device.

Why do usability?

Why do usability? Because very few things in life are truly intuitive. We — the totality of the Catholic Research Resources Alliance — have worked hard to build a tool facilitating Catholic scholarship. This tool is functional, and by that we mean it is always online, does not crash, and does not return incorrect information. In order for the “Portal” to be successful, it needs to go beyond functionality to usability. This means it needs to be easy-to-use, contain a limited amount of jargon, and be perceived by its intended audience as a time saver.

Expectations

Usability is an iterative process benefiting from the experience of many people. For these reasons, each institutional member of the Catholic Research Resources Alliance’s Digital Access Committee is expected to conduct usability studies against the Portal before the end of this calendar year. Using this posting as a framework, this means five or six usability studies will be done against the Portal before Christmas. The exact times when these studies will be done, and the exact way they are facilitated are left up to each institutional member as long as this document is used as a “recipe” and the balance of the Committee is consulted. This is a group process.

Locally required resources

Fiscally speaking, usability is not an expensive process. Instead, the greatest costs are measured in people’s time. In order to do this work each institutional member of the Digital Access Committee will need:

• a team of at least 2 people, but 3 for 4 is much better
• approximately 40 hours of time to be spent by the team
• money to compensate testers’ time
• usability software, optional

The most important resource is the people. At a minimum, two are required. One will facilitate tests. The other will record observations. But there are many tasks that need to be done besides facilitation and observation. It is better to have three for four people on the team. This will make scheduling testers easier, reduce the possibility of overwhelming anybody, and take advantage of different people’s individual talents.

The whole usability process will consume approximately 40 hours of staff time. If you assume 6 to 8 hour-long usability studies will be facilitated at your institution, and two people are doing the work, then actually doing the tests will “cost” about 16 hours of time. Evaluating the totality of the tests may take another 2 or 3 hours, so the cost is increased by 4 to 6 hours. Scheduling people to participate is one of the most difficult parts of usability and requires a great deal of coordination. Expect to spend another 4 hours just finding and getting qualified and representative people to participate to your study. As the year progresses, we expect the Portal to change, and consequently the usability studies will change. Time will need to be spent coordinating with the Digital Access Committee on these changes, another 3 to 4 hours. If you plan to take advantage of usability software, time will need to be spent purchasing the software as well as practicing with it. Finally, time will need to be spent documenting the experience so it can most effectively be shared with the Committee.

Usability studies cost the time of testers. It is customary to compensate these people for their time. The amount of compensation will be guided by your local policies, but things like food, gift certificates, small spending sprees at the local bookstore, or services are all examples. You might need to allocate as much as $10-$25 per usability participant for compensation.

Computer software exists to help facilitate usability studies. At the very least such software records how the participant interacts with the system being tested. More full-featured software also records the participants’ facial expressions and auditory responses. For the Macintosh we suggest Silverback which costs about $80. A popular Windows application is called Morae Recorder and costs just less than $200. Employing software in your studies makes it easier to be more thorough in your evaluation as well as enabling one to share individual studies. At the same time, the software adds a bit of complexity and expense.

Doing the work

Once the goals of usability and expectations are understood, and once the resources have been allocated, it is time to actually do the work. The process is more stratified and iterative than it is sequencial. In other words, it is not always necessary to complete one step before starting the next. The steps include:

• refining the usability tasks to be studied
• practicing with the technology, optional
• scheduling testers
• facilitating the studies
• evaluating the results
• reporting on the results

The following sections elaborate on each of these items.

Refining the usability tasks

Usability studies are done in an effort to learn how systems can be made easier-to-use, free of jargon, and perceived as a time saver for the intended audience. Since a primary focus of the “Portal” is to create “access to those rare, unique and uncommon research materials”, the usability study must test how well the Portal facilitates these goals. The initial set of usability studies done at Notre Dame included the following tasks:

1. Identify the library or archive holding the papers of Dorothy Day.
2. Find a record whose author is Graham Greene. Create an account, then add the Graham Greene record to your favorites, tagging it as “ggreene.”
3. Locate resources, including primary resources, on the Catholic Conference for Interracial Justice.
4. Find a set of records on the topic of “Catholic social action.” Choose 1-3 from the retrieved set and email them to yourself for future reference.
5. Locate materials on the topic of sermons and the Lutheran church.
6. Who owns “Our Sunday Visitor Records”? What telephone number would you call in order to schedule a time to visit the collection?
7. Which library has the most French-language materials in the “Portal”?
8. What is the most frequently used word in the pamphlet owned by Notre Dame entitled “Pastoral instruction for the application of the Decree of the Second Vatican Ecumenical Council on the Means of Social Communication”? (hint: see the record with the call number BV 4319).

Notice how the tasks touch on many different aspects of the Portal. They focus on the finding of diverse materials, identifying where they are physically located, and actually using some of them.

Since we expect to take the results of usability studies and apply them to the Portal interface as soon as feasible, the tasks outlined above may become moot over time. Consequently you will need to see how the Portal is evolving, discuss your institutional study with Digital Access Committee, and combine the result with your own personal experiences and skills to create a new set of questions.

You don’t want more than ten tasks in any set of usability studies. Any more than that and the tests take too long to facilitate and are difficult to evaluate. Put another way, create a list of tasks than can be studied in less than an hour.

Practice

If you opted to use computer software to help with your usability studies, then you will have to acquire it and practice with it. “Practice makes perfect,” and it makes you look good when facilitating the studies.

Scheduling testers

Scheduling usability testers has got to be one of the more difficult steps. Not only does it take a long time, but it also requires a lot of coordination and selectivity. First and foremost, it is important to schedule testers who are representative of our target audience — scholars. The Portal is intended to support research in all things Catholic. The material the Portal is archival in nature, leans towards primary literature, and requires previous knowledge of a deep nature in order to adequately interpret. In general, this not necessarily a system designed for undergraduates. Please make every effort to schedule faculty and graduate students working within the Portal’s subject domain.

Schedule the testers for no more than an hour at a time. Thirty to forty minutes will be spent on doing the tasks. The balance of the time can be spent on discussion and elaboration.

Between four and eight testers is usually considered enough for usability studies, but schedule about ten with the idea that some will unexpectedly drop out. It is much easier to ask people not to come than it is to find people to participate at the last minute.

Facilitating the study

You’ve created your list of tasks. You’ve practiced with the optional software. Your testers are arriving. The hard parts are now behind you, and it is time to actually do a study. The process is easy. Here’s how:

1. One person facilitates the study, and another person takes notes.
2. Thank the particpant for their time.
3. Remind the particpant that they are not being tested, but rather the Portal’s interface is being tested. There are no wrong answers.
4. Emphasis to the particpant the critical importance of thinking out loud. By doing so it will be much easier for you, the facilitator, to understand what is going on, and it will be easier for the note-taker to record the results.
5. When everybody is ready, go through the tasks one by one. Try really hard not to interfere with the completion of the tasks. If the particpant is really off base, then intervene but don’t do so too quickly. This part of the process is sometimes difficult to watch. Be patient. Remember, you are not being tested either, only the Portal’s interface.
6. After each of the tasks have been completed, have a discussion. Ask the particpant what they liked, disliked, and thought was easy or hard. Consider asking, “If you could change one thing about the interface, then what would it be?”
7. Thank the particpant for their valuable time, and don’t forget to give them their compensation.
8. After the particpant has gone, you may want to discuss the study among yourselves, and it a really good idea for the note taker to transcribe their notes for the evaluation process.

Evaluating the results

Once all the studies have been facilitated it is time to evaluate the results. The goal of the evaluation process is to come up with a prioritized list of things you think need to be improved with the Portal’s interface. The key word here is “prioritized”. We are sure there are many things that can be improved, but considering limited time and resources, some things need to be more important than others.

To create the prioritized list, try this:

1. Read the written notes, and review the optionally used software recordings. Based on our experience, we think it is easier to read the notes “across rather down”. In other words, we found it is easier to see patterns between the studies when we compared & contrasted the responses to each question from each participant as opposed to looking at each participant as a whole. For example, we looked at all the notes for task number #1, and then all the notes for task #2, and then task #3, etc.
2. Based on the notes, create a prioritized list of three to five items where each item is something you think needs to be addressed. Be prepared to cite which tester and which task demonstrates the issue you think needs to be fixed.
3. Based on your professional opinion, create a second prioritized list of three to five items where each item is something you think needs to be fixed.
4. Bring together your team of people — have a meeting.
5. Go around the room asking everybody for their prioritized items, first based on the notes and second based on professional opinion. Record everything on a whiteboard, and add tick marks to items repeated by team members.

In the end you ought to have a list of as many as a dozen issues to be addressed, and you ought to be able to sort them by the number of tick marks each received. Here at Notre Dame we had as many as five people on our team, and our list ended up looking like this:

• 6 – set search filter to off by default
• 5 – enable sending of more than one email at a time
• 4 – clarify difference between canonical and remote [files]
• 3 – remove autocomplete feature
• 2 – re-do text mining language
• 1 – tweak facets to be more descriptive or complete
• 1 – retain links of original EAD file in local EAD file
• 1 – respect my browser preferences
• 1 – remember [search] results after creating account
• 1 – make local EAD file the default
• 1 – implement authority control (cross-reference) functionality
• 1 – highlight search words in result [list]
• 1 – explain what facets are
• 1 – enable further search [refinements] after selecting “archival records”
• 1 – confirm adding to favorites
• 1 – add addresses and phone numbers to records

Based on these sorts of lists it is easy to see what are priorities and what are not. “Librarians love lists.”

Reporting on the results

This is the last step. Simply document what you discovered — most importantly your prioritized list of issues — and share it with the Digital Access Committee. As the results come in concerted effort will be made to address them as soon as feasible, or at least before the next round of usability studies to be conducted by other institutional members.

Summary and conclusion

Usability studies are an effective way of learn how software interfaces can be improved. They do not need to be expensive in terms of money, but they do require time and effort. Usabilities studies, like software, are never done. There are always things that can be improved. Consequently, usability studies are iterative. Implement something. Test it. Apply the results. Repeat. By working together we — the Catholic Research Resources Alliance — can share the load, draw from a wide variety of experiences, and ultimately create a better Portal interface. On our mark. Get set. Go!

In a previous posting called “Usability testing” (dated February 14, 2011) a set of eight usability questions was outlined. Since then Team Catholic Portal here at Notre Dame facilitated six usability studies made up of five graduate students and one faculty member. These participants were scholars in philosophy and theology. We used the simple facilitator/note-taker approach. We employed usability software (Silverback), but didn’t use it to evaluate our results. Using our notes as well as professional judgement, we evaluated the results and came up with the following prioritized list of things to be addressed with the Portal’s interface:

• 6 – set search filter to off by default
• 5 – enable sending of more than one email at a time
• 4 – clarify difference between canonical and remote [files]
• 3 – remove autocomplete feature
• 2 – re-do text mining language
• 1 – tweak facets to be more descriptive or complete
• 1 – retain links of original EAD file in local EAD file
• 1 – respect my browser preferences
• 1 – remember [search] results after creating account
• 1 – make local EAD file the default
• 1 – implement authority control (cross-reference) functionality
• 1 – highlight search words in result [list]
• 1 – explain what facets are
• 1 – enable further search [refinements] after selecting “archival records”
• 1 – confirm adding to favorites
• 1 – add addresses and phone numbers to records

Once we have finished migrating our existing “sandbox” implementation of the Portal to production hardware, I will see about implementing the changes. Some of them require changes to the underlying VuFind software. Some of them require changes in wording.

–Pat

March 30 All-Members Meeting and

March 29 Pre-meeting Events

Tuesday, March 29 – Villanova University, Philadelphia

1-4:30 Events at Falvey Memorial Library

6:30  Dinner with CRRA colleagues, place TBA

Wednesday, March 30 – Saint Joseph’s University, Philadelphia

10:00-2:00 CRRA All Members Meeting

and more

In this update …

• Welcome to Creighton University, our newest member.  Watch for more information about Creighton in the March 2011 Update.
• Congratulations to Duquesne’s Gumberg Library for winning an LSTA Grant Award to Digitize the Pittsburgh Catholic Newspaper!
• Collections Committee Update:  Portal Enhancements and more …
• CRRA at Upcoming Conferences and Outreach to Catholic Colleges and Universities, by Jennifer Younger
• CRRA Colleagues from the University of Dayton, Marquette University, and Dominican University braved the February weather for two days of tours, presentations, workshops, and conversation at the ND/CRRA Digital Humanities Forum and Workshops at Notre Dame.
• Mark your calendars! CRRA in Philadelphia (March 2011) To facilitate reservations and our hosts in providing refreshments, it would be helpful to have a count of expected attendees.  Please email Pat Lawton at plawton@nd.edu if you will be joining us for any or all of these events.

Congratulations to Laverna Saunders and Team at Duquesne’s Gumberg Library

for winning a $20,000 LSTA Grant Award to digitize additional years of the Pittsburgh Catholic Newspaper.  The Pittsburgh Catholic, America’s oldest Catholic newspaper, has been in continuous publication since March 16, 1844.

Through a pilot project, Gumberg Library completed digitization of the 1844-1864 issues, and seeks to complete digitization through the year 1900. “Duquesne University has continuously microfilmed the Pittsburgh Catholic since its inception in 1844. To convert the Pittsburgh Catholic to digital format, Gumberg Library started with volume 1 issue 1 and will continue to digitize the newspaper from the oldest volumes to the newest volumes.” Grant funds will allow the Gumberg to digitize the February 27, 1864 through January 3, 1900. Adding this content to our digital library will support our service to the local Catholic community and increase the depth of our unique content to be available through the CRRA (Catholic Research portal).”

Catholic newspapers were frequently cited as a source of great interest by participants in focus groups conducted last year at member institutions. We look forward to making the Pittsburgh Catholic available to portal users and encourage all to consider adding records and/or full text items for locally held Catholic newspapers, cited by scholars as an invaluable resource.  Kudos to Duquesne!

The Collections Committee Update: A View from the Collections Committee

Earlier this month, we looked at the enhanced portal functionality under development at http://vufind.library.nd.edu/ . You may remember that one goal in the Strategic Plan 2010-11 is to index finding aids in full EAD (Enhanced Archival Description) content for discovery.  With the EAD indexer and viewer in place, discoverability has been significantly increased.  A search for “Dorothy Day” now retrieves items not previously found in the “non-EAD indexer” portal.   For example, this record http://vufind.library.nd.edu/Record/cuaead_id2796180 was retrieved from the series level within the George Gilmary Higgins Papers. This exciting development underscores the value of deep indexing of archival finding aids in the portal.

It is also interesting to see areas in which a critical mass of items is developing along with growing member participation.  We searched four themes with the following results:

• Catholic social action – 1,203 responses in five institutions (and suggested topics such as “Church and social problems” and “Catholic Worker Movement” leading you to more)
• Vatican II – 1,270 responses in ten institutions
• Women religious – 491 responses in nine institutions
• Catholic education – 1,626 in 11 institutions

If you have rare or uncommon materials in these subject areas, these would be especially welcome additions to the portal. Of course, contributions relating to any of the twelve primary collecting themes are helpful and highly desirable. If you have any questions about finding aids or priorities, please contact Pat Lawton or any member of the Collections Committee.

CRRA Events at Upcoming Conferences
By Jennifer Younger, chair, CRRA Board of Directors

At the CRRA meeting at Georgetown University, we decided to hold the annual CRRA membership meeting in conjunction with the ACRL Conference in Philadelphia. We will not be holding a formal event during the ALA Annual Conference in New Orleans, June 2011.  Just as is the case at other conferences, however, there may be informal get-togethers with other CRRA members or prospective members.

There are CRRA participants at a number of different conferences, including those of the Catholic Library Association (CLA), April 26-28, in New Orleans; the American Theological Library Association (ATLA), June 8-11, in Chicago; and the Society of American Archivists (SAA), August 22-27, in Chicago and possibly at the Association of Catholic Diocesan Archivists meeting during the SAA conference on August 24, in Chicago.  [All events are posted to the CRRA calendar.]

At the CRRA meeting in San Diego, a couple of people asked about setting up a local team.  Despite not planning formal meetings, we hope we can always find opportunities to get together, share best practices and ideas for carrying out the CRRA mission.

Outreach to Catholic Colleges and Universities
By Jennifer Younger, chair, CRRA Board of Directors

Our membership outreach to Catholic colleges and universities is underway.  The directory on our home page lists the Catholic colleges, universities and seminaries and the directors of their libraries and/or archives. I have sent the About CRRA brochure as well as information on the collections policy, benefits and principles of membership, and making content available to the directors. About 200 of the Catholic colleges, universities and seminaries in the United States belong to the Association of Catholic Colleges and Universities (ACCU).

Several directors responded immediately. Michael Lacroix, Creighton University, was the first to ask for an invitation and Creighton is joining as the 20^th CRRA member. Others said they want to pursue membership. Still others said they have been carefully following previous news items since the CRRA began and are excited to hear from us. One director replied with information about the special collections at his institution, which I have sent to Bob O’Neill and the Collections Committee. With assistance from a good number of you, I am following up in conversations with all interested directors.  In addition, with directors and/or librarians from Marquette, St. Edward’s and Seton Hall, I have had the distinct pleasure of meeting directors, librarians and/or archivists at prospective member campuses.

We are on target to meet, possibly even to exceed, the goal in our strategic plan of 20 new members for next year, thanks to your good work. Your conversations with colleagues are the best way of reaching out to prospective members. Some of you will be attending the ACRL meeting in Philadelphia which follows the CRRA meetings on March 29-30.  Others of you are likely to be attending the Catholic Library Association meeting in New Orleans (April 26-28), the American Theological Library Association in Chicago (June 8-11) and/or the Society of American Archivists in Chicago (August 22-27). If there is anything that Pat or I can do to help you in reaching out to your colleagues, please do call on us.

February 24-25 ND/CRRA Digital Humanities Forum and Workshops at Notre Dame

Ann Hanlon, Rosemary Del Toro, John Jentz (Marquette University); Jillian Slater, Fred Jenkins, Fran Rice, Colleen Mahoney (University of Dayton) and Steven Szegedi of Dominican University braved the February winter weather and joined us for two days of tours, discussion, and conversation.  In addition to the planned events, Reference Librarian Bob Hohl provided us with a special look at the St. John’s Bible held by St. Mary’s College.

Thanks to all who made the trip!  This was our first meeting with many of you, and the event proved to be a great opportunity to learn about one another and our work. We learned about text mining and visualization techniques and how they are being implemented in the portal.  This ability to search across a large body of documents and “do stuff with them” (Eric Morgan) promises to be an important benefit for scholars interested in Catholic scholarship and to the CRRA mission.  As Bob O’Neill observed in a recent Collections Committee meeting, “text mining distinguishes the portal from other projects.”

*    *   *   *

All CRRA events and events of possible interest to members are posted to the CRRA calendar, please bookmark this link for future reference!

Check our progress and news on the CRRA blog: http://www.catholicresearch.net/blog/.

CRRA Update is an electronic newsletter distributed via email January-November to provide members with an update of CRRA activities.  Please contact us at 575.631.1324 or email plawton@nd.edu with your questions, comments, or news to share.

———
CRRA Calendar: http://tiny.cc/Calendar798
CRRA 2011 Philadelphia All-Events: http://bit.ly/CRRA2011
CRRA 2011 All-Members Meeting Saint Joseph’s: http://bit.ly/CRRA_SJU

Are you interested in learning more about Catholic digital scholarship and innovations in text mining and visualization to facilitate knowledge discovery within the Catholic portal?  Please join us virtually or in person this Thursday, Feb. 24 from 1:00 to 5:00 pm EST (noon Central, 10am Pacific).  For event details, see http://bit.ly/hWMU5j.

I am pleased to announce we are able to share this event with all of you via WebEx audio, video, and document-sharing technologies. Log-in at any time during the event and stay as long as you like.

For log-in information, just send an email to me at plawton@nd.edu.

We look forward to seeing you Thursday! –pat

—–
Pat Lawton
Digital Projects Librarian
Catholic Research Resources Alliance
574.631.1324 (office)
608.698.2519 (cell)
plawton@nd.edu

1. Identify the library or archive holding the papers of Dorothy Day.
2. Find a record whose author is Graham Greene. Create an account, then add the Graham Greene record to your favorites, tagging it as “ggreene.”
3. Locate resources, including primary resources, on the Catholic Conference for Interracial Justice.
4. Find a set of records on the topic of “Catholic social action.” Choose 1-3 from the retrieved set and email them to yourself for future reference.
5. Locate materials on the topic of sermons and the Lutheran church.
6. Who owns “Our Sunday Visitor Records”? What telephone number would you call in order to schedule a time to visit the collection?
7. Which library has the most French-language materials in the “Portal”?
8. What is the most frequently used word in the pamphlet owned by Notre Dame entitled “Pastoral instruction for the application of the Decree of the Second Vatican Ecumenical Council on the Means of Social Communication”? (hint: see the record with the call number BV 4319).
9. How would you describe the overall scope of the collection?

Wish us luck.

February 24-25, 2011 ND/CRRA Digital Humanities Forum

University of Notre Dame

Please note that we have reserved blocks of rooms at the Inn at St. Mary’s and the Morris Inn.  If you will be staying at either of these locations, please make your reservations by February 11, 2011.

We have reserved a block of rooms at the special rate of $107 per night. To make your reservation, call 1-877-567-1438 and mention you are part of the “Hesburgh Library block.”  Rooms will be held through February 11, we encourage you to make your reservations now!

—-

The Morris Inn is just across the street from the Eck Visitor Center.  For those staying at the Morris Inn, there is free parking.

We have reserved a block of rooms at the special rate of $132 per night, which includes full breakfast. To make your reservation, call 574.631.2000 and mention you are part of the “Hesburgh Library block.”

—-

Microtel South Bend Notre Dame University is about one mile from campus.  The hotel is brand new with a very convenient location and free Internet in room.

Arrival by air

The South Bend Regional Airport is about 15 minutes by car from the Notre Dame campus (flights should be booked to South Bend, Indiana — airport code SBN). From the airport, go east on Lincolnway West (left out of the airport) to downtown South Bend. Turn left on Indiana 933 (Michigan Street) and proceed about two miles to Angela Boulevard. Turn right onto Angela, and then turn left at the second stoplight (Eddy Street). Follow signs to visitor parking.

Visitors also can fly to Chicago and drive or take a bus to Notre Dame. The University is about two hours by car from Chicago’s O’Hare International Airport and about 90 minutes from Midway International Airport. From O’Hare, take I-190 east out of the airport, merge onto I-90 east (the Kennedy Expressway) toward downtown Chicago and merge with I-94 south (the Dan Ryan Expressway). Take the Skyway exit off the Dan Ryan and remain on I-90 to the Indiana Toll Road, which merges with I-80. From the Illinois border, it is about 75 miles to Exit 77 (the South Bend/Notre Dame exit).

Arrival by car

From the north: The University is located just south of the Indiana Toll Road (Interstate 80/90). Exit I-80/90 at Exit 77 and turn right onto Indiana 933. Make a left at the fourth stop light (Angela Boulevard), then turn left at the second stoplight (Eddy Street). Follow signs to visitor parking.

From the south: Take U.S. 31 north which becomes Indiana 933 just south of the city of South Bend. Stay on Indiana 933 through downtown South Bend to Angela Boulevard. Turn right onto Angela, and then turn left at the second stoplight (Eddy Street). Follow signs to visitor parking.

Arrival by train

The South Shore Line trains run directly from the Chicago Loop (corner of Michigan and Randolph) to South Bend Regional Airport in South Bend (about a two-hour trip). From the airport, the Notre Dame campus is approximately a 15-minute ride by car. Various transportation methods are available (e.g. taxi, rental car, limo).

Parking is available just west of the Hammes Bookstore, which is across the street of the Eck Visitors Center or – in the lot just east of the DeBartolo Performing Arts Center.   See Map.nd.edu.

Notre Dame Visitors Information

Please send your questions about this event or RSVP (not essential but helpful in our planning) to
Pat Lawton at plawton@nd.edu.

We look forward to seeing you in February!



The problem

There is an understandable need/desire to know how well the “Catholic Portal” is operating. But for the life of me I was not able to enumerate metrics defining success. On the other hand, Pat Lawton had no problem listing quite a few. Here are most of her suggestions:

• Are users looking at records?
• Are users searching in English? Other languages?
• Are users using field searches?
• Can we get a sense of the number of records viewed per search?
• Do we know how many searches resulted in zero hits?
• How many hits came from a google search result? Or other search engine?
• How many hits per day?
• How many times were each institution’s records viewed?
• How many times were the Web 2.0 things used?
• How many users set up an account?
• How often were the tabs at the top clicked on?
• Per searches where records were looked at?
• What is the average number of hits retrieved per search?
• What percentage of queries resulted in an error message?
• What sorts of search strings are entered?
• When are the peak periods of use? Is there a pattern?
• Where are users coming from?
• Which geographic locations and types of institutions?

If you know about Web (Apache) server log files, you know that that answers to many of these questions can be found there, sort of. If you are Web server administrator who deals with these log files, then you probably know about Analog, Webalizer, and Google Analytics. These tools can answer many of the questions, above, but the information would need to be “gleaned” from the reports. Time consuming at best and rather frustrating.

So the problem is, how do I generate regular or on-demand reports answering the questions listed above?

The solution

The initial solution was to write some sort of computer program regularly reading log files, and outputing the desired answers. Upon reflection, this would be tedious because the business logic — the questions needing answered — would either be hard-coded into the program, or the program would require an abunanced of command line switches. Complicated and not very flexible. Remember, good computer programs are programs that do one thing and do it well — the “Unix Way”.

Instead, the solution will be to first create a database — a “data warehouse” — containing log file content, and second, to provide a front-end to the database enabling people to query it. With this approach, counting the number of times anything occurs could be as easy as a single SQL (Structured Query Language) query as opposed to tabulating 10s of thousands of log file entries.

To date, the database is simple and defined by the following MySQL-specific SQL statement:

  CREATE TABLE IF NOT EXISTS `crra_logs`.`transactions` (
	`id`          INT           NOT NULL  AUTO_INCREMENT PRIMARY KEY,
	`host`        VARCHAR(128)  NOT NULL,
	`username`    VARCHAR(16)   NOT NULL,
	`password`    VARCHAR(16)   NOT NULL,
	`datetime`    DATETIME      NOT NULL,
	`timezone`    VARCHAR(8)    NOT NULL,
	`method`      VARCHAR(8)    NOT NULL,
	`request`     VARCHAR(1024) NOT NULL,
	`protocol`    VARCHAR(8)    NOT NULL,
	`statuscode`  VARCHAR(8)    NOT NULL,
	`bytessent`   INT           NOT NULL,
	`referrer`    VARCHAR(1024) NOT NULL,
	`useragent`   VARCHAR(1024) NOT NULL,
	`hosttype`    VARCHAR(16)   DEFAULT 'unknown',
	`requesttype` VARCHAR(16)   DEFAULT 'unknown'
  );

The astute Web server administrator will notice how the database’s structure mirrors almost exactly an Apache “combined” log file, with the following exceptions:

• id is a unique key
• datetime is a reformulation of the time stamp found in the Apache’s logs
• hosttype and requesttype are fields used to classify transactions, explained below

I then wrote a Perl script to read log files, parse each entry into fields, and stuff the result into the database. (“Thank goodness for regular expressions!”) Once this is done it is almost trivial to answer questions like this:

• How many different computers from the University of Notre Dame used the “Portal”? – SELECT COUNT(host) AS c, host FROM transactions WHERE host LIKE '%.nd.edu' GROUP BY host ORDER BY c DESC
• What are the 100 most popular requests sent to the server? – SELECT COUNT(request) AS c, request FROM transactions GROUP BY request ORDER BY c DESC LIMIT 100
• My computer’s address is lib-1234.library.nd.edu. What requests did I make against the Portal on December 13, 2010, and in what order? – SELECT datetime, request FROM transactions WHERE host = 'lib-1234.library.nd.edu' AND datetime LIKE '2010-12-13%' ORDER BY datetime ASC

Unfortunately, without some extra knowledge answering Pat’s questions is still problematic. For example, how does one count “hits” against the Portal when requests from Internet robots and spiders bloat the input? How does one accurately count searches for content and record views when so many of the requests include calls for images, javacript files, and cascading stylesheets?

The answers lie in the use of classification as well as the hosttype and requesttype fields. Many (most) of the “hits” on the Portal come from a computer named googlebot.com. I know this is a robot, and I can flag database records accordingly with the following SQL — UPDATE transactions SET hosttype = 'robot' WHERE host LIKE '%.googlebot.com'. Once I do this for all the robots hitting the Portal, I can accurately answer the question, “What computers operated by humans use the Portal the most?” — SELECT COUNT(host) AS c, host FROM transactions WHERE hosttype <> 'robot' GROUP BY host ORDER BY c DESC LIMIT 100.

Because VuFind uses HTTP GET methods almost exclusively, all transactions are saved in the Web server log files. These transactions have patterns. Searches contain the string “?lookfor=”. Record views all start with “/Record/”. Requests for supporting content contain things like “.gif”, “.css”, “.js”, etc. Consequently it is easy to classify the requests with SQL statements like this — UPDATE transactions SET requesttype = 'record' WHERE request LIKE '/Record/%'. Now it is really easy to count the most frequent record views by humans — SELECT COUNT(request) AS c, request FROM transactions WHERE hosttype <> 'robot' AND requesttype = 'record' GROUP BY request ORDER BY c DESC LIMIT 100.

Much of the work described above has been implemented in a handful of files — 4 SQL files and 1 Perl script — available for downloading. More classification work needs to be done but the foundation has been layed. The next big steps include automating the ingestion of new log file content and building a user interface to query the database.

Summary

Log file analysis will be greatly simplified through the use of data warehousing technqiues, and the consistently structured requests implemented by VuFind will make it much easier to learn who is using the Portal and how.

sponsored by
Hesburgh Libraries, the Center for Research Computing (CRC), and the
Catholic Research Resources Alliance (CRRA)

SCHEDULE OF EVENTS

Thursday, February 24, 2011
10:30               Optional tours of the Notre Dame campus and libraries (Meet at Eck Center)
11:30               Optional lunch at Legends on the ND campus

Forum draft Agenda                                                Eck Center Auditorium
1-1:30              Welcome, overview, context, and demo of the Catholic portal (Eric Morgan)
1:45 – 2:45       Crivella West
2:45 – 3:15       Break
3:15 – 4:15       Ron Snyder, JSTOR
4:15 – 5:00       Discussion, Q&A
5:00 – ?             Participants are invited to continue the discussion over drinks and/or dinner at Kildare’s Irish Pub, within walking distance of the meeting site.  Shuttles also available.

Friday, February 25, 2011.
Workshops with Ron Synder,  JSTOR                             Hesburgh Library, Room 248

Draft Agenda
9:00                Welcome, introductions (Eric Morgan)
9:15                 The first workshop is a “bibliographic instruction” type session where participants will      learn how to use  JSTOR’s Data For Research interface (http://dfr.jstor.org/).
10:15               Break
10:45               Using  JSTOR datasets.  Intended for computer programmers.  Ron Snyder and will walk participants through the use of the raw datasets extracted from searches.
12:00               Adjourn

About the presenters:
Crivella West (crivellawest.com) – Working closely with St. Michael’s College of the University of Toronto, Crivella West is applying text mining computing techniques to the Cardinal Newman archives for the purposes of providing enhanced understand of Newman’s writings and thought. Representatives from Crivella West will describe and demonstrate these techniques.

Ronald Snyder, JSTOR (dfr.jstor.org) – Snyder is a driving force behind JSTOR’s research and development efforts. As a technologist and data miner, he will discuss DfR (Data For Research) which allows one to search JSTOR, illustrate results with charts and graphs, and download resulting datasets for further analysis. Snyder will be discussing Dfr as content discovery tool supporting research generally, and the use of DfR for obtaining datasets, both in bulk form and programatically.

Eric Morgan (University of Notre Dame) – Eric will provide an overview and demo of the Catholic portal. The Catholic portal is a project of the Catholic Research Resources Alliance whose mission is to provide enduring global access to Catholic scholarly materials.

Participants are welcome to come and go at any time!

Refreshments will be served at all events.

Please send your questions about this event or RSVP (not essential but helpful in our planning) to Pat Lawton at plawton@nd.edu.

We look forward to seeing you in February!

Background

The content of the “Portal” is expected to be rare, infrequently held, and uncommon. More often than not, this type of material is held in library special collections and archives. Increasingly, this same material is digitized and stored in some sort of digital repository. Any repository worth its weight in salt supports some sort of API (application programmer interface) allowing computer programs to harvest and use the underlying metadata. ContentDM is one such repository application, and OAI-PMH (Open Archives Initiative – Protocol for Metadata Harvesting) is one such API.

Duquesne University recently became a memer of the CRRA (Catholic Research Resources Alliance), and it is my job to make their metadata stored in ContentDM a part of the Portal. The balance of this posting describes how I did that.

Implementation

The latest and greatest version of VuFind comes with a PHP-based utility to harvest content from OAI-PMH data providers. Using it is simple enough. Edit a configuration file. Run a program. Metadata (XML files) appear in a local directory. The utility is smart enought to keep track of harvest dates so OAI-PMH deletes and updates can be managed easily. For more detail see the section on OAI harvesting on the VuFind wiki.

VuFind comes with an second PHP-based utility to index the harvested metadata. Using it requires the developer to write XSLT files, edit another configuration file, and run the program. But since my PHP skill are not nearly as strong as my Perl skills, and since I had previously indexed other XML files in a different manner, I decided not to use the PHP indexer.

My implementation is based my previously written EAD indexing routines and described in “Indexing MARC and EAD in VUFind with Solr for the CRRA“. In a nutshell, the script:

• reads each harvested metadata file file
• maps the Dublin Core metdata to VuFind/Solr schema fields
• feeds the metadata to Solr

More specifically, I mapped the following Dublin Core elements to Solr schema fields like this:

• contributor -> author2
• creator -> author, author_letter
• date -> publishDate
• description -> description
• format -> format
• language -> language
• publisher -> publisher
• subject -> topic
• title -> title, title_auth, title_full, title_fullStr, title_full_unstemmed, title_short, title_sort
• type -> type

I populated additional Solr schema fields in different ways. Allfields is a concatonation of all the Dublin Core metadata elements. Fullrecord is a tiny XML file of my own design, similar to the one I created in the EAD implementation. Institution and building are presently hard-coded into the script but will later be pulled from a database containing all CRRA members. RecordType was filled with “oaidc”. Finally, the location of the remote digital object (dc:identifier) is inserted into the URL element of the fullrecord field.

Once the mapping is done a Perl WebService::Solr document object is created, filled with the metadata, and posted to Solr. The script is called oai_dc-index.pl and available for your perusal.

The final step was to write a VuFind record driver for the new record type — oaidc. The coding for this was trivial since much of the work had been done for the EAD files. I copied EadRecord.php to OaidcRecord.php, and changed the names of a couple of classes. These minor tweaks enable me to display Duquesne’s name, library, and URLs in VuFind.

The end result is a set of five additional records in the Portal [5], all pointing and providing access to digitized content from Duquesne University’s Gumberg Library.

Evaluation

The implementation is not perfect.

First of all, each of the five digitized items in Duquesne’s ContentDM implementation are books. All of the pages in the books are accessible individually, and each page has metadata associated with it. Unfortunately, the metadata is meager. Consequently I needed to delete hundreds of metatadata records from the OAI-PMH harvest and retain only the book-level metadata.

Second, the script currently includes a number of hard-coded characterisitcs, but when other OAI-PMH data repositories become available these hard-coded characterisitcs will be generalized.

Why is the important? There are a number of reasons. I believe a few of our CRRA members have ContentDM implementations. Harvesting and indexing their metadata will not only make the Portal richer, but it will also make it easier for students, teachers, and researchers to access the full text of the materials online.

From left to right: Eric Morgan (ND), Eric Frierson (St. Ed’s), Marta Deyrup (Seton Hall), Clay Stalls (Loyola Marymount), Kris Brancolini (Loyola Marymount), Jennifer Younger (CRRA), Tyrone Cannon (Univ of San Francisco), Janice Welburn (Marquette), Jean Zanoni (Marquette), Pat Lawton (CRRA), Alma Ortega (Univ of San Diego), Theresa Byrd (Univ of San Diego), Susan Ohmer (Notre Dame), Laverna Saunders (Duquesne), Diane Maher (U San Diego), Ed Starkey (U San Diego)

The San Diego meeting provided an opportunity for new and continuing CRRA members and friends to look at the enhanced portal, discuss future directions for the CRRA,  and last but not least,  to get to know one another.

]]> http://www.catholicresearch.net/blog/2011/01/crra-in-san-diego-january-6-2011/feed/ 0 http://www.catholicresearch.net/blog/2011/01/crra-in-san-diego-jan-6-2011/ http://www.catholicresearch.net/blog/2011/01/crra-in-san-diego-jan-6-2011/#comments Tue, 04 Jan 2011 23:26:06 +0000 plawton http://www.catholicresearch.net/blog/?p=227 We look forward to seeing many of you in San Diego for our upcoming meeting.  Full details follow and are on the web at http://tinyurl.com/crra-jan2011.

Portal development is a focal point for this meeting.  Many milestones have been met and Eric will demonstrate new portal functionality including Web 2.0 features of VuFind, an EAD indexing and display tool, and text mining techniques to facilitate discovery and creation of new knowledge.

For those of you unable to join us on-site, please join via the live webcast.  You may virtually join the meeting at any time, simply by clicking on this link: http://connectpro87278527.adobeconnect.com/crra/.

Jeff Rach (jrach@sandiego.edu ) is helping us with all things technical, and  has made the whole process quite seamless.  When you log in, you will see documents we are viewing, hear the discussion via audio streaming, and join in the conversation via textchat.

If you have any questions, please contact me via email or my cell: 608.698.2519.

Best wishes, safe travels, and see you online –pat

—————–

University of San Diego
Thursday, January 6, 2011
Agenda and details
5:30-8:00 pm (Eastern) 4:30-7:00 pm (Central) 2:30-5:00 pm (Pacific)

On-site
Joan B. Kroc Institute for Peace and Justice (IPJ)
Across from the Copley Library, University of San Diego
Campus map at http://www.sandiego.edu/maps/pdf_print.php#

Webcast
To join the meeting, follow the link:
http://connectpro87278527.adobeconnect.com/crra/
Invited By: Jeff Rach (jrach@sandiego.edu)
Contact Pat via cell: 608.698.2519

Agenda
2:00 Refreshments
2:30 Welcome, introductions (Jennifer)
2:45 Steps in building the portal (Pat)
3:00 Portal demo (Eric)
○ EAD viewer
○ Website navigability
○ Concordance demonstration
○ Use statistics
3:45 break
4:05 Discussion (Pat)
○ Hosting usability studies at member institutions
○ How CRRA can help members to make records available, i.e., how the
process works or may be facilitated for institutions
○ The NEH Challenge grant, future directions
4:45 Wrap up, closing remarks (Jennifer)
5:00 Adjourn

For those who will be in San Diego
In addition to the meeting at 2:30, tours of the University of San Diego (USD) campus will be
available to all CRRA attendees. Thanks to the generosity of Theresa Byrd, Director of the
Copley Library, lunch will also be provided. Events begin at 10:00 – feel free to attend one or all
events.

A map of campus is here: http://www.sandiego.edu/maps/pdf_print.php#

10:00 Tour of Copley Library and Archives
Meet at Copley Library
11:30 Lunch
Held at the Joan B. Kroc Institute for Peace and Justice (IPJ) 165 Conference Room F
(Board members will be meeting in IPJ 164 Conference Room G)
12:30 Walking tour of campus that will include a visit to a current exhibit on campus titled: |
Dreams & Diversions: 250 Years of Japanese Woodblock Prints: http://
www.signonsandiego.com/news/2010/sep/18/exhibit-of-japanese-woodblock-prints-includes/
Meet at Joan B. Kroc Kroc Institute for Peace and Justice (IPJ) 165 Conference Room F
2:00 Refreshments
Kroc IPJ 164 Conference Room G
2:30 Open Forum – CRRA all members meeting
Kroc IPJ 164 Conference Room G
5:00 Adjourn
6:00 Dinner (on your own) at C Level Restaurant

TRAVEL INFO
AIRPORT
The campus is a 10-minute cab ride (approximately $15.00).
Super Shuttle: http://www.supershuttle.com/

USD TRAM
Listed below is information about USD’s tram service which runs from the campus to the Old
Town Transit Center every half hour between 6:45am-10:15am and then from 3:00pm-7:30pm.
Conference participants can take the San Diego Trolley from downtown to the Old Town Transit
Center. Here is the link to the San Diego Trolley: http://www.sdmts.com/trolley/trolley.asp
Due to the completion of finals and classes not being held Wednesday, December 22, 2010, the Tram
Services Department will be running on a modified schedule. Please note the number of trams in service
will be reduced and the hours of operation will be limited to the following times:
Campus Routes (reduced service)
6:30 a.m. to 7:30 p.m.
Old Town (regular schedule)
6:45 a.m. to 10:15 a.m.
3:00 p.m. to 7:30 p.m.
Also due to the extended holiday period the Tram Services Department will be shut down completely from
December 23, 2010 thru January 2, 2010. There will be no trams servicing the campus, as well as trips to
and from the Old Town.
December 23, 2010 – January 2, 2010
No tram service due to the campus closure for the holidays.
For those participating via the Web
The San Diego meeting will be available to all members and friends via the Web using the userfriendly
application, Adobe Connect. Those joining the meeting via the Web will see and hear
meeting attendees in San Diego via video and audio streaming, will see shared documents,
and able to post comments and ask questions via text chat. Please join in for as much of the
meeting as you are able!
For those interested in the meeting, but unable to attend in-person or via the web, the session
will be archived.
To join the webcast:
Meeting Name: CRRA Meeting
Invited By: Jeff Rach (jrach@sandiego.edu)
To join the meeting:

http://connectpro87278527.adobeconnect.com/crra/

————
If you have never attended an Adobe Connect meeting before:
Test your connection: http://connectpro87278527.adobeconnect.com/common/help/en/support/
meeting_test.htm
Get a quick overview: http://www.adobe.com/go/connectpro_overview
Adobe, the Adobe logo, Acrobat and Adobe Connect are either registered trademarks or trademarks of Adobe Systems Incorporated in the
United States and/or other countries.
——-
Jeff Rach
Senior AudioVisual Technician
Joan B. Kroc School of Peace Studies
University of San Diego
5998 Alcalá Park-KIPJ 134
San Diego, CA 92110-2492
Phone: (619) 260-7810
Fax: (619) 260-7809
jrach@sandiego.edu

CONTACT INFO
Whether you are in San Diego or joining via the webcast, if you have any questions, please
contact me (Pat) via my cell at any time: 608.698.2519.

1. CRRA website – The good ol’ look & feel but wrapped around new content and functionality. (“Thank you, Eric Frierson!”)
2. Web 2.0 – All the Web 2.0 links (cite this, email this, favorite this) that did not work previously now function correctly.
3. EAD viewer – It is now possible to view EAD files locally or from the originating institution.
4. Item-level indexing – The content of EAD files is indexed at the item level making for finer-grained searching.
5. PDF display – Records linking to digitized versions of books now enable a person to get the full text. Examples include content from the St. Michael’s and the University of Notre Dame
6. Text mining – After extracting the full text from the PDF documents, it is possible to apply concordancing techniques to the full text for analysis.
7. Automated updating – The “Portal” can be updated automatically by harvesting metadata from member institutions, massaging it for the Portal, and re-indexing it on a regular basis.
8. Use statistics – Rudimentary Web server log file analysis as well as Google Analytics reports illustrate how the Portal is being used.
9. Blog – A running commentary on what’s happening with Portal development.

Naturally, I wrote a program to do this work — parse.pl. It is rather brain-dead and certainly not 100 percent accurate, but it goes generate a report of some value.

In the end, the Portal was queried approximately 18,000 from September to December in 2010. The report itself lists the top 100 queries and the number of times they were searched. The top 5 and the number of searches are:

1. Meditations (1462)
2. Cardinal virtues (918)
3. Newman, John Henry 1801-1890 (349)
4. Apostles (192)
5. Theological virtues (184)

The report also lists each query searched only once. Here’s a random sample:

“Christian saints Algeria Hippo (Extinct city) Biography.” * “Christopher Hollis” * “De rege et regis institutione” * “DeAndreis, John A. 1920-1979″ * “John Pearson (bishop)” * “John Pearson (cricketer)” * “John R. Cavanaugh” * “John R. Ryan” * “John Richard Parker” * “John Robert * Church year sermons Early works to 1800 * Church year sermons Early works to 1800 Indexes. * Self-esteem * Self-evaluation. * Seminary * Senigallia * Sermons, Chinese * Sermons, English * Sermons, German Early works to 1800 * pontificalia * portavoz * portrait * postmodernity * worldview * wrestling * yellow fever * younger * zill

I think the value of 18,000 queries is high. I will have to investigate that. Based on the queries, I believe most people are browsing the system and not necessarily entring specific queries. Why do I think this? Well, who puts in all of that syntax when searching?

    Who: Anybody and everybody across the University
   What: A set of presentations and workshops on
         digital humanities
   When: Thursday afternoon (February 24) and Friday
         morning (February 25)
  Where: (probably) Geddes Hall
    Why: Because it is about more than find and
         access, it is also about use and
         understanding

The Hesburgh Libraries, the Center for Research Computing (CRC), and the Catholic Research Resources Alliance (CRRA) are jointly sponsoring a set of presentations and workshops on the digital humanities Thursday afternoon (February 24) and Friday morning (February 25). While all of the details have yet to be ironed out, we expect there to be at least two presenters on Thursday:

1. Crivella West – Working closely with St. Michael’s College of the University of Toronto, Crivella West is applying text mining computing techniques to the Cardinal Newman archives for the purposes of providing enhanced understand of Newman’s writings and thought. We expect Crivella West to describe these techniques during the Forum.
2. Ron Snyder – Snyder is a driving force behind some of JSTOR’s research & development efforts. He will be discussing the digital humanities in general as well as demonstrating JSTOR’s Data For Research interface which allows one to search JSTOR, illustrate results with charts & graphs, and download resulting datasets for further analysis.

On Friday morning we hope to facilitate two hands-on workshops with Snyder. The first will be akin to a traditional “bibliographic instruction” session where participants will learn in detail how to use JSTOR’s Data For Research interface. This workshop is intended for scholars, researchers, and librarians. The second workshop is intended for computer programmers and it will deal with the in’s and out’s of using the raw datasets extracted from searches.

In the end the Libraries, the CRC, and the CRRA hope to raise awareness of digital humanities computing techniques. With the advent of so much full text, the Internet, and ubiquitous computing horsepower new methods for understanding the written word are manifesting themselves. The Forum will make these ideas concrete.

‘More later, but mark your calendars now.

DePaul is the CRRA’s newest member and we welcome and thank you!

Moreover, since the “sandbox” is runs version 1.0 of VUFind, many of the Web 2.0 links work correctly. In other words, things like emailing, tagging, citing, reviewing, etc. function correctly.

While we could move this whole thing into production, it may behoove use to associate hyperlinks with each found item to point back to hosting institutions to facilitate access. What do you think?

The problem

The University of Notre Dame owns a significant number of Catholic pamphlets. These materials have been cataloged and denoted as destined for the “Portal” in their MARC records with the letters “CRRA” in field 590$u.

The University’s library wants to digitize these materials, make the resulting PDF files freely available on the Web, apply optical character recognition against the PDF files, and support a text mining interface against the result. Bits and pieces of this work have already been done. The problem is gluing them together into functional workflow.

The solution

Here is an outline of a proposed solution:

1. scan the documents
2. convert them into PDF files
3. give them meaningful names
4. store them in a Web-accessible location
5. update their corresponding MARC records

Each of the sections below elaborate on the steps above.

Scan documents

Scanning the documents is the actual process of digitizing them. It can be done in-house with brute force using something like the Bookeye hardware. Or it could be outsourced. The technicalities of the process (dpi — dots per inch, color versus black & white, TIFF versus JPEG versus PNG,etc.) will be driven by collection development policies and the intended use of the materials. Personally, I think the pamphlets ought be scanned as black & white images at 300 dpi and saved as uncompressed TIFF images. Compared to original manuscripts or colorful out-of-copyright materials, I do not think the pamphlets warrant anything more substantial than that.

Convert to PDF

The original TIFF images function as archival surrogates. PDF files are intended for use. The next step is to concatenate the TIFF images representing a single pamphlet into a a single PDF file. During this process OCR (optical character recognition) should be applied to the images and saved inside the PDF files. This process can be done using our existing in-house facilities or an outside jobber. A word of caution, if the scanned images include two pages, then care will need to be taken when doing the OCR. Specifically, the OCR process needs to not go all the way across the page, but rather all the way down one image first and then down the next. Otherwise the resulting plain text will not be ordered correctly.

Give the files meaningful names

There is plenty of room for interpretation in the previous two steps, but there is little or no room for interpretation in this step. Save the TIFF and PDF files with names corresponding to the pamphlets’ MARC record field 001. The 001 field is a unique value across the library’s collection, and by using this value it will be easy to match a file with its description. For example, suppose the 001 field of a given pamphlet has a value of 00023459. Then assign the TIFF files names such as 00023459-001.tiff, 00023459-002.tiff, 00023459-003.tiff where everything before the dash (-) is the 001 value, and everything after the dash is a page number. Similarly, and most importantly, save the PDF files with names such as 00023459.pdf. One could save files in a single directory with a name corresponding to the 001 value, but the principle is the same. Explicitly associate the saved files with a MARC record. If the 001 field is not used during the file naming process, then updating the MARC records with URIs, below, will be considerably more expensive.

Store files

The next step is to store the files in a Web-accessible location. This could be as simple as putting them on a computer’s hard disk and providing access via the Web. Unfortunately, this process is not very scalable after 10′s of thousands of files. Consequently, it maybe be better to save the files in a repository-like application such as Fedora. The technology behind storing the files is not as important as the resulting URI/URL. It is very important to make sure the URL is constant and immutable. If it is not, then the risk of an ongoing maintenance nightmare is dramatically increased. It is also better if the URLs are shorter rather than longer. Put in the language of the Web, follow the principles of “cool URLs” when making the digitized content Web-accesible. [1]

Update MARC

Links now need to be made between the description of the pamphlets and their location on the Web. To do this one loops through all of the digitized pamphlets and updates the 856 fields of the corresponding MARC records with the URI/URL created in the previous step. This can be done manually, but it can be done programmatically if the files have been saved with values based on MARC field 001 values. Once this process is completed it will be possible for the patron to search the library’s catalog or “discovery system”, identify a Catholic pamphlet of interest, and then choose to retrieve it from Special Collections or download it from the Web.

Optionally, update MARC again

The process outlined above provides access to the materials, but if we want to make it easier for the patron to use and manipulate the materials, then an additional URI/URL will may need to be added to the MARC record. Specifically, an additional URI/URL in field 865 may need to be included pointing to a text mining interface. Just like the URI/URL pointing to the PDF file, this URI/URL needs to be as constant and immutable as possible. If this additional step is done, then not only will the patron have access to the materials in physical and digital form, but they will also be able to perform various text mining functions against it. Examples include: listing all the words starting with the letter z and the number of times they occur, listing the 50 most frequently used words, listing the 10 most frequently used two-word phrases, concordancing the results of the previous examples thus displaying them in context.

Summary

If the process outlined above is implemented, then the “Catholic Portal” software will be able to regularly and systematically harvest the Catholic pamphlet MARC records and integrate them into the CRRA in exactly the same way all of the other CRRA content is harvested.

Providing access and use of Catholic pamphlets is of interest to both the University of Notre Dame as well as the Catholic Research Resources Alliance (CRRA). Digitizing the materials, making them Web-accessible, and integrating their locations into our collection is a way of exploiting the current technological environment as well as meeting patron expectations. By providing text mining functionality we — the Libraries and the CRRA — will be exemplifying leadership in the wider community.

Notes

[1] “Cool URIs for the Semantic Web” — http://www.w3.org/TR/cooluris/

Jean McManus used a scanner to create PDF versions of a few Catholic pamphlets. Along the way, she also had the software to a bit of OCR. She then gave the PDF documents to me with filenames matching MARC 001 fields.

I saved these files to our local file system and used the venerable pdftotext application to extract the plain text. I then hacked my locally harvested MARC records describing the given pamphlets with two additional URLs. One pointing to the local PDF file. Another pointing to a rudimentary text mining interface. Finally, I reindexed the MARC records making the URLs visible. There were only three edited records, and you can see the fruits of these labors here:

• http://vufind.library.nd.edu/Record/undmarc_000537132
• http://vufind.library.nd.edu/Record/undmarc_000841024
• http://vufind.library.nd.edu/Record/undmarc_000941495

There are many things wrong with the implementation. The text mining interface points to invalid catalog records because they are hard-coded for University of Toronto content. The titles of the content include MARC field 245$c, but the older text mining interface did not expect this. Consequently, the title information for these newly added records is invalid. The PDF documents were scanned two pages at a time. This probably causes the extracted text to span both pages and thus invalidate every sentence. We will need to scan only one page per image to circumvent this problem.

Despite these difficulties, it is possible now to do a bit of analysis against the pamphlet, but there are many avenues for improvement. “Software is never done.”

The Problem

The heart of my problem is that the Portal is essentially a union catalog — an aggregation of metadata from a number of archives and libraries. Search results do not come from a single institution but a multitude. Search results need to display specific information about locations — information describing what library owns each item, what institution hosts the library, and the call number of selected items. Out-of-the-box VUFind is designed to dynamically query a library’s integrated library system (ILS), but I deemed this too complicated for our purposes. Too many different systems. Too expensive in both time and energy.

Consequently, my problem is, “How do I display location information in a multi-library environment?”

The Solution

The solution includes three parts: 1) exploiting the indexer (Solr) supporting VUFind, 2) creating and editing VUFind record drivers, and 3) editing Smarty templates.

Exploiting the indexer

The first part of the solution was documented in a previous posting where I described how metadata in the form of MARC records and EAD files is being indexed with Solr. To summarize, as each record is fed to Solr, three bits of information are saved in the index: 1) the type of data being indexed, 2) the name of the library holding the index item, and 3) the name of the institution hosting the library. This part works well and is pretty much divorced from the internal workings of VUFind.

Creating and editing record drivers

The second part of the solution was about creating and editing VUFind record drivers. VUFind is designed to branch to alternative sets of code based on the value of a Solr field called record type. In the case of our Portal, a record type may be “marc” or “ead”. VUFind handles MARC out-of-the-box. As documented in the VUFind wiki, to handle things other than MARC a record driver needed to be created. This is what I did, and in our case the driver is called EadRecord.php.

The astute PHP programmer will see that the driver inherits the vast majority of its functionality from the IndexRecord.php; EadRecord.php only overrides two methods. The first (getURLs) reads the value of the fullrecord field, uses an XPath expression to extract all the URLs from the fullrecord field, and returns the URLs as an associative array. The second method (getHoldings) is almost an exact duplicate of getHoldings from IndexRecord.php, but creates two new Smarty tokens (CRRALibrary and CRRAInstitution). Like the URLs, the values for these tokens are pulled directly from the fullrecord field.

Next I needed to edit the MarcRecord.php and IndexRecord.php record drivers. Editing MarcRecord.php was trivial. All I had to do was turn off a Boolean value. Specifically, I had to denote the value of summAjaxStatus as false because I did not want VUFind to try to query a remote ILS for holdings information. (I think this sort of thing ought to be configuration setting in config.ini, but that is for another time.)

Editing IndexRecord.php was almost as easy. Just like EadRecord.php, I needed to create two new Smarty tokens (CRRALibrary and CRRAInstitution) and stuff them into templates. This was done in the getHoldings and getSearchResult methods. Additionally, in getHoldings, I needed to create a third token — summCallNo to hold call numbers. Again, just like EadRecord.php, this was done by pulling values out of the fullrecord field. Very, very nice.

Editing templates

The last part of the solution was editing the Smarty templates. This was done by adding my newly created tokens (CRRALibrary, CRRAInstitution, and summCallNo) in both result.tpl and holdings.tpl. I find Smarty to be a bit obtuse because it requires the developer to understand YASS (“Yet Another Scripting Syntax”). Personally, I don’t think logic such as if-then statements should exist in template files. Logic belongs in code.

Observations

In retrospect, the whole process from indexing to display was pretty efficient. Gather metadata. Parse it. Feed it to indexer. Search. Get results. Parse (something else). Fill templates. Display. In order for this process to be maintainable, I need to hope things like IndexRecord.php and MarcRecord.php do not change over upgrades. As new versions of VUFind come along I will replace the IndexRecord.php and MarcRecord.php with my versions as well as drop in EadRecord.php. Unfortunately, I don’t think I can guarantee zero changes in IndexRecord.php and MarcRecord.php. These are gotchas I will need to expect. Similarly, I added new logic to my template files. Another thing to keep in mind but not as detrimental as changes to the record drivers.

Finally, I want to publically acknowledge Demian Katz’s assistance. He outlined what needed to be done clearly and suscinctly, both in writting as well as verbally. His description of VUFind’s architecture was very understandable, especially for this Perl programmer. He knows the application very well. Thank you.

The posting outlines how I have: 1) mirrored metadata and full text content from the Internet Archive, 2) made the mirrored content accessible through VUFind, and 3) implemented a rudimentary text mining interface against the mirror.

Background

The “Catholic Portal” is intended to be a research tool centered around “rare, unique, and uncommon” materials of a Catholic nature. Many of these sorts of things are older as opposed to newer, and therefore, many of these things are out of copyright. Projects such as Google Books and the Open Content Alliance specialize in the mass digitization of out of copyright materials. By extension we can hope some of the things apropos to the Portal have been digitized by one or more of these projects.

Very recently St. Michael’s College in the University of Toronto has become a member of the Catholic Research Resources Alliance, and consequently, they desire to contribute to the Portal. As it just so happens, the University of Toronto has been a big proponent of mass digitization. They have been working with the Open Content Alliance for quite a while. Much of their content, including content from St. Michael’s, has been digitized. Complete with MARC records, PDF files, and plain text these digital artifacts are freely available for downloading. Moreover, the availability of full text content opens up the doors to all sort of text mining and digital humanities computing techniques in library “discovery systems”. Collocations. Word clouds. Graphing and mapping. Concordancing. Etc. As an example of one such discovery system, the Portal not only provides access to the content, but it can also make the content useful.

With input from Dave Hagelaar, Pat Lawton, and Remi Pulwer I implemented all of the things above, to some degree. The balance of this posting describes how.

The Process

Dave Hagelaar from St. Michael’s College sent me a set of around 600 Internet Archive unique identifiers from their collection representing “rare, unique, and uncommon” materials. Based on previous work, I was able to harvest the metadata, mirror the content, and integrate the whole into our VUFind interface. The process included the following steps:

1. Convert identifiers – Each of the Internet Archive identifiers (keys) represent a Web page complete with metadata and links to digital content. The identifiers look something like this: delancienneetdel00rich. Given this information sets of URLs can be constructed pointing to locations at the Archive. Creating a set of URLs based on the list of keys was done with a trivial Perl script called keys2urls.pl. The resulting URL look like this:
   • MARC – http://www.archive.org/download/delancienneetdel00rich/delancienneetdel00rich_meta.mrc
   • PDF – http://www.archive.org/download/delancienneetdel00rich/delancienneetdel00rich.pdf
   • plain text – http://www.archive.org/download/delancienneetdel00rich/delancienneetdel00rich_djvu.txt
2. Mirror content – The next step was to copy the remote data locally — mirror it. This was done using the venerable wget program. Essentially, wget is called with a very long set of parameters as well as the output from Step #1. The result is a local cache of MARC, PDF, and plain text files. Since these files were saved in their own directory on an HTTP file system, each file has its own URL. To make life easier, the running of wget with all of its parameters was implemented as a simple shell script — mirror.sh
3. Enhance MARC records – Given the additional locations of the mirrored content, the MARC records harvested from the Internet Archive were not complete. They did not include URLs pointing to the Internet Archive, nor did they include the URLs pointing to the local cache. Consequently the next step was to enhance the MARC records. This was done with a second Perl script called updatemarc.pl, but the script does more. Since we hoped to provide text mining services against the full text, a third URL needed to be included in the MARC pointing to the text mining interface. Finally, since the text mining application needs a bit of metadata itself, a rudimentary database listing the full text items is created along the way. This entire subprocess was complicated by the fact that not all of the harvested MARC records were valid. Because of character encoding issues, some of them were not readable by my MARC record parser (MARC::Batch). Some of the records are structurally incorrect. Invalid leaders and misplaced record/field/subfield delimiters. Finally, some of the records apparently included invalid values for various indicators. To make sure the database was as clean as possible, any record generating any sort of error was not included in the final processing. This left approximately 400 of the original 600 records.
4. Index MARC records – The next step was to ingest the MARC records into VUFind’s underlying Solr index. This was done with a Perl script called marc-index.pl and described in a previous posting. With the completion of this step, the content provided by St. Michael’s College became available in the Portal. Search or browse the Portal for records. Find items from St. Michael’s. Click on a link to get the content from the Internet Archive. Click on another link to retrieve it from the local cache. For example, see the record for Letters of an Irish Catholic layman.
5. Support text mining – The final step in the process deserves a blog posting in its own right, and thus only a summary will be provided here. At its foundation, text mining surrounds the process of counting ngrams whether they be single letters, single syllables, multiple syllables, individual words, multi-word phrases, sentences, etc. Once these things are counted they can be measured. Once they are measured, patterns can be sought, and if patterns are found, then overarching descriptions can be articulated resulting in the creation of new knowledge or an increase in understanding. When coupled with concordances, ngrams can be placed within the context of the larger work to learn how they were used. Using two Perl modules (Lingua::EN::Ngram and Lingua::Concordance) a simple Web-based interface was written allowing the scholar to list the most frequent ngrams in a text, map their relative locations in it, and read snippets of text surrounding them. Using this technique it is possible to quickly and easily get an overview of the content of a document. The text mining application I created is initialized with an Internet Archive identifier. The application reads the identifier, looks up the location of the locally cached plain text file, reads it into memory, and allows the researcher to do “distant reading” against it. Unfortunately Lingua::Concordance only works sporadically against non-English files, but you can still see how the system works by using the concordance against Letters of an Irish Catholic layman.

Summary

The process outlined above described how full text content can be harvested from the ‘Net and integrated into the VUFind “discovery system”. The key to doing this easily was the existence of metadata (MARC records) describing the harvested items. Without this metadata the process would have been too laborious. The process also outlined how the harvested full text can be put to greater use through a simple text mining interface.

Software is never done. If it were, then it would be called hardware. Consequently, there are many ways the process can be improved. Examples include figuring out ways to repair broken MARC records, and updating Lingua::Concordance to work correctly with foreign language materials. Maybe I should call this job security.

Here goes:

1. Extract data – Open the spreadsheet. Activate the ACCU tab. Copy all of the data sans the “cool” data entry macros. Create a new spreadsheet. Paste all of the previously cut data into the new spreadsheet. Save the new spreadsheet for future reference with the name catholic_libraries.xls.
2. Extract more data – Repeat Step #1 for the tab labeled Tab 2, but save the newly created spreadsheet with the name atla.xls.
3. Clean – Open catholic_libraries.xls and delete columns so the only ones remaining are: last name, first name, school, address, city, state, zip code, and email address. Make sure the remaining columns are in the the order listed above. During this process the data may need further cleaning. For example, curly quotes need to be straightened. Carriage returns inside cells need to be removed. Make sure city and state values contain only… city and state values. No countries.
4. Sort – Sort catholic_libraries.xls in ascending order by school.
5. Save – Save the cleaned and sorted data as a tab-delimited text file with the name catholic_libraries.db. Make sure the resulting text file is Unix-based and not DOS- or Macintosh-based. Additionally, Excel often tries to do you a favor by surrounding fields containing commas with quotes. Remove the quotes.
6. Go to Step #3 – Repeat the process for the file named atla.xls, but include only the last name, first name, school, city, state, and email address in the saved data, and call the result atla.db.
7. Mount – Mount the saved database files (catholic_libraries.db and atla.db) by saving them to the Portal. They are expected to live in Y:\data\vufind\web\etc.

The result of this work should then be visible under the Directory tab.

This process is a bit tedious, but since the directory does not change very often, and now that I have documented the process, the next time the directory needs updating things will be easier. On the other hand, as the Portal grows, there will be a need for a real database, and it will be able to support additional functions, such as document delivery. Stay tuned.

• Ann Hanlon
• Demian Katz
• Eric Frierson
• Eric Morgan
• Kevin Cawley
• Pat Lawton
• Susan Leister
• Thomas Leonhardt

I did a bit of “Portal” show & tell demonstrating the work done to date on indexing EAD files. (See the previous blog posting.) We then discussed ways the indexing/display could be improved. Suggestions included:

• putting the words “Archival material” into the format field of the Solr index thus allowing better faceting
• reading the value of langmaterials and using it as the value for Solr’s language fields, again allowing for better faceting
• reading all of the fields associated with a given container-level element and putting them into Solr’s allfields field to improve indexing
• extracting the last value of our current “title”, using it as our title, and using the remaining values as some sort of supplemental description or alternatively, simply reversing the “title” string

We then brainstormed ways to resolve character encoding issues, the feasibility of making our metadata available via Web servers, and the status of the metadata guidelines.

We felt we had discussed it all, so the meeting was over.

Background

The Catholic Research Resources Alliance (CRRA) is a member-driven organization with the purpose of making available “rare, unique, and uncommon” research materials for Catholic scholarship. Presently the membership is primarily made up of libraries and archives who pool together their metadata records, have them indexed, and provide access to the index. My responsibility is to build and maintain the technical infrastructure supporting this endeavor.

A couple of years ago much of the CRRA metadata was manifested as MARC, and at that time VUFind was selected as the tool we would use to index, search, and display this content. About six months ago the Alliance realized the growing necessity of including EAD files as well. At the same time, the ability of accomodate non-MARC metadata was increasingly becoming a VUFind reality. New ground still had to be broken; processes needed to be implemented allowing VUFind (and the underlying Solr indexer) to understand how to work with materials which were not book-like.

The balance of this posting describes in greater detail how I am beginning to accomodate MARC as well as EAD metadata into VUFind’s interface with Solr.

Assumptions

The system runs on a number of assumptions. First, it is assumed it is the members’ responsibility to create and maintain their metadata. Second, it is my responsibility to index it and make it available for display. Moreover, it is assumed each metadata record incudes at least three values: 1) a unique identifier, 2) a human-readable description of an item, and 3) an address pointing to the location of the item. For MARC records, these things reside in the 001, 245, and 099 fields. For EAD files, they have been designated as the id attribute of unitid elements, the content of unititle elements, and the url attribute of the eadid element and from there the location of the item.

Additionally, it is assumed all metadata records, whether MARC or EAD, are available for harvesting from a Web server. In other words, each member who wants to have their MARC records available in the CRRA needs to export their records to a single file and make them accessible via a URL. Similarly, all EAD files which are intended to be indexed need to be in a single Web-accessible directory and the URL of the directory needs to be known. Making member metadata accessible via a Web server has three benefits: 1) it facilitates automation, 2) it distributes the responsibility of archiving metadata across the membership, 3) it enables the metadata to be harvested by other applications and used for other things. “Can you say ‘linked data?’”

Files and Perl scripts

Given these assumptions, the following sets of files and Perl scripts are used to do the work. The first set is core the both of the other two:

• libraries.db – A “database” of CRRA participants consisting of their names, libraries, and URLs where their metadata records can be found. This file is used by just about every other script in the system.
• subroutines.pl – A tiny library of Perl subroutines, mostly to read the contents of libraries.db.

This second set is used to index MARC metadata:

• marc-harvest.pl – Copies (mirrors) remote MARC files locally
• marc-add-code.pl – Validates and updates the values of MARC 001 fields making sure they exist and are unique
• marc-index.pl – Slurps up a Solr marc.properties template (template.txt), makes the appropriate substitutions, and indexes the MARC records associated with a given library
• marc-build.sh – A shell script used to run all of the MARC-based scripts. One ring to rule them all.

The third is used to index EAD files:

• ead-harvest.pl – Copies (mirrors) remote XML files locally
• ead-validate.pl – Makes sure the mirrored XML files are well-formed, conform to the EAD DTD, and include an eadid url attribute (done with a stupid stylesheet called geturl.xsl)
• ead-transform.pl – Makes sure each EAD container-level element includes a unitid with a unique id attribute, saves the result to a local cache, and transforms these same files into HTML. The first process is done with a stylesheet called addunitid.xsl. The second process is done with another stylesheet called ead2html.xsl.
• ead-index.pl – Indexes all the cached/transformed EAD files by parsing out container-level elements, creating an XML stream of records of my own design, parsing the result, and passing each record on to Solr. The heart of this script is a fourth stylesheet — ead2solr.xsl
• ead-build.sh – A shell script used to run all of the EAD-based scripts. Another ring to rule them all.

The “secret” to indexing EAD files is really no secret. I simply followed Demian Katz’s instructions. In a nutshell, to index non-MARC content the developer needs to:

• Parse the given metadata into records. I do this with ead2solr.xsl.
• Map each of the record’s values to as many of the underlying Solr fields as possible. Presently I only have titles and I do this through ead2solr.xsl as well.
• Create an XML snippet representing each record and map it to the Solr fullrecord field, described below.
• Denote a record type. I call mine ead.
• Save the whole thing to Solr, done with ead-index.pl.

Currently, my XML snippet (Item #3) looks like this:

  <record>
    <id>unaead_id2635150</id>
    <title>Catholic Church. Archdiocese of Detroit (Mich.)
      Collection -- Catholic Church. Archdiocese of
      Detroit (Mich.): Manuscripts -- Letters -- Bp.
      Baraga to his sister Amalia
    </title>
    <date>1836/1203</date>
    <url description='View remote, canonical version of EAD'>

http://archives.nd.edu/findaids/ead/xml/det.xml

    </url>
    <url description='View local version of EAD file'>

http://zoia.library.nd.edu/sandbox/crra-data/ead/una-det.html#id2635150

    </url>
  </record>

The VUFind application provides seamless access to the index through its search box, but a bit of work needs to be done to display search results. Specifically a “record driver” needs to be written to accomodate new record types (Item #4, above). This driver inherits methods from a parent driver, IndexRecord.php, and the developer needs to override some of the methods found there with methods considering the content of the fullrecord field. Presently, the only thing I have in my record driver (EadRecord.php) is a method to extract URLs. In the future I will need to include methods to extract names of CRRA members, names of their libraries, and additional descriptive metadata.

You can see the fruits of these efforts in the CRRA “sandbox” — something we are affectionately calling “The Green Interface”.

Issues

The whole process functions and could be run automatically from cron on a daily basis, but there is plenty of room for improvement. Issues include:

• speed – The indexing process is slower than I’d like. I think throwing more hardware thrown at the problem will make things faster.
• invalid data and stale URLs – A small percentage of the MARC and EAD files do not include the required metadata values. No unique identifiers. Malformed MARC leaders. Non-validating EAD files and/or eadid url attributes pointing to broken locations. This is where metadata maintenance comes in.
• character encoding – This is one of the bigger problems. Trying to figure out whether or not a MARC record has been exported as UTF-8 is difficult. Solr assumes UTF-8 and I don’t think it even knows about MARC-8. When MARC data is not encoded as UTF-8, search results look really ugly. Similarly, some of the EAD files, because of similar issues, really display poorly after they have been transformed, indexed, searched, and displayed.

None of these things are insurmountable. They will be addressed.

Next steps

My immediate next steps focus on richer search results. I need to extract additional information from the EAD files to supplement the content of my fullrecord field. After that I will explore the creation of “collection-level” records by indexing the headers of EAD files. These records will be fuller because they will have things like controlled vocabularies, scope notes/abstracts, and biographies from which to draw. Once the fullrecord fields are enhanced, I will need to go back to EadRecord.php and enhance its functionality. After that I will see about creating reports listing errors in metadata files. These reports will be designed to share with members making it easier for them to maintain their content.

All of that sounds like plenty to me. Wish me luck.

Assuming a Catholic Research Resources Alliance members place their EAD files in a HTTP-accessible directory, and those files have a .xml extension, then the following Perl scripts enable me to harvest and prepare them for indexing:

• harvest-ead.pl – reads remote HTTP-accessible directories and copies all of the .xml files found there to a local cache
• validate.pl – makes sure the cached XML files are well-formed and conform to the EAD DTD, and if not, then move the files to a different directory
• transform.pl – reads the validated XML files, adds id attributes to all unitid elements through the use of a stylesheet (addunitid.xsl), transforms the resulting XML into HTML using another stylesheet (ead2html.xsl), and saves the result to an HTTP-accessible directory

What was really cool and a huge time-saver was the use of ead2html.xsl. Originally named AAAv2002-HTML.xsl, found on a page called User Contributed Stylesheets, and submitted by Stephanie Ashley, this stylesheet took my id attributes and automatically made named anchors for me. Boy, did I get lucky. “Thank you, Stephanie!”

My next step is to revisit my indexing routines.

A few members of the Alliance were previously awarded a CLIR grant to catalog previously uncataloged special collections items. These members are now doing the work using EAD (Encoded Archive Description) with the intent of sharing the resulting metadata with the “Catholic Portal”. The purposes of the meeting were to build relationships between these particular Alliance members and to discuss progress on the grant. In attendance where people from St. Catherine University (Deborah Kloiber and Emily Asch), Marquette University (Matt Blessing, Ann Hanlon, Bill Fliss, and Jean Zanoni), and the University of Notre Dame (Pat Lawton, Kevin Cawley, and Eric Lease Morgan).

Of primary concern was the particular way people were using EAD and whether or not it would lend itself to indexing by the “Portal” software. Consequently, I spent a lot of the time describing the technical infrastructure of VUFind and how it interfaced with Solr, the underlying indexer/search engine. In short, the absolute need for unique identifiers, human-readable descriptions of items, and location codes were enumerated. The former two can be garnered from the unitid and unittitle elements of a EAD did elements. The later can be gotten from the url attribute of the eadid element. Everybody was confident their EAD files would contain these values.

We then went around the table doing a bit of show & tell against our EAD. The folks of St. Catherine’s were using the Archivist’s Tool kit to “catalog” their Ade Bethune collection. Marquette University was using a Microsoft Access database to “catalog” Dorothy Day content.

Time tables where then outlined. The whole CLIR project is expected to be finished by December of 2011. Participants in attendance thought their work would be done by the end of Spring 2011, and the remaining time would be spent on putting the content onto the “Portal” as well as doing various types of publicity (conference presentations, etc.).

The meeting was over around noon, and we all retired to the faculty club for lunch. (“Thank you, Marquette!”)

In retrospect, there may be two additional issues needing to be addressed. First, I originally planned to assign or replace unitid values with locally generated, “Catholic Portal” specific values, but I have since learned that unitid information is often times used as a sort of call number and therefore necessary for location. Replacing (removing) such values from the EAD files may make work down the line more difficult. Maybe I should be getting the unique values from an id attribute of the unitid element instead?

Second, as a group we may need to decide how to encode dates. Dates can be nested within unittitle elements as well as free-standing elements in the did. Just as importantly, they can take all sort of forms. In order to make sorting and faceting feasible, the Alliance may need to figure out ways to standardize and normalize dates.

You can (temporarily) see the fruits of these labors because all of the records have been associated with the Eric Lease Morgan Foo Bar Library. The result is a list of container-level records with very little additional information.

By the way, as of today I am running a version of VUFind as retrieved from the development trunk, specifically, revision 3029. When upgrading from revision to revision, it is important to retain one’s config.ini file and reindex. The process is not painful, if done infrequently. As time goes on I will also need to retain locally developed hacks, such as the ones I need to write below.

The next steps are to write the MARC record driver so it does not attempt to do automatic look-ups for call numbers, but rather extracts such information from of the local index. A second next step is to write an EAD record driver to accomodate the special cases of… EAD records.

In this update …

• CRRA Welcomes Three New Members
  • Duquesne University, Laverna Saunders
  • Loyola Marymount University, Kristine Brancolini
  • University of St. Michael’s College in the University of Toronto & Pontifical Institute of Mediaeval Studies, Jonathan Bengtson
• Digital Access Committee (DAC) News
• Member meetings in January and March – Mark your calendars!

CRRA Welcomes Duquesne University, Loyola Marymount University, and University of St. Michael’s College in the University of Toronto & Pontifical Institute of Mediaeval Studies!

We are pleased to announce the addition of three new members.   Following is brief information about our newest members, their collections and leadership.   A warm welcome to all!

[Watch for the welcome to Dominican University and University of San Francisco in October.]

New Member Highlights

Our new members bring a rich array of rare and unique resources to the CRRA.  Collection highlights and introductions to their member library deans/directors follow.  A warm welcome to all, we look forward to getting to working with you!

The Gumberg Library at Duquesne University (Pittsburgh, PA)

• Spiritan Collection http://digital.library.duq.edu/cdm-spiritan/.
  Many of the primary and secondary writings of the Congregation of the Holy Spirit.
• Papers of Cardinal John J. Wright and Vatican II http://www.duq.edu/archives/index.cfm
  Includes the Cardinal’s addresses, papers, sermons, writings, and personal library, with substantial material from Vatican Council II.
• Pittsburgh Catholic Newspaper http://digital.library.duq.edu/cdm-pc/ .
  Duquesne University has continuously microfilmed the Pittsburgh Catholic since its inception in 1844. To convert the Pittsburgh Catholic to digital format, Gumberg Library started with volume 1 issue 1 and will continue to digitize the newspaper from the oldest volumes to the newest volumes. Currently, the first 20 years covering from March 16, 1844 through 1864 are available by browsing or searching the full-text.

Laverna Saunders has served as University Librarian at Duquesne University since 2002 and worked at Salem State College, UNLV, DePauw University, Union College and Drew University over the course of her career.

She is a member of the PALCI Board and currently chairs LLAMA BES.  She has served on various ACRL committees over the years. She is a book reviewer for Technicalities, serves on the editorial board of Technical Services Quarterly, and has written articles and edited three books on the evolution of the virtual library. Her email address is lsaunders@duq.edu.

• 1481 printing of Dante’s Divine Comedy illustrated by Botticelli
• Rare Book Collection including Biblia Sacra Vulgatae editionis Sixti V. Pont. Max. (1603), A late 15th century example of a Book of Hours, probably printed in Paris.
• Jesuit Collection: 518 rare books by and about Jesuits
• Leaf from the Gutenberg Bible
• Catholic Human Relations Council Collection, 1958-1992
• Prominent Roman Catholic families in Los Angeles
• Reverend Harold F. Ryan, S. J., Catholic Authors Collection
• Saint Thomas More Collection
• Thomas G. Hanrahan Jesuit Drama Collection
• Ygnacio del Valle Family Collection: Eighteenth and nineteenth century Spanish-language devotional works, novels, and instructional works
• Early California Mission Vestments, liturgical clothing in use in California Catholic churches and missions during the early nineteenth century

Kristine R. Brancolini is Dean of the Library at Loyola Marymount University (LMU) in Los Angeles. Prior to her arrival at LMU in July 2006, she had been a librarian at Indiana University in Bloomington for more than twenty years, where she held a number of positions.  From 1998–2006, she was the Director of the Digital Library Program (www.dlib.indiana.edu); during that time she was principal investigator on numerous digitization projects with funding from the Institute of Museum and Library Services (IMLS), the National Endowment for the Humanities (NEH), and the U.S. Department of Education.  In August 2009, LMU opened a new library, the William H. Hannon Library, located on a bluff overlooking Marina del Rey and the Pacific Ocean.  The library has recently launched a number of new initiatives, including a Digital Library Program and a series of nearly 50 public programs in the library and elsewhere on campus.

• The Chesterton Collection includes over 3000 volumes connected with the life and work of the English journalist, G.K. Chesterton (1874-1936). It embraces virtually all the works of Chesterton. The collection also includes original sketches, complete microfiche of his personal papers, and the papers of John O’Connor (1870 – 1952), the Catholic priest who became the model for Chesterton’s character “Father Brown”.
• Henri J.M. Nouwen Archives and Research Collection documents the life and work of Henri J.M. Nouwen (1932-1996) and includes the vast majority of Nouwen’s manuscripts and published works, as well as secondary material in all formats about Nouwen.
• Archives of the Faith and Sharing Federation / Foi et Partage, a bilingual Catholic organization with a mandate to deepen and foster the experience of Christian community through week long retreats.
• The Counter-Reformation Collection includes over 3500 volumes of primary source materials showing the Catholic response to the Protestant Reformation up to the time of the French Revolution.
• The Pontifical Institute of Mediaeval Studies library has one of the largest primary and secondary source collections of medieval documentation in North America.

The library is also a lead partner in the development of the Humanities Knowledge Kiosk . The software has been applied to Newman’s works and is in development for Lonergan, Nouwen and Chesterton.

Jonathan Bengtson is the Director of Library and Archives for the University of St. Michael’s College at the University of Toronto and the Pontifical Institute of Mediaeval Studies, and a Fellow of the College and the Institute.  Educated in California, Oxford and London, he has held various senior positions in academic, research, and nonprofit libraries in Canada, the United States, and the United Kingdom—including Executive Director of the Providence Athenaeum (founded in 1753) in Providence, Rhode Island; Head Librarian of the Queen’s College, Oxford (founded in 1341); and, Associate University Librarian for Scholarly Resources at the University of Toronto. He is currently on the Board of Directors of the Society for the History of Authorship, Reading, and Publishing and the Multicultural History Society of Ontario. Since 2004, Jonathan has been the coordinator for the University of Toronto’s partnership with the Open Content Alliance and Microsoft Live Books mass digitization projects. He is actively involved, and one of the key collaborators, in a partnership with Crivella West Inc. of Pittsburgh to apply advanced textual linguistic analysis to public domain and in-copyright digital texts.  He is also writing a book for University of Toronto Press on an introduction to medieval manuscripts.

The Digital Access Committee met on September 2 to discuss topics 3 and (data input and ingestion; search functionality and display from the CRRA Strategic Plan Draft: Goals for 2010/2011.  (Minutes from CRRA meetings are available to all members.  Contact Pat for login and password to CRRA documents.)

Eric Frierson (St. Edward’s) and Wei Zhang (Georgetown) Join the CRRA Digital Access Committee (DAC)

The Digital Access Committee (DAC), under the leadership of Tom Leonhardt, welcomes two new members.  Eric Frierson of St. Edward’s University and Wei Zhang of Georgetown University bring technological expertise that will help us to move forward with development of the Catholic portal. Welcome, Eric and Wei!

Please reserve space on your calendars now – Thursday, January 6, 2011 and Tuesday, March 29, 2011. For the March 29, 2011 meeting, plan to join your CRRA colleagues for dinner on Monday evening and dinner on Tuesday as well for all who can.  We will organize the day to include both separate Board and committee retreats and plenary sessions for all.

Further details will be distributed in future Updates, the CRRA Blog, and email.

Check our progress and news on the CRRA blog: http://www.catholicresearch.net/blog/.

The problem

I am aggregating sets of EAD files from Catholic Research Resource Alliance members. I am expected to index these files at the most granular level possible — meaning at the did level. In order to satisfy both human and computer requirements, each indexed record needs at least a unique identifier, a human-readable descriptor, and a location code. The unique identifier can be gotten from the unitid element. The human-readable descriptor can come from the unittitle. The location code can be inferred from the url attribute of the eadid element.

Unfortunately, not all of the aggregated EAD files include a unitid, and when they do, they are not always unique. Additionally, the hierarchal nature of EAD files make the values extracted from unittitle elements almost meaningless unless they are placed within the context of their parent unittitle values. In short, indexing EAD files without some preprocessing makes the indexing process all but useless. What to do?

The solution

The solution includes: 1) adding and/or normalizing the unitid values, 2) constructing a more complete “title” based on previously enumerated unittitle values, 3) and outputting the whole thing to an XML stream easily indexable by Solr.

Adding and/or normalizing the unitid values (Step #1) can be accomplished with a stylesheet called addunitid.xsl. Essentially an identity transformation, the stylesheet loops through an EAD file using the generate-id() function to create or replace unitid values. The result is an enhanced EAD file.

Constructing more complete “titles” and outputting XML streams (Steps #2 and #3) is done by looping through the each did element, extracting the necessary metadata, creating a record describing each did-level element, and sending to STDOUT a rudimentary XML stream of my own design. The heart of this second stylesheet (ead2solr.xsl) is the ancestor::*/did/unittitle selector used to find all the parent unittitle values of a given did.

Finally, a simple shell script was written (clean.sh) making it easy to do the above transformations from the command line.

(I would not have been able to do this work if it weren’t for the XML4Lib mailing list and a few fine repondants to my pleas for help. Thanks go to MJ Suhonos, Tod Olson, Stefan Krause, and Alexander Johannesen. “Thank you!”)

Next steps

Software is never done. If it were, then it would be called hardware. Therefore next steps include:

• automatically adding the modified EAD files (the output of the first stylesheet) to Archon
• enhancing the output of the second stylesheet with scope notes, abstracts, etc.
• indexing the output of the second stylesheet

Fun with XSLT?

The problem

As the CRRA matures, I expect a greater amount of the metadata ingested into the “portal” will come from EAD files. In order to index EAD files meaningfully, I need to extract unique identifiers from each container-level element, a human-readable description of the container, and a location code. The identifier and human-readable description can easily come from unitid and unititle elements of did elements.

Unfortunately, unitid (and maybe unititle) are not required elements of did elements. While the CRRA could mandate the creation of such elements, it turns out to be almost just as easy to create them on-the-fly.

The solution

The good folks apart of the XML4Lib provided me with my solution — an XSLT stylesheet, below:

<xsl:stylesheet
  xmlns:xsl='http://www.w3.org/1999/XSL/Transform'
  version='1.0'>

  <!-- match everything and copy it -->
  <xsl:template match="node()|@*">
    <xsl:copy><xsl:apply-templates select="@*|node()" /></xsl:copy>
  </xsl:template>

  <!-- special case; match dids with no unitid -->
  <xsl:template match="//did[not(unitid)]">
    <xsl:copy>
      <!-- add a unit id -->
      <unitid><xsl:value-of select="generate-id()"/></unitid>
      <!-- continue copying -->
      <xsl:apply-templates select="@*|node()" />
    </xsl:copy>
  </xsl:template>

</xsl:stylesheet>

While not perfect, it certainly is a step in the right direction. Short and elegant. The next step will be to include some sort of parameter as input or to generate some EAD-specific identifier so each unitid value is unique across the corpus. (Actually, that is another issue I need to address.)

Thanks go to MJ Suhonos for the cool //did[not(unitid)] expression, Tod Olson for the idea of identity transformation (copying), and Stefan Krause for the use of generate-id.

Day #1

The Conference was divided into a number of presentations, group discussions, and informal social events. Joe Lucia (Villanova University) facilitated and opened the meeting with a number of general remarks surrounding libraries and the current environment:

The question is, “Who will fulfill the social mission of libraries in the future?” If libraries don’t do it, then some other institution will. Libraries represent a locus of knowledge for our communities and a place for cultural conversation. Open source software is rooted in this same social mission and congruent with the mission of libraries… Is Google Books and the HaitiTrust a new form of the “Information Commons”? Maybe, but maybe not… Cloud computing is a trend towards aggregation, concentration, and commercialization, but is that the best solution, since it too is not immune to proprietary lock-in… Software as service is also a current trend and we must ask ourselves, “Why not just build something based on the WorldCat APIs?” Public libraries are pointing a way towards the creation of knowledge spaces — a possible lead for academic libraries. Seen in this light, libraries may be new cathedrals.

Demian Katz (Villanova University) then shared how he has integrated VUFind with Serials Solutions’ Summon. After considering a number of options, he decided to go with single search and a two-column display. Do a search. Query the local VUFind (Solr) index. Simultaneously query the remote Summon index. Display both results in a common window with VUFind on one side and Summon on the other. Especially this means books are on the left and articles are on the right. “You can’t modify the Summon relevancy ranking, and thus you get a lot of noise. Merging the indexed content often places local materials lower in the relevancy ranked output.” There are a few things on Katz’s to-do list: the addition of social features, the highlighting of query terms, advanced faceting options, and a mobile interface. You can try this VUFind/Summon combination at library.villanova.edu/Find.

A similar presentation was given by Chris Spalding (Ex Libris) in his description of how VUFind can be integrated with Primo Central. “Through an API access to Primo Central content can be integrated with VUFind. We do two searches, get results, re-rank, and display. The key to the solution is the PC (Primo Central) add-on. We hope to do more collaboration and be as open as possible… We use AWS (Amazon Web Services) to host our content… We hope to share the code as soon as the end of the year, and we are sincerely trying to bootstrap the process of combining VUFind with Primo Central.” The approach described by Spalding is the approach I expected Katz to implement with Summon. Apparently there are problematic issues with both techniques.

Greg Pendlebury (University of Southern Queensland) then demonstrated a portable Javascript library called Anotar which is integrated with Fascinator (fascinator.usq.edu.au). Using CouchDB for a foundation, Anotar is intended to support the sharing of annotations across systems. Add comments to a Web page and have those comments syndicated across the ‘Net and accessible in other applications. The point for the community present was, “Maybe this sort of thing could be integrated into VUFind.”

Name & title authorities as well as controlled vocabularies was the focus of the next presentation, given by Katz. He first described how he experimented with prototypical Perl “hacks” found in a recent issue of Code4Lib Journal. These hacks exploit the WorldCat API to list authoritative names and subjects. He described another experiment where he integrated locally created authority content with the local VUFind (Solr) index. Finally he described a third possible solution taking advantage of the linked data provided by the Library of Congress. His next experiment will surround the use of the eXtensible Catalog Metadata Services Toolkit to munge and use authority records. “The use of authority lists make it possible for a person to do browse against the ‘catalog’.”

The group then broke into two or three smaller groups to discuss “birds-of-a-feather” sorts of ideas — breakout sessions.. Because of my interest in archival materials and EAD files, I went with the group called Beyond MARC. There we discussed things such as but not limited to the indexing of many different things: websites, EAD files, METS records, and full text. We also discussed the challenges of indexing hierarchical data, the content of boutique collections, and the provision of non-bibliographic services against metadata. In the end, we advocated for the greater use of VUFind record drivers, making it easier to support local customizations, and figuring out how to handle hierarchies.

Day #2

Working on a project called SWWHEP, Luke O’Sullivan (Swansea University) described how he hacked VUFind to work in a multi-ILS environment with the ultimate goal of providing reciprocal borrowing. Calling himself a “shambrarian” he described MARC as the “Dark Side of open source”. After being given sets of MARC records whose 001 fields had been modified for uniqueness, O’Sullivan essentially created a multitude of configuration files associated with each library system under his charge. When records were returned from searches his code looked at the 001 values and branched accordingly. Of all the implementations described during the Conference, O’Sullivan’s hack was the “kewlest”. See his good work at ifind.swwhep.ac.uk.

Birong Ho (Western Michigan University) was second up on the second day with a description of how she and her team exploited the use of Web Services computing techniques to communicate between VUFind and their local Voyager system. She uses these services to support holds, renews, etc.

I was then given the chance to describe a future for “next generation library catalogs”, a thing I call services against texts. In a nutshell, I advocated for discovery systems to go beyond find and move towards use, and with the increasing availability of full text content such a prospect is increasingly possible. “Quantitative metadata — as opposed to qualitative metadata — makes it easier to compare, contrast, and analyze individual items in collections or collections as a whole.” I then demonstrated how digital humanities computing techniques can be applied to full text content to discover underlying patterns.

We broke into small groups again — table talks — and brainstormed visions for VUFind 2.0. Some of the things we came up with at our table included: relevancy ranking based on social networking data, full text indexing, including content beyond books, personalization based on patrons’ characteristics or history, hooks to download full text from places like the Open Archives, the sharing of social data between VUFind implementations a la Ex Libris’s bX, tighter integration with Open Library, and an integration with VUFind into other applications through APIs.

Juicy quotes

Here is a short list of some juicy quotes I picked up from some of the attendees:

• “A plug-in architecture may be a good idea.” –Kun Lin
• “Consider bringing different views into VUFind instead of shelling out.” –Eoghan Ó Carragáin
• “Full text indexing is easily implementable as long as you tweak the boosting factor.” –Til Kinsler
• “Maybe part of the solution is to stop giving content to the vendor.” –Greg Pendlebury
• “Remember to exploit the record drivers in order to provide different services and views of content.” –David Lacy
• “Solr’s VUFind schema is currently flat but maybe the data model needs to be more flexible and maybe hierarchal.” –Till Kinstler
• “We are never going to have ‘one bucket’ searching.” –Joe Lucia

Observations and summary

The Conference was well-organized and provided a forum for plenty of discussion and idea generation. The setting was very nice and the food was plentiful. Everybody was able to participate. I heard a number of people say they were either implementing or toying with the idea of implementing Evergreen as their “catalog” and using VUFind as their “discovery layer”. I had not thought of this. Interesting. I appreciated the active participation of Chris Spalding. He was candid and sincere. It was very nice to put names from the mailing lists with faces, and thus the crowd was international. Blacklight was compared & contrasted with VUFind a number of times throughout the meeting. I believe both communities have something to learn from the other.

Alas, I was unable to stay for the fourth quarter of the event. I had a plane to catch, and I had made my reservations under the assumption the Conference would conclude at noon. I was wrong. Consequently I missed the last part of the meeting where next steps were to be articulated. If I had my druthers, two things would happen. First, I hope the development process becomes a bit more structured, complete with regular conference calls and software regression testing. Second, and along similar lines, I hope some entrepreneurial organization comes forward to provide commercial support for VUFind. Such a thing would make it more attractive to the libraries without local technical (computer) expertise.

Finally, I bounced my ideas regarding the indexing of EAD files off of as many people as I could. I think I am on the right track, even though few had experience with the same problem. Wish me luck.

At this time, we are putting together what promises to be a set of lively and informative discussions. This will be an opportunity to talk about CRRA activities taking place at your library, to discuss progress to date on the 2010/11 goals in the strategic plan, and to explore our readiness to promote the Catholic portal to librarians and scholars. VuFind 1.0 will be very near to being ready for implementation and this will be an opportunity to explore its functionality. Also, we will take a look at how the contents on the portal are growing particularly in regard to adding rare, unique and uncommon archival collections and other materials. The outlines of the proposal to be submitted to the NEH Challenge Grant will be ready for discussion. And, we want to hear from everyone – new and continuing members – how things are going at your library. Very importantly, this is an occasion to network and socialize with your CRRA colleagues.

Thursday, January 6, 2011, Copley Library, University of San Diego
• Noon to 2 p.m. Board of Directors (with both onsite and call-in participation for Board members)
• Noon to 2 p.m. Campus and library tours to be arranged
• 2:30 – 5 p.m. Open forum for all participants with refreshments provided by the Copley Library
• 5:30 Dinner for all participants at Le Gran Terraza which offers a fine dining experience on campus (your own treat)

Theresa Byrd, University Librarian, has graciously volunteered to host our group on campus at the University of San Diego. The University campus is situated on a mesa overlooking San Diego Bay. The Spanish Renaissance architecture and breathtaking views of Mission Bay, the Pacific Ocean, the community of Linda Vista and Tecolote Canyon make the campus a not to be missed destination in San Diego. The campus is also conveniently located near downtown San Diego. In addition to cabs (about $15), a regularly scheduled campus van to and from the sightseeing destination of Old Town San Diego offers an easy option for travel to campus. More information on location and travel will be sent at a later date.

Jennifer Younger, Chair, CRRA Board of Directors

The “Catholic Portal” uses VUFind as its “discovery interface” and sometimes I feel starved for people with whom to discuss issues surrounding the application. I then got wind of VUFind’s use at Western Michigan University (WMU) as well as the University of Michigan at Ann Arbor (U of M). Since WMU is half way between me and U of M I thought a “user’s group meeting” may be in order. A few calls were made, a few postings to a couple of mailing lists were written, and the meeting came to fruition.

There were nine of us in attendance:

1. Bill Dueber (University of Michigan)
2. Birong Ho (Western Michigan University)
3. Dean Lingley (Purdue University)
4. Eric Lease Morgan (University of Notre Dame)
5. Keith Kelley (Western Michigan University)
6. Matthew Riehle (Purdue University)
7. Roy Zimmer (Western Michigan University)
8. Scott Garrison (Western Michigan University)
9. Tod Olson (University of Chicago)

The good folks from Purdue suffered through the entire 3 1/2 hour event via Skype. “Kudos to Dean and Matthew.”

As a group we discussed quite a number of things, listed here in more or less chronological order:

1. straying from the code base – The hottest topic surrounded the difficulty of implementing VUFind version 1.0 given the fact that at least a couple of us have modified (“hacked”) previous versions to such a degree that implementing 1.0 was almost too much of a challenge. As one person said, “It might be easier to start all over with Blacklight rather than migrate my changes.” This does not mean anybody was dissatisfied with VUFind’s performance or many of its features. Record display is good. There is a distinct separation of inventory and OPAC. VUFind offers great flexibility, and public services staff seem very happy with the ease patron interfaces can be customized.
2. Blacklight – Given that, the discussion turned to a comparison between VUFind and Blacklight. While the group seemed to have minimal experience with Blacklight a number of things were definitely seen in Blacklight’s favor, such as: a more disciplined community complete with project management, the insistence of regression testing before code submissions were included into the base, and regular conference calls. Much of this was summed up as the “open source conundrum” — the differences between free software, open source software, and community source.
3. Solr – We then turned to a discusion of Solr since we all understood that VUFind and Blacklight were essentially client interfaces to the increasingly popular indexer/search engine. A number of us believed it was absolutely necessary to modify the underlying Solr schema in order to satisfy local needs. These modifications ran the gamut from what fields exist to how those fields are defined and filtered. We compared & contrasted the use of the stock query interface and the use of the Dismax handler. The indexing of data then led to a discussion how to handle diacritics, dates, and date ranges.
4. miscellaneous – As the discussion wound down we we talked about various things such as systems administration tasks, and whether or not to move the Solr indexer to another host or implement it under a servlet container other than Jetty.

I told the group I was going to attend the VUFind User’s Group Meeting taking place at Villanova in a couple of weeks, and I asked for a short list of things I ought to share there — take aways:

• governance – the VUFind community could use a bit more structure and the application of project management
• patches – member-submitted patches need to be incorporated to the code to a greater degree; a couple of us felt our contributions were not accepted
• authorities – a greater emphasis needs to be placed on integrating the profession’s good work done in regards to named authorities
• local customizations – a possible solution to the “straying” issue may be the implementation of some sort of local code base, something Blacklight apparent has
• “light” flavor – given the spectrum of programming skills available in libraries, some thought a VUFind “Light” may be in order
• repository – there is a need for a central place for the community to share local hacks, normalization routines, changes to the Solr indexer, etc.

In summary, the meeting was definitely a success. Discussion was thorough and focused. I believe we used our time wisely, and no one went away thinking it had been wasted. I do not think the group was representative of the whole VUFind community. We were more skilled than most. We agreed that VUFINd is not broken, but we did outline a number of ways it could be improved. We all agreed that the implementation of VUFind in our institutions represents a giant step forward compared to where we were at least a few years ago. oss++

Collection Policy Statement for the Catholic Portal

The purpose of the Catholic Research Portal is to provide global access to the wealth of research resources relating to the Catholic experience. Of primary interest are rare, unique and uncommon Catholic research materials. Because these resources are often uncataloged and little known outside their institutional repositories, the Portal seeks to encourage broad participation and to provide support to libraries, archives, and other institutions that wish to participate in this project but lack the resources to do so. The Portal will ultimately facilitate and assist researchers and students in identifying Catholic research resources and make Catholic scholarship more productive. In doing so, the Catholic Research Portal will contribute substantially to the generation of new knowledge.

The Portal will be the modern day bibliography of research resources providing access through a number of approaches, including author, title, subject, keyword, format, and holding institution. Resources will remain under the care of the owning institution. The Portal will identify the owning institution for non-digital resources and, where the resources exist in a digital format, it will link directly to the digital resource. Using international standards, the Portal will collect metadata from participating special and archival collections.

The Catholic Research Resources Alliance has identified twelve collecting themes:

• Catholic education
• Catholic intellectual life
• Catholic literary figures
• Catholic liturgy and devotion
• Catholic missions
• Catholic social action
• Diocesan collections, including papers of Bishops
• Men’s religious orders
• Peace-building
• Religion and citizenship
• Vatican II
• Women’s religious orders

These themes are intended to encourage the consideration and classification of institutional resources which may be suitable for the Portal. It is expected that the Portal will feature an initial emphasis on the above-named topics and that they will produce an early “critical mass” of research content collections. All contributors’ collections will be accepted for inclusion, however, provided that they are relevant to the study of Catholicism and can be deemed rare, unique or uncommon. All formats, including manuscripts, books, ephemera, photographs, and artifacts which meet these criteria, are of interest to the Portal.

This effort is being sponsored by the Catholic Research Resources Alliance (CRRA). Member institutions currently include: Boston College, The Catholic University of America, Georgetown University, Loyola University of Chicago, Marquette University, University of Notre Dame, St. Catherine University, St. Edwards University, University of San Diego, Seton Hall University, and Villanova University. As the Portal develops and expands, all Catholic colleges, universities, seminaries and archives in North America will be welcome to participate in this effort. Non-Catholic institutions with holdings of Catholic interest will also be welcome to contribute records.

Drafted by the Collections Committee of the CRRA, September 2009.

Approved by the CRRA Board of Directors on November 19, 2009.

View CRRA Members in a larger map

Step #1 – Make a list

Librarians love lists, and I am no exception. The process begins with a list (databases) of CRRA members who have MARC metadata to share. Each item in the list includes the following fields:

1. code – a unique three-letter identifier
2. institution – the name of the CRRA member
3. library – the name of the member’s library
4. URL – the location of their member’s MARC records

Right now, the name of this list is libraries.db. It is created by hand.

Step #2 – Harvest

The second step is to harvest content from each member library. This is done by looping through the list, extracting the URLs, and copying the remote MARC data sets to a local file system. This process is done with a script called harvest.pl.

Step #3 – Update

Because each record in the underlying Solr index must have a unique identifier, it is necessary for me to make each 001 value in each MARC record unique. To do this I loop through each of the harvested MARC records and prepend the three-letter institution code to each 001 field. This is done with a script called add-code.pl.

Step #4 – Re-index

The last step is to re-index the MARC records making sure the Solr index is as current as possible. This is done with a script called re-index.pl. It is the most complicated. This process is done by, again, looping through the database of CRRA members reading the institution code. The script then deletes all of the records from the index whose identifier begins with the institutional code. (“Thanks WebService::Solr!”). Each of the records from each of the institutions’ metadata files are then feed to Solr. Using this re-indexing process it is not necessary for me to manage overlays, duplicates, or deleted records. The whole index is wiped clean and refreshed anew.

VUFind

The whole process works pretty well, and because the Catholic Portal is based on VUFind, the whole process is extraordinarily flexible. VUFind cares not about the ingestion process allowing me to handle it in the manner I feel most useful. Let’s hear it for open source software!

Pat Lawton and I created this (updated) outline — a recipe — for getting CRRA member metadata records into the “Catholic Portal.” It is also available as a PDF document designed for printing.

1. Identify specialists – It takes many people with many skills to extract content for the Portal. It requires bibliographers (subject specialists) who know which materials located in your local institution fit the scope of the project. It requires catalogers (metadata specialists) who know how the local materials are described. It requires systems librarians (database administrators) who can extract metadata records from the underlying system(s).
2. Have a meeting – Bring together all the specialists from Step #1 and discuss Steps #3 through #10.
3. Understand the scope of the Portal – This is akin to understanding the purpose of the Portal, who is its intended audience, and what is its collection policy. In short, the Portal is intended to contain rare, unique, and/or uncommon materials, in all formats, useful for scholarly Catholic research.
4. Identify your resources and collections – List the resources and collections in your institution which fall into the scope of the Portal. Examples might include manuscripts, rare books, digitized images, sound recordings, the papers of famous individuals, the archives of leading organizations, pamphlets, newspapers, etc. This work will probably be led by bibliographers.
5. Articulate how your resources and collections are described – For each of your resources and collections identified in Step #4, determine which ones have metadata and which ones don’t. For those items which do have metadata, how they are denoted in your various computer systems? Are they all in a particular call number range? Do they comprise the totality of items in your “special collections” department? Are they all of the things encoded as EAD files? Do they all have some specific local note in your library catalog? Are they all saved in a particular local spreadsheet or database? etc. This work will probably be led by catalogers.
6. Flag records as “CRRA” – Once you have identified records appropriate for inclusion in the Portal, specifically denote them as such. For example, if your records are in MARC, then insert something like “crra” into a local note such as 590 subfield a. If your records are EAD files, you may want to insert “crra” into the <notestmt> within the <filedesc> element. This may be the work of both catalogers and systems librarians.
7. Validate records – Each and every record destined for the Portal must have three metadata characteristics. First, they must have a unique identifier. For MARC records this is the 001 field. For EAD files, this is a <unitid> element inside the <did> element. These unique identifiers are used by the Portal software as database keys.

   Second, each record must have some sort of descriptive title element. For MARC records this is usually 245 subfield a. For EAD files this is usually the <unititle> element inside the <did> element. These descriptive title elements provide a means for searching and put the object in context for the patron.

   Finally, every record must include some sort of location code or address pointing to the described object. For MARC records, this is often a call number in 099 or a URL in 856. For EAD files, this may be anything from a <note> denoting the postal address of your institution placed in the <did> element to URLs inserted into <extref> elements within <physloc> elements inside <did> elements. This may be the work of both catalogers and systems librarians.

   Please refer to the “CRRA Metadata Guidelines” for further guidance on requirements and best practices for maximizing the discoverability of your metadata records in the Portal.

8. Extract metadata records – Run a report against your computer system searching for all the records denoted by Step #6. Save the output to one or more files on a Web server, and tell us at Notre Dame the resulting URL. This process makes your metadata available for harvesting. For example, if your metadata records are in MARC, then query your integrated library system for “crra” in field 590 subfield a, and save the result as a single file of MARC records to an HTTP file system. If your metadata is stored as EAD, then find all the Portal-related EAD files and save them in a Web-accessible directory. In both cases, make sure your exported data is character encoded as UTF-8 and not MARC-8. This is the work of systems librarians.
9. Create a workflow – To ensure your records are continually added to the Portal it is necessary to repeat this process on a regular basis. For example, as new items are selected or come into your institution, bibliographers will need to immediately denote items destined for the Portal. You may do this by adding a special note to the acquisitions record. As the acquisitions are completed, the cataloger will need to immediately update the record(s) with “crra” flags. The systems librarian will need to extract the metadata on a regular basis and may consider writing a script that runs every night at midnight.
10. Repeat – This sort of work is never done. Go to Step #3 about twice a year, and go to Step #1 about once a year.

Finally, this “recipe,” like any good recipe, is only an outline of what needs to be done. There will surely be variations along the way, but based on our experience, this outline represents a good way to get started.

NTS (“note to self”), the account creation process did not work until I changed the value of RewriteBase in my httpd.conf file from /vufind to /.

For better or for worse, people increasingly rank their likes or dislikes of a website based on it’s graphic design. Certainly, usability, functionality, and completeness are very important qualities but so are aesthetics.

To create aesthetically pleasing websites a person today requires two specialized skills: 1) a formal understanding of graphic design (color, layout, typography, etc.), and 2) the ability manifest graphic design using HTML and cascading style sheet (CSS) technology. The former is embodied as an artistic flare. The later is akin to the painter’s brush or the bluesman’s guitar.

Alas, I do not have the necessary skills. After all, I think working both in terminal mode as well as at the command line are beautiful.

“sandbox” implementation

public interface

So, along with bibliographers who identify materials for the Catholic Portal, and along with catalogers who know how it has been organized, and along with systems librarians who know how to extract our metadata from our catalogs, we need competent Web designers.

Do you know of anybody in your institution who can help with this work? Like the other tasks surrounding the Portal, the work is ongoing but not constant. The short term goal is to give our current “sandbox” implementation of VUFind the look & feel of our “public” interface. Once completed we will be able to move the “sandbox” to production and dramatically increase the functionality of our interface.

At the present time you won’t find very much there except our indexed metadata records — about 60,000 of them. The next steps are to edit some of the underlying configurations to enable bits of functionality (call number displays, cover art, user reviews, etc.) The bigger issues to be resolved include: 1) giving our implementation the CRRA look & feel, and 2) indexing did-level data found in EAD files. More on these things later.

• CLIR grant announcements
• Focus group data analysis
• The CRRA website
• Kim Kelley
• Steve Connaghan
• Upcoming events

Enjoy and Happy New Year!

CRRA Update December 2009

December was a month of “hidden labor.”  We made good progress on a number of fronts (in addition to cookies, eggnog, and snow-shoveling), but most of our activity was behind the scenes. We made great strides on the following, yet the fruits of our labors may be made manifest only in the weeks and months to come.

CLIR grant announcements were sent to a number of venues. To date, the announcement has appeared in: Library Journal <http://www.libraryjournal.com/article/CA6709666.html>, ACCU’s Winter 2009 Newsletter <http://www.accunet.org/files/public/Winter_09_newsletter.pdf>, Marquette University’s website, and St. Catherine University website.

Look for future announcements in: Catholic Library World’s News Notes (March 2010), Midwest Archives (MAC) Newsletter (April 2010), ATLA-RC list and Newsletter, H-Catholic list, Cushwa Center Newsletter, Archival Outlook, and the Mid-Atlantic Archivist.

Focus group data analysis has begun. To date, four institutional members have conducted focus groups at their institutions, with a total of 34 participants. We hope to have data from at least two more institutions between now and Feb. 1.   The data so far has been rich with several discrete, emergent themes.

The CRRA website is being updated to include new members, participants, and an updated directory of library, seminary and archive directors at Catholic institutions in the United States and Canada.

Records from the VuFind test site have been moved into the CRRA website, bringing us to a total of 33,000+ records and counting.

Kim Kelley, Associate Provost for University Libraries and Dean, School of Library and Information Science, at the Catholic University of America announced that she would be leaving the CRRA Board of Directors, effective Jan. 1, 2010. Kim has been a great supporter of the CRRA and her calm and thoughtful input will be deeply missed.  We look forward to engaging Kim in future CRRA activities and wish her well in her new Wisconsin adventures!

Steve Connaghan, Acting Director of Libraries, will succeed Kim on the CRRA Board of Directors.  A warm welcome to Steve!

Upcoming events

ACCU 2010 Annual Meeting: Strategic Issues for Catholic Higher Education will be held at the Mandarin Oriental Hotel, Washington, DC, January 30 to February 1, 2010. For more information, visit: https://www.accunet.org/i4a/ams/publicLogin.cfm.

All CRRA events and events of possible interest to members are posted to the CRRA calendar, available at http://tiny.cc/Calendar798 and also accessible to members (login and password required) from the Admin area of the CRRA website.

CRRA Update is an electronic newsletter distributed via email the first Friday of each month to provide members with an update on CRRA activities. The Update is also posted to the CRRA Blog. Please contact us at 574.631.1324 or email plawton@nd.edu with your questions, comments, or news to share.

Happy New Year to all!

-    CRRA member news

-    Focus groups report

-    Progress report on our strategic plan

-    The CRRA Collection Policy Statement

-    News about the CLIR grant award

-    Upcoming events

I would like to include more news items by and about members in future updates, so if you have news items to share, please pass them on.

Pat Lawton
Digital Project Librarian
plawton@nd.edu

CRRA Update

NOVEMBER 2009

Focus Groups at Member Institutions
One of the goals from our annual meeting in July 2009 was to hold focus groups at all member institutions, to gather user feedback concerning the content of the portal.  In September, Notre Dame hosted four focus groups and documented their processes, providing a model for other institutions. As of November 30, Marquette University Libraries and Seton Hall University Libraries have completed their focus groups. With uniformity in approach and questions, we can realize a sizable dataset that promises to yield rich data from future portal users coast to coast.   Thank you to all who are in the planning stages or have completed their focus group interviews!

Once all member institutions’ focus group data is reported, data across institutions will be analyzed, summarized, and disseminated. The cross-institutional data gathered from the focus groups will serve to build a portal that aligns with users’ wants and needs.

Ed Starkey
CRRA Board Member, Ed Starkey, University Librarian of the University of San Diego,
announced he will be retiring the end of December, 2009.

Ed, together with Charlotte Ames, retired Notre Dame Catholic Studies Librarian, was the inspiring figure for the entire CRRA project.  Ed’s vision was to bring Catholic librarians and libraries together to advance Catholic scholarship and Catholic intellectual life around the globe; to encompass and create a place where scholars might find all materials related to the Catholic intellectual tradition in order to realize new knowledge.

Ed provided the inspiration, encouragement and knowledge that was so essential to the founding of the CRRA. We wish much joy to Ed in his retirement days.  He will be sorely missed and we will continue our work to realize the vision he has inspired.

Collection Policy Statement Adopted by CRRA Board of Directors
At the November 19 board meeting, Bob O’Neill, Chair of the Collections Committee (Boston College), moved to adopt the proposed collection policy statment with a change from “rare, unique and infrequently held” to “rare, unique and uncommon.”  The change was approved.

The newly adopted Collection Policy Statement is archived and available to CRRA members from the Admin area of the website and from this link:http://tiny.cc/Collection. (If you need a reminder of the password and login, send Pat an email at plawton@nd.edu.

New Portal Records Now Accessible at Catholicresearch.net
We are happy to announce that the goal to incorporate the VuFind interface and pilot project records  into our home site has been met, and well ahead of schedule!  At http://www.catholicresearch.net you will find the records that were formerly available only at the VuFind test site.

We welcome your comments about this update.  Pages are under construction to make them more inviting, more easily navigable, up to date, and informative.  Please send your comments or suggestions to Pat at plawton@nd.edu or Eric at emorgan@nd.edu.

Progress Report on the Strategic Plan
Thanks to our members’ hard work, we have made great progress on the objectives charted at our July 2009 Loyola meeting.  This link http://tiny.cc/LoyolaUpdate takes you to the “Loyola Plan” document with completed tasks highlighted.  We are well on track and even ahead of schedule in some areas.

• The pilot project to ingest 20,000+ records has been surpassed.
• Focus groups have been completed at Notre Dame, Marquette, and Seton Hall.  Four other institutional members have plans to host focus groups.
• Three CRRA member institutions collaborated to submit a successful CLIR grant proposal.
• We are working with new members to add their content to the portal and orient them to the CRRA.
• The board is working to identify new members and to develop policies and procedures to support a growing membership.

At the November 19, 2009 board meeting, Tom Leonhardt proposed that we establish CRRA Policies and Procedures to address a variety of issues from committee member roles and expectations to contributor expectations, rights and limitations, etc.  The board enthusiastically approved.  CRRA Committees will be involved in the drafting of policies and procedures.  We welcome your input!

In summary, we have made great strides in the four short months since our July meeting.  This is amazing progress for a largely volunteer organization.  Our congratulations and heartfelt thanks to all!

If there are aspects of the CRRA’s work in which you would like to have greater involvement, please contact a CRRA board member or Pat Lawton.

CLIR Grant Awarded to CRRA Member Institutions
CLIR Grant Awarded for the Catholic Social Action Access Project:  A Collaborative Project of Marquette University, Catholic University of America, St. Catherine University and the Catholic Research Resources Alliance

With funding by the The Andrew W. Mellon Foundation, The Council on Library and Information Resources (CLIR) has awarded Marquette University, Catholic University of America, St. Catherine University and the Catholic Research Resources Alliance (CRRA) a Cataloging Hidden Special Collections and Archives grant in the amount of of $149,964.  The grant will support the Catholic Social Action Access Project, one of only 14 selected from a total of 91 applications. More information about the 2009 CLIR awards is here: http://www.clir.org/hiddencollections/awards/index2009.html.

This collaborative project brings together three significant collections documenting US Catholic social action in the 20th century. St. Catherine University’s Ade Bethune Collection documents the career of a world-renowned liturgical artist and social activist who helped found the Church Community Housing Corporation to develop affordable housing in Newport County. The Catholic Charities, DC records (CCDC), held by the Catholic University of America document the CCDC’s leadership and support of progressive social legislation. Marquette University’s Dorothy Day-Catholic Worker Collection includes audio recordings of the voices of most influential Catholic social activists of the 20th century.

Jean Zanoni, Associate Dean of Marquette Libraries and Matt Blessing, Head of Special Collections and Archives, are Co-Principal Investigators.  Project collaborators are CRRA members and descriptions of project materials will be collocated within the CRRA’s Catholic Portal.

The award is effective January 1, 2010, and project activities will be completed by December 31, 2011.

The grant announcement above will be published in the Winter issue of ACCU’s newsletter. Other planned venues for the announcement include:   CLA’s Catholic Library World, the ATLA-RC listserv and newsletter; ACRL’s C&RL News; the SAA Newsletter; Midwest Archives Conference newsletter; and Mid-Atlantic Archivist newsletter.

We encourage you to spread the news among your local communities.  If you would like more information, contact Jean Zanoni, Matt Blessing, or Pat Lawton.  If you send an announcement about the grant, please let us know.  For purposes of grant reporting, we would like to track to whom announcements are made.

Thank you and congratulations to all grant participants!

Mark your calendars  …
Kevin Cawley (Notre Dame) will present “Investigating Change through the Catholic Research Resources Alliance” as a panelist at the Conference on the History of Women Religious, to be held at the University of Scranton June 27-30, 2010.

Of possible interest:  The American Catholic Historical Association will hold their ninetieth annual meeting from January 7 through January 10, 2010 at the ManchesterGrand Hyatt San Diego.  More information about the meeting is here: http://research.cua.edu/acha/meetings.cfm.

All CRRA events and events of possible interest to members are posted to the CRRA calendar, available at http://tiny.cc/Calendar798 and also accessible from the Admin area of the website.

-News about CRRA members
-An update on the focus groups at member institutions
- Notes from the Digital Access Committee meeting
- Future happenings
- Suggested readings

I would like to include more news items by and about members in future updates, so if you have news items to share, please pass them on.

Happy reading!

Pat Lawton
CRRA Digital Project Librarian
plawton@nd.edu

CRRA Update
October 2009

Member News

CRRA welcomes Loyola University Chicago and St. Catherine University!
The CRRA Board of Directors welcomed Loyola University Chicago and St. Catherine University as the newest members and participants in the CRRA. Bob Seal, Dean of Libraries, Loyola and Carol P. Johnson, Director of Libraries, Media Services, and Archives at St. Catherine accepted with pleasure the invitation to join the CRRA this year (2009/10).

Carol Johnson has been actively involved with the CRRA through her service on the Collections Committee and in partnering with us in authoring the CLIR grant proposal. Deborah Kloiber, Curator of the Ade Bethune Collection, played a significant role in the planning and drafting of the CLIR grant and has expressed her continued interest in CRRA activities. St. Kate’s houses a number of important print and digital collections. One highlight is their collection of the papers and art works of the liturgical artist, Ade Bethune. For more about St. Kate’s library and collections, see http://library.stkate.edu.
Bob Seal, Director of Loyola University Libraries, graciously hosted our July meeting. We enjoyed glorious Lake Michigan views and a tour of the newly completed Klarchek Information Commons. Kathy Young, University Archivist and Curator of Rare Books, participated in our July meeting and shared news of her work with a Chicago-based collaboration among libraries, universities, and archives, the Black Metropolis Research Consortium (BMRC). Loyola has a number of resources of interest to the CRRA, including the Jesuitica collection and the Women & Leadership Archives (WLA). Many of the WLA collections relate to Catholic women leaders; it will be a great resource for the CRRA. Dr. Beth Myers, Loyola’s Director for Women & Leadership Archives, will also participate in CRRA activities. For more about the library and collections at Loyola see http://libraries.luc.edu.

Ed Starkey Announces Retirement
CRRA Board Member, Ed Starkey, University Librarian of the University of San Diego, announced that he will be retiring the end of December, 2009. Ed provided the inspiration, encouragement and knowledge that was so essential to the founding of the CRRA. Ed will be sorely missed!

ARL/CNI Forum on Special Collections in Washington DC
Bob O’Neill (Boston), Jennifer Younger (Notre Dame), John Buchtel (Georgetown), and Pat Lawton attended the ARL/CNI Forum “An Age of Discovery: Distinctive Collections in the Digital Age” in Washington, DC. The forum included a wide array of speakers, including such notables as G. Wayne Clough, Secretary of the Smithsonian Institution; Don Waters, Program Officer for Scholarly Communication, The Andrew W. Mellon Foundation and Ian E. Wilson, Librarian and Archivist of Canada Emeritus, President of the International Council on Archives, and Strategic Advisor, University of Waterloo. Two themes dominated the forum: special collection are for use, and “[The] Online [Environment] enables the past to be put back together once again” (Ian Wilson). For more on the forum, see the Proceedings page with selected presentations.

CRRA News

Focus Groups to be Held at Member Institutions in November and December
Member institutions including Georgetown, Catholic, Villanova, Marquette and Seton Hall University are preparing for focus groups to be held in November/December. From this data, aggregate summaries of findings will be created and shared with all institutions. Focus group data will provide a systematic overview from potential portal users and may serve as a guide for future portal directions. Thank you to all who are gathering data! We look forward to your findings.

If you have yet to begin plans for focus groups at your institution, it’s not too late! To begin the process, call Pat Lawton and/or consult the documents from the focus groups held at Notre Dame. Documents include steps in planning, sample email announcements for focus group participants, key questions, and moderator notes. All documents are posted to the Admin area of the CRRA website.

Digital Access Committee (DAC) Meeting
The Digital Access Committee (DAC), chaired by Tom Leonhardt, met on October 12. Members identified the following as important next steps: create a data input form enabling members with no metadata records for particular collections to easily submit collection-level descriptions to the portal; identify existing common errors in the portal and gather suggestions for further improvements.

Meeting minutes are posted to the Portal admin area.

Suggested Reading
Two reports of possible interest to CRRA members were released this month.

SPARC Report on Income Models for Open Access
SPARC (the Scholarly Publishing and Academic Resources Coalition) examines the issue of sustainability for current and prospective open-access publishers in “Income models for Open Access: An overview of current practice,” by Raym Crow (http://www.arl.org/sparc/publisher/incomemodels/imguide.shtml). Although the report examines online journals in particular, by way of extrapolation, it is useful in thinking about models of sustainability for the portal. Read with an eye toward the CRRA, it presents a host of possible revenue streams for the portal, including sponsors, advertisers, and value-added services.

JISC Final Report on: “Digitisation of Special Collections: mapping, assessment, prioritization”
http://www.jisc.ac.uk/media/documents/programmes/digitisation/discmap_final_report_211009_final.pdf
This report may be useful in planning the process to identify collections of interest and prioritize the need for digitization. The methodologies look rather robust and the study provides a useful framework for identifying variables of interest to CRRA such as Curatorial environment, Age of collections, and Subject area. Also note that they surveyed intermediaries (librarians, archivists, etc.) as well as end users.

Interesting results include:
• Articulation of a method of user-driven prioritization
• Identification and articulation of the need to have collections accessible in one place (a comprehensive collection description and finding utility to support resource discovery)
• Recommendation that a “standard approach to collection description be adopted where the relationships between a collection and its ‘super-collections’ and ‘sub-collections’ are clearly presented”
• Finding that high quality metadata is essential for discovery

Looking ahead …
Eric Lease Morgan will be in Chicago November 14-16 for the Digital Humanities and Computer Science Conference.

Sister Jean Bostley of the Catholic Library Association and Malachy McCarthy of the Claretian Missionaries Archives, USA, have invited Eric and Pat to participate in a panel discussion at ALA in June 2010. The session is entitled Planning, Building, and Using Religious Archive Digital Sites: The USC Internet Mission Photography Archive and the Catholic Research Resources Alliance. Date and time to be announced.

Please send your news items for future CRRA Updates to Pat Lawton at plawton@nd.edu.

A Happy and Blessed Thanksgiving to all!

Please send suggestions for future updates to Pat Lawton at plawton@nd.edu. Thank you!

1. Identify specialists – It takes many people with many skills to extract content for the Portal. It requires bibliographers (subject specialists) who know which materials located in their local library fit the scope of the project. It requires catalogers (metadata specialists) who know how the local materials are described. It requires systems librarians (database administrators) who can extract metadata records from underlying system(s).

2. Have a meeting – Bring all the specialists together to discuss Steps #3-7.

3. Understand the scope of the Portal – This is akin to understanding the purpose of the Portal, who is its intended audience, and what is its collection policy. In short, the Portal is intended to contain rare, unique, and/or infrequently held materials useful for scholarly Catholic research.

4. Identify resources/collections – List the resources/collections in the library which fall into the scope of the Portal. Examples might include rare books & manuscripts, digitized images, sound recordings, the papers of famous individuals, the archives of leading organizations, pamphlets, newspapers, etc.

5. Articulate how identified resources/collections are described- For each of the resources/collections identified in Step #4 determine which ones have metadata and which ones don’t.

   For those items which do have metadata, list how items in the collection are denoted in your various computer systems. Are they all in a particular call number range? Are they the totality of items in your “special collections” department and/or encoded as EAD files? Have they all been cataloged with a local note in your integrated library system (ILS)? Are they all or a subset of items saved in a local spreadsheet or database? Etc.

   For “extra credit,” discuss ways the items which don’t have metadata can get some in the future.

6. Extract metadata records – Given the things discussed in Step #5, collect the metadata records from your system(s). For example, some sort of search might be done to extract all identified MARC records from an ILS. All EAD files describing materials apropos to the Portal might be saved to a directory. A report might be written against a database to create a tab-delimited text file. Etc.

   There are three things to remember when extracting the metadata. The first is something we are calling “MARC-ability“. For better or for worse, VuFind only accepts MARC records as input, and consequently, all metadata received for ingestion must be translated into MARC. Thus, “real” MARC records are easily accepted, but “tagged” MARC records are not. EAD files can be cross-walked to MARC and thus easily accepted. Some sort of delimited (CSV, tab, etc.) file works well because they are easily parsed. HTML files are poorly structured making any mapping process very difficult. The same goes for any word-processed file (Word, WordPefect, etc.). MARC, any flavor of XML, and delimited files work best.

   Second, each metadata record requires a number of specific fields. Each record requires a unique identifier. For MARC records this is a value in the 001 field. For EAD files, this is denoted by the identifier attribute in the eadid element. Next, each record requires a pointer to where the described item can be found. Generally speaking, these are either call numbers or URLs saved in the appropriate fields. The last requirement is not really a field but formatting. All data must be saved using the UTF-8 character encoding. Any other encodings (like MARC-8) are not readable. If not saved in plain ASCII or UTF-8, then diacritics display incorrectly and confuse the VUFind indexer.

   The third and final thing to remember is in regards to four levels of data integrity. The first level speaks to the way your data is structured. For XML files this means they are well-formed. For MARC records, it means the leader is 24 bytes long, the value of the first 5 characters of the leader equals the length of each record, fields are delimited with the appropriate ASCII characters, etc. The second level speaks to validity. For XML files it means the data conforms to a DTD or schema. For MARC records it means authors are in 1xx fields, the title is in 245, notes are in 5xx, etc. The third level of integrity is correctness. “To what degree is the value in 245 the title of the item? To what degree are the URLs not broken? Etc.” The last level of integrity is in regards to completeness. A metadata record’s completeness is directly proportional to its findabilty. The first two levels of integrity can be validated through computer technology. The second two levels are the domain of librarianship.

7. Send records to Notre Dame – After the records have been exported, email them to emorgan@nd.edu, and they will be ingested into VuFind.

You’re done! We will notify you via email when your records are available for viewing, giving you the opportunity to validate the process and examine the fruits of your labors.

Finally, this “recipe,” like any good recipe, is only an outline of what needs to be done. There will surely be variations along the way, but based on our experience, this outline represents a good way to get started.

Contact information

If you have questions along the way, don’t hesitate to contact Eric or Pat:
Eric Lease Morgan emorgan@nd.edu 574.631.8604
Pat Lawton plawton@nd.edu 574.631.1324

Together, the CRRA, Catholic University of America, Marquette University, and St. Catherine University libraries submitted a pre-proposal for CLIR’s “Cataloging Special Hidden Collections and Archives” grant program. Our pre-proposal was accepted! We may now submit a final proposal, due Sept. 4.

The focus of this funding opportunity is on cataloging. Proposed collections (or portions of) for cataloging include:  Ade Bethune (St. Catherine), Dorothy Day (Marquette), and Catholic Charities USA (Catholic). One of the innovative aspects of our proposal is that the cataloged collections will all be accessible via the Catholic Portal.

For further information about the program, please visit CLIR’s website
at http://www.clir.org/hiddencollections/index.html.

We still have a ways to go but we made it through the first round.

Congratulations to all collaborators!

We are a collaborative effort initiated by eight  Catholic colleges and universities to share their resources electronically with librarians, archivists, researchers, scholars, and the general public interested in the Catholic experience.

For more about the CRRA, see: http://www.catholicresearch.net.

The CRRA is currently engaged in building The Catholic Research Resources Portal.  The Catholic Portal provides access to rare, unique or infrequently held materials in academic libraries and seminaries’ special collections and archives. By electronically bringing together access to resources in many collections, the Portal will create easy, effective and global discovery of Catholic research resources.

We are currently in the midst of a pilot project to add member content into the VuFind interface.  The test database is here: http://vufind.library.nd.edu/.  Have a look around, and let us know what you think.

If you have any questions or comments about the CRRA, post them here or send them to me at plawton@nd.edu.

Welcome!

Pat Lawton
Digital Project Librarian for the CRRA