Catholic Research Resources Alliance (CRRA) Blog

Just less than 1,100 records from the Philadelphia Archdiocesan Historical Research Center (PAHRC) have been added to the “Portal” — http://bit.ly/uG92RG

Twenty-nine records from the Archives at the University of Dayton added to the “Catholic Portal” — http://bit.ly/weVl8h

Using VuFind’s inherent ability to index OAI metadata, I have successfully been able to index metadata coming from a PastPerfect implementation.

Starting somewhere near version 1.2, VuFind supports the indexing of arbitrary metadata types. Content from OAI repositories was the original example. Later, I figured out how to index EAD files. This was a break through for the “Portal”. Give credit to open source software.

With the addition of the Philadelphia Archdiocesan Historical Research Center (PAHRC) into the Catholic Research Resources Alliance, a new metdata format needed to be accepted — metadata other than EAD or MARC. PAHRC uses “cataloging” software called PastPerfect. From what I can tell, it is a sophisticated FoxPro/Microsoft Access database application. It provides the means for institutions to do data entry, and have their holdings searched, and ultimately displayed on the Web.

PastPerfect can export its metadata in a form of Dublin Core. After working closely with Shawn Weldon, Faith Charlton (both of PAHRC), and Brian Gomez (Past Perfect, Inc), the metadata exported by PAHRC was tweaked to be less ambiguous and more accurate. Once this was done I was able to harvest the metadata, parse it into something usable by VuFind’s Solr indexer, and make it available through the Portal. I did this with a script called pastperfect-index.pl. The result is a set of searchable records from PAHRC.

My current implementation is specific to PAHRC, and when other PastPerfect libraries/archives come on board, it will not be too difficult to abstract my implementation to support other institutions. That work is left to the future, when and if it occurs.

Fun with open source software!

There is some concern about duplicate records in the “Catholic Portal”, and this posting introduces the topic to a wider audience.

The “Catholic Portal” is intended to contain links to and content of a rare and infrequently held nature. Every once in a while search results return duplicate records. For example, yesterday, it was brought to our attention that there are five records with the title Life Of Mrs. Eliza A. Seton. On one hand, few if any of these records are duplicates because between the five of them they are held by two different institutions. And each institution owns multiple editions. In the sense of a “catalog”, this is perfectly acceptable, if not expected. On the other hand, the Portal is not a catalog but rather an index, and each of the five items are really a variation on a theme. Should these records be merged?

Demian Katz shared with me and the Portal’s Digital Access Committee a query that can be applied the Portal’s underlying Solr index, here, with carriage returns added for readability:

http://localhost:8080/solr/biblio/select/?
q=*%3A*&rows=0&start=0&facet=true&facet.mincount=2&
facet.limit=-1&facet.field=oclc_num&facet.field=isbn

The result of this query is a list of OCLC and ISBN numbers which occur in the index at least two times. According to the result, which only matches on the OCLC or ISBN keys, there are no records in the index appearing more than three times. Furthermore, there are about 1,100 duplicated OCLC numbers and about 300 duplicated ISBN numbers. Considering the total number of records (93,000) in the index, this represents a total duplication rate of approximately 1.5%. Is this value too high?

In an ideal world, there would be no duplicate records and/or duplicates would be merged into a single record. Unfortunately, the definition of “duplicate” is ambiguous, and a process for eliminating duplicates has not been implemented. To a Walt Witman scholar, the difference between various editions of The Leaves Of Grass is definitely significant. Thus, sometimes the differences in editions is very important. Other times and for other people, this is not always so important. In an ideal world, there would be no duplicates and a single record would warrant a de-duplication process, but the expense of de-duplicating that single record may be very high, especially if there is no de-duplication process in place. How many records — or what percentage of records — warrants a de-duplication process, especially considering the other things that have been set as priorities for the Portal? Honestly, I don’t know the answer.

Marta Deyrup and Martha Loesch, catalogers at (CRRA instution) Seton Hall University, and Pat Lawton, digital projects librarian for the CRRA, have released the results of their Survey of Digitized Rare Catholica held by Catholic universities, colleges, seminaries and archives in the U.S. and Canada. You may view the Summary Report of Results and the results data.

I was recently told to delete thousands upon thousands of records from the “Catholic Portal”, and through the magic of the Solr’s Web-based API and a full-featured HTTP client I was able to do this surgery with laser beam accuracy.

Specifically, I needed to delete all of the records in the Portal from the University of Notre Dame Archives because the Archives wanted to totally replace what finding aids were available. This meant deleting more than a 100,000 records from the underlying index. After a bit of investigation, I learned that at the following one-liner from the command line would do the trick:

curl http://localhost:8080/solr/biblio/update?commit=true -H "Content-Type: text/xml" --data-binary '<delete><query>id:unaead_*</query></delete>'

In short, curl is a command-line HTTP client. It is being told to first connect to the local host on port 8080. It is then told to find all the records matching the query “id:unaead_*” and delete them from the index named biblio. Once that is done, the underlying index is expected to commit the changes. Deleting these records took about ten minutes. I was then able to use my previously created scripts to harvest, validate, transform, and index the Archives’ content painlessly.

It is a pleasure when things work in the way they were designed! Now if I could only get my local indexing process to work faster.

On Wednesday, October 12, 2011 I had the opportunity to attend and present at the second annual VuStuff meeting held at Falvey Library, Villanova University (Philadelphia). This posting documents my experience there, but in a nutshell, this small and intimate meeting provided a venue for interesting discussion on the topic of modern librarianship.

Liberty Bell

cheese steak sandwich

Joe Lucia (Villanova University) initialized the meeting and set the stage by recommending a book called The Googlization of Everything. It advocates the creation of an open knowledge commons similar to the ones at the root of the fledgling Digital Public Library of America. To paraphrase his remarks, “Everything we do in our shop here embrases the open knowledge commons concept… Libraries are not just purveyors of content, but also creators of content — The New Resource Sharing. We [librarians] can become agents of information creation.”

The first presentation was given by Amy Baker Williams (University of Pittsburgh), and she described her process for conserving the maps of local coal mines. In the Pittsburgh (Pennsylvania) area there are many coal mines dating back as far as 1750. Some of the oldest maps of the mines date from 1850. A few years ago some miners were trapped in a mine, and if maps of the mines had been easily accessible, then rescue efforts would have been simplified. Since then concerted efforts have been made to preserve, digitize, and make accessible as many of these coal mining maps as possible in order to prevent similar accidents from happening in the future. I found the process used to flatten the maps to be the most interesting. Basically they are re-hydrated and unrolled. Moving the maps from the conservation lab to the scanning location was also interesting because, ironically, the maps are rolled up again for transportation as well as long-term storage. For more detail, see the website.

My presentation was next, and I shared with the audience how the Catholic Research Resources Alliance (CRRA) is using VuFind to implement the “Catholic Portal”. I first described the mission and history of the CRRA. I then outlined the Portal’s technical architecture as well as the process I used to index EAD files. Finally, I described how text mining functions have been integrated into the Portal’s interface emphasizing the possibilities for libraries in general.

Falvey Library

mural

During lunch we broke up into groups, and I sat with the folks interested in the digital humanities. For the most part we went around the table sharing common war stories. Most of our initiatives where fledgling, but there was plenty of enthusiasm.

After lunch a sort of “unconference” session was facilitated by David Upsal (Villanova University). The discussion topic that made itself apparent was the challenge of the profession to serve both traditional librarianship as well as librarianship in the current environment. If my memory serves me correctly, some of the suggested solutions included more resources (people and money), permission to “play” with new technology, a redefinition of library purpose, and greater collaboration between different types of libraries (public, academic, etc.)

The next presentation was given by Eric Zino (LYRASIS) who described how LYRASIS has been working with the Sloan Foundation and the Internet Archive to facilitate the digitization of 20,000,000 pages of library content. Approximately 160 libraries have been participating in the project with LYRASIS. Subsidized by the Foundation, partipants package up their content and ship it to the Internet Archive. The content gets digitized, returned to the owning library, and the digital versions are made accessible at the Archive. From my perspective, this is exactly how any other library works with the Archive, except in this case LYRASIS does a bit of hand-holding during the process. Not all media is digitized by the Archive though. Some things, such as microfilm, are scanned by a different vendor — Creekside Digital.

The last presentation of the day was given by Bob Behary (Duquesne University), and he shared with the audience how Duquesne is digitizing a newspaper called the Pittsburgh Catholic. The project was initiated by a Catholic order called the Spiritans (the founding order of Duquesne University) with evangelism at its root. At first digitized versions of the newspaper were put on CDs and distributed. This has evolved over time, and now the content is housed in a ContentDM system. The collection has proven useful in a number of ways, including: local & regional church histories, literature allusions (such as Emily Dickinson), and United States history. Behary listed a number of key considerations for any digitization effort: 1) get administrative support, 2) make sure the project fits within the mission of the institution, 3) make sure to use sustainable technology, and 4) ensure knowledgable research advocates are a part of the process.

Vuee Award

Art Museum staircase

I believe the meeting was attended by fifty to seventy-five people. Most were from the immediate area, and it offered a easy opportunity for professional development. Kudos to the folks at Villanova for hosting the event. Just before the meeting concluded I was awarded the second annual “Vuee” for best presentation. It is a small shoebox-sized container in the shape of a book. I was very flattered. “Thank you very much!”

Colleagues,

The registration deadline for this CRRA/Duquesne sponsored event is this Friday, October 15, 2011.  We are pleased that many of you have already registered for the event and if you have thought about registering, please do so now.  There is no fee to register.

The event features Catholic scholars, archivists, and librarians gathering together to consider the state of Catholic scholarship and how we can act together to advance and enhance freely available global access and discovery of important Catholic resources. The event will take place at Duquesne University (Pittsburgh) on Nov. 9-10.

We encourage librarians, scholars, and archivists interested in learning more about opportunities to make scholarly resources accessible to join in and meet new friends and colleagues.

A full roster of events and registration information is available at http://bit.ly/Duquesne_Symposium .   The registration deadline is this Friday, October 15, 2011.

We hope that you will join us in what promises to be a stimulating and productive conversation about Catholic scholarly research and the ways in which librarians and archivists support this research.

On behalf of Duquesne University and the Catholic Research Resources Alliance (CRRA),
•         Jennifer Younger, chair, Board of Directors at younger.1@nd.edu
•         Laverna Saunders, University Librarian, Gumberg Library, Duquesne University at lsaunders@duq.edu
•         Pat Lawton, CRRA Digital Projects Librarian at plawton@nd.edu