Parsing a Bibliography 2007

Anonymous Monk has asked for the wisdom of the Perl Monks concerning the following question:

Dear Perl Monks

I have about 10,000 untagged bibliographic citations in a series of Word documents, with the citations falling into roughly three types (monograph, anthology, article). Here are some examples:

Binkley, Marian. Set Adrift: Stories of Castaways and Other Survivors. Toronto: University of Toronto Press, 2002.
Brooke, Frances. Edited by Laura Moss. The History of The Bonzo Dog Band. Ottawa: Tecumseh Press, 2001.
Keahey, Deborah and Deborah Schnitzer, eds. Crocodile Cooking : 14 Recipes for Baked Crocodile. Calgary: University of Calgary Press, 2003.
Korinek, Valerie J. “'The longest journey is a thousand miles': Doug Wilson and the Politicization of a Province, 1975-83.” Canadian Historical Review 84, 4 (December 2003): 517-50.
Mckay, Sherry. “'Urban Housekeeping' and Keeping the Modern House.” BC Studies 140 (Winter 2003/04): 11-40.
Kealey, Linda. “North America, 1750-1914.” In A Companion to Native History, eds. Teresa A. Meade and Merry E. Wiesner-Hanks. Blackwell Publishing, 2003.

I have been trying to adapt the perl code from the 2004 Perl Monks thread "parsing a bibliography", but I'm confusing myself and thought I should seek wisdom.

I am wondering if this problem is best approached as a series of filtering steps, to create one output file for each main citation type, or if there is some other way to approach it. I've done some perl programming, but not a lot. The untagged Word entries are all in alphabetical order, and are not sorted by citation type. If I can get a largish percentage of these citations into a BibTex state requiring cleanup, I will be both lucky and pleased. Any direction or assistance would be much appreciated.

Here are my example bibliography entries, and the manually created BibTex output that would seem to be suggested by the citations:

Monograph
---------------
Binkley, Marian. Set Adrift: Stories of Castaways and Other Survivors. Toronto: University of Toronto Press, 2002.

@book{binkley2002,
author={Binkley, Marian},
title={Set Adrift: Stories of Castaways and Other Survivors},
address={Toronto},
publisher={University of Toronto Press},
year={2002},
}

Monograph - with Editor
-----------------------------------
Brooke, Frances. Edited by Laura Moss. The History of The Bonzo Dog Band. Ottawa: Tecumseh Press, 2001.

@book{brooke2001,
author={Brooke, Frances},
title={The History of The Bonzo Dog Band},
editor={Laura Moss},
address={Ottawa},
publisher={Tecumseh Press},
year={2001},
}

Anthology
--------------
Keahey, Deborah and Deborah Schnitzer, eds. Crocodile Cooking : 14 Recipes for Baked Crocodile. Calgary: University of Calgary Press, 2003.

@book{keahey2003,
editor={Keahey, Deborah and Deborah Schnitzer},
title={Crocodile Cooking : 14 Recipes for Baked Crocodile},
address={Calgary},
publisher={University of Calgary Press},
year={2003},
}

Article
---------
Korinek, Valerie J. “'The longest journey is a thousand miles': Doug Wilson and the Politicization of a Province, 1975-83.” Canadian Historical Review 84, 4 (December 2003): 517-50.

@article{korinek2003,
author={Korinek, Valerie J.},
title={'The longest journey is a thousand miles': Doug Wilson and the Politicization of a Province, 1975-83.},
journal={Canadian Historical Review},
volume={84},
number={4},
month={December},
year={2003},
pages={517-50},
}

Article - issue and date, but no volume number
---------------------------------------------------------------------
Mckay, Sherry. “'Urban Housekeeping' and Keeping the Modern House.” BC Studies 140 (Winter 2003/04): 11-40.

@article{mckayk2003,
author={Mckay, Sherry},
title={'Urban Housekeeping' and Keeping the Modern House},
journal={BC Studies},
number={140},
note={Winter 2003/04},
year={2003},
pages={11-40},
}

Article in a Book (Anthology)
------------------------------------------
Kealey, Linda. “North America, 1750-1914.” In A Companion to Native History, eds. Teresa A. Meade and Merry E. Wiesner-Hanks. Blackwell Publishing, 2003.

@book{kealey2003,
author={Kealey, Linda},
editor={Teresa A. Meade and Merry E. Wiesner-Hanks},
title={A Companion to Native History},
address={Toronto},
publisher={Blackwell Publishing},
year={2003},
}

Does it seem that a giant list of citations falling mostly into these citation types would best be processed multiple times to filter out each type?

Is my meagre, yet unsuccessful attempt to adapt the 2004 monks example code a pursuit worthy of monkish assistance?

Comment on Parsing a Bibliography 2007

Replies are listed 'Best First'.
Re: Parsing a Bibliography 2007 by educated_foo (Vicar) on Apr 12, 2007 at 23:00 UTC
This is a really hard problem if you want high accuracy (I've done this for money before). You probably want to start by trying to separate the individual citations without parsing them. Then you can iteratively come up with a library of regexes to pull them apart by starting with a few obvious and conservative patterns, examining the errors and misses, and developing more patterns. If you want to get really fancy, you can disambiguate by e.g. recognizing known authors' names as part of the author list rather than the title. Eventually you'll just have to throw up your hands and say "good enough"...	[reply]
Re^2: Parsing a Bibliography 2007 by ptum (Priest) on Apr 12, 2007 at 23:41 UTC
I agree with this approach. Assuming there is no way to definitively lock down all possible distinct formats, you'll end up iteratively building regexen and parsing your citations until 'enough' of them are recognized and the remaining unparsed subset is sufficiently small. Do you anticipate this as a one-time effort, or will you be regularly handling new citation forms?	[reply]
Re^3: Parsing a Bibliography 2007 by Anonymous Monk on Apr 13, 2007 at 00:09 UTC
This is a one time effort, and I can only hope to successfully parse out some percentage of them, but that's what I'm trying to do. I don't know how hopeful to be, and that's why I included the examples, as I thought I might be able to get some help parsing those either individually, or as a series of steps. For example, I imagine parsing all the monographs out first, which involves both finding the monographs and not finding the other stuff. If I then have a filtered result file with no monographs, I might be able to filter for articles. I'm trying to figure out how to parse each of those example lines I included. After I convert everything to BibTex, the citations should be importable into various tools such as Refworks. This is my hope. I don't need anything near perfection on the results, just something better than 100% manual tagging. Any assistance would be hugely appreciated.	[reply]
Re^4: Parsing a Bibliography 2007 by ptum (Priest) on Apr 13, 2007 at 14:25 UTC
Re: Parsing a Bibliography 2007 by derby (Abbot) on Apr 12, 2007 at 22:20 UTC
derby has come to the conclusion that due to the myraid of citation formats, parsing them is probably quicker and cheaper when done by offshore labor! -derby	[reply]


Syntactic Confectionery Delight
	PerlMonks