Re: Parsing a Bibliography 2007

This is a really hard problem if you want high accuracy (I've done this for money before). You probably want to start by trying to separate the individual citations without parsing them. Then you can iteratively come up with a library of regexes to pull them apart by starting with a few obvious and conservative patterns, examining the errors and misses, and developing more patterns. If you want to get really fancy, you can disambiguate by e.g. recognizing known authors' names as part of the author list rather than the title. Eventually you'll just have to throw up your hands and say "good enough"...

Comment on Re: Parsing a Bibliography 2007

Replies are listed 'Best First'.
Re^2: Parsing a Bibliography 2007 by ptum (Priest) on Apr 12, 2007 at 23:41 UTC
I agree with this approach. Assuming there is no way to definitively lock down all possible distinct formats, you'll end up iteratively building regexen and parsing your citations until 'enough' of them are recognized and the remaining unparsed subset is sufficiently small. Do you anticipate this as a one-time effort, or will you be regularly handling new citation forms?	[reply]
Re^3: Parsing a Bibliography 2007 by Anonymous Monk on Apr 13, 2007 at 00:09 UTC
This is a one time effort, and I can only hope to successfully parse out some percentage of them, but that's what I'm trying to do. I don't know how hopeful to be, and that's why I included the examples, as I thought I might be able to get some help parsing those either individually, or as a series of steps. For example, I imagine parsing all the monographs out first, which involves both finding the monographs and not finding the other stuff. If I then have a filtered result file with no monographs, I might be able to filter for articles. I'm trying to figure out how to parse each of those example lines I included. After I convert everything to BibTex, the citations should be importable into various tools such as Refworks. This is my hope. I don't need anything near perfection on the results, just something better than 100% manual tagging. Any assistance would be hugely appreciated.	[reply]
Re^4: Parsing a Bibliography 2007 by ptum (Priest) on Apr 13, 2007 at 14:25 UTC
I think you are on the right track with the 2004 discussion, particularly BrowserUK's commented response: Re^3: parsing a bibliography. I would recommend that you build a simple script with a regular expression that tries to sort the bibliography records into separate files based on their types. Recommend you follow other advice in that 2004 thread such that you dump records into an error file that don't match any of your rules so that you can fine-tune your regular expressions. Once you've got the records separated into distinct files, you can run them through another script (or subroutine) that contains a regular expression specific to that type of record. You may want to start with a list of rules that specify how you differentiate between record types, and then post a sample regex. I think you'll find plenty of monks who will be happy to edit your regex, but perhaps fewer willing to write the whole thing for you. :)	[reply]


Perl-Sensitive Sunglasses
	PerlMonks