Text::xSV quote problem

cfreak has asked for the wisdom of the Perl Monks concerning the following question:

After a long hiatus into the dark and scary world of PHP, I have returned with a question.

I have a project involving the parsing of a large (30MB) tab separated text file. The code was an existing project that I'm new to, the original author used Text::xSV for the parsing, so I don't really have the option to use something else.

The problem is the fields are not quoted in the text file but some of the data contains quotes. What i need is a way for Text::xSV to simply ignore them. I know I can use filtering to change them to some other character and then change them back once they are separated, but I'm hoping there's a better way.

I can't include the actual line due to non-disclosure agreements but its basically somethng like this

00001 Widget ACTIVE widget maker's inc 15"x15"

Here is the code I'm using to parse, bascially the same as the example in the pod

    my $csv = Text::xSV->new(sep=>"\t" );
    $csv->open_file("xxxxx.csv");
    $csv->read_header();

    # Make the headers case insensitive
    foreach my $field ($csv->get_fields) {
        if (lc($field) ne $field) {
            $csv->alias($field, lc($field));
        }
    }

    while( $csv->get_row() )
    {
       # save to a db
    }
[download]

Thanks!

Lobster Aliens Are attacking the world!

Comment on Text::xSV quote problem Select or Download Code

Replies are listed 'Best First'.

Re: Text::xSV quote problem
by tilly (Archbishop) on Oct 03, 2006 at 18:30 UTC

Text::xSV

my $file = "xxxx.cvs";
open(my $fh, "<", $file) or die "Cannot read '$file': $!";
my $line = <$fh>;
chomp($line);
my @fields = split /\t/, lc($line), -1;

while ($line = <$fh>) {
    chomp($line);
    my %row;
    @row{@fields} = split /\t/, $line, -1;
    # save to a db
}
[download]

[reply]
[d/l]

Re^2: Text::xSV quote problem

by cfreak (Chaplain) on Oct 03, 2006 at 22:15 UTC

I would, unfortunately the requirements are to use Text::xSV, lots of files in the project use it and they want to remain consistent. It is also possible that the format will change to something that will need it

I have solved the problem by using a filter function. Its kind of ugly but it works.

Lobster Aliens Are attacking the world!

[reply]

Re^3: Text::xSV quote problem

by derby (Abbot) on Oct 04, 2006 at 12:54 UTC

Hmmm ... I must find some proverb or pithy saying that covers this situation .... ignoring the advice of the module's [id://tilly|author] is probably not in your best interest. Maybe I'm being too harsh (or maybe it's the repetiviness of always saying this to my four year old) - but you should at least say thank you.

-derby

[reply]

Re: Text::xSV quote problem
by grep (Monsignor) on Oct 03, 2006 at 18:17 UTC

Text::xSV

Text::CSV_XS

docs for Text::xSV

set_filter

You did not explain exactly you're looking to do with these quotes but I would try '\' escaping the quotes first, to see if that works.

grep

One dead unjugged rabbit fish later

[reply]
[d/l]

Re^2: Text::xSV quote problem

by cfreak (Chaplain) on Oct 03, 2006 at 18:22 UTC

Yeah but I need the quotes, I'd have to change the back later. Not a big deal but I was trying to avoid that.

I tried escaping too ... it didn't make a difference.

Lobster Aliens Are attacking the world!

[reply]


good chemistry is complicated, and a little bit messy -LW
	PerlMonks