Beefy Boxes and Bandwidth Generously Provided by pair Networks
Keep It Simple, Stupid
 
PerlMonks  

Re^2: Fetching table from website using HTML::TableExtract

by sachin raj aryan (Acolyte)
on Mar 29, 2017 at 12:16 UTC ( [id://1186371]=note: print w/replies, xml ) Need Help??


in reply to Re: Fetching table from website using HTML::TableExtract
in thread Fetching table from website using HTML::TableExtract

actually not sure how to fix the data as data i am getting from webpage directly .
  • Comment on Re^2: Fetching table from website using HTML::TableExtract

Replies are listed 'Best First'.
Re^3: Fetching table from website using HTML::TableExtract
by shmem (Chancellor) on Mar 29, 2017 at 14:23 UTC

    Corion doesn't mean you fix the data in the webpage directly, but after you have extracted it with HTML::TableExtract which is properly inserting empty fields for data cells skipped due to colspan or rowspan. But you have multiline fields:

    The second row of your $ts->rows is

    Region,Level 31.03.2016,,Sanction/Renewal 01.04.2016 to 28.02.2017,,,Level 28.02.2017,,Sanction/Renewal During Current Month ,,,Level 26.03.2017,,Growth as on 26.03.2017,

    After Level 31.02.2016 there's an empty field because of colspan="2". The next field is

    Sanction/Renewal During Current Month

    so all you have to do is removing trailing whitespace/newlines from each field:

    foreach my $ts ( $te->tables ) { print "Table (", join( ',', $ts->coords ), "):\n"; foreach my $row ( $ts->rows ) { s/[\s\n]+\z/ for @$row; # <--- here # s/\n/ /gs for @$row; # uncomment if you want to convert # multiline fields into single line $OUT-> print( join( ',', @$row ), "\n"); } }
    perl -le'print map{pack c,($-++?1:13)+ord}split//,ESEL'
      Hi i tried your code somewhat working but still one issue below is the output

      Amount in Crores,,,,,,,,,,,,,,

      Region,Level 31.03.2016,,Sanction/Renewal01.04.2016 to 28.02.2017,,,Level 28.02.2017,,Sanction/RenewalDuring Current Month,,,Level 27.03.2017,,Growthas on27.03.2017,

      if you see multiple commas in first line and more than one in second row
        if you see multiple commas in first line and more than one in second row

        What do you expect?
        You are basically outputting a CSV file in which each line has the same amount of fields, separated by commas. Nothing between two commas is an empty field produced by expanding field spanning captions of the original table denoted as e.g. colspan="2" in html.

        If those fields were omitted, the placement of the captions would not correspond to the columns of the table data. Load the output file as a spreadsheet (e.g. in Excel) and you will see the effect.

        I don't know the purpose of what you are doing. If you want to remove empty fields from the output, proceed as follows:

        foreach my $ts ( $te->tables ) { print "Table (", join( ',', $ts->coords ), "):\n"; foreach my $row ( $ts->rows ) { s/[\s\n]+\z/ for @$row; s/\n/ /gs for @$row; $OUT-> print( join( ',', grep { length $_ } @$row ), "\n"); } }

        Again, load the data into a spreadsheet to see the difference. Compare to the original table on the website.

        perl -le'print map{pack c,($-++?1:13)+ord}split//,ESEL'

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://1186371]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others surveying the Monastery: (2)
As of 2024-04-25 20:19 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found