Beefy Boxes and Bandwidth Generously Provided by pair Networks
No such thing as a small change
 
PerlMonks  

Threads::Shared MultiLevel Hash and the Invalid value for Shared Scalar Error

by DFass (Initiate)
on Jan 16, 2014 at 15:50 UTC ( [id://1070818]=perlquestion: print w/replies, xml ) Need Help??

DFass has asked for the wisdom of the Perl Monks concerning the following question:

Ok...I've read up on the relevant postings here and elsewhere, and thought I got a handle on getting a locked, shared, thread to work. I know about the auto vivification trickery. But, I guess not...

It fails on the :unless (exists $$dataref{$sid}{"ALL"}){ line.

Here's what I have, so far...the relevant bits:

use strict; use warnings; use threads; use Thread::Queue qw(); use threads::shared; use DBIx::Connector; use Spreadsheet::WriteExcel; use Spreadsheet::WriteExcel::Utility; use Date::Simple qw(:all); use Date::Range; use Data::DumpXML qw(dump_xml); use HomeSharing qw(:all); use List::MoreUtils qw/ uniq /; sub setup_data($$%) { my $conn; my $currentyear; my $dataref :shared; my %data :shared; ( $conn, $currentyear, %data ) = @_; #my %data = %{$dataref}; my @CountiesList; @CountiesList = get_countiesList_multi($conn); push( @CountiesList, "OOC" ); my $loop; my $monthloop; #my $now=Date::Simple->new(); #my $currentyear0 = $now->year; my $id; for ( $loop = 1984 ; $loop <= $currentyear+1 ; $loop++ ) { #print "\nYear:$loop\t"; for ( $monthloop = 1 ; $monthloop <= 12 ; $monthloop++ ) { $id = $loop . "-" . $monthloop; unless (exists ($data{$id})){ my %a :shared; $data{$id}=\%a; } #print "\tMonth:$monthloop\n"; $data{$id}{"Hours"} = 0; $data{$id}{"InitialSeekers"} = 0; $data{$id}{"InitialProviders"} = 0; $data{$id}{"SharedLivingInterviews"} = 0; $data{$id}{"SeekerInterviews"} = 0; $data{$id}{"ProviderInterviews"} = 0; foreach my $county (@CountiesList) { #my $id = $loop."-".$monthloop; #print "\t\t\t$county"; unless (exists ($data{$id})){ my %a :shared; $data{$id}=\%a; } unless (exists $data{$id}{$county}){ my %a :shared; $data{$id}{$county}=\%a; } $data{$id}{$county}{"MatchesStarted"} = 0; $data{$id}{$county}{"MatchesEnded"} = 0; $data{$id}{$county}{"MatchesTotal"} = 0; $data{$id}{$county}{"ClientsStarted"} = 0; $data{$id}{$county}{"ClientsEnded"} = 0; $data{$id}{$county}{"HouseholdsProviders"} = 0; $data{$id}{$county}{"HouseholdsSeekers"} = 0; $data{$id}{$county}{"Households"} = 0; $data{$id}{$county}{"ClientsTotal"} = 0; $data{$id}{$county}{"MatchesPeopleStarted"} = 0; $data{$id}{$county}{"MatchesPeopleEnded"} = 0; $data{$id}{$county}{"MatchesPeopleTotal"} = 0; } ## end foreach my $county (@CountiesList...) } ## end for ( $monthloop = 1 ; ...) } ## end for ( $loop = 1984 ; $loop...) return %data; } ## end sub setup_data($$%) sub do_matches($$$$){ my $conn; my $maxFN; my $currentyear; my $dataref :shared; ( $conn, $maxFN, $currentyear, $dataref ) = @_; my $lastyear = $currentyear - 1; my $q = Thread::Queue->new(); my $num_workers = 25; my $num_work_units = $maxFN; # my %data = $$dataref; # Create workers my @workers; for (1..$num_workers) { push @workers, async { while (defined(my $unit = $q->dequeue())) { print("$unit\n"); my $stm2 = $conn->dbh->prepare( "SELECT FILENO,P_MUNCIPAL,MUNCIPALIT," . " STATE,ZIP,DATE_MATCH,DATE_ENDED, TOTAL_NUM" . " FROM MATCH WHERE (FILENO=?)" ); $stm2->execute($unit); $stm2->bind_columns( \my ( $fn, $where, $where2, $state, $zip, + $datem, $datee, $tn ) ); while ( $stm2->fetch ) { print "\tDoing $fn...\n\n"; lock($dataref); my $whichco = ""; my $whichco2 = ""; $whichco = get_county_multi( $conn, $where, $state, $zip, +1, $where2 ); $tn += 0; print "For File:$fn\nWhere = $where / $where2\n"; print "\tMatched = $datem \t Ended=$datee\n"; print "\tCounty = $whichco\tPeople = $tn\n\n"; print F "For File:$fn\nWhere = $where / $where2\n"; print F "\tMatched = $datem \t Ended=$datee\n"; print F "\tCounty = $whichco\tPeople = $tn \n\n"; my $startd = HomeSharing::convert_date( $datem, ymd( 1983, + 12, 31 ) ); my $endd = HomeSharing::convert_date( $datee, ymd( $curr +entyear + 1, 12, 31 ) ); print "Start = $startd\tEnd = $endd\n"; my $sid = $startd->year . "-" . $startd->month; my $eid = $endd->year . "-" . $endd->month; print "SID:$sid\tEID:$eid\n"; print F "SID:$sid\tEID:$eid\n"; unless (exists ($$dataref{$sid})){ my %a :shared; $$dataref{$sid}=\%a; } unless (exists ($$dataref{$eid})){ my %a :shared; $$dataref{$eid}=\%a; } unless (exists $$dataref{$sid}{"ALL"}){ my %a :shared; $$dataref{$sid}{$whichco}=\%a; } unless (exists $$dataref{$eid}{"ALL"}){ my %a :shared; $$dataref{$eid}{$whichco}=\%a; } unless (exists $$dataref{$sid}{$whichco}){ my %a :shared; $$dataref{$sid}{$whichco}=\%a; } unless (exists $$dataref{$eid}{$whichco}){ my %a :shared; $$dataref{$eid}{$whichco}=\%a; } unless (exists $$dataref{$sid}{"ALL"}{"MatchesStarted" +}){ my %a :shared; $$dataref{$sid}{"ALL"}{"MatchesStarted"}=\%a; } unless (exists $$dataref{$sid}{"ALL"}{"MatchesPeopleSt +arted"}){ my %a :shared; $$dataref{$sid}{"ALL"}{"MatchesStarted"}=\%a; } unless (exists $$dataref{$sid}{$whichco}{"MatchesStart +ed"}){ my %a :shared; $$dataref{$sid}{$whichco}{"MatchesStarted"}=\%a; } unless (exists $$dataref{$sid}{$whichco}{"MatchesPeopl +eStarted"}){ my %a :shared; $$dataref{$sid}{$whichco}{"MatchesPeopleStarted"}= +\%a; } unless (exists $$dataref{$eid}{"ALL"}{"MatchesEnded"}) +{ my %a :shared; $$dataref{$sid}{"ALL"}{"MatchesEnded"}=\%a; } unless (exists $$dataref{$eid}{"ALL"}{"MatchesPeopleEn +ded"}){ my %a :shared; $$dataref{$sid}{"ALL"}{"MatchesEnded"}=\%a; } unless (exists $$dataref{$eid}{$whichco}{"MatchesEnded +"}){ my %a :shared; $$dataref{$sid}{$whichco}{"MatchesEnded"}=\%a; } unless (exists $$dataref{$eid}{$whichco}{"MatchesPeopl +eEnded"}){ my %a :shared; $$dataref{$sid}{$whichco}{"MatchesPeopleEnded"}=\% +a; } $$dataref{$sid}{"ALL"}{"MatchesStarted"}++; $$dataref{$sid}{"ALL"}{"MatchesPeopleStarted"} += $tn; $$dataref{$sid}{$whichco}{"MatchesPeopleStarted"} += $tn; $$dataref{$sid}{$whichco}{"MatchesStarted"}++; $$dataref{$eid}{"ALL"}{"MatchesEnded"}++; $$dataref{$eid}{"ALL"}{"MatchesPeopleEnded"} += $tn; $$dataref{$eid}{$whichco}{"MatchesPeopleEnded"} += $tn; $$dataref{$eid}{$whichco}{"MatchesEnded"}++; print F "\n=============\nData:\n"; print F "For $sid Matches Started Data[All] = " . $$datare +f{$sid}{"ALL"}{"MatchesStarted"} . "\n"; print F "For $sid Matches Started Data[$whichco] = " . $$d +ataref{$sid}{$whichco}{"MatchesStarted"} . "\n"; print F "For $eid Matches Ended Data[All] = " . $$dataref{ +$eid}{"ALL"}{"MatchesEnded"} . "\n"; print F "For $eid Matches Ended Data[$whichco] = " . $$dat +aref{$eid}{$whichco}{"MatchesEnded"} . "\n"; #print F "DataCheck::Hours: " . $$dataref{$sid}{"Hours"} . + "\n"; print F "================\n\n"; } ## end while ( $stm2->fetch ) } }; } # Create work for (1..$num_work_units) { $q->enqueue($_); } # Tell workers they are no longer needed. $q->enqueue(undef) for @workers; # Wait for workers to end $_->join() for @workers; # return %data; } ## end sub do_matches($$$%) #-------------------------# # Main # #-------------------------# if ( -e "StatsAuto.xls" ) { unlink "StatsAuto.xls"; print "Excel File Deleted\n"; } ## end if ( -e "StatsAuto.xls"...) open( F, ">StatsDebug.txt" ) || die "Debug 1 failed"; open( G, ">StatsDebug2.txt" ) || die "Debug 2 failed"; my $workbook; #Excel Workbook my $worksheet; #Excel Worksheet my $conn_hs; #HomeSharing Database my $conn_stats; #Stats Database my $now = Date::Simple->new(); my $currentyear = $now->year; my $lastyear = $currentyear - 1; my $currentmonth = $now->month; my $Fudge = 0; if ( $Fudge == 1 ) { $currentyear = 2012; $lastyear = 2011; $currentmonth = 12; } ## end if ( $Fudge == 1 ) my %data :shared; my $MaxFN; my $MaxPS; my $today2; my $id; my $county; my $subkey1; my $subkey2; my ( %ranges, %monthlyranges ) = setup_ranges($currentyear); ( $workbook, $worksheet ) = setup_excel(); ( $conn_hs, $conn_stats ) = setup_databases(); my @CountiesList = get_countiesList_multi($conn_hs); ( %ranges, %monthlyranges ) = setup_ranges($currentyear); %data = setup_data( $conn_hs, $currentyear, %data ); print G "\nAfter Setup\n----------------------\n"; print_data($conn_hs,%data); do_hours( $conn_stats, $currentyear, \%data ); print G "\nAfter Hours\n----------------------\n"; print_data($conn_hs,%data); $MaxFN = get_MaxFN_multi($conn_hs) + 0; $MaxPS = get_MaxPS_multi($conn_hs) + 0; print "MaxFN = $MaxFN\tMaxPS = $MaxPS\n\n"; #( $today2, my %data2 ) = do_NOS( $conn_hs, $MaxFN, $currentyear, $cur +rentmonth, \%ranges, \%data ); do_matches( $conn_hs, $MaxFN, $currentyear, \%data ); print "\n\n"; print G "After Matches\n-------------------------------------------\n" +; print_data( $conn_hs, %data ); #do_clients( $conn_hs, $MaxPS, $currentyear, \%data ); #print G "After Clients\n-------------------------------------------\n +"; #print_data( $conn_hs, %data ); #do_calculate_data( $conn_hs, $currentyear, \%data ); #print G "After Calculate\n------------------------------------------- +\n"; #print_data( $conn_hs, \%data ); print "Excellizing...\n"; excelize_data( $conn_hs, $currentyear, $today2, $workbook, $worksheet, + %data ); #$a = dump_xml(%data); #print G "$a\n"; print "Done\n"; close(F); close(G);

Which begs the 'obvious' question of what's going wrong and how to fix it. I would like to be able to multithread it.

It works fine as an unthreaded program -- just, when you're dealing with 1500 match entries and 14,200 client entries -- it just takes a long time to process.

  • Comment on Threads::Shared MultiLevel Hash and the Invalid value for Shared Scalar Error
  • Download Code

Replies are listed 'Best First'.
Re: Threads::Shared MultiLevel Hash and the Invalid value for Shared Scalar Error
by BrowserUk (Patriarch) on Jan 16, 2014 at 17:03 UTC

    What do you think that unless (exists $$dataref{$sid}{"ALL"}){ is testing?


    With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.

      It SHOULD be testing the next level of the %data hash, passed by reference.

      It OUGHT to be already Shared, but, since I know that there is the auto-vivication trap in the module...thus, the test.

      The hash has 3 sub-hashes: ID (the year-month), COUNTY (either ALL, OOC, or a County name, and the VARIABLE...

      So e.g. it'd be $$dataref{"2013-1"}{"ALL"}{"MatchesStarted"}

        Wow. I did some work on this a month ago, but I doubt I still have the file in which I did it. I assumed you weren't coming back and probably ditched it. I sure don't relish recreating what I did :(

        But, since you've now answered the first question, try this one.

        What do you mean by "It fails on the :unless (exists $$dataref{$sid}{"ALL"}){ line."?

        Issues a warning? Dies with an error? Crashes with a core dump? Get bored and goes home?


        With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
        Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
        "Science is about questioning the status quo. Questioning authority".
        In the absence of evidence, opinion is indistinguishable from prejudice.

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: perlquestion [id://1070818]
Approved by ww
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others having an uproarious good time at the Monastery: (3)
As of 2024-04-19 23:06 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found