Beefy Boxes and Bandwidth Generously Provided by pair Networks
go ahead... be a heretic
 
PerlMonks  

Re: Automating Backup of a Google Map

by Sinistral (Monsignor)
on Jun 16, 2017 at 14:40 UTC ( #1192945=note: print w/replies, xml ) Need Help??


in reply to Automating Backup of a Google Map

You can automate the web interface in Google the Google Maps "My Maps", but there is an easier way to get to the raw KML that you might not have known about.

For the current owner of the map within the Google My Maps, use a web browser to choose "Export to KML" and choose "Entire Map" in the dropdown, and choose the checkbox for "Keep data up to date with network link KML (only usable online)". That export will give you a KMZ file. Use an unzip program (7zip, unzip, Winzip, anything that understands zip format) to see the contents. Within the zip file will be a doc.kml file. Examine the contents of the doc.kml in a text editor. There is a <link> element which has within it a <href> element. That is the URL you can use with wget/curl/LWP to to your backup file retrieval.

Replies are listed 'Best First'.
Re^2: Automating Backup of a Google Map
by jmlynesjr (Hermit) on Jun 17, 2017 at 00:28 UTC

    Sinistral, I saw this approach in Geo::Google::MyMap::KMLURL, but I wasn't sure what is was telling me.

    This looks like the best short term approach. Thank you very much!

    James

    There's never enough time to do it right, but always enough time to do it over...

Re^2: Automating Backup of a Google Map
by jmlynesjr (Hermit) on Jun 18, 2017 at 19:22 UTC
    Update:

    Feedback from the map Czar, Doug.

    "Below is the contents of the batch file with the wget. the italicized all caps words are things I took out to anonymize the information.

    cd /users/NAME/documents/FOLDER /Windows/GnuWin32/bin/wget http://www.google.com/maps/d/u/1/kml?mid=FI +LE ID --no-check-certificate

    This saves it to the folder in the batch file with the name kml@mid=FILE ID.NUMBER Then all you have to do is rename it NAME.kmz and it works perfectly. there might be a better way with the file names but this works fine for our purpose. it only has to be renamed if I need to restore the map. so far it is churning away as planned. I appreciate the help. you probably have more time in finding this solution than I do in setting up the map."

    Doug is running the .bat file hourly using Windows Scheduler. Thanks again to all that contributed.

    Edit: Callsign removed by request.

    James

    There's never enough time to do it right, but always enough time to do it over...

      There is a problem with storing too many files in a single directory as the directory itself gets fragmented and performance starts to suffer. There are indications that this starts at lower counts but i start to see it when directories have more than about 1000 files. If you were to store one file every hour this would start somewhere around 40 days in.

      a second problem with your method is is that "duplicate" files are stored. There realy is no need to save the newest file if it is the same as the last one.

      To solve the first problem i create a folder tree to store the files in. such as /users/NAME/documents/FOLDER/y2017/m06 or /users/NAME/documents/FOLDER/y2017/m06/d18 (i have a program that may store a new file every 3 min, this can mean 480 files a day)

      To solve the second problem i tend to compare the last and newest files and not put the new file into the history tree if they are the same. In your case the files inside the zip get a new datetime every pull, so you have to extract the relivant 'doc.kml' files and compare those.

      The following program should do both of these tasks

      use strict; use warnings; use Getopt::Long qw/GetOptions/; use LWP; use File::Basename qw/dirname basename/; use File::Copy qw/copy/; my $url ='http://www.google.com/maps/d/u/1/kml?mid=1Oa_e +gVdStSJBF5C7mpS6MXrkces'; #my $topdir ='/users/NAME/documents/FOLDER'; my $dir ='D:/goodies/pdhuck/down1/perl/monks/kmlbackup'; + my $ftype ='zip'; my $debug =0; my %optdef=("debug=i" => \$debug ,"url=s" => \$url ,"dir=i" => \$dir ); GetOptions ( %optdef ) or die("Error in command line arguments\n"); die $dir.' must exist' unless (-d $dir); my $lastdir=$dir.'/last'; # unless (-d $lastdir && -w $lastdir) unless (-d $lastdir) { mustdir ($lastdir); } my $now=time; my ($sec,$min,$hour,$mday,$mon,$year,$wday,$yday,$isdst) =gmtime($no +w) ; my $nicefmt='/y%04u/m%02u/d%02u-%02u-%02u-%02u-Z'; # last dir is mo +nth # my $nicefmt='/y%04u/m%02u/d%02u/%02u-%02u-%02u-Z'; # last dir is d +ay my $nice=sprintf($nicefmt,$year+1900,$mon+1,$mday,$hour,$min,$sec); my $lastfn=$lastdir.'/lastbackup.'.$ftype; my $nextfn=$lastdir.'/nextbackup.'.$ftype; my $ua=LWP::UserAgent->new(agent =>"libwww-perl-kmzbackup"); my $req = new HTTP::Request (GET => $url); my $request = $ua->request ($req); unless ($request->is_success) { die 'get failed for '.$url.' '.$requ +est->status_line;} open (my $nextout,'>',$nextfn) or die 'cant open '.$nextfn; binmode $nextout; print $nextout $request->decoded_content; close $nextout; my $aresame=1; my $compfile='doc.kml'; if (-f $lastfn) { use IO::Uncompress::Unzip qw(unzip $UnzipError) ; use IO::File; my $nextmember = new IO::Uncompress::Unzip($nextfn, Name => $com +pfile) or die "IO::Uncompress::unzip failed: $UnzipError\n"; my $lastmember = new IO::Uncompress::Unzip($lastfn, Name => $com +pfile) or die "IO::Uncompress::unzip failed: $UnzipError\n"; while ($aresame && ( my $nextline=<$nextmember>) && (my $lastli +ne=<$lastmember>) ){ unless ($nextline eq $lastline ) {$aresame=0} } if ($aresame && (my $nextline=<$nextmember>) ){ $aresame=0} if ($aresame && (my $lastline=<$lastmember>) ){ $aresame=0} close($nextmember); close($lastmember); } else { $aresame=0;} if ($aresame) { print "No change to file $compfile \n"; unlink $nextfn; exit; } my $endfn=$dir.$nice.'.'.$ftype; my $endfn0=basename($endfn); my $enddir=dirname($endfn); unless (-d $enddir) { mustdir($enddir); } copy($nextfn,$endfn) or die "Copy failed: $!"; print 'new backup:'.$endfn."\n"; copy($nextfn,$lastfn) or die "Copy failed: $!"; unlink $nextfn; exit; sub mustdir { my $dir=shift; return if (-d $dir); my $updir=dirname($dir); mustdir ($updir); mkdir $dir; unless (-d $dir) {die 'cant make dir:'.$dir; } } # mustdir

        There is a problem with storing too many files in a single directory as the directory itself gets fragmented and performance starts to suffer. There are indications that this starts at lower counts but i start to see it when directories have more than about 1000 files. If you were to store one file every hour this would start somewhere around 40 days in.

        Of course, the performance of directories full of files highly depends on the filesystem used. Caching (i.e. free RAM) and tuning of filesystem and operating system can influence the performance drastically, as can disk performance (compare an ancient harddisk with a high end SSD).

        Alexander

        --
        Today I will gladly share my knowledge and experience, for there are no sweeter words than "I told you so". ;-)

        Huck

        Thank you for sharing this concern and Utility. I will pass it on to Doug.

        James

        There's never enough time to do it right, but always enough time to do it over...

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://1192945]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others musing on the Monastery: (9)
As of 2020-05-25 09:28 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    If programming languages were movie genres, Perl would be:















    Results (145 votes). Check out past polls.

    Notices?