This meditation was my first introduction to CI, and I have have been using .travis.yml's based on this writeup for over a year, ever since I followed a link to this meditation. Recently, I found out about AppVeyor which provides free CI on Windows. I thus stole stevieb's test-brewbuild/appveyor.yml as a start... but I liked the multi-perl nature of his Travis-CI config, so I wanted to figure out how to get AppVeyor to do multi-perl as well.

Here is what I have so far, with a description of some of my "features" or "tricks". (You can see my experimental history in https://github.com/pryrt/avtest/commits/master/appveyor.yml, or you can skip to my conclusion in the "live" example at Math-PRBS/appveyor.yml.)

AppVeyor uses chocolatey for external package management, and that's how you get strawberry perl onto the VM. stevieb's starting config just downloaded the default strawberryperl package, but the --version option to choco / cinst allows picking from the Version History as well. I eventually came to the point where I make a matrix: of environment variables, setting the %perl% environment variable using perl: default or perl: 5.14.4.1 or similar (so that it matches the full version that chocolatey uses). For the 'default', I don't pass the --version %perl% argument, so chocolatey just installs the default 'strawberryperl' package. I kept all the strawberry versions I've tried as comments in the matrix, with notes of which ##failed## when Appveyor's chocolatey tried to install them; I haven't debugged what's wrong with those packages.

I use a trick in the build and test sections, to run the correct make: Strawberry Perl used to use dmake, but recent versions have switched to gmake, and there was even at least one intermediate strawberry version that used dmake.exe, but also provided gmake.exe, so I couldn't just test for existence of gmake.exe. That made a pure cmd.exe version of make test impractical. I ended up spawning the one-liner perl -MConfig -le "system $Config{make}, 'test'", to use whichever was the correct make for that particular strawberry (and it should continue to work, even if future strawberry versions switched to futuremake.exe).

I also added cover to my AppVeyor config (well, added my named-target testcover), to verify my coverage on Windows. (Originally, because coveralls only integrates with travis-ci, I didn't bother also doing coverage in AppVeyor. While writing this up, I realized having external confirmation of coverage on Windows seems nice, and provides public confirmation even before CPAN Testers Matrix is reporting for the Windows platform: I've seen it take a few days before any Windows results show up.)

In appveyor.yml, you have to install Devel::Cover if you want to run cover (just like in .travis.yml, where you have to use cpan-install --coverage): since it's not likely in your distro's prereqs, --installdeps doesn't grab it. Just using cpanm Devel::Cover worked (usually¹), but it took a long time, so I switched to cpanm --notest Devel::Cover. (Devel::Cover is a well-established module, and I am going to assume that it shouldn't fail tests on install...)

I had experimented with the cache of the C:\strawberry directory, to save time during builds. Unfortunately, if I've read the documents correctly, AppVeyor only gives 1Gb of cache with its free accounts, and each job of each build uses up a separate portion of that cache. Since each strawberry installation is roughly 0.3Gb, that's only about 3 strawberry installations across all my projects with all their jobs before it would be full. And it sounded like it would cause builds to fail if the cache was full. Maybe my interpretation is wrong, but I can handle an extra 1-2min per job before seeing the status to ensure that it won't fail the build just because the cache was too full.

Anyway, I hope this might be useful to somebody out there.

-- pryrt

PS: This post has so much chocolate 🍫 and strawberries 🍓: now I want dessert ☺.

¹: For some reason, with perl v5.18.4.1, Devel::Cover had test fails when I was first trying it with Math-PRBS, but that failure appears to have been a fluke, because it works now.