How I'm Earning My Beer, part 3

12 October 2015 by Caron WillsCaron Wills
The final piece is getting Wine Test Daily to run automagically. That means, it should run when I'm not here, whenever Wine has been updated. Sure, I can reach over and start all of the tests on each system here at CodeWeavers every day when I appear at work but the fact that I can have my systems setup to do it for me is more than appreciated. It's one of the best features of wt-daily, it checks to see if it needs to run. It stores the last git commit number in a file and whenever it's called, it compares the current git commit with the last one it used. If they are different, wt-daily runs a new set of tests.

WineHQ gets commits a little after I arrive at work. I'm happy to manually start the tests but often times losing my Macs for an hour to automatic tests is bad. I've setup the tests to take place after I leave for the day and am pleased to report that those tests have run overnight successfully several times. This part will be more technical on the Mac OS and only have a little work to do with wt-daily.

Let's finish setting up wt-daily first.

Note that most of this won't make any sense unless you've followed the prior two pieces:

How I'm Earning My Beer, part 1
How I'm Earning My Beer, part 2



Currently, my wt-daily.config file looks like this:

email="caron@codeweavers.com"
tag="cw-mbp-A370M-11-x"
tagmac="cw-mbp-A370M-11-m"
desc="OS X 10.11 dual GPU AMD Radeon R9 M370X and Intel Iris Pro with the X11 driver"
descmac="OS X 10.11 dual GPU AMD Radeon R9 M370X and Intel Iris Pro with the Mac driver"
exlude="ddraw:ddraw1 ddraw:ddraw2 ddraw:ddraw4 ddraw:ddraw7 user32:win"
excludemac="windowscodecs:tiffformat windowscodecs:converter user32:msg gdi32:dc"
Note that the dwrite:font test has already been fixed and the developer notified me as soon as the fix was in place, hooray!

I need to add an option to automatically run the tests when wt-daily is started. I need this because the tests are very polite and on launch ask if the user is ready to run them.


Automatically run the tests?

I want them to run unattended so I need to turn off that nice piece by adding a default option to my config file:
default_options="--auto-test"

With that in place, there is no more prompt when wt-daily is started. That is not the only change I want. I also want wt-daily to tell my system to go back to sleep when it's done. It will do that with this line:

shutdown="suspend"

Wt-daily is now configured to run with an automatic setup.

email="caron@codeweavers.com"
tag="cw-mbp-A370M-11-x"
tagmac="cw-mbp-A370M-11-m"
desc="OS X 10.11 dual GPU AMD Radeon R9 M370X and Intel Iris Pro with the X11 driver"
descmac="OS X 10.11 dual GPU AMD Radeon R9 M370X and Intel Iris Pro with the Mac driver"
exlude="ddraw:ddraw1 ddraw:ddraw2 ddraw:ddraw4 ddraw:ddraw7 user32:win"
excludemac="windowscodecs:tiffformat windowscodecs:converter user32:msg gdi32:dc"
default_options="--auto-test"
shutdown="suspend"




The easiest part of the OS X setup is to schedule a wakeup time for the Mac that won't interfere with normal workflow. Open System Preferences and choose the Energy Saver. In the bottom right corner, click "Schedule…". Schedule a time when the Mac will be free for roughly an hour.


Energy Saver Schedule



And now, the technical part. Convincing the Mac to run wt-daily at a specific time.

There are many paths to take with this. Automator, cron, various test suites, so many that we could mention. I chose launchd after reading through the documentation, noting that almost all documentation states that cron is deprecated but still acceptable. I went with launchd specifically because of this comparison in the documentation of launchd versus cron:

If the system is turned off or asleep, cron jobs do not execute; they will not run until the next designated time occurs.

If you schedule a launchd job by setting the StartCalendarInterval key and the computer is asleep when the job should have run, your job will run when the computer wakes up. However, if the machine is off when the job should have run, the job does not execute until the next designated time occurs.

Reading that was an ah-ha moment. I wanted that. I wanted the job to run on wakeup if - for some reason- it failed to run.
I began to create my plist file. A file like this that is created by the user (third party) should go in the user level ~/Library/LaunchAgents folder. If "LaunchAgents" is not present, create it from terminal:

cd ~/Library
mkdir LaunchAgents
cd LaunchAgents

Then, create a file with the extension ".plist"

nano com.wt-daily.yourname.plist

The first thing this file needs is to be be recognized by OS X when loaded. That means it needs all of this:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
</plist>

Technically speaking, that's enough for it to be recognized. Of course it's not enough to do much else. Now it needs some information, it needs to identify the task that is running as in: it needs a label. Best practice is to give it the same name as the file without the plist extension:

<key>Label</key>
<string>com.wt-daily.yourname</string>

We need to tell it what to run. For this portion we set ProgramArguments, a variable OS X recognizes when it starts a launchd job. It needs to be given an array that includes the string of the application being run:

<key>ProgramArguments</key>
    <array>
        <string>/Users/yourname/pathto/wt-daily/wt-daily</string>
    </array>

We need to tell it when to run it. For our purposes, we set StartCalendarInterval and then include the dictionary of when to run it. Be sure to tell it to start after the scheduled wake-up time:

<key>StartCalendarInterval</key>
    <dict>
        <key>Hour</key>
        <integer>19</integer>
        <key>Minute</key>
        <integer>38</integer>
    </dict>

And, because LaunchAgents are not invoked from the terminal, we need to teach our LaunchAgent where to find the libraries wt-daily will need. We do this by setting EnvironmentVariables and giving it a dictionary to read with PATH and a value for PATH (this is also a variable recognized by OS X, that's why it is being referenced in a specific way):

<key>EnvironmentVariables</key>
    <dict>
        <key>PATH</key>
        <string>/usr/X11/lib:/usr/lib:/usr/local/git/bin:/usr/local/autoconf/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/X11/bin</string>
    </dict>

The final product looks like this:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
        <key>EnvironmentVariables</key>
        <dict>
            <key>PATH</key>
            <string>/usr/X11/lib:/usr/lib:/usr/local/git/bin:/usr/local/autoconf/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/X11/bin</string>
        </dict>
        <key>Label</key>
        <string>com.wt-daily.yourname</string>
        <key>ProgramArguments</key>
        <array>
            <string>/Users/yourname/pathto/wt-daily/wt-daily</string>
        </array>
        <key>StartCalendarInterval</key>
        <dict>
            <key>Hour</key>
            <integer>19</integer>
            <key>Minute</key>
            <integer>38</integer>
        </dict>
</dict>
</plist>

Once all that is done, tell launchd that it's there:

launchctl load com.wt-daily.yourname.plist

If it needs to be changed after it was loaded, unload the file:

launchctl unload com.wt-daily.yourname.plist

Make changes and load the file again. When I first got this running, I did a lot of experimenting. I set the script to run two minutes in the future and waited to make sure it would run. Now I'm really pleased with the outcome and I'm on my way to having all of my systems set to run wt-daily every day.




And that's it! My fleet of Macs now wake up after I leave work. They invoke wt-daily and check to see if Wine has had an update. If it has, Wine's Conformance Tests run and post their results to this page:


Wine's Conformance Tests
See? There is already a block where we've run Mac tests every day there was something available to test. Anyone up for a beer?

About Caron Wills
Caron has been working in the computer software industry for over 10 years. She joined CodeWeavers in 2008 and became the Quality Assurance Manager for CodeWeavers in 2009. Contact Caron at caron@codeweavers.com and learn more about her professional accomplishments on LinkedIn.

About Caron Wills

Caron has been working in the computer software industry for over 10 years. She joined
CodeWeavers in 2008 and became the Quality Assurance Manager for CodeWeavers in 2009. Contact Caron at caron@codeweavers.com and learn more about her professional accomplishments on LinkedIn.

About CodeWeavers
Founded in 1996 as a general software consultancy, CodeWeavers focuses on the development of Wine – the core technology found in all of its CrossOver products. The company's goal is to bring expanded market opportunities for Windows software developers by making it easier, faster and more painless to port Windows software to Mac and Linux. CodeWeavers is recognized as a leader in open-source Windows porting technology, and maintains development offices in Minnesota, the United Kingdom and elsewhere around the world. The company is privately held.

The following comments are owned by whoever posted them. We are not responsible for them in any way.


New Comment

Rob Crittenden
Posted 2015-10-13 10:10
This is pretty cool. What do you do with the results of the tests? Are the failures triaged and assigned out, perhaps by area of failure, or is this used internally to benchmark failures in changes CodeWeavers makes against upstream to prevent regressions?
Caron Wills
Caron Wills
Posted 2015-10-21 13:46
Right now, we're stabilizing the tests (and the XQuartz project just put out an update so we're updating all of our systems).

In the future, each test that has failures will ideally be looked at to see if the test is too stringent or if there is another reason for the failure.

Today, we ran tests and found that several were resolved by the update to XQuartz, in fact, there's a group that timed out previously that now passes. Hooray!

We also look to see if there are regressions. Usually, every patch is run through WineTest when it is submitted to wine-patches and if it causes a problem, it is flagged for review by the author. Sometimes, WineTest's bot doesn't find the regression and the test results will instead show it. That information is useful because we (anyone adept at Wine) can go back and look at what was committed to the test that day to see if there's something causing the failure.

Essentially, it's a useful tool that helps. It isn't the only tool, but it is one of many.
Please Wait...
eyJjYXJ0IjowLCJ0enMiOi01LCJjZG4iOiJodHRwczpcL1wvbWVkaWEuY29kZXdlYXZlcnMuY29tXC9wdWJcL2Nyb3Nzb3Zlclwvd2Vic2l0ZSIsImNkbnRzIjoxNTM4NzY3MzM4fQ==