How I'm Earning My Beer, part 2

7 October 2015 by Caron WillsCaron Wills
After completing the first part; Wine's Conformance Tests ran and that was all well and good and fun. There was a flurry of flashes on the screen, various windows were named after various tests, sound played, I rejoiced. It was an exciting accomplishment!

But the report did not work. My results did not show up and I was quickly frustrated. This is where I considered giving up.

I don't need to have a beer, right?
I mean, that's really a simple carrot and nothing else. I wanted to submit the results but if it won't run, it won't run; and it did run but for some reason it didn't report.

This is where the community comes in. This is where we often don't realize how many people are around us willing to help. I dropped a hint at my frustrations into chat and rather than getting the quick response of "that's too bad", I got a question:

How close are you?

And I don't know. And I have to be walked to the log file that wt-daily has very helpfully left in my folders. The location is:

~/wt-daily/winetest/wine/

The log file is called:

wt-bot-(yourtag).log

It's lines and lines of useful information about all the tests that ran. The first thing I learn is that WineTest really did run and it really did submit results to WineHQ. It took it almost 30 minutes to run... and it had 54 failures.

I report the number of failures and I get another question:

How many skips?

I confess I don't know. I didn't skip anything.

Now I get the explanation:

Wine's Conformance Tests are set to accept and publish results with 50 or less failures and 10 or less skips. Both requirements must be met to be featured on the results page. It sounds like you didn't skip anything, add a line to your configuration file and skip a test or two that fails and you will get it.

I'm skeptical because I've already put a bit of effort into getting this to run and I've watched a few other people grumble about libraries and dependencies but there is encouragement in that explanation: you will get it.

I started at the bottom of the logfile and looked for the failing tests. They were fairly easy to identify and so I added some tests to skip to my configuration file.

Look for lines like this:

Running tests - 53 failures
Running: windowscodecs:tiffformat (541 of 579)
2015-10-05 11:28:16.135 wine[44040:808186] void *__CFPasteboardAllocatorAllocate(CFIndex) : Lock timeout
Assertion failed: (sizeof(uint64)==8), function TIFFClientOpen, file tif_open.c, line 99.
wine: Assertion failed at address 0x000b:0x9b5a062a (thread 0063), starting debugger...
2015-10-05 11:28:16.139 wine[44040:808186] void *__CFPasteboardAllocatorAllocate(CFIndex) : Lock timeout
2015-10-05 11:28:16.141 wine[44040:808186] void *__CFPasteboardAllocatorAllocate(CFIndex) : Lock timeout
2015-10-05 11:28:16.144 wine[44040:808186] void *__CFPasteboardAllocatorAllocate(CFIndex) : Lock timeout
2015-10-05 11:28:16.147 wine[44040:808186] void *__CFPasteboardAllocatorAllocate(CFIndex) : Lock timeout
2015-10-05 11:28:16.149 wine[44040:808186] void *__CFPasteboardAllocatorAllocate(CFIndex) : Lock timeout
2015-10-05 11:28:16.151 wine[44040:808186] void *__CFPasteboardAllocatorAllocate(CFIndex) : Lock timeout
2015-10-05 11:28:16.154 wine[44040:808186] void *__CFPasteboardAllocatorAllocate(CFIndex) : Lock timeout
2015-10-05 11:28:16.156 wine[44040:808186] void *__CFPasteboardAllocatorAllocate(CFIndex) : Lock timeout
2015-10-05 11:28:16.158 wine[44040:808186] void *__CFPasteboardAllocatorAllocate(CFIndex) : Lock timeout
2015-10-05 11:28:16.160 wine[44040:808186] void *__CFPasteboardAllocatorAllocate(CFIndex) : Lock timeout
2015-10-05 11:28:16.163 wine[44040:808186] void *__CFPasteboardAllocatorAllocate(CFIndex) : Lock timeout
2015-10-05 11:28:16.165 wine[44040:808186] void *__CFPasteboardAllocatorAllocate(CFIndex) : Lock timeout
2015-10-05 11:28:16.168 wine[44040:808186] void *__CFPasteboardAllocatorAllocate(CFIndex) : Lock timeout
2015-10-05 11:28:16.170 wine[44040:808186] void *__CFPasteboardAllocatorAllocate(CFIndex) : Lock timeout
2015-10-05 11:28:16.173 wine[44040:808186] void *__CFPasteboardAllocatorAllocate(CFIndex) : Lock timeout
2015-10-05 11:28:16.175 wine[44040:808186] void *__CFPasteboardAllocatorAllocate(CFIndex) : Lock timeout
2015-10-05 11:28:16.177 wine[44040:808186] void *__CFPasteboardAllocatorAllocate(CFIndex) : Lock timeout
2015-10-05 11:28:16.180 wine[44040:808186] void *__CFPasteboardAllocatorAllocate(CFIndex) : Lock timeout
2015-10-05 11:28:16.182 wine[44040:808186] void *__CFPasteboardAllocatorAllocate(CFIndex) : Lock timeout
Running tests - 54 failures

Look for a line where the test failures increases. The amount of failures before the test is 53 and after windowscodecs:tiffformat runs, there are 54.

This is where the "windowscodecs:tiffformat" test failed. Adding a skip to that test looks like this in wt-daily's configuration file:

email="your email address here"
tagmac="a-tag"
descmac="A description of your system for Wine developers"
excludemac="windowscodecs:tiffformat"

I need more than one test skipped to get to that 50 or less mark so my configuration file ends up looking like this:

email="caron@codeweavers.com"
tagmac="CWtestbox004"
descmac="OS X 10.11 dual GPU AMD Radeon R9 M370X and Intel Iris Pro"
excludemac="windowscodecs:tiffformat windowscodecs:converter user32:msg gdi32:dc dwrite:font"

And it worked:

CWtestbox004 on WineTest

This is cause for great celebration. Not only is WineTest running on my system, it's reporting on WineTest. I made several Macs run and after a few reports came online, those watching the results come in realized that the name "CWtestbox###" was really horrible for identifying which box was which at a glance. We needed to summarize the box in 20 characters or less and still be able to tell who was who among our Macs.

This is the information we wanted to know at a glance:

  • Where is the report coming from?
  • Which type of Mac?
  • What hardware is onboard?
  • Which OS version?
  • Which test ran; the Mac Driver or the X11 driver?

My boxes are reporting from CodeWeavers, so "cw-" is the first piece of the new name. That will be consist from each box that comes from systems I setup. Then, the type of Mac. We have MacBook Airs, MacBooks, MacBook Pros, Minis, iMacs; we have a decent collection to choose from and setup. We decided we could summarize them fairly easily with "mba-, mb-, mbp-, mini-, imac-".

Then, identifying the hardware. We needed to know something about the hardware. For us, the easiest and most informative way to describe the Mac seems to be the graphics card vendor and number of the most advanced card in the system: "a###-, i###-, or n###-"; as in AMD, Intel, Nvidia.

We needed to know what version of the Mac operating system was in use. This is simply expressed as the .## version. Mavericks is 10.9 so it would be 9, Yosemite is 10.10 so it would be 10, Snow Leopard is 10.6 so it would be 6.

Finally, we have the option to run with the Mac Driver or the X11 driver in these tests. We left this simple, either "m" or "x".

Now, the same system above is expressed in the test results pages as:

  • cw-mbp-A370M-11-m
  • cw-mbp-A370M-11-x

This suggests I got X11 to run with WineTest in addition to the Mac Driver. This is true. Getting those results to run is as easy as adding some lines to the configuration file:

email="caron@codeweavers.com"
tag="cw-mbp-A370M-11-x"
tagmac="cw-mbp-A370M-11-m"
desc="OS X 10.11 dual GPU AMD Radeon R9 M370X and Intel Iris Pro with the X11 driver"
descmac="OS X 10.11 dual GPU AMD Radeon R9 M370X and Intel Iris Pro with the Mac driver"
excludemac="windowscodecs:tiffformat windowscodecs:converter user32:msg gdi32:dc dwrite:font"

There is a set of lines for running the Mac Driver and for running the X11 driver. It means that wt-daily will run two sets of tests, back to back. Sadly, this will not be enough to run WineTest on the X11 driver because X11 crashes during a run on several of the ddraw tests and in user32. This is consistent on all of the Macs we've setup to date and with the crashes, WineTest cannot continue, so we skip those tests with an exclude line:

email="caron@codeweavers.com"
tag="cw-mbp-A370M-11-x"
tagmac="cw-mbp-A370M-11-m"
desc="OS X 10.11 dual GPU AMD Radeon R9 M370X and Intel Iris Pro with the X11 driver"
descmac="OS X 10.11 dual GPU AMD Radeon R9 M370X and Intel Iris Pro with the Mac driver"
exlude="ddraw:ddraw1 ddraw:ddraw2 ddraw:ddraw4 ddraw:ddraw7 user32:win"
excludemac="windowscodecs:tiffformat windowscodecs:converter user32:msg gdi32:dc dwrite:font"

With those tests skipped, WineTest will successfully run both the Mac Driver and X11 driver.

This still isn't enough for a beer... because I'm human and I'll forget to run the tests. It is enough to get results on the test page.


About Caron Wills
Caron has been working in the computer software industry for over 10 years. She joined CodeWeavers in 2008 and became the Quality Assurance Manager for CodeWeavers in 2009. Contact Caron at caron@codeweavers.com and learn more about her professional accomplishments on LinkedIn.

About Caron Wills

Caron has been working in the computer software industry for over 10 years. She joined
CodeWeavers in 2008 and became the Quality Assurance Manager for CodeWeavers in 2009. Contact Caron at caron@codeweavers.com and learn more about her professional accomplishments on LinkedIn.

About CodeWeavers
Founded in 1996 as a general software consultancy, CodeWeavers focuses on the development of Wine – the core technology found in all of its CrossOver products. The company's goal is to bring expanded market opportunities for Windows software developers by making it easier, faster and more painless to port Windows software to Mac and Linux. CodeWeavers is recognized as a leader in open-source Windows porting technology, and maintains development offices in Minnesota, the United Kingdom and elsewhere around the world. The company is privately held.

The following comments are owned by whoever posted them. We are not responsible for them in any way.


New Comment

Eveline
Eveline
Posted 2015-10-07 22:13
Mavericks (note the "k") happens to be 10.9 :)
....just saying....
<ducks and runs away>
Caron Wills
Caron Wills
Posted 2015-10-08 08:50
Eh.

You are correct, of course. Mavericks is 09, Yosemite is 10. I'll update that. My apologies.
Please Wait...
eyJjYXJ0IjowLCJ0enMiOi01LCJjZG4iOiJodHRwczpcL1wvbWVkaWEuY29kZXdlYXZlcnMuY29tXC9wdWJcL2Nyb3Nzb3Zlclwvd2Vic2l0ZSIsImNkbnRzIjoxNTIxMDQwODU5fQ==