Giovanni Mascellani
Hello world!
First time writing here, so let's start with a little bit of introduction! My name is Giovanni, and I live in Italy with my wife and my little daughter. In a past life, about a year ago, I was a mathematician who enjoyed a bit of Wine hacking every now and then in his spare time. In November 2020 I started working at CodeWeavers, and made Wine development my daily job. It is a joy to work with great people, and develop free software for the benefit of everybody.
This morning, out of curiosity, I looked back to see what I accomplished this past year. It turns out I've touched many different areas of Wine and made a lot of games work better in CrossOver and Proton.
I know Wine code looks somewhat magical to the untrained eye; sometimes it seems magical even to software developers! After all, it’s a complex piece of software with hundreds of libraries. I remember how puzzled I was when I first uncorked Wine and became familiar with the innerworkings. Rest assured there is nothing mystical about it, just Wine Developers patiently crushing grapes (and bugs). Read on, and you’ll find out about some of my discoveries in the vineyard.
We have an internal bug tracker which, well... tracks bugs from various sources (e.g., the public Proton GitHub, or CodeWeavers’ phenomenal QA team). My task is simple: go there, pick one bug and beat it until it gives up.
Risk of Rain 2, aka “My very first bug”
The first bug I worked on was for Risk of Rain 2. The game crashed when users tried to enter a multiplayer lobby. Debugger at hand I set to work. Very quickly I saw that this was due to a NULL pointer dereference, and that it was related to lsteamclient. What is lsteamclient? Well, Wine is a program that straddles two worlds. On the one hand Linux (or macOS), which is the host operating system, and on the other the emulated Windows environment. When a Steam game is launched, it contacts the Steam program to have a number of services, for things like saving achievements or running multiplayer. But the game is inside the Windows world, and, when using Proton, the Steam program runs in the outer Linux world. lsteamclient sits across the two worlds, taking the requests back and forth between the game and Steam.
There are times a game provides Isteamclient a callback, which Steam uses to signal that something has happened (e.g., a remote user initiated multiplayer). In the case of Risk of Rain 2, the game was providing a NULL callback. This is not invalid: it simply means that the game doesn’t want to be called back for some specific kind of events. In turn, lsteamclient, assuming the callback was valid, tried to call a NULL pointer. Calling a NULL pointer is, of course, illegal and led to a crash. As is often the case, once the problem is identified, the fix is simple. There you have it, my very first Proton patch.
XCOM: Chimera Squad, aka “One Wine bug down, a gazillion more to go”
Take Wine, add a few local patches and a dab of magic glue and you have Proton. This means that bugs in Proton are often bugs in Wine. So after debugging and fixing some problem with Proton, chances are that the patch is to be sent to the upstream Wine project, and then added to Proton. This was the case with a bug I worked on in the XCOM: Chimera Squad launcher. The launcher worked fine with Proton 5.0, but was nothing but a black window in Proton 5.13. How did I find the dastardly bug responsible for this travesty, and dispatch it from my sight? With one of the most important and powerful tools available to the mighty Wine Hacker, the logging system (if you wanna hack… you gotta learn it!). By enabling the appropriate channels, you can see every step the application took and where it failed.
Looking at the logfiles for this bug, I noticed a thread was stuck (waiting for an event that wasn’t firing) in the winegstreamer library. As its name suggests, winegstreamer uses the popular GStreamer library to provide video processing services to the game (e.g., if the game wants to decode a video it calls the API provided by Windows for video processing). Wine implements those APIs by forwarding them to GStreamer and then returning the result to the game. In this case, winegstreamer asked GStreamer to initialize a video decoder on behalf of the application and waited for the decoder… and waited… and waited. No response. While winegstreamer was waiting for the all clear for the video decoder, it wasn’t noticing that GStreamer was signaling an error, so winegstreamer remained stuck... waiting for the green light, and forever missing that big red light. Then it was just a matter of putting together a small patch, sending it on to Wine and pushing it to Proton (thus benefiting vanilla Wine, CrossOver, and Proton users -- three birds with one stone!). Sounds simple? Well, yeah, the patch was simple. But it took days of work gathering information to pinpoint the bug in the first place.
F1 2020, aka “When the bug is not in Wine”
Believe it or not, my dear friends, but sometimes the bug is not in Wine (or Proton). Sometimes the bug is in the game itself. Sometimes the game does not conform to the library specifications it is calling. But, alas, we have no power over the game. We don't have access to its source code, and no way to ask the developers to change it. Fear not, for Wine does not break, it bends, and in these cases we bend Wine to accomodate what the game doesn't do properly. Look at this F1 2020 glitch.
You can easily see some artifacts on the left side of the character, both in the big picture and in the bottom left preview. This took quite a lot of time to debug: graphics cards draw 3D scenes by running lots of tiny programs called “shaders”, which compute the color of every single pixel in the frame. One shader will, for instance, take into account the textures of the drawn objects and the positions and intensities of the lights. The output of this shader is the exact shade of red to each individual pixel belonging to, as in the picture above, a character's jacket.
What was happening here, I discovered painfully reading the compiled shader code, was that close to the edge of the character some lights are not being taken into account by the shader, which clearly result in a darker color. But that’s not enough to solve the issue: why was the shader failing to take some lights into account in the first place? I discovered there was another shader, that precomputed which lights were relevant for each part of the screen. For performance reasons, you don’t want to go through all the lights when you know that some of them are not relevant; therefore the game considers the screen divided in many tiles, each 16x16 pixels large, and first runs a shader that inspects a tile and decides which lights are to be computed, and then runs a second shader that actually does the computation (this also explains why the glitch has this jagged shape). But again: why was that happening? Sometimes investigating a bug means going through a long chain of whys, until you find the root cause.
In order to complete its task a shader is executed in parallel (once for each pixel). Sometimes one execution must wait for a result to be provided by another parallel execution. This only works properly when a synchronization operation occurs between the value being generated and the execution requiring that value. And what if a synchronization operation goes amiss? Well, what happens can depend on many factors, like the driver, the operating system and the graphics card itself. The NVIDIA and AMD drivers on Windows do well to hide the effects of a missing synchronization operation, but the Linux drivers do not. This doesn’t mean that the drivers are buggy: they have the right to expect a synchronization operation in such a situation. It’s the shader which is faulty here.
The game is responsible for the bug and we cannot fix the game, so we come up with a hack to do what the game is not. Once I found what the issue was, I discovered it was already known. A workaround (aka “hack”) to put additional synchronization points in place for games that were missing them had already been deployed for DXVK, the component that (among many other things) converts DXBC formatted Windows shaders to the SPIR-V Linux format. Since this kind of hack can get in the way of other properly behaved programs, they are usually only enabled for games that require them. In the end, the fix was as simple as adding F1 2020 to the list of games requiring this hack.
Talk about an emotional roller coaster! I spent a lot of time researching this bug, and I was very excited when I found the solution and wrote a patch. A patch that, frustratingly, was for naught, because a hack to correct the bug already existed. So that’s a lesson to learn: you can travel a hundred miles and still stand where you are.
I hope you enjoyed this jaunt through the vineyard. There are tales of hard work and toil where these came from, so let me know in the comments if there is a bug you’re curious to learn more about.
We're always hiring talented developers for Wine, CrossOver and Proton. Strong C language skills are a must. To learn more about our Wine Developer / Wine Hacker / Open-Source Programmer position, please visit our jobs page.
About Giovanni Mascellani
Giovanni lives in Parma, Italy with his wife and two little daughters. In a previous life he was a mathematician. Now he is a Wine Hacker, and since he joined CodeWeavers in November 2020 he regularly goes hunting for bugs.
