SLI Sync Limit reached (redux)

[Nofew]

Hello! Sorry for being away for so long! Life happens!

 

I tend to get odd hardware bugs that nobody else on Planet Earth appears to get. I've been struggling with one issue for the past year and a half now, at least, and the last hope I had to fix it appears to have retired.

 

I'll cut to the chase: I have three Titan Black cards by Asus, stock vbios. (I'd /love/ a custom one and that might fix the issue I'm having, but none appear to exist anywhere on T|I.) Currently I only have two in the case, but knowing that I have three at my disposal is important. I'll explain why shortly.

 

Out of the blue one day, I couldn't get above a few hundred FPS when benchmarking with SLI turned on. At the surface, that's not too interesting. I get that. It gets downright *weird*, though.

 

At all times during these events, I've only had two GPUs installed at the same time. The first thing I thought was that I had a bad SLI cable, so I ordered a new one and tried that. No dice. Tried swapping the position of the cards, no luck. Tried ordering a third GPU and trying every combination of two cards in case one was broken, but all of those combinations had the same result.

 

Things got weirder when I started paying more attention to the numbers, reporting tools and did some math. When running at 2560:1440, I get about 300 FPS. It doesn't fluctuate; it hits a cap. The number is incredibly steady; even the decimal point only moves by .1 between frames and hangs within .1 of where it settles after 3Dmark gets going for more than a second or two. The scene's complexity doesn't matter -- I can be running Ice Storm at the most basic settings or Cloud Gate, and it'll still get stuck at that limit. Single GPU, Ice Storm hangs around 1,600-1,800 FPS depending on my CPU clock.

 

My first thought was that the cards weren't using the SLI cable anymore and were using the motherboard instead, thus saturating the PCIe lanes. It turns out MSI Afterburner can measure them, and this indeed appears to be the situation SOMETIMES; PCIe bandwidth goes to 50% (I assume this is really 100%) during some benchmarks where it used to hang around 2-5%, but tests like Fire Strike don't saturate it.

 

Upon seeing this of course the issue seems obvious, but there's one major inconsistency that might rule this out: Even in cases where the PCIe bus isn't saturated, two cards always perform worse than one card now. Even during Fire Strike Extreme, where I get maybe 30 FPS with one card, I used to get 60 FPS with two cards. Now it's around 26 with two, and the PCIe lanes aren't saturated.

 

Seeing this, I decided to see if MSI Afterburner could tell me more. I've always had every reporting option turned on, and I usually know what they look like. One changed dramatically: "SLI Sync Limit". I don't know what this is and Googling it never turns up useful results, and there's not even any documentation I've found that explains what this is in any appreciable detail. All I know is, until things started going weird, it was always at 0. Never changed. Ever. Now, whenever I run anything in SLI except the Windows desktop, it goes to 1. It seems to be a binary indicator since I've never seen it go anywhere else.

 

Seeing as PCIe saturation might be a factor here, I decided to try and reduce stress on the PCIe bus. One of my thoughts was that the cards were sending their framebuffers over PCIe rather than SLI, so I changed the resolution. Lo and behold, changing it from 2560:1440 to 1920:1080, which is roughly half the pixels if you do the math, brought the framerate cap up to about 600! Changing it to 720 brings it to around 1,200. I'm fairly convinced they are indeed sending their framebuffers over PCIe at this point, and that that's causing some pretty major issues.

 

 

Now, I know nVidia used to support doing SLI without an SLI cable with a driver setting; head to the control panel, pick an option from a dropdown or something, apply, OK, all happy. It'd just make the cards talk via PCIe rather than SLI, and it worked. They eventually removed this option from the control panel, but after emailing nVidia about the issue it turns out that the feature still exists, but it's hidden from users.

 

So, my first thought was that I somehow toggled the option in some fashion. I tried reinstalling windows, downloading fresh drivers, no overclocks, no nothing, and the same issue presented its self right off the bat. This is where I'm stumped.

 

I've tried everything that I could think to get my hands on -- I've updated my motherboard's BIOS, I've done a microcode update for my CPU (i7-3930k), I've tried using different GPUS (the third Titan Black I mentioned, not other models), different screens, HDMI, DisplayPort, older drivers, newer drivers, beta drivers.. None of it works. Any useful help that I haven't already gone over would be appreciated! (Context: 

Spoiler

I asked here for help last year, but nobody smart came along; I say that because they asked me questions that I already answered in my first post. I think about three people replied and basically said "Go update your drivers and reinstall windows lol" -- Really? x.x... This is TechInferno, not Yahoo Answers. x.x Pay attention, people!

)

 

 

EDIT: Huh! Weird! I just got the idea to try running the benchmarks windowed! Weird stuff happened! With SLI off, fullscreen, I get about 1,400 FPS during Ice Storm. With it on, fullscreen, about 338. With it on, and windowed, about 1,200! PCIe still goes way up but SLI Sync doesn't trip. It's still worse than one card by its self (Which never used to be the case; I almost broke 2,000 once!), but there's something to running things fullscreen vs running it windowed!

 

PS: It's not 3Dmark. This happens with every benchmark and game out there, with the same limit of 338.

 

 

EDIT2: I found https://www.reddit.com/r/techsupport/comments/3lrjhy/sli_sync_limit_is_spiking_from_0_to_1/ . Appearntly SLI Sync Limit appears even on non-SLI setups! Seriously, what /is/ this thing?!

 

 

EDIT3: It seems to be related to GPUBoost 2.0. I finally found a custom vbios over at http://www.overclock.net/t/1503215/titan-black-bios-and-tweaking and SLI Sync Limit doesn't trip anymore, but somehow the framerate is even worse now; I can only get about 177 FPS on Ice Storm at this point, and PCIe is still going nuts. What the heck!? Also, I apparently achieve a stable 1,150 mhz at 1.05 volts. From everything I've read I shouldn't be able to get that under 1.212 volts. Either I've got the most golden sample _ever_, or something's being reported inaccurately. I'm not sure if this is related.

Edited January 2, 2016 by Nofew
EDIT3