ASUS Z87 Pro desktop + GTX780Ti@16Gbps-TB2 (AKiTiO Thunder2 via ThunderEXII) + Win8.1

[DrNewcenstein]

So I first heard about e-gpu *somewhere* about a year or so ago and of course Google led me here, where I read a few posts about various implementations and ideas and such, and then went off to contemplate and research it more, and to wait around for it to catch on enough for it to be "a thing" you could just buy and be done with.

 

Obviously, aside from Alienware and whoever else's proprietary idiocy, it's not "a thing" you can just go and buy.

 

So I read some more, but the one thing different about my situation was that I wanted to implement this on my desktop, not a laptop, for 3D rendering (Daz Studio with iray).

As well, I wasn't using a Mac, so there were certain things about it that simply did not apply to my situation.

 

I have an ASUS Z87 Pro motherboard, which doesn't have Thunderbolt built-in, but does support it via the ASUS Thunderbolt EX II add-in card.

I picked up an Akitio Thunderbox 2, and noted that it was aimed at Mac, according to the box. Still, I tried. I had to take the guts out of the enclosure because a GTX 980 is too big for it.

I skipped the powered riser thing and just got the Akitio and a PSU to power the card.

 

At any rate, since I didn't know about the paperclip trick to hotwire the PSU, and it wouldn't turn on without being attached to a motherboard, the whole thing sat on a shelf for about a year while I looked around a bit more.

 

Today I decided to give it one last try, and for the last several hours I've been doing just that.

 

I dug an old tower out of the closet and put in my PSU, hooked it to the mobo so it'd power on, and hooked up the Akitio.

 

I was able to get it showing in Device Manager, and of course got Code 12. However, rather than use a command prompt to hack my system into allocating resources, it dawned on me that I have plenty of available resources already being used by things in my motherboard I don't use. Bluetooth, WiFi, RealTek HD audio, etc. These are useless to me, since all my gaming sound comes from my GTX 980, and I'm hardwired to the router. As well, the on-board Intel HD graphics stuff is going unused.

So, in Device Manager, I uninstalled these items, which of course freed up resources.

 

I did make a few BIOS edits in regards to those items in particular: disabled onboard audio, onboard video, onboard wifi and Bluetooth. I also set the PCI speed to Gen2 instead of Auto (read that somewhere today).

The Thunderbolt BIOS options unlocked only after the TB add-in card was installed, and there were a few tweaks I made there, as well. Notably, Enabling the Option ROM for TB during POST (it's worded backwards - the Disabler for it is Enabled, so I had to Disable the Disabler, which Enables it). There was another one, just above that I believe. I'd have to go back and see what it was, but basically it enabled resources for TB or somesuch. It didn't work by itself when I first tried it, so I know that alone wouldn't do it. I don't know if it helps, though.

 

Not sure which one of those is and is not necessary, but upon reboot, I no longer had Code 12, Daz Studio, EVGA Precision X, GeForce Experience, and NVIDIA Control Panel see 3 cards - the 2 internal (980 and 780 TI) and the external (another 780 TI) - and I'm rendering noticeably faster (10 minutes faster) with all fans blazing away.

 

The instructions for the TB add-in card state to connect the Display Port of the card to the DP of the MOBO, which I did at first, but then moved it from the MOBO to the external GPU. I figured that was just so the MOBO video is routed to the TB card, which would then be connected to a TB display. Ergo, connect the video card that's actually being used. That part probably doesn't matter anyway since I'm only using the egpu for rendering power, not video output.

 

I'll have to go into the BIOS and write down the exact settings if anyone's got a similar setup and similar issue, but basically the TL;DR is that you can avoid using the DSDT edit trick for Code 12 by uninstalling mobo features that you're not using and disabling them in the BIOS.  

 

I'd take a picture, but it's a fire hazard of noteworthy proportions.

Edited July 13, 2016 by nando4
Title

[DrNewcenstein]

Ohhh I get it now, an admin must be editing the title to cover the system specs. Makes sense.

 

However, the 980 is not on the Akitio. Lemme 'splain:

 

Since it's a desktop system and not a laptop, the external monitor thing is a given. The dual 780 TIs are used for 3D rendering, and the 980 is for primary display and of course gaming, since there's no way to tell a game "process through this card, but output through this one" as you can with 3D software.

 

At the start of this latest round of "trying to make it work", the 980 was in the Akitio, and the dual 780 TIs were in the tower, as the tower only has room for 2 double-width cards.

However, since I knew I was not going to feed video to the monitor out of the external GPU, the 980 went back into the tower, and one of the 780s went into the Akitio.

 

So it's  ASUS Z87 Pro desktop + GTX780TI@16Gbps-TB2 (AKiTiO Thunder2 via Thunderbolt EXII) + Win 8.1

 

For my next trick, I'll see if my ASUS NV550JV laptop can interface with the 780TI egpu for mobile rendering power, since it has a TB port. However, the lack of a proper housing for the card and not being able to drag a PSU around with me might actually make that a bad idea. Still, if I know I'm going somewhere, like work, where I can find a wall socket and plenty of space...

[DrNewcenstein]

UPDATE: Today I got in a Titan Z, and replaced the 780 TI on the Akitio box with it. I thought that since I was removing one card from the system, that it would just replace it in the Resources. Wrong! I had to disable even more resources on top of what had already been disabled.

 

Since each card has its own Audio functions, each Audio function gets assigned a set of Resources via the PCI bridges. It was not enough to disable/uninstall the Audio functions of the two cards I'm not using for Audio (the Titan and remaining 780 TI), I had to uninstall their PCI Bridges.

 

In Device Manager, select View>Devices By Connection and you'll find the Audio functions and their bridges easily. Be aware that there are PCI bridges parented to PCI bridges, which look like:

PCI Bridge

>PCI bridge
>>PCI bridge for Audio function
>PCI Bridge

>>PCI bridge for display and other functions (CUDA processing, etc)

 

If you uninstall "PCI bridge" you lose the card entirely, so what you'd want here is to uninstall the first ">PCI bridge" which leads to the ">>PCI bridge for Audio function". You'll free up those resources which the new card will immediately consume, and then it'll work fine. If not, uninstall another PCI audio bridge (remember, each card in the system has one! Keep the one for the primary display card and uninstall the others)

 

[DrNewcenstein]

Further along the rabbit hole, I now have the following setup:
Same base PC as stated before (CPU and RAM and MOBO)

Titan X Pascal - Primary display in PCIe Slot 0

GTX 980 in Slot 1

Titan Z on Akitio Thunderbox via ASUS ThunderEX II PCIe card

 

I'm still using this setup primarily for GPU-based rendering in Iray in Daz Studio, and it works.

However, I'm learning as I go, and have found that the Titan Z, which is 2 6GB cards with 2880 cores each, doesn't appear to be functioning fully. In fact, using GPU-Z and Nvidia Inspector, these apps only show half the card. When rendering, the Titan X's VRAM usage goes as high as 10GB, but the Titan Z can't get above 4GB.

Unfortunately, I wasn't as knowledgeable about any of this when I first slapped it together and declared success, so I can't tell if this is a recent issue or if it's been this way all along.

 

I know Akitio says this box is not for GPUs, but since that hasn't stopped anyone else here, I wasn't letting it stop me.

 

I've also just run LuxMark, but I'm not finding other Titan Z results to compare mine to, I don't think. I found one entry that lists 2 Titan Zs, and judging by the VRAM numbers, it does look like it's simply listing each half of the card as opposed to being two physically-separate cards, but there are no system details given.

On the whole, it does appear my Titan Z is not running at full capacity, though as I said, I don't know if that's due to the Akitio box, or the Thunderbolt add-in card.

 

[DrNewcenstein]

aaaaand after further examination, the Titan Z was indeed only running at half capacity. Apparently, when freeing up Resources in Device Manager, disabling the Audio portion of the card took half the card with it.

 

This only appears to affect dual-GPU cards, though, as my 980 was not negatively affected by disabling its Audio function.

 

However, to get the other half of the Titan Z to work, I did have to remove the 980 entirely, and still spent hours assigning Resources and reloading drivers. I've also got an HDD "missing", and it looks like one of my CPU's PCI bus controllers is permanently disabled.

I could be facing a complete teardown and rebuilding from the ground up here.

 

I've got an Amfeltec eGPU cluster coming, and that will hold the Titan Z, 980, and my 2 780TIs, so hopefully I can get my system straightened out by then.