Shadow of Mordor Performance – Resolution Shenanigans and Benchmark Madness

Middle-Earth: Shadow of Mordor (which I'm just going to call Shadow of Mordor from here on) sort of came out of nowhere and has become the sleeper hit of the month. Created by Monolith Productions -- the company behind "classics" like Blood and Shogo: Mobile Armor Division, not to mention more recent titles like F.E.A.R. and F.E.A.R. 2, No One Lives Forever, and Condemned -- I don't know that anyone was really expecting much from yet another Middle-Earth game. It's not that games based on Tolkien are all bad, but there have been so many over the decades and more often than not they've been at best mediocre. Anyway, Shadow of Mordor plays a lot like the Batman: Arkham Asylum/City/Origins games, or the Assassin's Creed games, or probably any number of other stealth/beat-em-up third person games. I've enjoyed quite a few of those titles, and you can definitely add Shadow of Mordor to the list.
While the game itself has received many good reviews, Shadow of Mordor really shakes things up when it comes to benchmarks and resolutions. I'm not going to try and review the game (since others have already done that and I generally agree with them), but let me talk for a bit about the resolution shenanigans going on in Mordor. It's either a brilliant idea or horribly stupid, depending on your perspective -- or perhaps a little of both. I think a major part of the resolution problem also ties directly in with the console origins of the game -- not that it was designed solely for consoles, but targeting the Xbox One and PlayStation 4 can create some interesting compromises that seem to have been carried over to the PC release.

Resolution Upscaling and Downscaling

The crux of the problem with the resolutions that Shadow of Mordor exposes is this: it will always display at your monitor's native resolution, but the game can render at either lower or higher resolutions to improve frame rates or quality (respectively). If you have a 1920x1080 LCD like many people, it may not really matter much, but in my case I test gaming performance at multiple resolutions on a 30" 2560x1600 display. That display lets me test performance at 2560x1440, 1920x1080, 1600x900, and 1366x768, and so far it has never been a problem... until Shadow of Mordor came along. Because my display is natively a 16:10 aspect ratio, the only options available via the resolution scaling are also 16:10 resolutions...and not even common resolutions at that! Want to run 2560x1600? No problem. 1920x1200? Um, sorry; the closest I can get is 2048x1280 or 1792x1120. What, you've never seen a display with those resolutions before? That's because they don't exist!
ShadowOfMordor-Resolutions-(1) ShadowOfMordor-Resolutions-(2) ShadowOfMordor-Resolutions-(3) ShadowOfMordor-Resolutions-(4) ShadowOfMordor-Resolutions-(5) ShadowOfMordor-Resolutions-(6)

ShadowOfMordor-Resolutions-(7) ShadowOfMordor-Resolutions-(8) ShadowOfMordor-Resolutions-(9) ShadowOfMordor-Resolutions-(10) ShadowOfMordor-Resolutions-(11)

2560x1600 Scaling Options (Resized to 1280x800)
The way things work is that Shadow of Mordor has eleven different resolution scaling options. There's of course native (100%) scaling, and for those that want a higher image quality the game can render at 150% or 200% as well and provide super-sampling (SSAA) as the image gets scaled down to your native resolution. It's actually not going to be playable on most PCs with 200% scaling, because CrossFire and SLI support is currently broken, but at some point that should be fixed. The other eight scaling options are 90%, 80%, 70%, 67%, 50%, 40%, 33%, and 20% -- and yes, those percentages are apparently all fixed and so far I have not found a way to override them.
The idea is that if your GPU isn't sufficiently fast to render at native resolution, you can render at a lower resolution and have the game upscale the output to your native resolution. It sounds sort of clever, except in practice it may result in worse image quality and/or lower performance than if you let your display or GPU do the scaling. It also means you're locked in to percentages of your display resolution, so even if you have a high-end display you may not be able to test performance at "standard" lower resolutions.
The good news is that this is really only a problem if you want to compare performance between different configurations. That's obviously something I do a lot, but for your typical gamers the only thing that really matters is how the game runs on their particular system. Is it playable? Great! Not running well? Turn down the quality setting or use a scaling factor (and introduce aliasing/jaggies). There's also a problem if you want to improve image quality, as you can only go so far; a 4K display like the Acer XB280HK (28-inch G-Sync -- yes, I have one now!) will allow up to a rather insane 7680x4320 setting, while a typical 1080p display will only allow up to 3840x2160; my old 30" display meanwhile can go up to 5120x3200. Again, it's not consistent and can be a bit confusing.

Shadow of Mordor - Performance and Analysis

Moving on to performance, I have to shake things up a bit this time. My normal benchmarking resolutions are 2560x1440, 1920x1080, 1600x900, and 1366x768, but I can't run all of those on Shadow of Mordor with my current selection of displays. I ended up using the 28" Acer 4K display and testing at 2560x1440 (66.7% scaling), 1920x1080 (50% scaling)... and then 1536x864 (40%) and 1280x720 (33.3%). As long as I keep the resolutions the same, it's not a huge concern, but do note that the Medium and Low settings in Shadows of Mordor are slightly less demanding than in other games, at least as far as resolution is concerned.

Shadow of Mordor is an NVIDIA TWIMTBP title (The Way It's Meant To Be Played), which means among other things that NVIDIA has likely helped out with some performance optimizations and perhaps some libraries (e.g. GameWorks) to handle certain effects. Unlike the Batman games, there's no PhysX option in Shadow of Mordor, but the NVIDIA help does manifest particularly well in one area: minimum frame rates. Put bluntly, NVIDIA's drivers generally offer stable frame rates while AMD's GPUs with the current 14.9 AMD drivers have a huge amount of jitter. Take a look at the following image and pay particular attention to the zigzag nature of the chart in the top-right. I've also produced a chart showing a short 200 frames sample of FPS from the R9 290X and GTX 980 at our 1080p High settings on the right.
shadow-of-mordor-amd-jitter Shadow-Of-Mordor-AMD-vs-NVIDIA-FPS-Comparison
The net result is that while AMD's GPUs can produce decent average frame rates overall, the constant jitter makes the game feel a bit choppy. If you enable VSYNC that might help a bit, but with frequent drops below 60 FPS what you'll end up with is oscillations between 60 and 30 FPS. It's times like this where the decision of Ubisoft to cap the FPS of The Evil Within at 30 FPS makes a bit of sense, at least as far as providing a uniform player experience is concerned. However, the real problem here is with AMD's drivers; they simply need to be fixed to properly handle Shadow of Mordor and produce reasonably consistent frame rates.
In terms of overall performance, NVIDIA's GTX 980 is at the top of the charts as usual, and without working CrossFire or SLI support it will remain there for a while. In fact, based on the poor minimum frame rates with the Radeon cards, the GTX 970 and GTX 780 are also ahead of the R9 290X at our Ultra settings, and at 1080p High I'd still take the GTX 770 over the current R9 290X experience. Older AMD GPUs like the R9 280X and R9 280 have the same issue with jitter, and until that's fixed you won't have smooth frame rates with any of the AMD GPUs.
And really, that's why we test new releases: to keep the GPU vendors honest. NVIDIA didn't officially release a Game Ready driver for Shadow of Mordor, and SLI support is still broken (though I've heard the problem has to do with how the game is handling rendering internally and that it needs a slight patch to fix both SLI and CrossFire support), but even without a Game Ready driver the GeForce cards provide a much better experience than the Radeon cards. How long will it take for AMD to release a Hotfix driver for Shadow of Mordor? Perhaps a week or two, perhaps a month or more; we don't really know until the driver arrives. When it does, and when SLI and CrossFire are both working, we'll be sure to revisit the topic of Shadow of Mordor performance.
Quickly looking at options for lower end GPUs, the Intel HD 4600 is basically unable to post playable frame rates at our 1280x720 Low setting, which isn't too surprising but it's nonetheless discouraging. Batman: Arkham Origins is in a similar state with HD 4600, and until Broadwell appears next year you really have to have a discrete GPU to play most games properly. AMD's Radeon R7 250X is able to break 30 FPS at both our Low and Medium settings, but minimum frame rates are still below 20 at Medium quality -- the GTX 750 and GTX 750 Ti would be a safer bet right now.
Update: there are two things to address that have happened in the three weeks since we initially posted this story. First, AMD released a beta driver (14.9 Beta 2) that adds mostly working CrossFire support for Shadow of Mordor. Second, we've gone back and tested at 4K resolutions for good measure. Interestingly, the R9 290X ends up as our fastest single GPU, just barely edging out the otherwise more expensive GTX 980 -- in an NVIDIA TWIMTBP title, no less! However, all is not perfect in AMD land; jitter has improved with the beta drivers, but CrossFire minimum frame rates aren't much better than a single GPU, and the game does seem to have some jitter or at least slowing down and speeding up going on. We'll likely need to revisit the game next month when AMD has non-beta drivers out (and when we have time!) and retest both AMD and NVIDIA GPUs to see what if anything has changed.

Shadow of Mordor Image Quality

Shadow-Of-Mordor-IQ1-1-Low Shadow-Of-Mordor-IQ1-2-Medium Shadow-Of-Mordor-IQ1-3-High Shadow-Of-Mordor-IQ1-4-Ultra

Shadow-Of-Mordor-IQ2-1-Low Shadow-Of-Mordor-IQ2-2-Medium Shadow-Of-Mordor-IQ2-3-High Shadow-Of-Mordor-IQ2-4-Ultra

Left to Right: Low, Medium, High, and Ultra
Shadow of Mordor image quality is pretty typical of what we see with modern gaming engines. At maximum quality, even with the HD texture pack, the improvements over Very High are extremely small, and even dropping to High doesn't change things much. There's a drop in shadow map quality that you won't notice much when playing, and at Medium the Ambient Occlusion difference can be seen in quite a few areas; there's also a drop in texture quality as you reduce settings, but in motion it's not as apparent. And then as usual, at Low quality SSAO is disabled and shadows mostly consist of pre-calculated shadows on buildings. There's also a difference in the amount of vegetation generated at lower detail levels, but nothing that would really make or break the game.

How to Benchmark Shadow of Mordor

Shadow of Mordor has a built-in benchmark that's available in the display settings. We use the Low/Medium/High/Ultra presets (with the HD Content Pack installed), and we do our best to target our standard resolutions. However, as discussed above the way things work in Shadow of Mordor means we can't quite test at 1600x900 or 1366x768 -- at least, not without a display that runs those resolutions.
We use a 4K display on the desktop, running at 67% (2560x1440), 50% (1920x1080), 40% (1536x864) and 33% (1280x720). For laptops that have 1080p displays, the scaling options we use are 100%, 80%, and 67% -- which use the same resolutions as on our desktop. If you have something else (e.g. a 3K 2880x1620 laptop like the MSI GS60), 67% will give you 1920x1080, but there's no available scaling option that will give anything close to 1536x864 or 1280x720. The solution (if you're able) is to use an external DisplayPort LCD, assuming your laptop supports one and you have the necessary display, which is what we did.
Beyond the selection of resolution, we use the standard built-in settings: Ultra, High (no Very High), Medium, and Low. The built-in benchmark has some real flaws, however. First, it apparently logs frame rates during the loading screen, and there are voice clips that vary in length that will impact the overall average. Then the first second or two once the real benchmark starts has a lot of jitter as the graphics finish loading, resulting in highly variable "minimum" frame rates.
To get around both problems, we use FRAPS to log the frame rate during the benchmark. We start right as the roof goes out of view in the bottom-right and stop just before the scene finishes fading to black. As with Sleeping Dogs and Sniper Elite 3, we then use Excel to calculate the average FPS of the lowest 3% of frames. Considering all of this extra work, the "realistic" minimum FPS in Shadow of Mordor on AMD cards is particularly bad. Anyway, here are the specific quality settings the game applies at each level:
Shadow-of-Mordor-Settings-1-Ultra Shadow-of-Mordor-Settings-2-Very-High Shadow-of-Mordor-Settings-3-High Shadow-of-Mordor-Settings-4-Medium Shadow-of-Mordor-Settings-5-Low
One other noteworthy addition is the Nemesis system, where leaders of the Orc armies called Uruk are scattered throughout the map that you'll occasionally battle. These are much more difficult "boss" type battles, and they add a nice dynamic to the game. What's particularly entertaining is that the Uruk will come back repeatedly, unless you behead them during your battle, and each time they show more scars than before. Also, if you happen to get killed, the Orc that kills you is automatically promoted to Uruk status and then fights for position in the Orc army. So basically, any time you die, the enemy that killed you levels up; I'm not sure if there's a maximum level for Uruk, but I could see some of them becoming quite tough -- all the more reason to attempt beheadings as much as possible (though reportedly even that isn't guaranteed to work 100% of the time).
Overall, I've really enjoyed playing Shadow of Mordor so far, but again I have to note that it's very similar in terms of game play to the Batman and Assassin's Creed games -- though obviously with a different setting. If you're a fan of the Lord of the Rings and Hobbit movies and books and have dreamed of hacking and chopping Orcs to pieces, this is about as good as it gets. The only real problem is that the built-in benchmark along with the choice to upscale/downscale graphics to your native resolution makes this a less than ideal game for comparing hardware for end users. If that's the worst thing I can say about the game then it's certainly worth a look.

Leave a Reply