HOME | BENCHMARKS | FORUM| CONTACT |
 
DAWbench - Reference Benchmarks :.

   

Cross Platform DAW Performance - Part III :

   

With the Steinberg specific testing out of the way, I wanted to address some concerns that were raised by several punters across forums that the reports were biased due to the well know fact that Cubendo performed measurably better on Windows over OSX, despite the fact that I clearly stated in the first report that it was simply a starting point and that we needed to start somewhere. I was initially going to move to Protools LE/MP/RTAS , but on giving it further consideration I decided to instead move this round of testing to applications more in line with Cubendo to see if the performance variable was actually specific to the Steinberg applications themselves.

The other cross platform applications most similar to Cubendo that used
VST Plugins and ASIO/Core Audio were Presonus StudioOne and Cockos Reaper , both were relatively fresh to the scene compared to Cubendo, but had already developed good reputations cross platform. I also expanded the testing to 3rd party plugins only to give as even a playing field as possible.

 


Preparing for Battle :.    

No changes or adjustments were required in regards to hardware, however the expanded DAW application list and added 3rd party plugins proved to be the biggest hurdle preparing for this round of testing as I needed to ensure that not only all of the respective applications were updated , stable and running well, but also all of the plugins performed as expected across the 3 applications.

There were definitely a few quirks that needed to be resolved, one of the more serious being an ongoing issue with the WaveArts MD5 and StudioOne in OSX where the session would randomly bomb on loading - reporting a stream of errors about presets not being able to be created , very frustrating to say the least - which ended up being a resource allocation issue between the app and the plugin

 

I reported that to the dev's at Presonus who have acknowledged the issue and are reportedly trying to resolve it with collaboration from WaveArts. I have not had further confirmation on whether that has or can be resolved

StudioOne OSX could also be very particular with naming the sessions as well , having a point in the name - i.e. 1.5, would cause it to truncate the name of the session on first save for example. Not a major issues, but annoying when simply trying to navigate around basic functionality.

There was also a looping glitch with Reaper on OSX that we were unable to resolve before print unfortunately, but it has been reported to Justin over at Cockos and a possible solution is currently being worked on.

 

DAW Benchmarks:
DAWbench DSP.

DAW Application Details :
Cockos Reaper : Version 3.6.x
Presonus StudioOne : Version 1.5.x
Steinberg Nuendo : Version 5.0.x

Reference System Detail:
Intel i7 920 Quadcore/ 2.66 GHZ/
Intel X58 / 6 GB DDR3-PC10600.

Audio Hardware Detail:
RME : HDSPe : Driver 3.0x

O.S Detail:
Windows 7 x64 / OSX 10.6.2

 

dawbenchdsp-multi-md5

Round 1 : DAWbench DSP - WaveArts MD5 Multiband Compressor - VST2

Lets take a closer look at the cross platform comparative results for each respective DAW application.

Nuendo 5 : @ 032 - performed better on Win7 by 240% , @ 064 - performed better on Win7 by 57% , @128 - performed better on Win7 by 21%, @ 256 - performed better on Win7 by 6 %

StudioOne 1.5 : @ 032 - performed better on Win7 by 89% , @ 064 - performed better on Win7 by 68% , @128 - performed better on Win7 by 57%, @ 256 - performed better on Win7 by 45 %

Reaper 3.6 : @ 032 - performed better on Win7 by 47% , @ 064 - performed better on Win7 by 16% , @128 - performed better on Win7 by 20%,
@ 256 - performed better on Win7 by 19 %

On Windows the MD5 , as noted on the previous report , had always been a very stable and consistent performer in past testing , no matter what DAW application was being used , and again the results across the 3 respective DAW's this time maintained that .

 

Reaper scaled a little better across all latency settings than both StudioOne and Nuendo 5 , which is consistent with past results, but there really isn't a lot separating the 3 applications.

On OSX the results were not as clear cut across the 3 DAW's. At 032 samples both StudioOne and Reaper easily outperformed Nuendo, but as the latency was increased StudioOne dropped off quite substantially while both Nuendo and Reaper continued to scale significantly better , especially above 064 samples.

Reaper was a bit of a stand out on OSX at the lower latencies, but the empirical results do not show the whole story. I encountered a small glitch as the playback approached the loop point, which is the identical issue I had with Reaper on Windows a few years back.

It is associated with the read ahead buffering and Anticipative multithreading routines of the application, and although the issue in Windows was resolved in collaboration with the dev's at the time , we couldn't pinpoint a solution this time around.

 

This has effected the end results for Reaper on OSX as I needed to back off a number of plugins to clear the slight glitch at the loop point, despite playback of the session apart from that being 100% clear.

Considering the cross platform comparative performance of Reaper on this session is already substantially better than the other 2 applications - even with the results being hampered by the loop glitch - it has the most potential to get closer to even par performance across both OSX and Windows at low latency, as long as that can be maintained with the other 3rd party plugins

We do need to note that despite the potential, the performance variable between Windows and OSX , especially at the lower latencies , is at varying degrees present in all 3 applications, so its not simply a case of pointing the finger at Steinberg's coding, as some have continued to maintain since the initial report.

     
dawbenchdsp-multi-csp
Round 3 : DAWbench DSP - URS Channel Strip Pro - VST2

With the Channel Strip Pro being one of the new kids on the block for DAWbench, this was the first time I had tested across multiple applications , so again, lets take a look at the results for each respective DAW application.

Nuendo 5.0 : @ 032 - performed better on Win7 by 201% , @ 064 - performed better on Win7 by 84% , @128 - performed better on Win7 by 48%,
@ 256 - performed better on Win7 by 31 %

StudioOne 1.5 : @ 032 - performed better on Win7 by 103% , @ 064 - performed better on Win7 by 104% , @128 - performed better on Win7 by 85%, @ 256 - performed better on Win7 by 81 %

Reaper 3.6 : @ 032 - performed better on Win7 by 40%, @ 064 - performed better on Win7 by 47%, @128 - performed better on Win7 by 47%,
@ 256 - performed better on Win7 by 49 %

 

On Windows the results were again very consistent and close to par across all 3 applications , with the results across all latencies being within 5 plugins. All scaled very well , nothing really to report past that

On OSX, the results again were a bit of a mixed bag across the 3 applications, but a few patterns were starting to emerge

Like the previous MD5 results, @032 both StudioOne and Reaper clearly outperformed Nuendo , but StudioOne then quickly dropped off at all latencies above that - consistent to the last test session. Nuendo's and Reaper again scaled more effectively, but not to the extent that they did with the previous plugin.

We do need to note that the Reaper results are still being hampered by the loop glitch.

 

Quick analysis shows that all of the applications on OSX did not scale as well with this particular plugin compared to the previous, possible variables being the multiple and therefore more complex DSP processing required , or simply better code optimization on OSX for the previous plugin.

What is again evident and is remaining consistent , is that the performance variable between Windows 7 and OSX across all the applications is being maintained.

This does not rule out that code optimizations for the respective DAW's are not consistent across both platforms , nor that the optimization for the respective plugin's themselves are comparable , but there is a pattern emerging the more elements we introduce into the testing .

         
dawbenchdsp-multi-emp
Round 4 : DAWbench DSP - Elysia mPressor - VST2

As I noted in the last report, the Elysia mPressor adds another variable, that being an inherent delay, so it would be interesting if that effected the results in the other 2 DAW app's that hadn't been tested previously.

First off, the respective results across the 3 DAW applications

Nuendo 5.0 : @ 032 - performed better on Win7 by 237% , @ 064 - performed better on Win7 by 98% , @128 - performed better on Win7 by 71%,
@ 256 - performed better on Win7 by 71 %

StudioOne 1.5 : @ 032 - performed better on Win7 by 136% , @ 064 - performed better on Win7 by 96% , @128 - performed better on Win7 by 70%, @ 256 - performed better on Win7 by 97 %

Reaper 3.6 : @ 032 - performed better on Win7 by 61%, @ 064 - performed better on Win7 by 52%, @128 - performed better on Win7 by 48%,
@ 256 - performed better on Win7 by 49 %

 

As on the previous CSP results, on Windows the results were again close to par across all 3 applications , all latencies being within 5 plugins. The inherent delay may have come into play more so with Reaper as it was a touch below the other 2 app's on this session, which is different to all the others, but we are realistically splitting hairs trying to separate the Windows results.

On OSX the patterns that were emerging in the pervious runs were evident again with the results for StudioOne and Reaper being clearly better than Nuendo at 032 samples, but then dropping off dramatically while both Nuendo and Reaper scaled measurably better at the higher latencies.

The overall scaling with the mPressor is a lot closer to the Channel Strip Pro than the MD5, so my initial thoughts of the multiple DSP processors being a factor in the CSP not scaling as well doesn't seem to be panning out.



 

An analysis of the results maintains the consistency that has developed over the course of the testing where the variable between OSX and Windows 7 is measurably in favour of the later - actually it has widened in this specific instance which may indicate that plugins with inherent delays do take a harder hit on OSX.

Of course that is not conclusive by any stretch, we would need to test a bunch more plugins to get a clearly idea on that specific.

All of the testing thus far has indicated that there is a distinct variable between the cross platform performance of the current testing pool, whether that shows a consistency in the lack of optimization for all the DAW's and plugins on OSX , or whether the actual O.S and its inherent subsystems come into play is the question that remains.

 

         
Conclusion :

I have tried to address as many of the initial qualms that were raised by several members of the community in respect to the performance variable being specific to Steinberg products.

With the variable remaining consistent with the other 2 DAW applications to some degree , that absolves Steinberg of being the only party that has a distinct cross platform performance variable , but of course then raises a whole bunch of other questions , is it specific to VST Plugins, is it related to Core Audio , is each application equally optimized for both platform, etc, etc

There was speculation doing the rounds on some of the forums that Steinberg did not have native Core Audio support, that in fact it used a wrapper , and that was one of the main causes of the performance variable.

This started circulating after a poster had analyzed an error dump and saw reference to a component called CoreAudio2ASIO.bundle , and of course came to the conclusion that Steinberg were using a wrapper, which accounted for the extra 2 cycles per buffer setting that I noted in my first report.

 

Well needless to say unless the other applications are using the same approach , then that assumption isn't really holding much water. Also my information regarding the extra cycles being required by Core Audio had nothing whats ever to do with Steinberg and its alleged wrapper, but was specifically related to Core Audio itself.

On further investigation , information from a reliable source confirmed that there was a related component in the Windows version of Cubendo, and that it is simply an interconnect between the Steinberg BAIOS audio engine and the respective audio driver for each platform

Of course all of the above can be taken with a grain of salt because it will be near impossible to get a Steinberg developer to openly discuss the details of that component, but I thought it was worth noting.

One qualm that will no doubt be raised again is that I have left out Logic in this round of testing, some even go as far as suggesting that no Apple testing is valid without including Logic ?

 

Considering that my focus is Cross Platform DAW Performance , its needless to say that Logic misses a vital component of the required criteria , also as I detailed in the first report , its hybrid engine makes a head to head comparative using the methodology near impossible. I will do some specific testing and analysis at a later date, but it is not a priority for me.

The next report is covering Protools LE/MP Native scaling, which shifts the testing to RTAS v VST , ASIO/CoreAudio v DAE. With all of the recent rumblings about PTHD Native and the rumours surrounding PT Native 9 ( which is apparently replacing both LE and MP ) , it will be a good springboard for doing head to head with the newer versions as they come on line , as well as some head to head with the competing plugin and driver formats.

Vin Curigliano
AAVIM Technology
October 28 2010

Part I | II | IV | V | VI

© AAVIMT 2010