Blog to discuss Midnight Coders products, features, ideas and trends in development of Rich Internet Applications

Thursday, March 08, 2007

Flex RemoteObject vs WebService benchmark

It is not an earth shattering discovery that Flex/Flash Remoting is faster than WebServices. Pretty much everyone knows that (although there are still a few who try to prove the opposite). The real question is how much faster. We set up a simple benchmark test comparing invocations of the same C# class as an http service, SOAP web service and a remoting service exposed via WebORB. There are three benchmark tests, each invoking one of the following function in the backend object in a tight loop:

// argument and return type - primitive value - string
public String echoString(String str)
{
return str;
}

// argument and return type - complex type
public Car echoCar(Car carObj)
{
return carObj;
}

// argument and return type - array of complex types
public Car[] echoCars(Car[] cars)
{
return cars;
}

The Car class is defined as:
public class Car
{
public String make;
public String model;
public int mileage;
public String driverName;
}

The client side code reuses the same function to issue invocations for RemoteObject and WebService objects, since both extend from the AbstractService class. This results in testing of the pure performance for the underlying plumbing.

And now, here's the live benchmark application (http/web services, beg for mercy, cuz RemoteObject is about to beat the heck out of ya):


You can download the code for the example and run it locally. Make sure to download WebORB for .NET so you can expose the backend class as a remoting service (client code, server code). The webservice WSDL is available here.

Technorati tags: , ,

15 Comments:

Blogger Dave said...

Is it really that big a difference though? If you run one invocation, which is the more likely real world application scenario (how many screens require 100 remote service calls which is the default number of invocations in this example) the results come out something like this.

[ComplexType]
1) WebService/HTTP Service = 328ms
2) RemoteObject = 349ms

[PrimitiveType]
1) WebService/HTTP Service = 372ms
2) RemoteObject = 283ms

So in the best case RemoteObject is about 100ms faster on average for a single service call, or about 1/10 of one second. In a more real world example using a ComplexType they are actually about the same (well WebService comes out faster but not statistically different).

Yes, if I up the number of invocations I can magnify this difference and the magnification isn't linear. That isn't surprising either since we now have a thread concurrency/thread synchronization effort going on which adds a multiplier to the delta as the VM context switches all the threads.

The reality is in most application the delta isn't even recognizable by the user. Heck, I don't think most drag race drivers have 1/10 of a single second reaction times. The net effect is they both perform about the same

What these tests DO draw out is the huge impact having an API that is out of fidelity with the user experience has as it requires the UI to manage and context switch multiple threads in parallel when the UI requires multiple service calls to render a single view. This is why we/I go on and on about the importance of the fidelity of the UI with the SOA and why we espouse a Front to Back development approach.

Don't take this out of context though. There are as of Flex2 *really* strong advantages of using RemoteObject especially with CompelxTypes with regards to strongly typed stubs and the work required to typecast WebService calls. However I continue to contend the delta in performance is negligible and insignificant compared to the impact of a strong alignment between the experience and the API.

4:24 PM

 
Blogger Mark Piller said...

Dave, the difference is huge. Granted if you're building a "hello world" app with a single call, it will not make any difference whether you use RemoteObject or WebService [you'd probably spend 2x time developing a webservice-based app, but it might be a good thing for the billable hours :)]. Indeed, who cares if an invocation takes 20ms or 200ms??? However, in a real-world scenario where an application could have hundreds of clients, it will make a lot of difference. Just knowing that a server (and clients) spend so much more time processing the requests would not make me feel comfortable delivering such a solution to a customer. Imagine a trading desk app, where hundreds of messages must be processed concurrently. The 100ms difference you're pointing out will be much more than that, and might result in significant financial losses. Btw, you do not have to up the number of invocations. Just try the same 1 iteration test with the "arrrays of complex types" test. Remoting is at least 4x faster than Web Services for a single invocation. In my opinion that's so much closer to a typical real-world scenario, and btw, I do know a lot about web services (i worked on a SOAP stack for Java in my past life - Glue by TME). Real-world apps create very large soap packets and that's where real the performance degradation with XML serialization kicks in.

5:35 PM

 
Blogger Chaitu said...

If the invocations are decreased to 1 and the length of the string is increased the webservices seems to work faster. Why would that be?

5:41 PM

 
Blogger Mark Piller said...

The first invocation with Remoting is usually slower than all subsequent ones. On average, remoting comes out faster though.

5:49 PM

 
Blogger Dave Wolf said...

Mark,

Thanks for the response. I appreciate it because I think this is a subject filled with more myth and speculation then fact and the tests you posted absolutely do draw out some intrinsic differences and do so with real fact and not theory or guesswork. Its honestly very helpful to the community. Thanks.

I do indeed remember TME and their SOAP stack. You guys were acquired by WebMethods back like 03? In terms of background in this space it is also worth noting that I was the a Principal Architect for Sybase's J2EE app server and the Lead Product Manager for B2B and EAI at MSFT (including as their liaison to RosettaNet & ebXML). There's no doubt we've both had our battle scars from the evolution of remoting!

You are exactly right that the final test is actually the most accurate one at showing off where SOAP struggles the most. The reality is that using SOAP for nothing but remoting is serious overkill. Using a binary encoded packet protocol like AMF makes much more sense. SOAP is a 7bit ASCII protocol that is bulky and its deeply nested structure is a burden on the Flash player. (Worth noting, on the financial app pushing thousands of messages per hour, the vast majority of the time those messages would be small discrete messages not wouldn't resemble returning thousands of rows.....)

Where it becomes a less simple choice becomes a question for the customer and for their business. Back in the CORBA days we had IIOP (which is striking similar to AMF) and yet today SOAP is king. Why? In the end it was a mixture of market adoption, and cross platform, cross business support. If you are building a true enterprise solution you have more than just a Flex/Flash UI and you have trading partners on every platform and using every technology. In the end building a server infrastructure using a protocol that cannot be consumed by a C++ client, or a VB Client or a PB client or a Swing client or WPF/E or whatever might come down the pike next week isn't an option for some classes of applications. If such interop weren't an issue IIOP and RMI and the dizzying array of other binary encoded platform specific protocols would hold the kind of mind share SOAP does today.

What I wanted to point out though is that the blanket statement that SOAP is slow is not accurate. There are a thousand things you can trip up in designing your application that will cause 1000X more performance impact then the wire protocol. We have fortune 100 clients including several of the largest banks in the nation that are using Flex applications with SOAP based services in production today and they scream. That said, we have many projects we work on where AMF is the wire protocol and for the most part we drove the choice of AMF.

I guess to do a better job at getting my main point out, making a blanket statement saying that SOAP is slow and AMF blows its socks off isn't entirely accurate nor a complete answer.

8:39 PM

 
Blogger Jamie said...

Exposing my ignorance a little here - does FDS use Remoting or something else ? If something else, would it be possible to add it as a fourth contender ?

Jamie.

3:09 AM

 
Blogger Nate said...

I agree with Dave.

For the complex object example you should reformat your objects and put them in a simpler array in the C# webservice. When I did this on my flex app you could visibly tell a huge difference. The code is simple too, about 5 lines. A simple copy and paste can add it to any service.

I would like to see the tests redone with this in mind.

-Nate

11:22 AM

 
Blogger Mark Piller said...

Nate, what is a "simpler array"? How can the array in the current implementation, (which is just a Car[]) can be any simpler??

11:27 AM

 
Blogger dorkiedorkfromdorktown said...

a lot of people don't see the 200ms to 15000ms as much a speed advantage until they put it to use in an organization that gets 1 million hits a day. that bandwidth can save the company thousands of dollars. with that many users the speed of the applications become apparent. the sooner the server can finish sending the stream the sooner it can get on to another service call. i agree at very very small isolated singular calls, which another poster mentioned, the advantages aren't as apparent but thats not real world. real world is everyone tapping the server on and off over many different lines of communication. that speed advantage remoting brings really is appreciated in real world applications.

11:04 PM

 
Anonymous Anonymous said...

Is the point here that the technologies are chosen for a range of advantages/disadvantages? Speed is but one factor. If speed is the ultimate design requirement, perhaps remoting is best. If interoperability is the design requirement then web services allow greater flexibility at a performance hit.

Most design decisions are usually more complex than a 'one or the other' approach.

I also suspect that the authors preference to flex is the motivating factor in favour of these tests. You could just as easily argue that a desktop app is faster than a web app, therefore why use web apps at all, however once again, the design considerations probably rule it out, so a balanced decision is required.

A straight performance test isn't usually so helpful unless it is seen as one small item in a greater picture.

7:59 PM

 
Blogger turbidity said...

Hi. Thanks for this post and for the article. I thought I'd let you know that the RemoteObject tests are returning this error to me: "got fault Send failed". The other two types of tests are working just fine.

3:10 PM

 
Blogger William Cook said...

I'm curious what you think of this:

http://www.cs.utexas.edu/~wcook/Drafts/2006/WSvsDO.pdf

If you use remote objects with typical OO design approaches, you can actually get a less scalable solution than web services. It has to do with the possibility for a "document oriented" model in which services. Your benchmark doesn't take this into account.

2:46 PM

 
Anonymous Anonymous said...

There's some interesting questions raised here.

1: Are the SOAP web services using document/literal or RPC/literal? It matters because the resulting data sets can look very different, be of dramatically different sizes, and require more or less parsing effort on the client side. This will have a (possibly dramatic) influence on performance.

2: Exactly what server software is generating the responses at the back end, and how fast is it? If the back end is slower at generating the responses then this might lead to the appearance of other factors in the test appearing to be slow. Ideally the back end for web services should be GSOAP which is an incredibly fast back end implementation.

3: Compression is everything for tests like this. Compressed versus uncompressed timing tests are worlds apart. There's no information here about whether the responses are compressed or not. Uncompressed SOAP/web services tend to be fat and slow. A real world application probably should be using GZIP compression of responses, and uncompressed responses end up being very large which would skew these results. Is there a way to identify if compression is on? Even better would be to include an option to try each test with and without compression.

It's a very interesting test but without more detail it wouldn't be right to draw conclusions about the performance of soap web services versus AMF.

Thanks

Andrew

9:42 PM

 
Anonymous Anonymous said...

Thanks for the response. I appreciate it because I think this is a subject filled with more myth and speculation then fact and the tests you posted absolutely do draw mp3 indir out some intrinsic differences and do so with real fact and not theory or guesswork. Its honestly very helpful to the community. Thanks.

4:14 PM

 
Anonymous Anonymous said...

Another consideration is the enabling of larger result sets to the client. This allows us to take advantage of local sorting, filtering, and other manipulations. And while shipping and handling performance is a critical factor, don't forget the time it takes to populate a grid, for example. In my observations, when the number of rows gets above 1K, the XML result sets start to exhibit serious slowdowns in population of a grid, while the AMF performance seems unaffected. I've tested with 5K rows and found XML to take over a minute while AMF data was ready in a few seconds. Ultimately I assume the reason is that the AMF data is immediately ready for use while XML data still requires heavy parsing work.

1:27 PM

 

Post a Comment

<< Home