SOAP vs Flash Remoting Benchmark

SOAP vs. FLASH REMOTING BENCHMARK

NOTE: The following benchmark was executed before Macromedia released Flash 7.01 updater. The update has presumably fixed the memory leak described in the article. We are in the process of verifying if the update indeed addressed the memory leak issue. We will post our conclusion as soon as the re-evaluation is complete.

Introduction

With the recent release of Flash MX 2004 Professional Edition and the SOAP support built into the Flash player, there is a lot of questioning whether Flash Remoting became obsolete. Indeed, it appears that Macromedia did not do anything to enhance the Flash Remoting feature, but instead made a significant stride to beef up support for SOAP web services. The noticeable presence of the WebServiceConnector component as well as the API for programmatically invoking web services makes a lot of developers wonder whether Flash Remoting is overshadowed by SOAP now. Anyone familiar with both approaches would admit that the core difference between them would be performance, though products such as FlashORB add a variety of features that significantly enhance the Flash Remoting approach. In this article we analyze the results of a set of benchmark tests designed to demonstrate the performance differences between SOAP and Flash Remoting.

Test Setup
Server-size Service Class
Benchmark Client Application
Client Design
Tests Results
Benchmark Observations
Conclusion

Test Setup

The tests were designed to demonstrate system performance for transferring simple type, complex type, array of simple types and array of complex types. Additionally a set of tests was included where a combination of the above listed types was transmitted in the same test run. Each test run was executed first against the SOAP service and then the same test was ran against a Flash Remoting service. While a test was running, the benchmark client measures the performance of the client-server communication and reports results like "total message throughput" and "number of messages per second". The same class was exposed as a web service as well as Flash Remoting service. FlashORB was selected as the Flash Remoting hosting server and Apache Axis was selected as the SOAP stack selected for the SOAP tests.

Why we chose FlashORB: The most obvious reason is that we developed FlashORB and we strongly believe that it is the fastest Flash Remoting server on the market (a set of benchmarks that were ran internally, clearly supports our claim).

Instructions for setting up Flash Remoting service with FlashORB

Why we chose Apache Axis: Axis is one of the most commonly used SOAP stack implementations. It is written in Java (just like the Remoting opponent) and is running under the same environment constraints as FlashORB.

Instructions for setting up SOAP web service with Axis

Server-side Service Class

The server-side service class includes a set of methods corresponding to each type of test:

echoString - this method accepts a string object argument and returns the same string object
echoPerson - this method accepts a data structure representing a person. Each data structure has a set of fields. The method then returns the same object received as an argument
echoPersonArray - this method accepts an array of Person objects and returns the same array.
echoArray - this method accepts an array of string objects and returns the same array.

Below is the source code for the class exposed as a SOAP web service and Flash Remoting service:

// copyright (c)2003 Midnight Coders, LLC
package benchmarks.flashorb;

/**
* <tt>TestService</tt>
*
* @author <a href="http://www.flashorb.com">Midnight Coders, LLC</a>
*/

public class TestService
{
public String echoString( String string )
    {
    return string;
    }

public int echoInt( int integer )
    {
    return integer;
    }

public Person echoPerson( Person person )
    {
    return person;
    }

public Person[] echoPersonArray( Person[] person )
    {
    return person;
    }

public String[] echoArray( String[] array )
    {
    return array;
    }
}

// copyright (c)2003 Midnight Coders, LLC
package benchmarks.flashorb;

/**
* <tt>Person</tt>
*
* @author <a href="http://www.flashorb.com">Midnight Coders, LLC</a>
*/

public class Person
{
public String name;
public int age;
public String streetAddress;
public String city;
public String state;
public String zipCode;
}

The WSDL document describing benchmarks.flashorb.TestService exposed as a SOAP web service can be found here.

Benchmark Client Application

The benchmark client application consists of five sections, each corresponding to one of the methods in the remote service. Each section has one or more fields to configure the argument(s) for the appropriate method. For example the "echo Int" section has a text field where the number for the argument can be entered. The "echo Person Array" and "echo String Array" sections use the values from the "echo Person" and "echo String" sections for the actual data, but they also contain a field where the size array can be entered. Each section has a check box, which if selected specifies that the method should be executed in the test run. This way one can easily select the methods that should be invoked in a test. At the bottom of each section, there is a test summary area. The "Messages Total" fields provide information about the number of messages sent and received during a test. The "Messages/Second" fields automatically calculate the throughput for the running test. At the very bottom of the client UI, there are two buttons responsible for starting a test. The "Test Duration" field can be used to set the duration of the test.
Only one test can be executed at a time - once a test is started, no other test can be executed. The recommended approach is to run one type of a test first (SOAP or Flash Remoting) and once it completes, execute the other type of the test for the same test duration and method selection.

Client Design

The client application contains an invisible movie clip (invoker) responsible for running a test. In the first frame of the invoker movie clip, 2 proxies are created: one for the SOAP web service and the other for the Flash Remoting service:

#include "NetServices.as"
import mx.services.WebService;

// create web service proxy
var service:WebService = new WebService( "http://localhost:8080/axis/services/TestService?wsdl" );

// create flash remoting proxy
var gwConnection = NetServices.createGatewayConnection( "http://localhost:8081/flashorb" );
var frService = gwConnection.getService( "benchmarks.flashorb.TestService", this );

stop();

Frame 1 of the Invoker movie clip

As soon as one of the buttons kicks off a test, button click handler delegates the execution to the second frame of the invoker movie clip, which checks the methods that should participate in the test. For each selected method an operation is added to an array:

import mx.services.PendingCall;

// create a new array to contain the operations for this test
this.servicesToCall = new Array();

// if echoInt method is selected, add an operation to the array
if( _root.testEchoInt.selected )
{
// create a function which will be invoked in the test
var invokeFunction:Function = function( service )
    {
    // this takes place when the function is invoked in the test engine (frame 3)

    // if this is a soap test, execute the method on the web service,
    // otherwise, execute the function on the flash remoting service
    if( _global.soapTest )
      {
      // execute echoInt on the web service and assign return handler
      var callObj:PendingCall = service.echoInt( _root.echoIntField.text );
      // assign return handler, so we can count the number of processed messages
      callObj.onResult = echoIntCallback;
      // increment the number of the invoked messages
     _global.echoIntCallsSent++;
      }
    else
      {
      // invoke the method on the remoting service. return handler is set in frame 1
      service.echoInt( _root.echoIntField.text );
      // increment the number of the invoked messages
      _global.echoIntFRCallsSent++;
      }
     }

// add the operation to the array
servicesToCall.push( invokeFunction );
}

// repeat similar steps for all other operations
....
....
....

Frame 2 of the Invoker movie clip

Once the servicesToCall array is set up, the Invoker movie clip enters a tight loop (frames 3 and 4). The code continuously invokes methods on the remote service (SOAP or Flash Remoting), until the time in the test duration elapses.

// while the test is running, continue invoking methods
if( _global.running )
{
// invoke each function is the servicesToCall array.
// make sure to pass the right service object into the function call
for( var i = 0; i < this.servicesToCall.length; i++ )
this.servicesToCall[ i ].call( null, _global.soapTest ? this.service : this.frService );
}
else
{
// stop once the test is done (or interrupted)
stop();
}

Frame 3 of the Invoker movie clip

Frame 4 of the Invoker movie clip sends the control back to frame 3. Full source for the benchmark client can be downloaded here.

Tests Results

A number of tests were executed to identify the performance of both of the approaches. The tests (and the client application) are designed in a way that would demonstrate raw performance of transferring a homogenous data type (primitive, complex or array) or a combination of data types. Each message invoked on a remote service includes both transferring an argument to the service (serialization) and receiving the same object as a response (deserialization). Below is a description of each of the executed tests with the analysis of the results. Each test was executed with the client and server on the same computer to avoid network latency. However this could have impacted the results since both client and server compete for the same CPU and memory. For your convenience, we host a live web service and Flash Remoting service on our web site so you can run the benchmark at your convenience (coming shortly).

Test Method(s)

Test Description

echoInt
test duration - 60 sec

In this test a single integer is passed an argument and is returned by the service. With both SOAP and Flash Remoting the size of the on-the-wire message is relatively small, though when compared to each other, the SOAP message is proportionally larger. The results of the test are shown below:

	SOAP	Flash Remoting
Messages Sent	3727	3895
Messages Received	3727	3895
Messages/second	62	65
Memory allocation	Before the test - 26.5Mb After the test - 110Mb	Before the test - 26.1Mb After the test - 26.7Mb

echoString
test duration - 60 sec

In this test a string object is passed into a remote method as an argument and is returned by the service back to the client. For smaller strings, the size of the SOAP message is significantly larger, but as the size of the string argument increases, the difference in size between SOAP and Flash Remoting on-the-wire messages becomes insignificant.

String size - 20 bytes

	SOAP	Flash Remoting
Messages Sent	3701	3896
Messages Received	3701	3896
Messages/second	62	65
Memory allocation	Before the test - 19.5Mb After the test - 103Mb	Before the test - 19.4Mb After the test - 20Mb

String size - 200 bytes

	SOAP	Flash Remoting
Messages Sent	3796	3893
Messages Received	3796	3893
Messages/second	63	65
Memory allocation	Before the test - 19.3Mb After the test - 106Mb	Before the test - 19.3Mb After the test - 20Mb

echoPerson
test duration - 60 sec

In this test a complex type (Person object) is passed as an argument into a remote service method and is returned from the method as a return object. The Person class is rather ‘simple’ complex type. It has 6 string fields, however the results of this tests should be rather indicative of the performance with even more complex data types.

	SOAP	Flash Remoting
Messages Sent	3474	3893
Messages Received	985	3893
Messages/second	16	65
Memory allocation	Before the test - 19.3Mb After the test - 194Mb	Before the test - 19.3Mb After the test - 20Mb

echoArray
test duration - 60 sec

In this test an array of string objects is passed to the remote service and is returned by the remote service. The array size and the size of the string in each cell of the array are configurable.

Array size - 20 elements

	SOAP	Flash Remoting
Messages Sent	3010	3894
Messages Received	1138	3894
Messages/second	19	65
Memory allocation	Before the test - 20.5Mb After the test - 292Mb	Before the test - 19.1Mb After the test - 21Mb

Array size - 2000 elements

	SOAP	Flash Remoting
Messages Sent	87	1215
Messages Received	87	1215
Messages/second	< 2	20
Memory allocation	Before the test - 19.1Mb After the test - 330Mb	Before the test - 21.8Mb After the test - 22Mb

echoPersonArray
test duration - 60 sec

In this test an array of 20 elements each containing an instance of Person class is sent as an argument to a remote service. The service returns the argument back to the client as a return type.

Array size - 20 elements

	SOAP	Flash Remoting
Messages Sent	535	3871
Messages Received	535	3871
Messages/second	9	65
Memory allocation	Before the test - 19Mb After the test - 282Mb	Before the test - 19Mb After the test - 21Mb

echoInt
echoString
echoPerson
test duration - 60 sec

In this test 3 operations were repeatedly invoked in the same frame for the entire duration of the test. This test is meant to demonstrate performance when both primitive types and a complex type are passed to a remote service.

	SOAP	Flash Remoting
Messages Sent	1298	3885
Messages Received	1295	3885
Messages/second	22	65
Memory allocation	Before the test - 18.8Mb After the test - 110Mb	Before the test - 19Mb After the test - 20Mb

echoPerson
echoArray
echoPersonArray
test duration - 60 sec

In this test 3 operations were repeatedly invoked in the same frame for the entire duration of the test. This test is meant to demonstrate the performance of invoking remote methods and passing large data structures.

Array sizes - 20 elements

	SOAP	Flash Remoting
Messages Sent	302	3876
Messages Received	302	3876
Messages/second	5	65
Memory allocation	Before the test - 20 Mb After the test - 250 Mb	Before the test - 19Mb After the test - 20Mb

Benchmark Observations

Flash 7 SOAP implementation has a significant memory leak. The duration of each of the executed tests was set to 60 seconds. During the execution of a test, the memory allocated by the browser (IE 6.0) has increased between 400% (echoInt test) to 1700% (echoArray test).
Tests passing just primitive arguments or strings had comparable performance between SOAP and Flash Remoting approaches. This can be explained by the rather simple format of the on-the-wire SOAP messages for these data types. As the complexity (and size) of the SOAP messages increases, the performance significantly drops. For example, the throughput of passing an array of 20 elements (each containing a Person object) is 7.22 times higher than with Flash Remoting (65 messages/second vs. 9).
The SOAP approach experienced a close-to-failure condition with a very large array of strings (2000 elements). Throughout the test, the browser process became very sluggish and non-responsive resulting in a throughput of less than 2 messages per second. The Flash Remoting approach carried out the same test significantly better, with an average throughput of 20 messages/second.

Test Conclusion

Macromedia has done an outstanding job making its SOAP integration as simple as possible. Our experience using the WebServices APIs was very positive. However, there are several drawbacks in the current Flash Player SOAP implementation. The most noticeable drawback is the memory leak, which makes the approach to be impractical. The Flash Remoting approach is by far more stable and has better performance. The SOAP approach results in larger payloads and greater CPU processing resulting from the XML Schema validation and XML parsing. The Flash Remoting approach benefits from a binary representation of request/response as well as the invocation-batching feature. For any application that requires moderate to high remote invocation volume and/or involving complex data types and large arrays, Flash Remoting is a better fit.

Discuss the article on FlashORB discussion forum