Rethinking Flex Data Management - part 3

This is part 3 in a series dedicated to the subject of Flex Data Management. In the first post I gave an overview of FDMS, the second part talked about some of the challenges and limitations associated with the FDMS approach. Now let's talk about WebORB Data Management for Flex (WDMF) - a new and innovative framework we have been working on. (WDMF will become available in the WebORB Professional/Enterprise distribution in late March/early April timeframe.)

What is WDMF? It is a new data management framework we are adding to WebORB. The framework is designed to do one thing extremely well - make developer lives easier creating data-driven Flex applications. Our to goal is to empower developers with tools and technologies so they can focus more on solving core application problems and less on writing integration or 'plumbing' code.

WebORB Data Management for Flex consists of:

1. Design-time code generator integrated into the management console.
   
2. Runtime data management framework. This is a set of APIs and events available on both client and server sides.
   
3. Object relational mapping engine.

The code generator accepts a configurable data model which may consist of one or more data tables from one or more databases. Generated code includes a set of ActionScript v3 classes as well as all the supporting backend code. The framework includes rich support for all fundamental database operations (CRUD), as well as the ability to dynamically compose data retrieval queries using the dynamic method composition technique. The technique has been borrowed from the Ruby on Rails ActiveRecord pattern and elegantly adapted to the ActionScript environment.

The code generator automatically compiles and deploys the generated server code. The code for the client can be expanded from the generated ZIP into a Flex Builder project and immediately put to use. No additional configuration changes or custom coding tasks are required. The object model is very simple and intuitive. A row for every table from your data model is represented with a class. The class provides access to all basic data management operations. For example, consider the following table:

Among other classes supporting the framework, there is a generated ActionScript class representing a row from the Customers table would have the following structure:

public class Customer extends ActiveRecord
{
public var CustomerID:int;
public var CompanyName:String;
public var ContactName:String;
public var ContactTitle:String;
public var Address:String;
public var City:String;
public var Region:String;
public var PostalCode:String;
public var Country:String;
public var Phone:String;
public var Fax:String;
}

The class above and all its instances become the central point for any operation related to the Customers table. For instance, the APIs to perform customer searches, create new customer records, delete or modify existing records are accessible through the Customer class. Additionally, as a client makes a change or deletes a record, WDMF notifies all other clients in possession of the same record about the modification, thus realizing client synchronization.

The API to work with remote data is simple and very easy to use. For example:

retrieving a list of all customers:
ActiveRecords.Customer.findAll();

retrieving a list of all customers from 'Tokyo":
ActiveRecords.Customer.findByCity( "Tokyo" );

retrieving a list of all customers from USA with the contact title as "Owner":
ActiveRecords.Customer.findByContactTitleAndCountry( "Owner", "USA" );

creating a new customer:
var newCustomer:Customer = new Customer();
newCustomer.CompanyName = "Acme Brick";
newCustomer.ContactName = "James Bond";
newCustomer.ContactTitle = "Chief Development Officer";
newCustomer.City = "Dallas";
newCustomer.save();

retrieving a record by primary key and deleting it:
var c:Customer = ActiveRecords.Customer.findByPrimaryKey( 123 );
c.remove();

It is important to note that the operations shown above will automatically be propagated to all other connected clients which happen to contain the same Customer record. Additionally, none of these examples require any modifications in the generated server code. However, the server code establishes a very flexible framework support supporting a variety of customizations.

There are a lot of cool features I plan to cover in the subsequent posts, including support for data binding, extensibility , transactions and security. Until then, check out the FAQ:
http://www.themidnightcoders.com/weborb/dotnet/wdmf-faq.shtm

Also, if you are attending the 360Flex conference, I will be demonstrating WDMF during my session next Monday between 4:00 and 5:20 in the Integration track.

Upcoming Flex events - where to meet us

There are several upcoming conferences we will be participating in. Here's a list of events to meet us at and the presentations to attend:

• 360Flex Conference
  San Jose, CA, March 5-7, 2007
  Harris and I are traveling to the conference and will conduct two sessions: "Building Flex Apps Faster with WebORB" on March 5th and "Flex and .NET" on March 6th.
  
  
• Spring </ br> Conference
  Athens, OH, March 22, 2007
  The title of my session is "Data-driven Flex applications in 5 minutes". I will be talking about and demonstrating our new (absolutely fantastic) data management system for Flex client applications.
  
  
• Flex Maniacs
  Washington DC, June 25-26, 2007
  I will be presenting on the subject of "Flex without Java". The session will focus on Flex integration with .NET, PHP and Ruby on Rails.

Technorati tags: Flex, WebORB, Flex conference, Flex seminar

rethinking Flex Data Management - continued

This is part 2 of the series of posts dedicated to Flex Data Management and our implementation of the functionality. In part 1 I gave a high level overview of various FDMS moving parts, their configuration and some basic client-side API. To review, a typical FDMS solution would include the following:

1. Managed data - traditional RDBMS or an in-memory data store.
   
2. ValueObject (VO) class - a class representing a row in the targeted data table.
3. Data Access Object (DAO) class - a class responsible for the execution of the basic data store operations (CRUD), materialization and serialization of the value objects.
4. Data assembler - an interface between FDS and the user's data management application on the server-side. Typically a class encapsulating high level operations requires by FDMS.
   
5. Destination configuration - contains a client-side facing representation of the data assembler as well as metadata used by FDS to enable client synchronization and some additional features.
6. Client-side ValueObject class - maps directly to the server-side ValueObject (2). Represents a row of managed data.
7. Application specific client-side data management logic - this is the code you add to handle view updates, synchronization and conflict resolution events.
8. Generic client-side logic data management logic - encapsulated in the form of the mx.data.DataService class as well as some auxiliary classes and APIs.

The diagram below demonstrates how these components relate to each other:

As you can see, FDMS is a fairly complex system with a lot of moving parts. The system is designed to provide an end-to-end data management solution with some very powerful features like multi-client synchronization, conflict resolution and support for the disconnected mode. As I mentioned in the previous post, WebORB provides a basic implementation of the Flex Data Management functionality. Since we introduced support for FDMS last year, we have been running into multiple scenarios where it was rather hard to explain and justify the benefits of the approach shown above. The resistance is mostly due to the following factors and limitations:

1. Significant development effort. A developer has to write a lot of supporting code: server value objects, DAOs, assemblers, port the server VOs to the client side (which may need to be done by a client-side developer). Indeed, from the list shown above, the following items are developers responsibility: 2, 3, 4, 5, 6 and 7.
2. Lack of programming flexibility. We hear this one over and over. Data driven rich internet applications DO require some flexibility to create custom data management operations. Consider a scenario where a single request from a client must update multiple data stores, perform a select query, find a record and link it to some other data object. Oh, and all of this must happen in the context of the same transaction. It may be possible to achieve all this, but it is definitely not straight-forward.
3. Non-expressive API. Client-side interface with the FDMS system is encapsulated mostly in the mx.data.DataService class. The class is rather compact and includes support for all basic data management operations. Since the API is centralized in a single class, it makes it less intuitive and not as expressive. It especially becomes obvious for anyone familiar with the frameworks like Ruby on Rails' ActiveRecord.

With these limitations in place, if you were to put side-by-side a Rails developer and a Flex server developer tasked to come up with a similar solution for their respective environments, the Rails developer would be much further along in the same amount of time. The main reason is Rails does a lot more to get you started and then let's you stay super productive. There is no reason why Flex developers cannot enjoy the same.

So we had an idea to make something that would nicely fit the Flex programming model, provide the same great features available in FDMS, address the problems shown above, offer the best features available in most popular data management frameworks like Rails's ActiveRecord and Hibernate and best of all be available for all server platforms and programming environments. The next post will go over what we came up with for our new data management solution for Flex. Stay tuned.

Technorati tags: WebORB, Flex, FDMS, Flex Data Management

Breaking the 10,000 RTMP connection limit - WebORB Mach 3

We have been running scalability tests for the Enterprise Edition of WebORB. We started with a simple setup - multiple RTMP clients on the same computer as the WebORB server. Even though the setup is not representative of a real-world scenario, it is good to see the server operating in a more stringent environment where the resources are shared. This pushes us to come up with a better server solution which is not hungry for memory and CPU.

Shortly into the test we ran into an interesting limitation where new connections would be rejected at 3976 concurrent connections. Turns out Windows allows ports 1024-5000 for the short-lived TCP-IP connections by default. The difference between these numbers is 3976! Fortunately there is a way to overcome that limitation. There is a KB article on the Microsoft's site describing the solution.

Once we increased the limit, the server could easily reach the goal we have established internally - 10,000 concurrent RTMP connections. The next step is to calculate how multiple connections correlate to the performance overhead and add a monitoring panel to our Management Console.

Technorati tags: WebORB, RTMP, Flash messaging

Flex and .NET Integration FAQ

We just published an FAQ page for WebORB for .NET. The page addresses most frequent questions we answer on daily basis. The plan is to evolve the FAQ into a technical knowledge base for many Flex and .NET integration topics. You can access the FAQ at:

http://www.themidnightcoders.com/weborb/dotnet/faq.shtm


Technorati tags: WebORB, Flex, Flex and .NET

Let's talk about Flex Data Management...

We have been very busy working on a very cool and exciting subsystem for WebORB. The new module is our own implementation of the Data Management Services functionality from FDS. We already implement a subset of Flex Data Management Services, but decided to create an alternative. The new implementation is very different and, in my opinion, friendlier and easier to use from a developer perspective. Before I delve into the new data management system, let's review how FDMS works. Since WebORB already implements a subset of FDMS, the overview below applies to both WebORB and FDS unless noted otherwise.

One of the core concepts in FDMS (and the same is true with Remoting and Messaging) is the destination. A data destination represents a collection of operations applied to a data entity. A data destination is configured in WEB-INF/flex/data-management-config.xml and encapsulates a lot of important information:

1. Server-side class responsible for processing data fetching (fill) and data synchronization (create, update, delete) operations. The class deals with value objects representing destination's entity.
   
2. Identity description. Provides a definition of the entity properties representing entity's identity.
3. Association information. Destination's entity may be correlated to other entities. This configures the relations and cardinality. (Note: this applies only to FDS, WebORB's FDMS implementation does not support associations).
   
4. Paging configuration. Specifies if the data returned by (1) is pageable and configures page size.
5. Data management function declarations for (1). This is an optional configuration item. If (1) does not implement a particular interface, fill and sync methods must be declared in the destination.
   

Below is a sample data destination definition taken from the WebORB distribution:

<destination id="contact">
<adapter ref="dotnet-dao" />
<properties>
<source> Weborb.Examples.DataManagement.ContactAssembler </source>
<scope>application</scope>

<metadata>
<identity property="contactId"/>
</metadata>

<network>
<session-timeout>20</session-timeout>
<paging enabled="true" pageSize="2" />
<throttle-inbound policy="ERROR" max-frequency="500"/>
<throttle-outbound policy="REPLACE" max-frequency="500"/>
</network>

<server>
<fill-method>
<name>loadContacts</name>
</fill-method>

<fill-method>
<name>loadContacts</name>
<params>System.String</params>
</fill-method>

<sync-method>
<name>syncContacts</name>
</sync-method>
</server>
</properties>
</destination>


Configuration shown above defines the "contact" destination. The destination is mapped to the Weborb.Examples.DataManagement.ContactAssembler class. The class contains the loadContacts and syncContacts methods responsible for loading data from the server and performing data updates (see the source code below):

[ReturnType( "samples.contact.Contact" ) ]
public object loadContacts()
{
ContactDAO dao = new ContactDAO();
return dao.getContacts();
}

public void syncContacts( ChangeObject change )
{
if( change.IsUpdate )
doUpdate( change );
else if( change.IsCreate )
doCreate( change );
else if( change.IsDelete )
doDelete( change );
}

private void doUpdate( ChangeObject co )
{
ContactDAO dao = new ContactDAO();
dao.update( (Contact) co.NewVersion, (Contact) co.OldVersion );
}

private void doCreate( ChangeObject co )
{
ContactDAO dao = new ContactDAO();
Contact contact = dao.create( (Contact) co.NewVersion );
co.NewVersion = contact;
}

private void doDelete( ChangeObject co )
{
ContactDAO dao = new ContactDAO();
dao.delete( (Contact) co.OldVersion );
}

As you can see from the code above, all top level functions exposed through the destination delegate to a Data Access Object (DAO) class. The DAO class is the one that implements the actual database integration, includes all the queries and connection management. It is also responsible for entity object materialization and serialization.

The diagram below shows the relationship between the entity class (Contact), DAO class (ContactDAO) and the class exposed through the data destination:



Now that you have an idea of what's going on on the server, let's review the client-side:

The key component on the client is the mx.data.DataService class. The class does not come with the Flex SDK and is available only with the FDS distribution. To connect to a data destination create a DataService instance as shown below:

// connect to the "contact" data destination
var ds:DataService = new DataService( "contact" );

The client side uses ActionScript counter part to the server's Contact class. When the server code returns a Contact object, it is mapped to a client-side representation of the same value object.

To retrieve a list of contacts from the server, use the fill() method:

ds.fill( myCollection );

The method call above populates the myCollection object (which must be an instance of mx.collections.ListCollectionView) with a collection of contact objects returned from the server's loadContacts() method shown above.

To create a new contact record, instantiate the client-side Contact class, populate it with data and then invoke DataService's createItem as shown below:

var myContact:Contact = new Contact();
myContact.name = "James Bond";
myContact.phone = "5551212";
ds.createItem( myContact );

To delete an existing contact use the deleteItem method();

ds.deleteItem( myContact );

The DataService API is quite rich and offers a lot of possibilities. You can see the API documentation here.

This should give you a high level overview of the Flex Data Management Services. In the next post, I will describe the issues with the FDMS approach and the reasons for creating an alternative implementation in WebORB.