Archive for the 'prevayler' Category

Serialization: IExernalizable and Evolving Classes

What is evolving a class? It's the inevitable:

• addition of a field
• change the superclass
• remove a field
• change the name of a field
• change a field to static (from non-static)
• change a field to transient (from non-transient)
• change the type of a field type (e.g. int to float)

As soon as you do any of the above whatever you've previously saved something to disk, be it local or remote will break upon loading back in.

This isn't common in most web based flash development which lives in isolation on a webpage and doesn't save anything, but introduce any sort of saving. With Desktop Flash apps (AIR, Zinc), and RIA's, offline online applications, it IS becoming something that is important.
There are 3 different strategies to serializing objects, only one is particularly useful for evolving classes.

1. Direct: writeObject(yourClassObject);
2. Customize with IExternalizeable: iterate over each attribute of your object and write each of those to the stream.
3. Customize with IExternalizeable, iterate over each attribute of your object write it with a key to a Dictionary / HashTable.

Approach 1 is great for simple uses and first passes. However it doesn't scale well or at all. Approach 2 Is better but as the stream is sequential, this can make dealing with multiple versions difficult or impossible as you have to know the exact order in which to parse. Approach 3 is the best, it uses a Dictionary as an intermediate map, it's only a bit more code to write, and is a *whole* lot easier to parse. This is also the approach that Remoting uses when talking to the server with the ASObject, and in someways writing to disk is identical to writing to the filesystem.

Here is the list of imports:

Here is the version number, we'd need to increase everytime a significant data change has happened, and some of the static register class necessary to get the class to serialize and deserialize correctly:

Here are the modified IExternalizable functions. Note that the read from the stream has to deal with all permutations for a given version to version, as this number of changes gets large, the normalization can be quite large, thankfully good architecture up front can prevent much of this.

AS3 Customized Serialization and Deserialization, IExternalizeable to the rescue!

This and original off Pete's blog is great article on a little known feature critical for de/serialization in flash, and the knowledge is not just for client/server as the  The flex documentation on the subject implies. Anywhere the AMF format is used: shared local objects, local connection, in AIR using fileStream. cloning objects using ByteArray etc. So many gems (might as well be easter eggs!) in the AS3 code, sadly barely documented. It's using IExternalizeable for custom serialization, which mimicks Java's already proven approach.

What's that mean? It means you can better control how objects are written and read back from the byteStream.
What if I don't want X to be written?
To compliment a great follow up on using the [Transient] metadata by Darron, with both the clone and when serializing to disk or the web. I've got to say the metadata format makes sense but also wierds me out.

How do I get my class type back

registerClassAlias is necessary for local cloning and maintaining Class type, when written to a ByteArray they forget which class they belong to. I would have hoped that the RemoteClass metadata would have worked but that was not the case. Anyway you can do this via a static var:

private static const REG:* = registerClassAlias("reflection.IntrospectableObj",IntrospectableObj);

this is called during static initialization, once which is all that's really needed as the linkage is static, not per instance.

What if you need to transcend versions of code?

basic approach to using serialization to deal with different versions of the class is
1) write a version number to the stream first, then a) save to hashmap, or b) iterate over the members that should be saved.
2) read the version number back from the stream, then reconstitue the members from a) the Dictionary (like Remotings's ASObject) or b) in the order they were written to the stream. Vary the parsing response upon the version number.

Summary of Serialization Tips:

1) Use IExternalizable to customize how your objects are written.

2) Use the [Transient] metadata to NOT serialize objects if you don't want them there, and are using writeObject().
3) registerClassAlias is necessary for local cloning and maintaining Class type. You can do this via a static var,
private static const REG:* = registerClassAlias("reflection.IntrospectableObj",IntrospectableObj);

4) keep a unique version number in the code, write a version first in the output stream then when reading it back perform different behavior based on that stream. The use of a Dictionary for key/values can be helpful too.

AS3 and Serialization, Memento Pattern

Starting to work on porting the persistence aspect of Prevayler to Flash/AS3. It's a perfect fit with Flash, as Prevaylence assumes that everthing can fit into RAM (which flash does, aside from loading new things in), and changes have to be synchronized, which seeing as it's a primarily a single thread doesn't have to worry about synchronizing multiple writes.
Prevaylence relies heavily on understanding the serialization of objects, both the commands and the business object state should be save and restoreable. Since ByteArray supports AMF packets (as does shared object) this means that private vars and Class definitions, functions should be able to be written to disk and brought back from dead...but this leaves the larger question, just like the dinosaurs ...should they be brought back?
It's certainly tempting.  Java has faced this for years in it's serialization routines. In general built in serializing is good, but doing it across version is bad, as it's inevitable that the Class definition will change, adding, removing variables and methods ...even changing types, during refactoring. When reconstituting things may not serialize properly much as you don't fit into your GI Joe pajamas from kindergarden (well hopefully). It's unpredictable what the Flash player will do when forced to put version 1.2 into 1.3 shoes, or potentially version 1.4 into 1.2.  So it's important to know exactly what is being saved and how flexible it is.

Since I was working on Trace.me this week, and private, protected, etc variables didn't show up in the output, (which is proper Object encapsulation), I was curious as to if the variables were invisible or lost. I remembered  hitting similar things when working with and on openAMF, so I dug a bit, and just found a document I wrote a few years ago, on openAMF, ah good times...

http://openamf.cvs.sourceforge.net/*checkout*/openamf/openamf/docs/understanding.htm?revision=1.1

There the issue there was different, was actually whether java's accessors e.g. get someValue and set someValue translated across the wire. The serialization in Java openAMF calls the accessors and puts them into a hashmap, before sending across the wire. Flash does something similar when sending back to the server, skipping the accessors as back then it didn't really know what those were. But I wasn't sure if this was the case when targetting ByteArray and Shared Local Object for output.

I have an dummy object called IntrospectableObject, it has all manner of public, private, protected etc vars of all different types (well at least enough to test), I've been using to test. I've added a few accessors and a toString method which outputs everything.

The code to test ByteArray as a serialization is simple

import flash.net.SharedObject;
import com.troyworks.prevayler.*;
import flash.utils.ByteArray;
import flash.net.registerClassAlias;
import flash.utils.describeType;

////////////// BYTE ARRAY TESTS ////////////////

var bA:ByteArray = new ByteArray();
//registerClassAlias("com.troyworks.prevayler.IntrospectableObj", IntrospectableObj);
var iObj:IntrospectableObj = new IntrospectableObj();

bA.writeObject(iObj);
bA.position = 0;
var aObj:Object = bA.readObject();
trace("aObj " + aObj.toString());

trace("XML " + describeType(iObj).toXMLString());
trace("attempting to introspect ");
for (var i:String in aObj){
trace( i + " " + aObj[i]);
}

It outputs:

new Introspectable //consturctor of source
aObj [object Object]
attempting to introspect
pubStrVar public testString
pubBooTrueVar true
pubVar undefined
pubUIntVar 1
pubIntVar -1
pubNumVar 0.5
pubBooFalseVar false
pubDateVar Wed Aug 29 02:40:48 GMT-0700 2007

which reveals no accessors are called, and nothing but public vars are called, which is to be expected. Uncommenting:

registerClassAlias("com.troyworks.prevayler.IntrospectableObj", IntrospectableObj);

We get something completely different. As it's creating a new class upon reconstitution, and then calling it's toString() which has access to the private/internal so can see more. This means that ByteArray does get the whole picture even if it doesn't share all of it.

new Introspectable //constructor of source
new Introspectable  //constructor of reconstructed
aObj pubStrVar public testString
pubNumVar 0.5
pubIntVar -1
pubUIntVar 1
pubBooFalseVar false
pubBooTrueVar true
pubDateVar Wed Aug 29 03:00:12 GMT-0700 2007
priStrVar priTestString
priNumVar 0.2
priIntVar 2
priUIntVar 2
priBooFalseVar false
priBooTrueVar true
pritDateVar Wed Aug 29 03:00:12 GMT-0700 2007
attempting to introspect
--none--- //can't see anything as it's now class based.

What interesting is simliar to AMF, the constructor is called by Flash/AS3, but the values of the items as serialzed overwrite, bypassing the encapsulation layer (and thus any events). This is good if sometimes tricky to remember. It's resetting the snapshot of the object's state. Which compared to broadcasting a dozen+ change events makes sense, as it really hasn't changed, it's just been restored. This is basically what we want but as prior this may hit the shoes don't fit problem, break on newer swfs with different code coming out.

What this indicates to me is that serialization for Prevayler like approaches, should be done via the memento pattern, and only the mementos should be serialized. Wikipedia has a good article on it.
http://en.wikipedia.org/wiki/Memento_pattern  which interestingly links to a large page on finite state machines. which uses the memento pattern to save state/shallow history, with extended variables.
http://en.wikipedia.org/wiki/Finite_state_machine

Both the business logic and the commands should implement the pattern.  Mementos immutable read-only ideally, with all values passed into the constructor and only accessible via getters. But if the number of attributes are large, I could see a Object/struct with purely public variables may be better for useability and performance, as Mementos shouldn't do anything. In addition, Mementos should have a version number of the 'file format', and preferrably some mechanism can faciliate putting the data in via calling getters on the originator object, if they aren't marked as non-serializable (e.g. calculated results like totals shouldn't be serialized). Which I often like to put into a separate class, as the serialization/deserialization may be for different formats, XML, and they aren't really 'business object' concerns. One could even use the namespace feature to manage which one is being used.
This is exciting to me as Shared Local Object is cheap and fast and shared..a great place for multiple flash movies potentially in different web pages, maintaining a common log. Inside AIR's using it's DB, is another good place to store offline/online commands that get synched back up..potentially against other users making changes at one, when connected again.
This would take Flash RIA's closer to conventional applications, even if they are microcontent. It's always annoyed me how long running flash games how some don't remember where you are, using cheat codes so you can jump to a particular level, and that undo is so problematic, so it's either being petrified to explore for fear of making a mistake or having to do the same thing over and over again. There are many games I'd play longer if these capabilities existed. Flash makes these features easy,  it's just that most applications aren't written to support.
Since ByteArray can be compressed and written to Strings, this means they can be files, and thus to disk somewhere, be it SaveAs in AIR,  POST, XML. Sockets, meaning that a saving a application and then logging into another machine and getting everything back would be easy, or potentially opening and saving multiple versions so people can try things or create things easily.

August 29, 2007 at 6:50 am   and Filed Under as3, flash, introspection, java, performance, persistence, prevayler, serialization  | Add Comment

Prevayler and AS3, ByteArray

Since I have a background in Java Server side architecture, and web app scalability. One of the areas that interests me is applying the many tools that made Java serverside so easy/power (that is when correctly used) into the Flash world.
I have used RDMBS: Oracle, MYSQL, PostGress, SQLServer, XML, OODBs: Versant, Poet and ObjectStore. The latter being a large departure in working with after working with RDBMs for so many years, as there wasn't any brittle Java<->SQL mapping code, just more complicated serialziation/deserialization.

The easiest way to think about Object Oriented databases is if your familiar with Virtual Memory Management, but when applied across multiple machines. Virtual Memory Management makes it appear to the Operating System that there is far more RAM in the system than there actually is, so you can open up that 500MB photofile, and photoshop and Windows all in 512MB or RAM and tack on that brittney spears head..though it might be sluggish, unless you have enough memory. Relatedly this sluggishness is also what happens to web apps, as they are constantly going back to disk to return a few values to the pipe. What happens is and the programs unaware are constantly saved to disk (paging) when they aren't being used and reloaded into memory when they are needed. In some cases the ENTIRE ram is written to disk, this is how hibernate works in Windows when you have a laptop. Or in the case of virtual machines, the entire PC and hardware state is written to a file/disk. Generally this process is transparent to the end users and the application inside. What an OODB does is act like a multiple machine memory manager, with gargantuan virtual memory space, where the RAM is actually on multiple remote machines. If you imagine the alphabet as our database, and say that each server could only hold 3 letter of it, that our servers might have this in memory:

1. ABC
2. ABC
3. ABC
4. ABC
5. ME
6. THE

One of the key to performance is keeping frequently used items like ABC in memory (in this case being popular we've saturated several machines with them) and things not used (like Z) on disk. Almost all application and network accelerators use a variant of this memory first, caching routine. RAM access times for reads and changes are 1000x or more that of the fastest harddrives. RAM has gotten so cheap that this becomes the predominate web scaling strategy, and now it's quite easy to hold 16GB of ram on a single pc. Couple this with decreased processing as there is less mapping code...if any, plus 'pointer' navigation instead of expensive table joins, it's easy to see how OODB's can be order of magnitudes faster than RDBM's on disk. That said OODB's are expensive to use, and skillsets familiar with using them hard to find. With updates to the object model you have to migrate the version 1.0 stored on disk to the version 1.2. The virtual memory just like a harddrive can get fragmented.
Anyway, A year or so later I discovered Prevayler, and it made sense for many if not most applications. It's a one machine 'poor man's OODB. This is a good article on using it.
http://www.onjava.com/pub/a/onjava/2005/06/08/prevayler.html

So what's this to do with Flash? With AS3, and the ability to clone items deeply, I think there is the ability to apply the paradigm and thus benefits to Flash.
http://www.kirupa.com/forum/showthread.php?p=1897368

http://www.richapps.de/?p=34

http://www.stimuli.com.br/trane/2007/may/09/as3-sad-bits-1-deep-copying-or-cloning/

http://niko.informatif.org/blog/2007_07_20_clone_an_object_in_as3
Prevalyer, is the Command Design Pattern, with a command log allowing undo/redo, with the addition snapshots of entire application state periodically. This is similar to tweening in flash when you have keyframe (snapshots) and between frames. Except in this case we are dealing with data instead of visual asset state.

The command log could be stored locally via sharedObject, AIR's ability to write to the file system, or on the server, AMF/Remoting, or XML. In addition if the commands share a similar value to the server they could be used on multi-user applications, that have group editing, replication/failover and undo at a reasonable price. Which would be really cool!

This has a good graph on how it works architecturally.

http://www.ibm.com/developerworks/library/wa-objprev/