Archive for the 'serialization' Category

XHTML + E4X + Namespaces

Namespaces in flash are cool and...pretty pesky.

• for unknown reasons, querying a E4X's queries in the same block apparently have to be individually namespaced. Neither putting the default namespace in the class or function worked.
• anything that tries to get at the xml while it's being passed around has to have a copy of the namespace. This isn't particularly efficient when you are serializing between objects and XHTML.

Steven Sachs has a good tutorial on some of the issues faced when working with XHTML and Flash.  In particular the namespace.  However after losing about 2hrs I've come up with an alternate solution...just stripping the namespace out as it's introducing more code chaos than the namespace would have solved.

In the next rev of kidthing.com's website we are going to XHTML for SEO and CMS, deeplinking etc reasons.   In addition we are consuming Wordpress, initially as RSS and eventually as fully skinning the main pages in Wordpress.  Since Flash's support for HTML is limited we run things through a 'cleaner' that replaces quotes, superscript, some bullets with the appropriately embedded characters.  It was easy to add the following to strip it out on the complete load of the XHTML

res = res.split('xmlns="http://www.w3.org/1999/xhtml"').join('');

I switched over from String manipulation to XML manipulation to get at selective DIV's that get mapped to textfields in the flash templates (Flash is like CSS on steroids)  E4X here really shines.

Why AMF Rocks and beats XML

Ted made a great summary of why AMF is Flash's file format of choice.

http://www.onflex.org/ted/2007/11/abcs-of-amf.php

While I've touted the benefits about persistence and serialization of native classes in previous posts (see persistance or serialization categories). Often XML is preferred for data interchange with servers or text editor compatible configuration. Here Ted had a gem in the comments, comparing why using XML inside of AMF over XML in a text file, rocks!

E4X is super handy for querying the data and returning collections after parsing to xml. That said XML is a native object and can be included in AMF3. The data is stored as a string zlib compressed and when deserialized becomes an XML object again.

bytes.writeBytes( new XML('') );

This gives you zlib compression and benefits of e4x in one go. zlib is very effective on xml and can compress documents as much as 20X, I have seen 300Kb XML files become 7K given tag redundency in the ASCII text.

The XML human readable is important as often for games and such that I build, but frequently it's doesn't make sense to build a configurator given the many excellent XML editors out there. In addition it's very easy for most servers to generate, where they can't generate AMF as easily. But there are two compelling benefits to going that route:
1) As the XML grows, compressing things by up to 20x can improve the responsiveness of the application (like startup of a website), and reduce network costs.

2) XML stored this way is native data type to Flash not a string...meaning it's already an XML accessible to E4X instead of a random string that has to be parsed into XMLNodes. Reconstituting the data out of the AMF stream still takes overhead, but it should be much faster than parsing it from interpreting tags and attributes from the plain text, as most the marshalling code is in C inside the player instead of actionscript. Read better experience for end users.

What occurs to me now is that either a very simple AIR app act in a similar fashion to Winzip for compressing/decompressing text for consumption by flash, or act as a simple text editor, just using a text field.
I'm sure that Adobe will open up the AMF API so that servers can generate raw AMF without using remoting or RMTP. This will open up the possibilities for CMS driven websites to prerender sites daily out of database requests.

Using ByteArray to Serialize AS3 Classes for AIR, Web and Everywhere!

When using AIR's FileStream to write AS3 classes to ByteArray, I've found that the same file format works for loading in via URLLoader, as well as what's uploaded using FTP. This isn't surprising, but wanted to test anyway.

What this means is that ByteArray is a groovy ubiquitous format for storing full classes and dependencies and references, via ByteArray.writeObject and ByteArray.readObject (provided registerClass is used), without nececessarily having to parse them manually though that is still suggested for versioning and class evolution. Meaning you can create a file using AIR , send that file to a friend in email, and have them open it up in a flash based webpage, and they operating in a webpage could Http POST the change back to you. The ByteArray can have rich complexity, classes, self references, collections, etc. This could be used to save whole game states (even massive multiplayer, NPC brains, etc). Or in my target case debugging outside other non-AIR runtimes to speed development.

How to save to the same directory the swf AIR is running from

This took me a bit to figure out. Primarily as I'm using AIR to write the files in Flash CS3 and it's not immediately apparent in the docs, how to target the 'self' directory of the swf. This isn't recommended in the docs (which I agree) but in the case I'm using it for it makes sense to do so. The API for AIR is here

This would be swf._url in AS2, or in AS3 DisplayObject.loaderInfo.url , as when using relative links in URLLoader to load whatever you're saving this will resolve properly in AIR. Turns out flash.filesystem.File.applicationDirectory is the droid you are looking for. It's not fully qualified (e.g. file:C:\somedirectory...) when traced it looks like:

"app-resource:/yourSwfNameGoesHere.swf"

unless you use the file.nativePath which is required in order to actually write to the directory. So you can use still this in old school string name parsing the swf name out, so you don't have to use File.applicationResourceDirectory if your looking to make a single swf for both AIR and Web use, and on the web, the AIR libraries won't be accessible.

Here's two handy scripts to help inspect ByteArray's, the first looks at the 1's an 0's, the other looks at the objects in the stream, using the getQualifiedClassName, and if it finds an Array or sub ByteArray attempts to inspect it.
/* converts byteArray to binary like

TraceBits--------------------
0 = 110
1 = 1011
2 = 1001000
3 = 1100101
*/

Local Shared Objects…not so Shared. Cross Domain Projector Issues.

We have a a few Desktop Flash applications: One is a main application, the other an update utility for the main app. They use a Flash Cookie to find each other, as the main app could be installed anywhere (USB, C), so it writes to the cookie where it's installed at. The apps are based in Director using the Flash Xtra.

Say we start with a completely empty SharedObjects folder

Running the main app sets a cookie into a folder like

....Flash Player\#SharedObjects\BFYLPV7P\localhost\02000.sol

via a call like

var so : SharedObject = SharedObject.getLocal ("02000", "/");

where "/" should see the root of the Flash Cookies.

then say Pandora (or any other flash based website using cookies) runs it generates a new sandbox

....Flash Player\#SharedObjects\2S7GJDMQ\pandora.com

and then all future read requests are against that new '2S7GJDMQ' folder instead of the "BFYLPV7P" folder, and then neither the app or the updater, can see the old shared local object folder anymore. Which in the case of it containing lots of persistent configuration data, is the equivalent of clearing the Browser cache and forgetting everything.

Trying to understand how it's created, I can delete the contents of the shared Objects folder and then republish in the Flash IDE and get a new hash for the folder every time so it seems to be either random or time based. It's not as far as I can tell the behaviour documented, it's certainly not behaving as I would expect.

As mentioned here. It used to be that one could work around this via:
SharedObject.getRemote("my_so", "rtmp:/./..", "/");

sadly it no longer works in Flash 8, or 9.

I got this reply on the FlashCoders list:

>Movies running in different sandboxes cannot access shared objects of
>each other.

Yes, that makes sense, except for 2 things:

1) As far as I can tell, they should NOT be in separate sandboxes, they
use the exact same code, and are installed in next to the same folder. As far
as I can tell there is no way to specify the domain for a projector to get the
desired behavior.

2) It's also inconsistent. First run it WILL use the same sandbox as the Flash IDE/etc, then later something will trigger the generation of a new sandbox (install a second version of the app, or view pandora) then all further requests go against the new sandbox instead of the original. Meaning the Flash Player changes it's mind about which domain it thinks it belong to!

My best guess, likely due to it running in a projector without access to URL information, it's making stuff up.

Our first attempt at solving, required upgrading the Flash Xtra in both projectors to Flash8, apparently having one at Flash7 and the other Flash8 is enough to trigger the domain.

Also interesting is the Flash IDE never really seems to have this problem, presumably because it is using the Flash Player EXE as the handler for all swfs via the registry. So one potential workaround was firing a Flash Projector with the sole point in life to read/write the local Shared Object, with local connection object to read how flash sees it and then shut down.

Flash on the Desktop…everywhere. Flash CS3 supports AIR output

This week has been week of Desktop Based Flash. 1) Using Director + the Buddy API Xtra, 2) a client with a C++ and F-in-Box, and just finishing up save capability 3) AIR Local File Access for Flash
Testing applications inside of these projectors can really slow down application development as it's compile in flash, and then test in the projector.

Still I'm quite stoked to see that Flash IDE has an update to support building AIR apps in the IDE. This rocks for several reasons as many apps I make are based off XML configuration or could benefit from making tools to help generate the data.  Here is a good post on actually using the update to create a hello world example.

This is after spending a week battling with bizarre Shared Object behavior. Memo to self, don't depend on SharedObjects for reliable desktop storage!

Serialization: IExernalizable and Evolving Classes

What is evolving a class? It's the inevitable:

• addition of a field
• change the superclass
• remove a field
• change the name of a field
• change a field to static (from non-static)
• change a field to transient (from non-transient)
• change the type of a field type (e.g. int to float)

As soon as you do any of the above whatever you've previously saved something to disk, be it local or remote will break upon loading back in.

This isn't common in most web based flash development which lives in isolation on a webpage and doesn't save anything, but introduce any sort of saving. With Desktop Flash apps (AIR, Zinc), and RIA's, offline online applications, it IS becoming something that is important.
There are 3 different strategies to serializing objects, only one is particularly useful for evolving classes.

1. Direct: writeObject(yourClassObject);
2. Customize with IExternalizeable: iterate over each attribute of your object and write each of those to the stream.
3. Customize with IExternalizeable, iterate over each attribute of your object write it with a key to a Dictionary / HashTable.

Approach 1 is great for simple uses and first passes. However it doesn't scale well or at all. Approach 2 Is better but as the stream is sequential, this can make dealing with multiple versions difficult or impossible as you have to know the exact order in which to parse. Approach 3 is the best, it uses a Dictionary as an intermediate map, it's only a bit more code to write, and is a *whole* lot easier to parse. This is also the approach that Remoting uses when talking to the server with the ASObject, and in someways writing to disk is identical to writing to the filesystem.

Here is the list of imports:

Here is the version number, we'd need to increase everytime a significant data change has happened, and some of the static register class necessary to get the class to serialize and deserialize correctly:

Here are the modified IExternalizable functions. Note that the read from the stream has to deal with all permutations for a given version to version, as this number of changes gets large, the normalization can be quite large, thankfully good architecture up front can prevent much of this.

AS3 Customized Serialization and Deserialization, IExternalizeable to the rescue!

This and original off Pete's blog is great article on a little known feature critical for de/serialization in flash, and the knowledge is not just for client/server as the  The flex documentation on the subject implies. Anywhere the AMF format is used: shared local objects, local connection, in AIR using fileStream. cloning objects using ByteArray etc. So many gems (might as well be easter eggs!) in the AS3 code, sadly barely documented. It's using IExternalizeable for custom serialization, which mimicks Java's already proven approach.

What's that mean? It means you can better control how objects are written and read back from the byteStream.
What if I don't want X to be written?
To compliment a great follow up on using the [Transient] metadata by Darron, with both the clone and when serializing to disk or the web. I've got to say the metadata format makes sense but also wierds me out.

How do I get my class type back

registerClassAlias is necessary for local cloning and maintaining Class type, when written to a ByteArray they forget which class they belong to. I would have hoped that the RemoteClass metadata would have worked but that was not the case. Anyway you can do this via a static var:

private static const REG:* = registerClassAlias("reflection.IntrospectableObj",IntrospectableObj);

this is called during static initialization, once which is all that's really needed as the linkage is static, not per instance.

What if you need to transcend versions of code?

basic approach to using serialization to deal with different versions of the class is
1) write a version number to the stream first, then a) save to hashmap, or b) iterate over the members that should be saved.
2) read the version number back from the stream, then reconstitue the members from a) the Dictionary (like Remotings's ASObject) or b) in the order they were written to the stream. Vary the parsing response upon the version number.

Summary of Serialization Tips:

1) Use IExternalizable to customize how your objects are written.

2) Use the [Transient] metadata to NOT serialize objects if you don't want them there, and are using writeObject().
3) registerClassAlias is necessary for local cloning and maintaining Class type. You can do this via a static var,
private static const REG:* = registerClassAlias("reflection.IntrospectableObj",IntrospectableObj);

4) keep a unique version number in the code, write a version first in the output stream then when reading it back perform different behavior based on that stream. The use of a Dictionary for key/values can be helpful too.

AS3 Serialization and Deserialization: ByteArray BitmapData registerClassAlias

Grr. Flash's registerClassAlias does not work with BitmapData (I wonder ?), so while you can save it to the ByteArray and wherever that can go... you can't get it back! As BitmapData complains about the constructor getting no args, presumeably that memory allocation is fixed upon the constructor.
This means you'll have to roll your own, which isn't too hard but it when having the BitmapData as members of other classes it might be a pain as each will have to serialize/deserialize the BitmapData using the IExternalizable.
Another option might have been overridding, but while BitmapData can be extended there isn't a no arg constructor, so the constructor can't really be overridden efficetively. A Proxy could be created for this.
Andrew has a good workaround, iterating over the pixels, though I've opted to write the width and height into the stream as well. You can use writeObject for the height and widht without problems (versus writing writeInt)
For additional details and others experiencing this problem: search terms "registerClassAlias BitmapData" "deserialize Bitmapdata"

• http://www.google.com/search?q=deserialize+BitmapData&ie=utf-8&oe=utf-8&aq=t&rls=org.mozilla:en-US:official&client=firefox-a
• http://www.adobe.com/cfusion/webforums/forum/messageview.cfm?catid=641&threadid=1255888#4547075
• http://osflash.org/pipermail/red5_osflash.org/2007-April/011091.html

Solutions

• [1] http://www.cynergysystems.com/blogs/blogs/andrew.trice/BitmapData/SharedDesktop/srcview/
• [2] http://www.barncar.com/writejpg.mxml (sample code using corelib to save as Jpg)
• http://www.bytearray.org/?p=90 (a more flexible snapshot of video/swf to jpg using corelib, which is a completely different solution path)
• corelib is here http://code.google.com/p/as3corelib/

NOTE:

if you do need to use registerClassAlias you can do this trick:

private static const REG:* = registerClassAlias("reflection.IntrospectableObj",IntrospectableObj);

AS3 and Serialization, Memento Pattern

Starting to work on porting the persistence aspect of Prevayler to Flash/AS3. It's a perfect fit with Flash, as Prevaylence assumes that everthing can fit into RAM (which flash does, aside from loading new things in), and changes have to be synchronized, which seeing as it's a primarily a single thread doesn't have to worry about synchronizing multiple writes.
Prevaylence relies heavily on understanding the serialization of objects, both the commands and the business object state should be save and restoreable. Since ByteArray supports AMF packets (as does shared object) this means that private vars and Class definitions, functions should be able to be written to disk and brought back from dead...but this leaves the larger question, just like the dinosaurs ...should they be brought back?
It's certainly tempting.  Java has faced this for years in it's serialization routines. In general built in serializing is good, but doing it across version is bad, as it's inevitable that the Class definition will change, adding, removing variables and methods ...even changing types, during refactoring. When reconstituting things may not serialize properly much as you don't fit into your GI Joe pajamas from kindergarden (well hopefully). It's unpredictable what the Flash player will do when forced to put version 1.2 into 1.3 shoes, or potentially version 1.4 into 1.2.  So it's important to know exactly what is being saved and how flexible it is.

Since I was working on Trace.me this week, and private, protected, etc variables didn't show up in the output, (which is proper Object encapsulation), I was curious as to if the variables were invisible or lost. I remembered  hitting similar things when working with and on openAMF, so I dug a bit, and just found a document I wrote a few years ago, on openAMF, ah good times...

http://openamf.cvs.sourceforge.net/*checkout*/openamf/openamf/docs/understanding.htm?revision=1.1

There the issue there was different, was actually whether java's accessors e.g. get someValue and set someValue translated across the wire. The serialization in Java openAMF calls the accessors and puts them into a hashmap, before sending across the wire. Flash does something similar when sending back to the server, skipping the accessors as back then it didn't really know what those were. But I wasn't sure if this was the case when targetting ByteArray and Shared Local Object for output.

I have an dummy object called IntrospectableObject, it has all manner of public, private, protected etc vars of all different types (well at least enough to test), I've been using to test. I've added a few accessors and a toString method which outputs everything.

The code to test ByteArray as a serialization is simple

import flash.net.SharedObject;
import com.troyworks.prevayler.*;
import flash.utils.ByteArray;
import flash.net.registerClassAlias;
import flash.utils.describeType;

////////////// BYTE ARRAY TESTS ////////////////

var bA:ByteArray = new ByteArray();
//registerClassAlias("com.troyworks.prevayler.IntrospectableObj", IntrospectableObj);
var iObj:IntrospectableObj = new IntrospectableObj();

bA.writeObject(iObj);
bA.position = 0;
var aObj:Object = bA.readObject();
trace("aObj " + aObj.toString());

trace("XML " + describeType(iObj).toXMLString());
trace("attempting to introspect ");
for (var i:String in aObj){
trace( i + " " + aObj[i]);
}

It outputs:

new Introspectable //consturctor of source
aObj [object Object]
attempting to introspect
pubStrVar public testString
pubBooTrueVar true
pubVar undefined
pubUIntVar 1
pubIntVar -1
pubNumVar 0.5
pubBooFalseVar false
pubDateVar Wed Aug 29 02:40:48 GMT-0700 2007

which reveals no accessors are called, and nothing but public vars are called, which is to be expected. Uncommenting:

registerClassAlias("com.troyworks.prevayler.IntrospectableObj", IntrospectableObj);

We get something completely different. As it's creating a new class upon reconstitution, and then calling it's toString() which has access to the private/internal so can see more. This means that ByteArray does get the whole picture even if it doesn't share all of it.

new Introspectable //constructor of source
new Introspectable  //constructor of reconstructed
aObj pubStrVar public testString
pubNumVar 0.5
pubIntVar -1
pubUIntVar 1
pubBooFalseVar false
pubBooTrueVar true
pubDateVar Wed Aug 29 03:00:12 GMT-0700 2007
priStrVar priTestString
priNumVar 0.2
priIntVar 2
priUIntVar 2
priBooFalseVar false
priBooTrueVar true
pritDateVar Wed Aug 29 03:00:12 GMT-0700 2007
attempting to introspect
--none--- //can't see anything as it's now class based.

What interesting is simliar to AMF, the constructor is called by Flash/AS3, but the values of the items as serialzed overwrite, bypassing the encapsulation layer (and thus any events). This is good if sometimes tricky to remember. It's resetting the snapshot of the object's state. Which compared to broadcasting a dozen+ change events makes sense, as it really hasn't changed, it's just been restored. This is basically what we want but as prior this may hit the shoes don't fit problem, break on newer swfs with different code coming out.

What this indicates to me is that serialization for Prevayler like approaches, should be done via the memento pattern, and only the mementos should be serialized. Wikipedia has a good article on it.
http://en.wikipedia.org/wiki/Memento_pattern  which interestingly links to a large page on finite state machines. which uses the memento pattern to save state/shallow history, with extended variables.
http://en.wikipedia.org/wiki/Finite_state_machine

Both the business logic and the commands should implement the pattern.  Mementos immutable read-only ideally, with all values passed into the constructor and only accessible via getters. But if the number of attributes are large, I could see a Object/struct with purely public variables may be better for useability and performance, as Mementos shouldn't do anything. In addition, Mementos should have a version number of the 'file format', and preferrably some mechanism can faciliate putting the data in via calling getters on the originator object, if they aren't marked as non-serializable (e.g. calculated results like totals shouldn't be serialized). Which I often like to put into a separate class, as the serialization/deserialization may be for different formats, XML, and they aren't really 'business object' concerns. One could even use the namespace feature to manage which one is being used.
This is exciting to me as Shared Local Object is cheap and fast and shared..a great place for multiple flash movies potentially in different web pages, maintaining a common log. Inside AIR's using it's DB, is another good place to store offline/online commands that get synched back up..potentially against other users making changes at one, when connected again.
This would take Flash RIA's closer to conventional applications, even if they are microcontent. It's always annoyed me how long running flash games how some don't remember where you are, using cheat codes so you can jump to a particular level, and that undo is so problematic, so it's either being petrified to explore for fear of making a mistake or having to do the same thing over and over again. There are many games I'd play longer if these capabilities existed. Flash makes these features easy,  it's just that most applications aren't written to support.
Since ByteArray can be compressed and written to Strings, this means they can be files, and thus to disk somewhere, be it SaveAs in AIR,  POST, XML. Sockets, meaning that a saving a application and then logging into another machine and getting everything back would be easy, or potentially opening and saving multiple versions so people can try things or create things easily.

August 29, 2007 at 6:50 am   and Filed Under as3, flash, introspection, java, performance, persistence, prevayler, serialization  | Add Comment