Leonardo's blog

Click here to download.

Edit: If the above url doesn't work due to Geocities limitations. Please, download the file from this link.

A couple of weeks ago, I was talking with a friend about a software he uses to book bus trips, he told me his company needs something similar, also he thinks I could adapt one of my applications to work like this program. The software is installed on more than one hundred sites, and it's connected to a central server over the Internet.

What resulted interesting to my friend was its speed, it works like if it was connected to a database in a local network (LAN), but it isn't. Also he added the server's Internet access is a cheap cable connection with ~300kb/s upload speed, and the clients connects using standard DSL or Cable connections. I asked what database and programming language the developers used. It's connected to a Firebird database and was programmed in C#, he replied. Then, I decided to try to replicate the scenario with a simple Delphi program.

My first attempt was a Delphi client connected to a Firebird 2.1 database using IBX controls. In my LAN, the app screams, but when I connect it over Internet it is as slow as a turtle. It seems to be that Firebird sends too much information over the network and it slows down the connection.

In my second attempt, I changed my app to use RemObject's SDK, I created an application server connected to the same database, it improved a lot...but if I'd choose the RO approach, I'd must rewrite the whole application.

My application is based on classes derived from TPersistent, and uses TCollection/TCollection Item to store data, it doesn't uses any DbAware component and to persist it's data, it dynamically loads a Dll containing what I call a Database Access Driver who knows, thanks to RTTI and a (not so) complex mechanism, how to talk to the database. The Driver it's using currently, contains the IBX controls to talk to Interbase/Firebird, and I thought it could be great if I can create a similar Driver to use RemObjects SDK, but it turned out to be a real pain (and I hadn't time to investigate RO SDK in detail).

After the failed attemt to use RemObjects, I decided to create a simple Application Server, based on Indy's TCP Server control. The server should receive a request from the client asking for an object, or collection of objects, and the params to filter the result, then the server dynamically creates an instance of the requested class, fill its data, then converts it to XML and sends the result to the client, who must un-serialize the XML to reconstruct the instance.

I created the server side of the application, and found that the XML serialization part was too slow, resulting in more than 5 seconds for a collection of 500 items, maybe the serializer I used isn't optimized, and I didn't have time to improve by myself, so I decided to find a new serialization method...Thanks god and Roland Beenhakker I found this article on how to write and read the content of a TPersistent descendant to/from a stream. The first time I tested the trick Roland exposed in his article, I was shocked, it serialized my collection instantaneously, the de-serialization was also super fast.

How the server store the objects in the database?, it uses the same Data Access Driver the older Client Application used to connect to the database.

Knowing the Serialization/Deserialization method I'll use for the application, I focused on writing a new Data Access Driver for connecting to the Application Server. The task was pretty easy, just serialize objects, then use the WriteStream and ReadStream method of Indy to send and receive the streamed objects to/from the server.

When I finished writing the new Data Access Driver, I decided to try it in a real environment, connecting the client from a remote machine to my local PC by using my home Internet connection. The results where amazing, the whole round trip only took 1.5 seconds!, impressive.

Knowing that the serialization/deserialization process is instantaneous, and the only possible bottleneck is the network. I decided to compress the streams sent over the network, and used InflateStream and DeflateStream from the great TurboPower Abbrevia library. The reduced stream size improved even more the results.

As the data sent over the network is binary, it is also very (if not impossible) difficult to be intercepted and used by a unauthorized people. And if you are very concerned about security, after compressing the stream you can encrypt it.

One negative side effect of this approach, is that it only works with clients and servers written in Delphi and C++ Builder. To me, this is not a problem, because I maintain both, the server and the client, but if the server must be accessed by some other language, you could improve the protocol to let the server determine in which format the data should be transferred.

In the next post, I'll show a minimalistic version of the program.