Wednesday, March 18, 2009
Transferring objects over the network (Part I)
Introduction
A couple of weeks ago, I was talking with a friend about a software he uses to book bus trips, he told me his company needs something similar, also he thinks I could adapt one of my applications to work like this program. The software is installed on more than one hundred sites, and it's connected to a central server over the Internet.
What resulted interesting to my friend was its speed, it works like if it was connected to a database in a local network (LAN), but it isn't. Also he added the server's Internet access is a cheap cable connection with ~300kb/s upload speed, and the clients connects using standard DSL or Cable connections. I asked what database and programming language the developers used. It's connected to a Firebird database and was programmed in C#, he replied. Then, I decided to try to replicate the scenario with a simple Delphi program.
My first attempt was a Delphi client connected to a Firebird 2.1 database using IBX controls. In my LAN, the app screams, but when I connect it over Internet it is as slow as a turtle. It seems to be that Firebird sends too much information over the network and it slows down the connection.
In my second attempt, I changed my app to use RemObject's SDK, I created an application server connected to the same database, it improved a lot...but if I'd choose the RO approach, I'd must rewrite the whole application.
Persistence
My application is based on classes derived from TPersistent, and uses TCollection/TCollection Item to store data, it doesn't uses any DbAware component and to persist it's data, it dynamically loads a Dll containing what I call a Database Access Driver who knows, thanks to RTTI and a (not so) complex mechanism, how to talk to the database. The Driver it's using currently, contains the IBX controls to talk to Interbase/Firebird, and I thought it could be great if I can create a similar Driver to use RemObjects SDK, but it turned out to be a real pain (and I hadn't time to investigate RO SDK in detail).
The application server
After the failed attemt to use RemObjects, I decided to create a simple Application Server, based on Indy's TCP Server control. The server should receive a request from the client asking for an object, or collection of objects, and the params to filter the result, then the server dynamically creates an instance of the requested class, fill its data, then converts it to XML and sends the result to the client, who must un-serialize the XML to reconstruct the instance.
I created the server side of the application, and found that the XML serialization part was too slow, resulting in more than 5 seconds for a collection of 500 items, maybe the serializer I used isn't optimized, and I didn't have time to improve by myself, so I decided to find a new serialization method...Thanks god and Roland Beenhakker I found this article on how to write and read the content of a TPersistent descendant to/from a stream. The first time I tested the trick Roland exposed in his article, I was shocked, it serialized my collection instantaneously, the de-serialization was also super fast.
How the server store the objects in the database?, it uses the same Data Access Driver the older Client Application used to connect to the database.
The client's Data Access Driver
Knowing the Serialization/Deserialization method I'll use for the application, I focused on writing a new Data Access Driver for connecting to the Application Server. The task was pretty easy, just serialize objects, then use the WriteStream and ReadStream method of Indy to send and receive the streamed objects to/from the server.
The real test
When I finished writing the new Data Access Driver, I decided to try it in a real environment, connecting the client from a remote machine to my local PC by using my home Internet connection. The results where amazing, the whole round trip only took 1.5 seconds!, impressive.
A little improvement
Knowing that the serialization/deserialization process is instantaneous, and the only possible bottleneck is the network. I decided to compress the streams sent over the network, and used InflateStream and DeflateStream from the great TurboPower Abbrevia library. The reduced stream size improved even more the results.
Side effects
As the data sent over the network is binary, it is also very (if not impossible) difficult to be intercepted and used by a unauthorized people. And if you are very concerned about security, after compressing the stream you can encrypt it.
One negative side effect of this approach, is that it only works with clients and servers written in Delphi and C++ Builder. To me, this is not a problem, because I maintain both, the server and the client, but if the server must be accessed by some other language, you could improve the protocol to let the server determine in which format the data should be transferred.
In the next post, I'll show a minimalistic version of the program.
Comments:
<< Home
We have something similar: our solution uses DXSock for the server and Indy for Client, serialize the data with our own data-to-stream in a binary fashion, then compress with abrrevia plus encrypt with Lockbox. To be faster we cache data locally in the client side, with Nexus (or FlashFiler)
# posted by 
: 1:21 PM
: 1:21 PM
Hola, gran post (y gran trabajo).
Desarrollo con Delphi desde hace bastante tiempo y he usado bastante DataSnap para soluciones de este tipo, aunque es totalmente distinto a lo que planteas dado que no usas componentes data-aware.
Ahora, te pregunto, decís que usas IBX con Firebird 2.1 ¿Cómo te ha funcionado esa combinción? ¿Que versión de los IBX (y de Delphi) estás usando?
Desarrollo con Delphi desde hace bastante tiempo y he usado bastante DataSnap para soluciones de este tipo, aunque es totalmente distinto a lo que planteas dado que no usas componentes data-aware.
Ahora, te pregunto, decís que usas IBX con Firebird 2.1 ¿Cómo te ha funcionado esa combinción? ¿Que versión de los IBX (y de Delphi) estás usando?
# posted by : 5:45 AM
: 5:45 AM
Uso los IBX 7.04 con Delphi 7, pero da lo mismo usar otros componentes como Zeos que los uso cuando necesito conectarme con Postgres u Oracle.
Hola, muy interesante tu articulo, tengo un problema similar con delphi 7.1, firebird 1.5 y uso dbexpress (dbx) que viene con delphi...que tipo de optimizacion podria hacer para bajar el consumo de ancho de la transmision de datos.
De antemano muchas gracias.
De antemano muchas gracias.
# posted by : 8:03 AM
: 8:03 AM
Con respecto a las optimizaciones de Firebird yo probaría lo siguiente:
1) Prueba con Firebird 2.1, tienes que hacer un backup de tu base de datos y luego un restore con el 2.1 y listo, todo funciona perfectamente. En este artículo (http://www.firebirdsql.org/rlsnotesh/rlsnotes210.html#rnfb210-global-remote) se explican las mejoras en cuanto a conexiones remotas.
2) Optimiza las consultas. No uses "select * from xxxx", sino "select campo1, campo2, campoN from xxx" explicitando sólo los campos necesarios.
3) Controla que el problema no esté en otro lado, por ej. en la interfaz gráfica.
4) Por último, si los puntos anteriores no ayudaron deberías intentar el método propuesto en este post.
Saludos,
Leonardo.
1) Prueba con Firebird 2.1, tienes que hacer un backup de tu base de datos y luego un restore con el 2.1 y listo, todo funciona perfectamente. En este artículo (http://www.firebirdsql.org/rlsnotesh/rlsnotes210.html#rnfb210-global-remote) se explican las mejoras en cuanto a conexiones remotas.
2) Optimiza las consultas. No uses "select * from xxxx", sino "select campo1, campo2, campoN from xxx" explicitando sólo los campos necesarios.
3) Controla que el problema no esté en otro lado, por ej. en la interfaz gráfica.
4) Por último, si los puntos anteriores no ayudaron deberías intentar el método propuesto en este post.
Saludos,
Leonardo.
Leonardo,
Con respecto a la combinación IBX/Firebird 2.1 me refería si te funcionan sin problema dado que el API de Firebird 2.x cambio y los IBX estan desarrollados sobre la base de Interbase. Como te comentaba, yo uso principalmente Datasnap, un middle-tier server hecho con Delphi 6 y Firebird 1.5.
Apuntaba más a si lo usaste en producción.
Saludos.
Maxi.
Con respecto a la combinación IBX/Firebird 2.1 me refería si te funcionan sin problema dado que el API de Firebird 2.x cambio y los IBX estan desarrollados sobre la base de Interbase. Como te comentaba, yo uso principalmente Datasnap, un middle-tier server hecho con Delphi 6 y Firebird 1.5.
Apuntaba más a si lo usaste en producción.
Saludos.
Maxi.
Maxi, la respuesta es Sí, lo uso actualmente en producción y no tuve ningún tipo de problemas. Es más, en una aplicación simplemente actualicé la versión de la base de datos y no tuve siquiera que recompilar para comenzar a usarlo.
Igualmente te aclaro que el uso que yo hago de esos componentes es muy limitado, simplemente instancio un TIbQuery por código, hago la consulta correspondiente, luego relleno mis TCollections y libero el dataset. Quizás si se lo usa más intensivamente haya algún tipo de problemas, yo por suerte no tuve ninguno.
Igualmente te aclaro que el uso que yo hago de esos componentes es muy limitado, simplemente instancio un TIbQuery por código, hago la consulta correspondiente, luego relleno mis TCollections y libero el dataset. Quizás si se lo usa más intensivamente haya algún tipo de problemas, yo por suerte no tuve ninguno.
You may look at OverByte Midware
http://www.overbyte.be/frame_index.html?redirTo=/products/ics.html
for object transfert over internet, DBOvernet has taken the same code a little farter for databases
http://www.dbbridge.com/dbovernet/dbovernetindex.htm
(I think dbOvernet is no longer alive)
http://www.overbyte.be/frame_index.html?redirTo=/products/ics.html
for object transfert over internet, DBOvernet has taken the same code a little farter for databases
http://www.dbbridge.com/dbovernet/dbovernetindex.htm
(I think dbOvernet is no longer alive)
# posted by : 2:00 PM
: 2:00 PM
Thanks fro sharing this. What is great, from what I understand to this point, is that I can do this without a 3rd party component. Looking forward to your example.
I too use IBX with firebird and find it works great for me... so far. However, I just started a new project that is solely using FIBPlus.
- Lou
I too use IBX with firebird and find it works great for me... so far. However, I just started a new project that is solely using FIBPlus.
- Lou
# posted by : 7:58 AM
: 7:58 AM
You should try HTTP + RESTful approach to access your data in clean,uniform way. And then there is no limit to use only Delphi on client side, it can even be javascript based rich internet app.
If you need security then add HTTPS.
For faster serialization you might consider JSON instead of XML, there are JSON libs in Delphi as weel. But it's rather trivial format, so writing your own parser is trivial.
Anyway, I'm coming from Java world and people are doing stuff like this for ages....
If you need security then add HTTPS.
For faster serialization you might consider JSON instead of XML, there are JSON libs in Delphi as weel. But it's rather trivial format, so writing your own parser is trivial.
Anyway, I'm coming from Java world and people are doing stuff like this for ages....
# posted by : 7:57 AM
: 7:57 AM
Thanks for your comment. In the Delphi world, the data transfer using XML, or any other kind of text transfer also is used for ages. The goal of the article was to show a "faster" way to share objects between Delphi applications.
BTW, I tested this approach in FreePascal and doesn't work if the server AND client aren't created using the same compiler, due to differences between Delphi's RTL and FPC RTL.
BTW, I tested this approach in FreePascal and doesn't work if the server AND client aren't created using the same compiler, due to differences between Delphi's RTL and FPC RTL.
Is there a part 2 (and 3 etc etc.. ) :-) still sounds like an interesting concept
Post a Comment
# posted by : 1:20 AM
: 1:20 AM << Home
