Another interesting webinar the other day. Omneon described their grid storage system. Not only does their systtem have RAID type features which allow the data to be backed up, but they have the facility to have the data on the system several times over. What this gives is the ability for multiple users to access these multiple versions in parallel.
The client software asks the metadata server to tell it where to find the data files. The clients then access these blocks of data directly, so the load on the metadata server/s is kept to a minimum. If the blocks of data are kept as duplicates, spread around the storage devices, the load on the data servers can be spread as much as you like. There could be several copies to feel the need of many clients. By splitting the files into blocks it gives minimum impact in time of a failure. The data blocks can simply be rebuilt from another copy on the system. It’s similar to a RAID sysem in that respect but with the growth potential to serve huge numbers of clients.
I suppose that all this is similar to buying storage at a huge facility like Amazon. They do all the work of storing the data and allowing access. They take care of backup and recovery automatically. Data storage is growing at such a huge rate. This type of system must be ideal.
I don’t suppose that I’ll have to do much of this, but it struck a chord and could have parallels in the area of creating multiple objects. The central controller can just direct a client to a factory that produces the object. Several factories could balance the load. They could even be talking to one another through an external API. This separates the idea from just a clustered server into a system that can be plugged into and filtered.
I’m just rambling here. I thought it was interesting…