Digital Media Mass Storage Testing
Video eats storage. Lots of it. For breakfast, lunch and dinner. In copious quantities – an hour of professional footage of a quality drama production easily eats a terabyte. We easily need some 10 hours, split over a 1000s shots or so, to just produce one episode. And we work with it fast – we scroll or ‘jog’, frame accurate, at many times real speed. And during the final craft edit phase – this implies accessing some 100’s of very large files, extract a few 10’s of gigabytes and combine it all in fractions of seconds. And multiply this by the 100’s of programmes the ³ÉÈË¿ìÊÖ completes every day – and you can see that there is a little engineering challenge here.
Ìý
Ìý
Ìý
Now seemingly storage has become a commodity item, available from OEMs, from interchangeable, generic suppliers and specialist suppliers. In our world – reality could not be more different. In fact – we’re not even close to having generic storage perform at the levels needed for SD production, yet alone HD. Right now – our storage is bespoke, heavily tailored for broadcasting and hence expensive.
So this is where our R&D department comes in.
Now on its own, storage performance is determined by the distribution of data on hard disks and how fast the read and write heads can move around the spinning disks to access chunks of a file. Storage providers can somewhat optimise the physical setup of the storage, but for specific types of applications. For example, storage optimised for database access would perform poorly when accessing large video files.
For file-based production, storage is no longer just about size, disk speed and network speed. It is determined by the end-to-end system and anything through which data is moved. All the way from inter DMA and bandwidth on the motherboard and PCI busses up to the network buffers, the behaviour of the operating systems and the video software. And unfortunately – each tries to game the other – with little holistic optimisation.
It is difficult to know in advance how well storage will perform with different production applications, without lengthy testing using real production tools. And this is not something we can do routinely (we need those production people and expensive tools to make TV programmes) – or which would give us reproducible results.
As there were no commercial or open source tools for testing storage performance under proper, realistic, production conditions, ³ÉÈË¿ìÊÖ R&D developed the media storage meter in 2004. The media storage meter uses multiple clients to emulate the behaviour of production tools. This let us compare large media storage systems fairly.
We open sourced this tool in 2006 and we’ve just today released an updated version. This was so that manufacturers, operating system vendors and basically just anyone could use it. We did this for several, quite selfish, reasons. We hope that vendors would get a more realistic idea of the actual performance of their products. But most of all – we hope that it inspires vendors, researchers, system integrators, entrepreneurs and hackers to seriously improve commodity based storage hardware. To help the industry create new, novel, cost effective products which are suitable for professional broadcasting based on generic storage.
The media storage meter software consists of a control application and a meter client.Ìý The control application runs on a Windows PC and controls multiple meter clients that can run on different operating systems.
Users can specify access profiles to match the behaviour seen when using real production tools, such as video capture, scrubbing, editing and rendering. Now instead of saying the storage seemed a bit slow when we made such and such a program, it’s possible to quantitive measure how blistering fast or not the storage is. Also, because everyone really hates doing testing, test scripting has been added. Users can click a button to run multiple tests over night and not interfere with the day to day production activities. This makes tests repeatable, a major plus when trying to explore the plethora of storage options available.
If you want try out the new Media storage meter, and prove that your new storage design has what it takes - the software and instructions are .
Comment number 1.
At 16th Jun 2011, Kit Green wrote:There are two types of storage.
1) New
2) Full
More seriously it seems that ³ÉÈË¿ìÊÖ S&PP have the same problem as everyone else, clients that delay sign off so ever increasing amounts of storage are required as deletion is delayed.
I hope you are liaising with them to find cost effective solutions.
Complain about this comment (Comment number 1)