Follow Us

We use cookies to provide you with a better experience. If you continue to use this site, we'll assume you're happy with this. Alternatively, click here to find out how to manage these cookies

hide cookie message

Case Study: Storing map data for ever

UDO and HT FIleStore form Ordnance Survey's bridgehead

Article comments

The UK's main map supplier, the Ordnance Survey, is storing its digital map data on Plasmon UDO disks With BridgeHead HT FileStore software. The data has to be kept permanently.

Ordnance Survey

Back in 1791 the British Government realised that in planning adequate defences to repel any invasion, the South Coast of England needed to be comprehensively and accurately mapped. So it instructed its Board of Ordnance – the defence ministry of its day – to speed the necessary survey work. That decision led to the mapping of the whole of Great Britain in detail, and is also the source of the intriguing name 'Ordnance Survey' – an organisation which has grown to become the world's leading map-maker and a major provider of digital geographic information.

Today it is a self-financing £100-million-a-year civilian organisation producing computer data products and paper maps for business, leisure, administrative and educational use. It is still part of Government, but it covers its operating costs by selling its products and services or licensing others to use its copyright material.

Although best known for its paper maps, computer data already accounts for more than 90 percent of Ordnance Survey’s business. Digital product customers include the Land Registry, Forestry Commission, Police and Fire services throughout Britain and other Government departments. It is also involved in a collaborative project to produce a geographical blueprint of Great Britain, the Digital National Framework.

Digital mapping

During most of its 200-year existence the Ordnance Survey has jealously safeguarded its paper map data. That style of map-making has been swept aside in a digital flood of aerial and satellite imagery.

Ordnance Survey is creating the definitive national geographic database, digitally mapping the changing face of Britain for perpetuity. Its considerable investment in global positioning systems, digital aerial photography and new surveying techniques enables it to constantly update, enhance and maintain the database with at least 5,000 changes every day.

High definition digital aerial photography provides a significant part of this project, with camera data taking up more than 20 percent of Ordnance Survey’s primary disk storage space in 2005. This is expected to grow by 20 percent annually.

This digital data flood is huge and constant. Dave Lipsey, the information systems infrastructure manager said: “We anticipate an average of over 40TB of raw data will need to be archived each year, with files of over 1GB not uncommon. The archive solution we chose had to be capable of managing such a high volume of files of this size, for the longest time period possible."

The more than 40TB of data consists of new cartographic, photographic and administrative information. These images and records provide the foundation for the range of mapping products and geographic services offered to leisure, commercial and government customers. Ordnance Survey views the need to cost-effectively retain and access an authentic version of these digital records for an indefinite period of time as a critical aspect of its business. This mapping data is its set of crown jewels. If it is lost the Ordnance Survey is lost.

The data has to be kept effectively for ever. It cannot be lost, degraded or mis-placed. It's store has to be the equivalent of a storage Fort Knox and provide an utterly reliable and long-lasting storage facility.

The data is special; it is unchanging fixed content.

SAN not up to the job

Historically, Ordnance Survey used online disk storage, adding hardware to absorb the data growth, a strategy based on the falling price of disk. However, with the increased use of digital graphical data, the SAN-based storage with Cisco MDS 9000 switches, soon felt the strain. Not only is Ordnance Survey creating millions of new images each flying season, but also the underlying files are massive; each raw file is 700 megabytes (MB) and edited JPEGs are a minimum of 90MB.

The SAN storage method was not going to be affordable and sustainable over the long term. It estimated that the disk-based method was costing about £2,500 per Terabyte (TB) of stored data.

The continual creation of image files that needed to be retained forever was accumulating on the SAN faster than it could be backed up. The system was starting to get more errors with unavailable tapes and backup time windows were being exceeded, putting other systems at risk. Ordnance Survey also estimated that within four to six months, the data being received would surpass the backup capabilities, putting the organisation at risk in the event of a disaster.

It had almost come to a physical hard stop and it had reached capacity limits of its online disk hardware. In addition, although the cost of online storage was coming down, Ordnance Survey did not benefit from the cost reductions as the data management costs soared.

Something had to be done, meaning a new archive system chosen.

Lipsey said: "We knew that throwing capacity at this problem was not the answer. The best solution for reducing the burden of the growing digital geographical data storage needs was automated archiving to long life removable media. A new system would need to operate openly, integrate into our existing infrastructure, and provide multiple copies for backup and robust disaster recovery."

Media choices

The media/drive choices were relatively simple: tape; hard drives, optical disks, meaning archive-quality and not consumer CD/DVD-class media. There is only one archive-class optical media and that is Plasmon's UDO. Nothing else is available now with UDO's 50-plus year lifespan.

By deploying UDO, Ordnance Survey could comfortably operate on an 8-12 year data migration cycle, which is much greater than that of any disk or tape archive. This extended media lifecycle helps to secure record integrity while significantly reducing its long-term cost of operation.

A magnetic tape library could match the storage life of UDO but only by periodically refreshing the tapes. UDO can be written to in WORM format and left alone. It's much cheaper to look after a UDO archive than any tape format archive.

Any hard drive-based solution has to deal with the basic fact of HDD life; hard drives break. That meant some form of RAID protection would be necessary with the result that the per GB storage cost was higher than UDO's. Disaster recovery (DR) needs meant two archives would be needed anyway.

Ordnance Survey worked with OptoMedia, a storage systems integrator, to deploy a Plasmon G638 UDO library with 19TB of online storage capacity for the primary facility, and a second, smaller Gx24 UDO library for the DR site.

Software choices

Having alighted on the media the next choice was the software

BridgeHead Software’s HT FileStore software was selected for its ability to automatically classify file data and select it according to policy for inclusion in the UDO-based archives.

Lipsey said: "Magnetic disk and tape can’t match the media life provided by UDO, and with BridgeHead’s storage management software, we can safely archive, manage and migrate our data for ever.”

HT FileStore provides a robust level of “protected data lifecycle management” for the secondary storage repositories it manages.

Its self-replicating archive feature eliminates nearly all the expense of performing backup and disaster recovery procedures. Secondary archive media is stored offsite at the disaster recovery location, which can be quickly accessed without time-consuming recovery in the event of a major failure at the primary facility.

HT FileStore writes copies of data to both UDO locations simultaneously, providing a DR capability. At the same time, the system ensures that primary disk space is used to its optimum, that archived data is accessible either transparently or actively, depending on Ordnance Survey’s needs, and that it is fully searchable.

HT FileStore provides a robust level of “protected data lifecycle management” for the secondary storage repositories it manages.

BridgeHead’s HT FileStore provides long-term data management and uses intelligent automated policies to locate new files and archive them to the on-site and off-site Plasmon G-Series UDO libraries. After a given time, it checks that the archive is good and then leaves a stub impression of just 1KB on the server in place of the original larger raw file, thus reducing the huge cost and management burden of primary storage and at the same time allowing fully transparent access to the file in the archive.

The stubbed files are also indexed so users can perform in-depth searching of the archive. Once found a file or files will be restored to the required location, allowing them to be worked on. Users see, in effect, a single tier of storage.

Lipsey said: "Simply put, it’s like an automatic valve, controlling the flow of data in, out and within the system policing the data flow through the defined policies. Rather than manually guessing which files are available to be archived or deleted, the process in now completely automated."

HT FileStore also manages data migration to new media in the future, providing enduring resilience. Ordnance Survey will be using more than 1,300 pieces of 30 Gigabyte (GB) UDO media to archive historic and future information and has plans to implement 60GB UDO2 media when it is released in 2007.

"Working with Ordnance Survey has shown us the true meaning of storage forever," commented Tony Cotterill, managing director of BridgeHead Software. "One should never underestimate the importance to any business of having easy access to its data, whenever that data might be needed in the future."

Cheaper and better storage

The Bridgehead/Plasmon system provides a long-term and sustainable storage platform. It has already reduced the cost of storage to just £1,200 per TB with a further decrease to £600/TB anticipated in the future.

Ordnance Survey has already saved 20TB of capacity in the first few months of operation.

"The combined solution from Bridgehead Software and Plasmon is the latest in archiving software and long term storage media technology, and will save Ordnance Survey almost £500,000 in storage costs," said Lipsey. "The project will breakeven in four years, with an investment return rate of 13.5 percent, a figure that is well above the Government minimum of 5 percent and our own benchmark of 7 percent."

He also said: "Plasmon’s UDO storage media has a life expectancy of 50 years, unparalleled in technology available today. And BridgeHead’s HT FileStore enables us to securely migrate data to any future technology as and when it becomes available, no matter which vendor develops it."

Takeaways

1. A SAN is not well-suited for the job of long-term archiving of WORM data.
2. Nor is tape.
3. You need archive software to integrate the archived data with daily operations and transfer data back and forth.
4. One archive is not enough. You need a second for DR purposes.
5. Having a virtual single tier of storage for users makes their life easier.


Share:

More from Techworld

More relevant IT news

Comments




Send to a friend

Email this article to a friend or colleague:

PLEASE NOTE: Your name is used only to let the recipient know who sent the story, and in case of transmission error. Both your name and the recipient's name and address will not be used for any other purpose.

Techworld White Papers

Choose – and Choose Wisely – the Right MSP for Your SMB

End users need a technology partner that provides transparency, enables productivity, delivers...

Download Whitepaper

10 Effective Habits of Indispensable IT Departments

It’s no secret that responsibilities are growing while budgets continue to shrink. Download this...

Download Whitepaper

Gartner Magic Quadrant for Enterprise Information Archiving

Enterprise information archiving is contributing to organisational needs for e-discovery and...

Download Whitepaper

Advancing the state of virtualised backups

Dell Software’s vRanger is a veteran of the virtualisation specific backup market. It was the...

Download Whitepaper

Techworld UK - Technology - Business

Innovation, productivity, agility and profit

Watch this on demand webinar which explores IT innovation, managed print services and business agility.

Techworld Mobile Site

Access Techworld's content on the move

Get the latest news, product reviews and downloads on your mobile device with Techworld's mobile site.

Find out more...

From Wow to How : Making mobile and cloud work for you

On demand Biztech Briefing - Learn how to effectively deliver mobile work styles and cloud services together.

Watch now...

Site Map

* *