Case Study: Parallel Internet: Inside the Worldwide LHC computing Grid
Face to face with the network that will help scientists discover the origins of the Universe.
By Jon Brodkin , Network World | Techworld | Published: 13:00, 28 April 2008
Raw data is sent over dedicated 10Gbips optical fibre connections to the CERN Computer Centre, which is known as "Tier-0" in the LHC Computing Grid. Here, raw data is sent to tape storage and also to a CPU farm which processes information and generates "event summary data." Subsets of both the raw data and summaries are sent to the 11 Tier-1 sites, including Brookhaven National Laboratory on Long Island, NY, and Fermilab in Illinois.
Each of the 11 Tier-1 centres are connected to CERN via a dedicated 10 gigabit per second link, and the Tier-1 centres are connected to each other by a general purpose research network. Each Tier-1 centre receives only certain subsets of information. Brookhaven, for example, is dedicated to ATLAS, one of several large detectors housed at the LHC, while Fermilab handles data from the CMS (Compact Muon Solenoid) detector.
The Tier-1 centres are responsible for reprocessing raw data, which is then kept on local disk and tape storage and distributed to Tier-2 centres, which are located in most parts of the world.
Tier-2 centres are connected to Tier-1 sites and each other by general purpose research networks, such as the US Department of Energy's Energy Sciences Network. Tier-2s are located mainly in universities, where physicists will analyse LHC data. Ultimately, about 7,000 physicists will scrutinise Large Hadron Collider data for information about the origins and makeup of our Universe, according to CERN.
The LHC collisions will produce 10 to 15 petabytes of data a year, says Michael Ernst of Brookhaven National Laboratory, where he directs the program that will distribute data from the ATLAS detector. Brookhaven, as a Tier-1 site, will be responsible for filtering data so it can be easily readable by scientists located at the more numerous Tier-2 facilities, Ernst says.
Brookhaven has about 1,200 multicore x86 servers dedicated to the LHC, along with disk and tape storage that holds seven petabytes of data. Ernst says Brookhaven will have to scale that storage up significantly by 2012, when he expects to be storing 13 petabytes of Large Hadron Collider data.
Worldwide, the LHC computing grid will be comprised of about 20,000 servers, primarily running the Linux operating system. Scientists at Tier-2 sites can access these servers remotely when running complex experiments based on LHC data, Pordes says. If scientists need a million CPU hours to run an experiment overnight, the distributed nature of the grid allows them to access that computing power from any part of the worldwide network, she says. With the help of Tier-1 sites such as Brookhaven, the goal is to make using the grid just as easy for universities as using their own internal networks, according to Pordes.
Asked if the LHC project is the most complicated thing he's ever worked on, Ernst gave a quick laugh and said, "Yeah, I would say so."
This article first featured in Network World.