High energy physics team captures network prize

Researchers from the U.S. Department of Energy's Fermi National Accelerator Laboratory and Stanford Linear Accelerator Center recently joined an international team in shattering the world network speed record. To capture first place in the SC|05 Bandwidth Challenge, the team of high energy physicists, computer scientists and network engineers led by the California Institute of Technology transferred physics data at a rate of over 150 gigabits per second--equivalent to downloading over 130 DVD movies in one minute.

The Bandwidth Challenge was held in Seattle November 12-18, 2005 at SC|05, the International Conference for High Performance Computing, Networking, Storage and Analysis. The contest challenged scientists and networking engineers to create the best and most advanced techniques for using vast amounts of data and showcasing it on advanced networks. Bandwidth Challenge entries were evaluated based on their ability to support the mission of science and to maintain sustained data throughput.

In preparation for its third consecutive win, the Caltech-led team transferred 475 terabytes of high energy physics data in 24 hours and sustained average data rates of more than 100 gigabits per second for several hours. Data from particle physics experiments were transferred from laboratories and universities worldwide, routed through the SC|05 exhibit hall in Seattle, and stored either in the hall or at other institutions around the globe. The official peak throughput was 131.6 gigabits per second, measured during the Challenge on 17 of the 22 optical fiber links used by the team.

"This is much more than measuring how quickly we can send bits," said Fermilab's Don Petravick. "Our Bandwidth Challenge entry used applications that transport actual physics data at high rates, essential to the high energy physics experiments that we support. We brought together people and resources from universities and laboratories on four continents, demonstrating the value of collaboration and our ability to achieve records while transferring data from running experiments."

SLAC transferred data from its BaBar experiment, and Fermilab transferred data from its CDF and DZero experiments, the CMS experiment at the CERN laboratory in Switzerland, and the Sloan Digital Sky Survey through seven 10-gigabit optical links in the Fermilab/SLAC booth at the SC|05 exhibit hall. Software and systems developed at the two laboratories aided the team's high-speed disk transfers, some of which took place over transoceanic network links.

"Demonstrating sustained application throughputs of well over 100 gigabits per second for real physics data over international links will help open the doors for the next generation of data-intensive science," said Les Cottrell, assistant director of Scientific Computing and Computer Services at SLAC. "We learned valuable lessons about the current limitations, bottlenecks and instabilities in today's high-performance components that can be applied to improving management, monitoring and diagnosis and identifying future requirements."

The next generation of data-intensive science will arrive in 2007, when the world's highest-energy particle accelerator, the Large Hadron Collider, begins operating at CERN in Geneva, Switzerland. The Bandwidth Challenge entry was part of the preparation for the torrent of data that will come from the LHC experiments. Thousands of physicists around the globe will need to access and analyze LHC data in their quest to solve the mysteries of the universe. The processing, distribution and analysis of the data will be completed using high-speed optical networks, software to monitor and manage the data flows across the networks, and grid computing.

"This demonstration allowed us to preview the globally distributed grid system of more than 100 laboratory- and university-based computing facilities that is now being developed in the U.S., Europe, Asia and Latin America in preparation for the next generation of high energy physics experiments at the LHC," said Caltech's Harvey Newman, head of the Bandwidth Challenge team.

To achieve extraordinary rates of data transfer in the Bandwidth Challenge, the team used technology developed by SLAC and Fermilab researchers.

As high energy physics data is made up of many event files, accessing all the necessary events from disk every time they are requested takes longer than scientists would like. The PetaCache, a cluster of machines with large amounts of memory developed at SLAC, ships high energy physics data 10 times faster than from disk. SLAC data were transferred using the bbcp and xrootd applications developed at SLAC and INFN-Padova in Italy.

Data from Fermilab were transferred using the dCache data transport application developed by Fermilab and the DESY laboratory in Germany. Lambda Station, also developed at Fermilab, was exhibited as a means to route high-volume data flows between the high-speed optical networks and their ultimate endpoints.

The winning Bandwidth Challenge team also included researchers from CERN in Switzerland; the University of Manchester in the UK; the Universities of Michigan and Florida, Vanderbilt University and Brookhaven National Laboratory in the U.S.; the University of Sгo Paulo, Sгo Paulo State University and Rio de Janeiro State University in Brazil; Kyungpook National University and the Korea Institute of Science and Technology Information in Korea; and the KEK Laboratory in Japan.

Source: Fermi National Accelerator Laboratory

Related stories:

California Scientists Demonstrate How to Use Advanced Fiber-Optic Backbone for Research
(PhysOrg.com) -- How can super-fast networking among research institutions in California help scientists make new discoveries? Researchers, campus administrators and networking infrastructure officials converged on the University of California, San Diego in September to find out.
Physicists Set New Record for Network Data Transfer
An international team of physicists, computer scientists, and network engineers led by the California Institute of Technology, CERN, and the University of Michigan and partners at the University of Florida and Vanderbilt, as well as participants from Brazil (Rio de Janeiro State University, UERJ, and the State Universities of São Paulo, USP and UNESP) and Korea (Kyungpook National University, KISTI) joined forces to set new records for sustained data transfer between storage systems during the SuperComputing 2006 (SC06) Bandwidth Challenge (BWC).
Challenging the World's Largest Computing Grid
Enough data to fill 17,000 CDs were transferred from Edinburgh University to the CCLRC Rutherford Appleton Laboratory (RAL) in nine days, as part of the latest networking challenge by particle physicists. Delegates to the UK e-Science All Hands Meeting in Nottingham will be able to watch the progress of the challenge in real time on a screen showing data being exchanged between sites worldwide.
Researcher Looks For Better Way to Kill Cancer Cells
Physics Professor Diandra Leslie-Pelecky brought more with her when she arrived at UT Dallas than expertise in nanotechnology and shiny behemoth lab equipment. She brought an award for $84,000 from the National Institutes of Health via the Cleveland Clinic.
Spallation Neutron Source sends first neutrons to 'Big Bang' beam line
New analytical tools coming on line at the Spallation Neutron Source, the Department of Energy's state-of-the-art neutron science facility at Oak Ridge National Laboratory, include a beam line dedicated to nuclear physics studies.
Scientists create superconducting thin films
(PhysOrg.com) -- One major goal on the path toward making useful superconducting devices has been engineering materials that act as superconductors at the nanoscale -- the realm of billionths of a meter. Such nanoscale superconductors would be useful in devices such as superconductive transistors and eventually in ultrafast, power-saving electronics.
Broken symmetry: Answering the solace of quantum
Humans like the comfort of symmetry -- the identical image in the mirror, the matching wings of the baroque mansion, the equal numbers in opposing football teams.
World's biggest computing grid launched
(PhysOrg.com) -- The world’s largest computing grid is ready to tackle mankind’s biggest data challenge from the earth’s most powerful accelerator. Today, three weeks after the first particle beams were injected into the Large Hadron Collider (LHC), the Worldwide LHC Computing Grid combines the power of more than 140 computer centers from 33 countries to analyze and manage more than 15 million gigabytes of LHC data every year.

News discussion:

Physics news