Case Study: MPLS pays off

"The devil is in the details," says Gordon Bither, senior vice president for integration network solutions at State Street, the world's largest institutional investment company, with $9.8 trillion in assets under custody.

Such an attitude led State Street to create a MPLS network for carrying converged voice, data and video simultaneously over two carrier networks, dubbed an active/active architecture. With active/active, traffic travels the shortest routing path, jumping from one carrier network to the other as necessary.

A focus on detail also led State Street to create six standardised architectural models for funnelling traffic from global offices to its data centres and eight models for client connectivity. Among the client models are Gold and Silver options. Gold, the highest-availability service model, provides two routers for dual-carrier connectivity, says Steve White, vice president of network architecture and design at the Boston company. With Silver, two routers connect to a single carrier network, with ISDN for recovery, he says.

As of April, State Street has tied its eight data centres, 90 global offices and more than 250 clients to the new network, with nary a hiccup.

Validation first, contract second
Bither and State Street network engineers were thorough in plotting State Street's migration from the multicarrier frame-relay network it managed internally to the MPLS architecture it would rely on carriers to manage. Before they signed a contract, they wrote an architectural white paper for carriers, conducted 115 reference calls and spent 10 months gathering data on the carriers' global networks. They selected AT&T and MCI in November 2003, but didn't ink the contract until four months later. In the interim, State Street officials completed terms and conditions and set aggressive service-level agreements with the partners.

In another eight months of heads-down engineering, the MPLS team replicated the State Street WAN with a 72-router lab environment, and reverse-engineered the carriers' MPLS networks on that simulated backbone. Before it turned on the first site in October 2004, the project team validated all design models and thoroughly tested the MPLS architecture. "We had a very structured understanding of what we wanted to do," White says. "When the carriers' core engineers came here, we solved complex problems together, but we drove everything."

From the get-go, the project team understood that this new network would have to rely on enabling technologies, such as QoS. That way, if a business manager asked for videoconferencing, White could say, "No problem. We can do that today because we have QoS, and availability is baked into the architecture."

Framed
In the frame-relay days, a videoconferencing request could have sent engineers scrambling to come up with a custom approach for involved sites. The frame-relay network had become overly complex and inefficient, growing as a series of these point solutions, Bither explains. Ultimately, about 18 service providers - one international and many regional carriers - provided the connectivity for that aged hub-and-spoke architecture.

In 2002, the frame-relay network situation had become untenable. The WAN had become a constant in the company's daily report on IT problems, and it was negatively affecting business. Bither relates his experience: "[The report] would come out at 8 a.m., and I would get a call from the CIO shortly thereafter to address the WAN issues."

As network executives began thinking about architectural options, MPLS topped everyone's minds. "MPLS was always our desired goal, with convergence in mind," Bither says.

Also clear to Bither and the executive management team was that State Street needed to get out of the business of managing its WAN services. "By out-tasking WAN [moves, adds, changes, deletes] work, addressing and those types of jobs, I was able to hire higher-level resources who understand the overall architecture and the business," he says. "We maintain the thought leadership and allow others to do the repeatable process as a managed service."

With its months of net modelling, State Street made sure its managed-services providers could handle the job. "The success of a managed-services environment is to design your environment around the managed service. Your models and your architecture need to support the service provider's ability to provide what you're looking for," White says.

But Bither didn't want to be too rigid: "You can't demand something of a service provider that a service provider can't provide," he says.

A standardised, supercore model
State Street's due diligence in designing the fully managed, dual-carrier, active/active MPLS network has paid off, time and again. More than once, either the AT&T or MCI portion of State Street's global MPLS network has failed, with no impact on the customer, because traffic is load balanced across the networks, Bither says. "No one has known, and I like it that way," he adds.

Plus, with the day-to-day tasks out of the way, Bither's network specialists can focus on how to capitalise on MPLS. They're implementing multicasting and VoIP from Cisco, for example.

State Street leverages a supercore architecture interconnecting redundant hubs in eight global locations. Each node comprises various Cisco switches - from the high-end Catalyst 6500s to the low-end 3700s. Traffic from metro Ethernet LANs is consolidated within the supercore hubs, which themselves are linked over a dual-carrier dense wavelength division multiplexing network. This model facilitates State Street's ability to control routing within a region while maintaining any-to-any connectivity across the globe, White says. "Although there is complexity associated with this, we've kept the design as simple as possible and have driven out human intervention" by establishing highly structured repeatable processes and the 14 standardised architectural models, he adds.

The MPLS project team rarely needs a custom approach for any business request, a huge improvement over the frame-relay days, Bither notes. Obviously, this also means productivity has skyrocketed - "all that's required is a quick look to make sure the bandwidth is properly engineered," he says. What's more, in the six-plus months since the network went live, it hasn't once appeared in a daily problem report, and response times have improved 10 percent to 25 percent globally.

State Street's project proves again that a no-short-cut approach is often the fastest route to improvement.