MSN Hotmail Group Controls Rate of Change and Establishes ...



Overview

Country or Region: United States

Industry: Technology

Customer Profile

The MSN® Hotmail® group at Microsoft develops and maintains the Internet-based e-mail service for nearly 220 million users worldwide.

Business Situation

In the competitive world of Web-based e-mail, the MSN Hotmail group wanted to better control the rate of change in its environment, yet maintain its flexibility and agility to respond to market demand.

Solution

The MSN Hotmail group adapted Microsoft® Operations Framework best practices to fit the unique aspects of its Internet-based product environment.

Benefits

■ Controlled change

■ Improved visibility into operations

■ Increased confidence level

■ Reduced errors and outages

■ Improved efficiency

| | |“These team members have proven that MOF processes and procedures can work in this environment. Consistent processes have actually made them more rather than less efficient.”

Tony Stilling, MSN Hotmail Lead Program Manager, Microsoft

Tony Stilling, MSN Hotmail Lead Program Manager, Microsoft

| |

| | | |The MSN® Hotmail® group at Microsoft provides Internet-based e-mail service to more than 215 million |

| | | |users worldwide. With a user base that was growing exponentially and pressure to respond quickly to |

| | | |market demands, the group’s challenge was to control the rate of change in its environment. Processes|

| | | |for managing change were inconsistent across the group’s seven teams, and isolated changes often were|

| | | |made without management fully understanding the impact organizationwide. To resolve the issue, the |

| | | |group implemented change control processes as outlined in Microsoft® Operations Framework (MOF), |

| | | |adapting the processes to fit the unique aspects of the group’s online-based environment. With new |

| | | |processes in place, management is now able to control the rate of change in the environment and has |

| | | |renewed confidence in the group’s ability to attain aggressive performance and availability goals. |

| | | | |

| | | |[pic] |

| | | | |

Situation

Less than a decade ago, Internet-based e-mail was a novel concept that offered users access to e-mail from anywhere in the world using a Web browser—a convenience that most private e-mail systems couldn’t provide. Since 1997, Microsoft has provided Internet-based e-mail service to millions of users through MSN® Hotmail®. In just eight years, subscriptions have grown at phenomenal rates: 358 percent from 1997 to 1998; 285 percent from 1998 to 2000; and 282 percent from 2000 to 2005. Today, MSN Hotmail has 215 million users, placing Microsoft in the number-one position in the Internet-based e-mail marketplace.

For the MSN Hotmail team at Microsoft, the ability to accommodate the needs of a rapidly growing user base was just one of many hurdles it has faced over the years. Some of the group’s greatest challenges have been in establishing processes and procedures that are appropriate for developing an Internet-based “product”—actually a service. With a company for which the majority of products are boxed and packaged applications that are sold, an Internet-based service is somewhat of an anomaly. Typical processes and procedures that work well for other Microsoft product development teams do not necessarily translate well to the MSN Hotmail group.

To begin with, the desktop-to-server ratio of a traditional product doesn’t apply to Web-based applications. A Microsoft® Exchange Server installation, for example, has a certain number of desktops per server. When users have an issue, they contact the Help Desk, which offers one-on-one assistance. “With MSN Hotmail, we are dealing with literally thousands of servers and millions of users around the world that we can’t just reach out and touch,” says Tony Stilling, MSN Hotmail Lead Program Manager for Microsoft. “The only way we know when there is a problem is when a customer escalates an issue or when our monitoring alerts us.”

Secondly, the product development cycle for MSN Hotmail is much shorter than for typical boxed software. With fierce competition in the Web-based e-mail market, there is great pressure to respond quickly to customer demand for more and better features—for instance, larger inboxes and fewer restrictions on the size of attachments. “We need to have an intuitive, fast, and reactive attitude toward change in this environment,” says Stilling. Traditional processes used in IT organizations are not necessarily conducive to that agile, reactive mindset.

Underlying the day-to-day tasks of product development, the MSN Hotmail group has also dealt with the daunting project of migrating its product to the Microsoft Windows® operating system. (The original Hotmail program, which Microsoft acquired in 1997, had been developed on FreeBSD and UNIX platforms.) To minimize disruption and maintain ongoing business, the group took a phased approach to the migration project. Although limited migration efforts continue today, the environment has been quite stable for the past year.

Even so, in the absence of consistent processes, the group continued to struggle with managing change in the environment. Although engineers wrote implementation plans for proposed changes, and a change advisory board (CAB) reviewed and signed off on those plans, often the key questions of “What are we doing?” “Why are we doing it?” and “When are we doing it?” were not being thoroughly asked and answered.

The more crucial problem was that the overall impact of proposed changes was not being fully investigated or understood. Many different engineering teams developed different components of the system, yet there was too little interaction or communication between teams. Often one team’s proposed changes could have serious impact on another team. In addition, it was too easy for an individual to propose a change to a single engineer and completely bypass the CAB approval process. Ad hoc changes could cause errors that required many hours of labor to rectify or, worse yet, interrupted service to users. One such case resulted in a widespread outage that lasted several hours.

To avoid these types of pitfalls, the management team believed it needed to establish some consistent processes and procedures groupwide, particularly in the area of change control. “We knew we needed to establish consistent processes, but we wanted to ensure that they would be appropriate for the peculiarities of our environment and product,” says Stilling. At the same time, engineers were fearful that instituting change control processes would be restrictive and time-consuming, and cause the group to lose the agility that was so crucial to remaining competitive in the marketplace.

Solution

Recognizing that many of its customers face the same types of issues, Microsoft provides best practices guidance for IT organizations that run systems based on Microsoft environments through its Microsoft Operations Framework (MOF). The framework includes a process model that defines four “quadrants” or general areas of operation: Changing, Supporting, Operating, and Optimizing. Within each quadrant, an IT organization addresses processes and procedures for specific service management functions (SMFs). In the Changing quadrant, the service management functions are configuration management, release management, and change management.

Although MOF is designed to provide high-level guidance for any type of IT organization, some Microsoft customers who develop Web-based products weren’t convinced that the model was appropriate for their circumstances. “Much of the MOF implementation focuses on a desktop-heavy environment—that is, desktops being served by a few servers. For teams that work on Web-based products and services, the environment is server-heavy, and therefore the business and architectural models are very different,” says Stilling. “Many customers had tried to run their online systems using MOF processes, and it just wasn’t fitting. For instance, MOF describes how the service desk should work, but these types of groups typically don’t have service desks,” adds Stilling.

Stilling, who had successfully applied MOF principles in a different group within Microsoft, believed that MOF processes could be tailored to fit a server-heavy environment. In June 2005, he launched a project to apply MOF principles for change control in the MSN Hotmail group.

Stilling began by conducting an informal assessment with the group’s engineering managers and leading engineers. The purpose was to identify the weaknesses in the current processes and determine what should be done immediately to strengthen those processes. They decided to focus specifically on controlling the rate of change in the environment. While they would always need to remain agile and able to react quickly to market pressure, they wanted to be in control of when and how change happened within the group.

Engineers initially resisted the proposal to institute change control processes. They were fearful that they would get bogged down in administrative paperwork and become less efficient in their jobs, which could jeopardize the product’s competitive edge in the marketplace. To allay those fears, management decided to deploy change control procedures within each of the seven MSN Hotmail teams one at a time. Stilling worked individually with the frontline engineers to implement the new procedures and answer questions one on one. To further dispel engineers’ concerns about administrative red tape, management took a less restrictive view of change control than MOF outlines. For instance, MOF suggests that the CAB meet regularly to review proposed changes. The CAB for the MSN Hotmail group, however, reviews all proposed changes using e-mail, which allows them to respond quickly to the needs of the business.

Stilling is careful to make the distinction between change management, as defined by MOF, and change control. Change control is a first step that entails establishing sound processes for getting control of changes in the environment. Change management involves delivering a change management system. “People are often too anxious to jump right in and start using a change management tool before assessing the suitability of the tool to the group’s needs,” says Stilling. “The problem with that is that they end up tailoring their processes to the tool. It is critical that groups first test the processes manually and get that part right, then tailor the tool to those processes.”

The MSN Hotmail group already had a “ticketing system” in place that it used for incident management and release management. To test the new change control processes, the management team decided to use the existing ticketing system for service request management as well. “It made sense, because it links right in with our incident management system,” says Stilling. “When there is an alert on a system, a ticket is generated. Now we can tailor our work requests specifically to the information provided by the alert.” But the ticketing system is only a temporary tool. In the future, once the change control processes are stabilized across the entire MSN group, the tools team will develop a tool specifically designed for change management.

Benefits

The MSN Hotmail group has benefited significantly from change control measures it has instituted. The group has not only achieved its original goal of controlling the rate of change, but more importantly, has proven that MOF processes can work and be successfully tailored to an Internet-based development environment. “We've evolved MOF and have successfully deployed processes traditionally designed for workstation-heavy environments into a server-heavy environment,” says Stilling.

Controlled Change

The Web-based development environment of MSN Hotmail is fast-paced; the group needed to focus on controlling not simply change, but the rate of change in its environment. Before the group established consistent change control processes, an engineer could make an unplanned change—often thought to be minor—without receiving CAB approval. Such changes often had far greater impact on other development teams and the overall system than anyone realized. To control the rate of change, MSN Hotmail established a change control process that required all proposed changes to be reviewed and approved or rejected by the CAB. The CAB then scheduled those changes so as to avoid conflicts and minimize impact on development teams across the organization.

The group recognized that requiring CAB review and approval for every change would not be efficient or cost-effective long term, but that it was a necessary interim step. Once the CAB was able to get control of the rate of change, it established a Change Library, which allows certain changes to be applied without CAB approval. To further facilitate efficiency and reduce ”process overhead,” the group will soon implement new processes that define three levels of change: known, tested changes that can be automatically approved; minor changes that require only change manager approval; and major changes that still require CAB approval.

Improved Visibility into Operations

Using new change control procedures, everyone—from engineering managers to team members—is able to view proposed changes, approved changes, and the schedule for implementing changes. “Using the ticketing system empowers people to keep an eye on a service and the way it’s running. It also enables engineers who are troubleshooting to retrospectively look at the Change Record and review the history of changes that have been made,” says Stilling. “We simply couldn’t do this before without searching through e-mail messages or meeting with someone who could recount the history of a service or an incident.”

Increased Confidence Level

As a direct result of the improved visibility into operations, change is far more predictable now. That predictability has been a critical factor in raising management’s level of confidence in the system. “Before, management was reluctant to raise performance and availability goals, because it was difficult to predict how change would affect the environment overall,” says Stilling. “Now, with control over the rate of change, management is far more confident not just in raising our goals, but in our ability to achieve them.”

Reduced Errors and Outages

Before the MSN Hotmail group had consistent processes and procedures in place, errors and outages were inevitable. Now that the CAB reviews all proposed changes, approves or rejects each one, and then schedules implementations, the incidence of errors and outages has been greatly reduced. “In this business, we want to avoid outages at all costs because so many millions of users depend on our service daily,” says Stilling. “Change Control has already allowed us to avoid potential outages because implementation errors are spotted at the peer review and CAB approval levels. As a result, we’ve already enjoyed increased availability.”

Improved Efficiency

Five months after testing, refining, and establishing the new procedures, management held an open-house meeting for all engineers and group members to voice their opinions. The crucial question management asked was: “If you had the opportunity to go back to the old environment before change control was in place, would you?” Almost unanimously, the answer was a resounding no. “Everyone has worked very hard and made tremendous progress in a very short time. These team members have proven that MOF processes and procedures can work in this environment. Consistent processes have actually made them more rather than less efficient,” says Stilling.

The change control project has been a great success for the MSN Hotmail group. Now that change control procedures have been tested, proven, and stabilized, the next step is to have the tools team build a change management tool to automate these processes. In 2006, the group plans to address configuration management, another service management function defined by the MOF Changing quadrant. The change control processes established by the MSN Hotmail group will be deployed across other teams within MSN.

-----------------------

| |Solutions

■ Microsoft Operations Framework | | |

© 2006 Microsoft Corporation. All rights reserved. This case study is for informational purposes only. MICROSOFT MAKES NO WARRANTIES, EXPRESS OR IMPLIED, IN THIS SUMMARY. Microsoft, Hotmail, MSN, and Windows are either registered trademarks or trademarks of Microsoft Corporation in the United States and/or other countries. All other trademarks are property of their respective owners.

Document published January 2006 | | |

For More Information

For more information about Microsoft products and services, call the Microsoft Sales Information Center at (800) 426-9400. In Canada, call the Microsoft Canada Information Centre at (877) 568-2495. Customers who are deaf or hard-of-hearing can reach Microsoft text telephone (TTY/TDD) services at (800) 892-5234 in the United States or (905) 568-9641 in Canada. Outside the 50 United States and Canada, please contact your local Microsoft subsidiary. To access information using the World Wide Web, go to:

“It is critical that groups first test the processes manually and get that part right, then tailor the tool to those processes.”

Tony Stilling, MSN Hotmail Lead Program Manager, Microsoft

| |

“Now, with control over the rate of change, management is far more confident, not just in raising our goals, but in our ability to achieve them.”

Tony Stilling, MSN Hotmail Lead Program Manager, Microsoft

| |

................
................

In order to avoid copyright disputes, this page is only a partial summary.

Google Online Preview   Download