Operations Run Book - WordWonk



Operations Run Book

Enter Client’s Name Here

|Prepared By: |Managed Services, Champion Solutions Group |

|Version: |1.00 |

|Publication Date: |10/03/05 — 2:05:18 PM |

|DOCUMENT IDENTIFICATION INFORMATION |

|Document Name: |Operations Run Book for Enter Client’s Name Here |

|Version: |1.00 |

|Date Created: | |

|Created By: | |

|Date Published: |October 3, 2005, 2:05 PM |

|Security Classification: |Restricted for use by IBM/CSG Managed Services Clients |

|Creation Software: |Microsoft Word 2003 |

|Contributors: |MSOC Team |

|CHANGE HISTORY |

|Ver. |Date |Change Description |Approval |

|0.01 | |Initial draft for review by MSOC team | |

|1.00 | |Initial version for publication | |

| | | | |

| | | | |

| | | | |

| | | | |

|DOCUMENT REVIEW |

|Name/Title |Signature |Date |

| | | |

| | | |

| | | |

DISTRIBUTION LIST

IBM Managed Services

Champion Managed Services

All IBM/CSG Managed Services Clients

DOCUMENT LOCATION

This document is available via the Champion Portal at .

|Client’s Final: |The Client’s copy is stored on the CMS portal under their specific document area. |

|Template |To obtain the internal template from the CMS portal, please click here, or enter the following URL in your |

|(CMS Use Only): |Web browser: |

| |. |

Table of Contents

Introduction 1

Contact Information 1

Champion 1

IBM 1

Escalation Process 2

Champion Group / IBM Web Portal 3

Champion Group / IBM Phone Support 3

Infrastructure 3

Facility Overview 3

Site restrictions 3

Fire and emergencies 3

Shipping To The Facility 3

Directions 4

Shared Common Areas 4

Hardware Configuration 5

Operating Procedures Overview 6

Introduction 6

Remote Accessibility 7

Purpose 7

Scope 7

General Policy 7

Requirements 7

Enforcement 8

Operating System Permissions 8

Purpose 8

Scope 8

Policy 8

Ownership and Responsibilities 8

General Configuration Guidelines 9

Compliance 9

Server Setup 9

Overview 9

Scope 9

Policy 9

Ownership and Responsibilities 10

Backup Configuration 11

Software 11

Policies 11

Data Restoration Process 12

Purpose 12

Overview 12

Incremental Backups 12

Incremental Restores 12

Database Restores 12

Tape Handling and Retention 12

Retention Policies 12

DRM Tape Handling 12

Responding To Alerts 14

Change Management 14

Types Of Change Requests 14

Severity And Priority 15

Change Request Classifications 15

Scheduled Client Change 16

Scheduled CSG Change 16

Emergency Changes 16

Who Is Authorized To Request A Change? 16

How Is A Request Submitted? 16

Scheduled Maintenance Windows 17

Change Control Board 17

Decision Categories 17

Turnaround Time 17

Problem Management 17

Business Hours 17

Contacting Support (Champion MSOC) 17

Telephone 17

Champion Portal 17

Severity And Priority Levels 17

Trouble Ticket Workflow 20

Monitoring Standards 21

PURPOSE 21

REFERENCE 21

OVERVIEW 21

Thresholds 21

IIS Services 21

SQL Server 2000 23

Storage Area Network Switches 24

Webservers 24

Network Intrusion Detection System 24

Purpose 24

Scope 24

General / Policy 24

Enforcement 24

Operating System Patches / Service Packs 25

Microsoft 25

AIX 25

Linux 25

Disaster Recovery 25

Appendix A — Windows Server Security Checklist 26

Linux Security Checklist 33

Introduction

Welcome to the Managed Services Operations Center (MSOC) for Champion Solutions Group (CSG). As an IBM business partner, the MSOC has been established for the purpose of providing managed services for customers. This document serves as a centralized repository for all policies, procedures, and supporting documents that are associated with the day-to-day operations of the MSOC. The administrators and engineers are provided the ability to quickly and easily navigate to documentation needed to perform assigned duties accordingly.

Contact Information

Champion

|Managed Services Operations Center (To submit a request for service) |

|Telephone: |(888) 997-7789 | |

|Web Portal | | |

IBM

|Name & Title |Telephone |E-mail |

|Enter PM Name Here |Enter PM’s Phone # Here |Enter PM’s E-mail Here |

Escalation Process

The escalation process describes the information flow in case of non-compliance with minimum service levels. This escalation process applies to severity 1 calls only.

The following escalation sequence is to be utilized if a service is not delivered in a specific timeframe.

ESCALATION

|15 Minutes — MSOC Call Center | | |

|888-997-7789 | | |

| | | |

|30 Minutes — MSOC Supervisor, |AND |IBM Project Manager, |

|Eric Schneider | |Enter PM Name Here |

|561-251-6240 | |Enter PM’s Phone # Here |

| | |Enter PM’s E-mail Here |

| | | |

|45 Minutes — MSOC Manager, | | |

|Jay Kobert | | |

|954-646-2784 | | |

| | | |

|1 Hour — President, Managed Services |AND |IBM Project Executive, |

|Ian Sutcliffe | |Enter PE’s Name Here |

|561-997-2900, XT 262 | |Enter PE’s Phone Here |

| | |Enter PE’s E-mail Here |

Champion Group / IBM Web Portal

Champion Managed Services has designed a web portal called ‘The Pulse’, for our customers intended to provide various types of information, such as but not limited to:

1) Customer Infrastructure Documentation

2) Procedures/Processes

3) System Monitor Tools

4) On-line Service Requests

Champion Managed Services Portal URL:



Champion Group / IBM Phone Support

Customer’s can directly contact the Managed Services Operations Center directly via the telephone by dialing:

(888) 997-7789

Infrastructure

Facility Overview

The customer’s environment is maintained in the Champion Managed Services facility located in the IBM Atlanta BellSouth eBHC (eBusiness). For the purpose of this document, we will refer to eBHC as the “facility”.

The facility maintains several security features for your protection. Security technology may include biometric readers, cyberlocks, and interior and exterior motion-activated video surveillance cameras in selected areas.

Site restrictions

Smoking is not allowed in the facility. Unauthorized recording devices, including cameras and video recorders, are not permitted.

Fire and emergencies

The center maintains a fire suppression system. Emergency announcements are made by the facility manager. During a fire emergency, all visitors must report to the front parking lot and wait for the Onsite Operations staff to give a fire status. Emergencies should be reported promptly to the Onsite Operations staff.

Shipping To The Facility

Any request for shipments must be submitted through the Champion Managed Services Operations Center (MSOC). The details for requesting service (submitting a ticket) are located in the procedure titled “Creating A Request For Service” on Champion Managed Service’s portal ().

Be prepared to provide the following shipping information to the MSOC when scheduling the delivery:

• Name of carrier

• Way bill number

• Expected date and approximate time of arrival

• Number of packages

• Approximate weight and dimensions

• Specific handling instructions

If the shipment is going to be delayed, contact the MSOC to modify the shipping information.

All carriers must be instructed that all deliveries must indicate Inside Delivery.

Shipments must be addressed to:

BellSouth® c/o IBM Site Manager

Customer name/identifier

BellSouth® Trouble Ticket Number

675 W. Peachtree Street NW

Atlanta, GA 30308-1989

Directions

The address is:

675 W. Peachtree Street NW

Atlanta, GA 30308-1989

From Atlanta Hartsfield Airport

1. Follow the airport exit signs to Camp Creek Parkway.

2. Merge onto I-85 N toward I-75 N/ATLANTA.

3. Take the US-19/SPRING STREET exit (exit number 249D) toward US-29/W. PEACHTREE STREET.

4. Take the ramp toward US-19/US-29 N/US-78/W. PEACHTREE STREET.

5. Turn SLIGHTLY RIGHT onto LINDEN AVENUE NW.

6. Turn LEFT onto W. PEACHTREE STREET NW.

Shared Common Areas

The facility has a common area located past the mantrap. The common area is shared by all customers of the IBM e-business Hosting Center and has the following amenities:

• Eating area

• Vending machines

• Coffee machine

• Restrooms

• Conference room

Hardware Configuration

|Part No. |Qty. |Description |Server Name |Operating System |

| | | | | |

| | | | | |

| | | | | |

| | | | | |

| | | | | |

| | | | | |

| | | | | |

| | | | | |

| | | | | |

| | | | | |

| | | | | |

| | | | | |

| | | | | |

| | | | | |

| | | | | |

| | | | | |

| | | | | |

| | | | | |

| | | | | |

| | | | | |

| | | | | |

| | | | | |

| | | | | |

| | | | | |

| | | | | |

| | | | | |

| | | | | |

| | | | | |

| | | | | |

| | | | | |

| | | | | |

| | | | | |

| | | | | |

| | | | | |

| | | | | |

| | | | | |

| | | | | |

| | | | | |

| | | | | |

| | | | | |

| | | | | |

| | | | | |

| | | | | |

| | | | | |

| | | | | |

| | | | | |

Operating Procedures Overview

Introduction

Please note that the procedural content of this section is presented on a general, high-level basis. Please refer to the Champion Managed Services portal () for the detailed, step=by-step procedures.

The following serves as an overview of policies, procedures, and supporting documents that are associated with the day-to-day operations of the Managed Services Operations Center (MSOC). It is made available to the administrators and engineers, and provides them with the ability to quickly and easily navigate to the documentation that is needed to perform assigned duties accordingly.

Each procedure is structured to lead the engineer and management through steps to ensure the rapid and efficient completion of a particular task. In addition to the steps, general overviews are provided for clarity. After having completed a specific procedure several times, and have become familiar with its background, you will be able to use the document as a reference guide and proceed directly to the steps required.

Remote Accessibility

Purpose

The purpose of this policy is to define standards for connecting to Champion Managed Service's network and any hosted network environment that Champion manages from any host. These standards are designed to minimize the potential exposure to Champion Managed Services, and managed network infrastructures, from damages which may result from unauthorized use, out-dated / insecure encryption methods, and unsupported methods of connection to Champion Managed Service’s resources. Damages include the loss of sensitive or company confidential data, intellectual property, damage to public image, infrastructure device and/or Operating system configurations, and damage to critical Champion Managed Service’s internal systems.

Scope

This policy applies to all Champion Managed Service customers, customer clients, employees, contractors, vendors and agents that require connection to the Champion Managed Service network and customer-hosted network environments. Remote access implementations that are covered by this policy include, but are not limited to, dedicated internet circuits, dial-in modems, frame relay, ISDN, DSL, VPN, SSH, and cable modems, etc.

General Policy

1. The following policies outline details about different methods of accessing network resources via remote access methods, and acceptable use of Champion Managed Service's managed networks:

A. Acceptable Encryption Policy

1) ESP-3DES

2) Hash / ESP Authentication = MD5, SHA, AES-128, AES-192, AES-256

3) D-H group = 2

B. Virtual Private Network (VPN) Policy

1) Site to Site = IPSEC

2) Remote Access VPN = PPTP (Microsoft Client)

2. Based on business and application requirements for administration; the following additional methods are acceptable once a secure tunnel has been established or requests from the customer with acknowledgement of their insecurities can be established.

|A. Microsoft Terminal Services / Remote Desktop Protocol |

|B. Secure Shell |D. PCAnywhere |

|C. Telnet |E. RealVNC / VNC |

Requirements

1. Secure remote access must be strictly controlled. Control will be enforced via submission of a change request through the MSOC from authorized personnel from the customer.

2. At no time should anyone provide their login or email password to anyone.

3. Customers must submit all encryption details with the specific source to destination for the customer network. Details include the following: Peer IP Address, Pre-Shared Key, Specific host / network to Specific host / network destination.

4. Firewall change requests must be submitted to the MSOC. No firewall change requests will be completed without the completion of a Firewall Rule Request Form.

5. Frame Relay must meet minimum authentication requirements of DLCI standards.

6. Non-standard hardware configurations and security configurations must be approved by Champion Managed Services MSOC.

7. All hosts that are connected to Champion Managed Services managed environments and networks via remote access technologies must use the most up-to-date anti-virus software. This includes personal computers. Third party connections must comply with requirements.

8. Customer personal equipment that is used to connect to Champion Managed Service's managed networks is not supported.

Enforcement

If any of the above requirements are not met, Champion Managed Services will be responsible for damages that may be caused from the misuse of remote access policies. Service requests that do not comply with the policies in this run book may be subject to rejection by Champion Managed Services.

Operating System Permissions

Purpose

The purpose of this policy is to establish standards for the base configuration of server equipment that is supported by Champion Managed Services. Effective implementation of this policy will minimize unauthorized access to customer’s proprietary information.

Scope

This policy applies to server equipment supported by Champion Managed Services.

Policy

Champion Managed Services recommends the utilization of the server security best practices (see Appendix A). All server security considerations for application management will be defined by the customer, which may or may not affect SLA availability credits.

Ownership and Responsibilities

• All servers supported by Champion Managed Services are owned by the MSOC. The MSOC is divided into technical verticals to ensure efficient problem resolution. The technical verticals include:

o Network

o Server

o Storage

o Data Management

• Servers are registered within Champion’s enterprise management system. At a minimum, the following information is required to positively identify a given system:

o Server contact(s) and location, and a backup contact

o Hardware and Operating System/Version

o Primary functions and applications

• Information in the enterprise management system is kept up-to-date.

• Configuration changes for production servers follow the appropriate change management procedures.

General Configuration Guidelines

• The latest security patches must be installed on all systems as business permits.

• Trust relationships should be avoided whenever possible.

• Always use standard security principles of “least required access” to perform a function.

• If a methodology for secure channel connection is possible, privileged access must be performed over secure channels, such as encrypted network connections using SSH or IPSec).

• Servers are physically located in an access-controlled environment.

Compliance

• Audits will be performed on a regular basis by Champion Managed Services.

• Audits will be managed by the internal audit group.

Server Setup

Overview

The purpose of this policy is to establish standards for the base configuration on server equipment that is managed by Champion Managed Services. Effective implementation of this policy will minimize server setup time and ensure stability across environments.

Scope

This policy applies to server equipment owned by the customer and managed by Champion Managed Services. It defines the process of how the logical operating systems are setup in the Champion managed environment and prepped to accommodate customer SOR.

Policy

1. Champion Managed Services support and install the following operating system manufacturers and versions.

A. Microsoft

i. Windows 2000 Server

ii. Windows 2000 Advanced Server

iii. Windows 2003 Standard Server

iv. Windows 2003 Web Server

v. Windows 2003 Enterprise Server

B. Linux

i. Red Had, versions 9.0, AS 2.1, and AS 3.0

C. AIX, version 5.1 or later

D. VMWare

i. ESX

2. Once a server has been integrated with all work order hardware resource allocations (such as Processors, RAM, NIC’s, HBA’s, etc.), the following processes are followed:

A. Surveys of hardware chassis light indicators are conducted prior to operating system logical configuration.

B. Successful posts with bios confirmation of installed hardware on servers are conducted prior to operating system logical configuration.

C. Bios, RAID, etc. firmware versions are verified and updated with the latest available and recommended versions from the manufacturer.

D. Disk partitions are built as necessary based on customer requirements regarding OS partitions, and/or data partitions. (Pre-determined by customer and IBM/Champion Managed Services technical teams prior to build dates.)

E. Servers are installed with the requested Operating System. (Pre-determined by customer and IBM/Champion Managed Services technical teams prior to build dates.)

F. At the time of server operating systems installation, all available security and critical updates from the OS manufacturer are applied unless otherwise agreed to in writing by the customer and IBM/Champion.

G. IP’s will be bound to the network interface cards with verified server name instances per the customer requirements.

H. Operating system resources will be kept on the servers for necessary OS-based applications. For example: i386 directory will be kept on a windows 2003 server root drive unless customer requests otherwise.

I. For monitoring reasons, a local or domain account is created for perfmon statistics to be used for proactive monitoring. The use of a local account or domain account is determined by the projected network environment that the server will be participating in.

Ownership and Responsibilities

Champion Managed Services (CMS) maintains the server hardware and operating system instances. All applications are the responsibility of the customer.

In order to proactively manage the environment, a standard maintenance window, between 3:00 AM and 6:00 AM Eastern Standard Time each Sunday, is reserved for internal, Champion maintenance. The time at which Client-requested changes are made is determined during the change request process.

Backup Configuration

The policies discussed in this section are established standards, but can vary per a mutual agreement to a customer request.

Software

Server

TIVOLI Storage Manager (TSM) 5.x

Client

AIX, TSM Client Module  5.x

AIX, Agent Client  3.x

AIX, ORACLE (TDP Agent)

Linux, TSM Client Module  5.x

Linux, Agent Client  x.x

Linux, ORACLE (TDP Agent)

Windows, TSM Client Module 5.x

Windows, Agent Client x.x

Windows, ORACLE ***

Licensed, installable physical media is required for ALL of the above.

Policies

Policies are rules that are set at the IBM Tivoli Storage Manager server to manage client data. Policies control how and when client data is stored, for example:

• How and when files are backed up and archived to server storage

• How space-managed files are migrated to server storage

• The number of copies of a file and the length of time copies are kept in server storage

The standard policy consists of a standard policy domain, policy set, management class, backup copy group, and archive copy group. The attributes of the default policy are as follows:

Backup Policy

• Daily incremental backups will be taken

• An incremental backup is performed only if the file has changed since the last backup.

• Up to two backup versions of a file on the client’s system are retained in server storage. The most recent backup version is retained for as long as the original file is on the client file system. All other versions are retained for up to 30 days after they become inactive.

• One backup version of a file that has been deleted from the client’s system is retained in server storage for 60 days.

DRM Policy

• Duplicate copies of daily backups will be sent to offsite storage.

• The offsite storage retention policy follows the backup retention policy above.

Data Restoration Process

Purpose

The purpose of this section is to describe the steps necessary to restore data from backups.

Overview

In order to understand how to restore data from backups it is necessary to understand how backups are organized. Backups are organized on all systems with the following concept. All non-database related files are backed up under a nightly incremental backup. Database related files are backed up either using the associated TDP (Tivoli Data Protection) agent, or cold database backup.

Incremental Backups

Tivoli Storage Manager (TSM) uses a incremental strategy where the first time an incremental backup is run against the server everything is backed up. Everything means all data except database related files, which are excluded in a list maintained in the TSM client path. After the first backup of all files that have had their timestamp updated are incrementally backed up. This incremental method ensures that a full restore is available at all times.

Incremental Restores

Incremental restores may be requested, and must include:

|Source |Path |

|Destination |Date of source and path |

|Node |Estimated amount of data |

Database Restores

Oracle Database restores in an UNIX environment is handled by the customer via custom RMAN scripts. The RMAN script connects to the TSM Server and opens channel(s) to tape drives and passes the restore request between Oracle and TSM.

Tape Handling and Retention

Retention Policies

Policies are rules that are set at the IBM Tivoli Storage Manager server to manage client data. Policies control how and when client data is stored, for example:

• How and when files are backed up and archived to server storage

• How space-managed files are migrated to server storage

• The number of copies of a file and the length of time copies are kept in server storage

The standard policy consists of a standard policy domain, policy set, management class, backup copy group, and archive copy group.

DRM Tape Handling

Champion's Managed Service facility in Atlanta uses the services of Iron Mountain for the storage of media offsite. This procedure explains how to send tapes from the Atlanta facility to off site storage, and how tapes are retrieved from the off site storage vault when the retention policy expires.

Champion Managed Services standardizes on IBM LTO2 tape technology.

The application that is used to manage these libraries is the Disaster Recovery Manager (DRM) within the Tivoli Storage Manager (TSM), which enables these libraries. A copy is made of all backups that are scheduled, which are controlled by the DRM.

The retention policies set in TSM controls the DRM process that:

1. Sends tapes off site, and

2. Requests tapes to be returned from vault storage.

Sending Tapes Offsite (Onsite-to-Offsite)

To send tapes offsite, the following tasks must be performed:

1. Copies are made of daily scheduled backups.

2. Disaster Recovery Manager (DRM) sets the state on these copies indicating they are ready for offsite storage.

3. An automated process will run at 13:00 daily and move the DRMedia to the I/O trays of the libraries.

4. An Onsite to Offsite Report will be e-mailed to Champion Managed Services Division and the IBM On-site System Administration (OSSA) team daily at 13:00.

5. IBM OSSA team will verify the tapes in the I/O ports with the DRMedia Report upon removal.

6. IBM OSSA team will transfer the DRMedia to Iron Mountain for offsite storage.

7. IBM OSSA team will provide a hard copy of the DRMmedia Report to the Iron Mountain courier.

8. IBM OSSA team will fax a copy of the signed Iron Mountain report to Champion Managed Service Division. The report indicates that the DRMedia has been received.

Retrieve Tapes From The Vault

To retrieve tapes from the vault (Iron Mountain), the following tasks must be performed:

1. A daily Vault Retrieve Report will be automatically generated at 13:00 and e-mailed to Champion Managed Services Division and the IBM OSSA team. (See list above.)

2. Using the Iron Mountain online tool, the IBM OSSA team will make the appropriate entries to order the tapes described in the Vault Retrieve Report. Iron Mountain will return the requested DRMedia the following day.

3. Upon the receipt of the tapes from Iron Mountain, the IBM OSSA team will verify that the DRMedia that is retrieved from the vault (Iron Mountain) matches from the previous day’s Vault Retrieve Report.

4. IBM OSSA team will log all tapes received in the Daily Tape Transfer Log.

5. IBM OSSA team will send a copy of the Daily Tape Transfer Log to Champion.

6. IBM OSSA team will place the Vault Retrieve DRMedia in a storage cabinet, which is provided by Champion.

7. Champion personnel will load the Vault Retrieve DRMedia into the tape libraries.

8. Champion personnel will run the DRM program which processes the DRMmedia volumes into the proper state in the libraries.

Responding To Alerts

1. Verify that the alert is a legitimate alert and not just a monitoring time out.

2. If it is during working hours, check with team to see if anyone is working on the system.

3. Create a ticket in heat.

4. Contact customer to make them aware of the alert and to see if they are doing anything to the system. In the event that the alert is acknowledged by the customer as a valid alert, proceed with the following:

• Request from the customer if alert constitutes a priority one ticket.

• In the event that it is a priority one ticket, contact the corresponding IBM Project Manager and inform them of the status of the ticket.

• Enter priority into Heat.

5. Notify the customer that you are creating a ticket and assigning it to an engineer.

6. Contact the appropriate engineering team to notify them of the alert.

NOTE: In the event of the alert being a Priority 1, maintain an hourly contact with the engineer and customer with status updates until the alert is cleared.

Change Management

The Managed Services division of Champion Solutions Group (CSG) performs monitoring services for Clients. These services are performed through CSG's Managed Services Operations Center (MSOC), which is staffed around the clock by MSOC engineers whose primary responsibilities are to:

• Monitor, maintain, and protect the MSOC, the Client's environment and networks, and the equipment

• Fulfill the Client's requests for modifications and problem reports as needed

• Perform backups and other procedures in compliance with appropriate service level agreements

There are occasions or circumstances that require changes to the Client's environment or monitoring needs. In order to maintain Client satisfaction by providing timely responses to such requests, this section provides an overview of the change management policy, and offers guidelines for preparing a change request.

For the steps required for accepting and monitoring a Client's change request, please refer to the appropriate procedures that reside on the Champion Managed Services portal ():

• Creating A Request For Service

• Change Control Process

Types Of Change Requests

Change requests may be submitted by the Client, IBM, or CSG. Depending on the service level agreement, the types of changes cover the following environments:

|Operating System |Network | |

|Storage |Database | |

The Champion Managed Service Operations Center (MSOC) provides the Client with the ability to submit service requests by utilizing the Helpdesk System on the MSOC Portal via the Internet, or by telephone. It is recommended that priorities 1 and 2 should be submitted via telephone.

Either process provides the capability of submitting any of the three types of requests:

Administrative Request (AR) — Does not affect production systems, and implementation times are minimal. For example: new user, change password, change rights . . .

Problem/Fix Request (PF) — This is similar to an administrative request except that you may be experiencing some outage problems. For instance, a server or a network may be down.

Change Request (CR) — A change request is associated with a change in the environment. For example, if you have a piece of hardware that is already installed, but is not currently configured in the infrastructure, a change request is required in order to make the hardware part of the steady state infrastructure.

Severity And Priority

When you submit any of the three types of service requests, you are required to assign a priority level, which identifies criticality of the problem and the level of support that is needed. The four priority levels and their definitions are:

|Priority 1: |Critical Impact/Urgent The product, service, or network is not usable or affects the customer's core |

| |business. It is Champion’s objective to respond to all priority 1 requests within fifteen (15) minutes. |

|Priority 2: |Emergency Request The Client needs a quick response for reasons defined by the requester. It is Champion's |

| |objective to respond to all priority 2 requests within four (4) hours. |

| |Reminder: Per contract, the Client is permitted only three priority 2 requests per month, which are included |

| |in the Client's monthly fees. There will be additional charges if the Client exceeds the limit of three |

| |priority 2 requests per month. |

|Priority 3: |Major Impact An important function or service is not available, however the environment can still be used. |

| |It is Champion's objective to respond to all priority 3 requests within 48 hours and escalated appropriately.|

|Priority 4: |Minor Impact/Informational The product, service, or network is not seriously affected and is not currently |

| |affecting the customer’s core business. Or, this level of severity comprises shortcomings, suggestions, or |

| |questions. It is Champion's objective to respond to all priority 4 requests within one week and escalated |

| |appropriately. |

Change Request Classifications

Change Management includes any type of change that affects the Client's environment. Examples of this include emergency reboots, scheduled reboots, and other changes performed by either the Client or CSG that affect the server, network, or storage during, or outside of, a normal maintenance window.

Scheduled Client Change

When changes are known in advance, the customer should notify the MSOC manager. The MSOC manager begins the Change Management process by opening a ticket in Heat. The MSOC manager is the primary contact for the Client and continues to monitor and track the change request until it is completed.

Scheduled CSG Change

When CSG requires the utilization of the regular maintenance window, the MSOC manager begins the Change Management process, including opening a ticket in Heat. The MSOC manager contacts the Client to inform them of the upcoming change. The MSOC manager is the primary contact for the Client and continues to monitor and track the change request until it is completed.

Emergency Changes

If a Client needs to make modifications that require a reboot of the system, the Client should call the MSOC manager to implement this change immediately. The MSOC manager verifies that the person calling is authorized to request changes of this nature. Once the Client is verified, a ticket is opened to track the change. The Client receives a ticket number for tracking purposes. The MSOC manager is the primary contact for the Client and continues to monitor and track the change request until it is completed.

NOTE: With proper authorization, an MSOC Administrator may perform this function for the MSOC manager.

Who Is Authorized To Request A Change?

The individuals who are authorized to request a change differ depending on which entity originates the request.

|Client |The Client's authorized requesters are pre-defined during phase IV of the post-sales process. During this phase, the |

| |MSOC manager is responsible for contacting the Client to obtain the name(s) of the individual(s) who are authorized to |

| |request changes. |

|IBM |For a current list of the IBM authorized requesters, see Contact Information on the Champion portal |

| |(). |

|CSG |The CSG authorized requesters include: |

| |Operations Manager |

| |President, Managed Services |

| |NOTE: Emergency requests will be handled on an ad hoc basis and must be thoroughly documented to include the |

| |requester's complete identification, which may require a specific authorization code. |

How Is A Request Submitted?

Most change requests are to be submitted via the CSG Managed Services Web site at . However, emergency requests can be handled on an ad hoc basis, via the telephone, but must be thoroughly documented to include the requester's complete identification, which may require a specific authorization code.

NOTE: A request for change must be received no later than end-of-day on Wednesday to be reviewed and considered by the Change Control Board on the Thursday of the same week. Any request received after end-of-day Wednesday will not be addressed until Thursday of the following week.

When a request is received, use the Requests For Service procedure to accommodate the Client's needs.

Scheduled Maintenance Windows

Champion Managed Services (CMS) maintains the server hardware and operating system instances. All applications are the responsibility of the customer.

In order to proactively manage the environment, a standard maintenance window, between 3:00 AM and 6:00 AM Eastern Standard Time each Sunday, is reserved for internal, Champion maintenance. The time at which Client-requested changes are made is determined during the change request process.

Change Control Board

The Change Control Board (CCB), which meets each Friday, consists of personnel from IBM and CSG. The current members are:

|CSG |Operations Manager |

| |MSOC Supervisor |

| |Technical Leads |

| |Senior Application/Web Development Lead |

|IBM |IBM Project Manager |

| |Project Manager - Business Partners |

| |Site Project Executive |

NOTE: A request for change must be received no later than end-of-day on Wednesday to be reviewed and considered by the Change Control Board on the Thursday of the same week. Any request received after end-of-day Wednesday will not be addressed until Thursday of the following week.

AUTHOR’S NOTE:

The NOTE above refers to a specific timetable for holding the CCB meeting(s). We need to clarify the frequency at which they are actually held. Is it on Monday, Wednesday, and Friday? Is it everyday?

There are other sections that may require modifications too. AND, the policies and procedures will require the same modifications.

Decision Categories

There are three categories into which a decision may be defined:

|Approved |The Client is informed when the change will be performed. |

|Declined |The Client is informed why the change was declined. This may require that a completely new sales process |

| |begin in order to define specific requirements. |

|Postponed |The Client and the CCB mutually agree to postpone the change until a later date. This may require that a |

| |completely new sales process begin in order to define specific requirements. |

Turnaround Time

While an immediate acknowledgement is provided to the Client, the turnaround time for scheduled changes is within one week from the day the request is reviewed by the CCB. Many tasks associated with a request may be performed the same day, or within one week from the day the request is approved.

A request for change must be received no later than end-of-day on Wednesday to be reviewed and considered by the Change Control Board on the Thursday of the same week. Any request received after end-of-day Wednesday will not be addressed until Thursday of the following week.

NOTE: Emergency requests will be handled on an ad hoc basis and must be thoroughly documented to include the requester's complete identification, which may require a specific authorization code.

Problem Management

Business Hours

Champion’s Managed Service Operations Center (MSOC) is available 24 hours per day, 7 days per week. The MSOC provides you with the ability to submit service requests by utilizing the Helpdesk System on the MSOC Portal via the Internet, or by telephone.

Contacting Support (Champion MSOC)

The following contact information is provided for the customers for the purpose of creating a request for service and incident reporting:

Telephone

Toll Free: 888-997-7789

Local / Direct: 561-997-7789

Champion Portal

1. Log on to the Champion Portal.

2. Click the Support button at the top of the page.

3. From the left navigation panel, click Helpdesk System. In an effort to protect your records, you may need to enter your user name and password to enter this restricted area.

Instructions for “Creating A Request For Service” is provided on the portal under Documents.

Severity And Priority Levels

When a service request is submitted, a priority level must be selected and assigned to the request. Each level identifies the criticality of the problem and the level of support that is needed. The four priority levels and their definitions are:

|Priority 1: |Critical Impact/Urgent The product, service, or network is not usable or affects the customer's core |

| |business. It is Champion’s objective to respond to all priority 1 requests within fifteen (15) minutes. |

|Priority 2: |Emergency Request The Client needs a quick response for reasons defined by the requester. It is Champion's |

| |objective to respond to all priority 2 requests within four (4) hours. |

| |Reminder: Per contract, the Client is permitted only three priority 2 requests per month, which are included |

| |in the Client's monthly fees. There will be additional charges if the Client exceeds the limit of three |

| |priority 2 requests per month. |

|Priority 3: |Major Impact An important function or service is not available, however the environment can still be used. |

| |It is Champion's objective to respond to all priority 3 requests within 48 hours and escalated appropriately.|

|Priority 4: |Minor Impact/Informational The product, service, or network is not seriously affected and is not currently |

| |affecting the customer’s core business. Or, this level of severity comprises shortcomings, suggestions, or |

| |questions. It is Champion's objective to respond to all priority 4 requests within one week and escalated |

| |appropriately. |

Trouble Ticket Workflow

[pic]

Monitoring Standards

PURPOSE

The purpose of this document is to outline the thresholds that are associated with standard monitoring activities performed for the Champion Managed Services Clients.

REFERENCE

Please contact the Manager of Managed Services Operations Center to clarify any process within this procedure and for any concern beyond its scope.

OVERVIEW

Champion Managed Services provides monitoring services based on the standard, global thresholds for the activities listed below under Thresholds. While the standard monitoring thresholds are listed below, Champion Managed Services is able to provide modified settings for the alerts associated with particular monitoring activities. Any alternative parameters will be assessed on an as-needed basis.

Thresholds

The following are the current standard monitoring functions and their respective thresholds.

IIS Services

The following are the thresholds for IIS Services:

 

|PerfMon Status |

|Warning Level Alert |Critical Level Alert |

|  |Down >2 Minutes |

 

 

|Page File |

|Warning Level Alert |Critical Level Alert |

|>= 90%   5 Minutes |>= 99%   5 Minutes |

|Memory Usage |

|Warning Level Alert |Critical Level Alert |

|>= 80%   5 Minutes |>= 90%   5 Minutes |

 

 

|Disk Usage |

|Warning Level Alert |Critical Level Alert |

|= 95%   3 Minutes |

 

 

|DNS Server Response |

|Warning Level Alert |Critical Level Alert |

|>= 10000msec   5 Minutes |>= 10000msec   5 Minutes |

 

 

|PortMonitor Status (LDAP, Web, E-mail) |

|Warning Level Alert |Critical Level Alert |

| |Down >5 Minutes |

 

 

|Ping |

|Warning Level Alert |Critical Level Alert |

| |Down >3   Minutes |

 

 

|Ping Response Time |

|Warning Level Alert |Critical Level Alert |

| |>=300msec   3 Minutes |

 

 

|Switch SNMP Status |

|Warning Level Alert |Critical Level Alert |

| |Down   Immediate |

SQL Server 2000

The following are the thresholds for SQL Server 2000:

 

|PerfMon Status |

|Warning Level Alert |Critical Level Alert |

| |Down   5 Minutes |

 

 

|Processor Time |

|Warning Level Alert |Critical Level Alert |

|>= 80%   5 Minutes |>= 90%   3 Minutes |

 

 

|Buffer Cache Hit Ratio |

|Warning Level Alert |Critical Level Alert |

|= 80%   5 Minutes |>=97%   5 Minutes |

 

Storage Area Network Switches

The following are the thresholds for SAN switches:

 

|Port Status |

|Warning Level Alert |Critical Level Alert |

|  |>=2   Immediate |

 

Webservers

The following are the thresholds for Webservers:

 

|Available Memory |

|Warning Level Alert |Critical Level Alert |

| ................
................

In order to avoid copyright disputes, this page is only a partial summary.

Google Online Preview   Download