THE DELPHI METHOD

[Pages:10]THE DELPHI METHOD by Gail Fann Thomas, Associate Professor Graduate School of Business & Public Policy

Naval Postgraduate School

Maybe you'd like to know what issues are most likely to emerge in homeland security over the next 5-10 years. Or, maybe you'd like to examine the pros and cons of a specific homeland security policy. Both of these questions could be answered with a research technique called the Delphi Method, an iterative survey process that uses experts to answer a question.

Today, we'll take a look at when and how you might use this particular tool. After getting an idea of how it works, you may find it's something you'd like to use to conduct your thesis research. It also might prove to be a helpful tool in your job as a homeland security professional.

Learning Objectives

Let's look at an overview of the module and what you're expected to know when you've finished this session. First, we'll tell you a little bit about the origin of the Delphi Method so that you understand why it was developed. Next, we'll talk about when you would use this particular method and describe its primary characteristics. After that you'll learn about the process. In other words, how would you go about conducting a Delphi study? What are the steps? Finally, we'll show you an example of a Delphi study related to homeland security.

As you proceed through the module you'll notice that we've designed this session to give you a brief overview of the Delphi method, but you'll probably need to do some additional reading before you actually tackle this method yourself. In that case, you'll find a list of resources accompanying the learning module. One set of references addresses the actual method. The other set of references lists studies that were done using this technique. By looking at some of these examples, you'll see the wide variety of ways that this technique can be used.

When you've finished viewing this module, we hope you'll be able to:

? Describe the purpose of the Delphi method, ? provide a brief overview of the process, and ? write a research question that would use the Delphi Method.

Okay, let's get started.

1

History of the Delphi Method The Delphi Method began as an Air Force-sponsored Rand study in the early 1950's. The original study used expert opinion to estimate the bombing requirements for a hypothetical Soviet attempt to disable the USA's production capacity. The study also allowed officials to predict the USA's capacity to maintain the war effort. Using this new Delphi method of structured questionnaires followed by controlled written feedback, researchers were able to learn from a group of experts on an issue that was critical to national security. At that time in our history, an alternative method would have required a costly data collection process which would have been very difficult given 1950's computer capabilities. Since the original study, the Delphi method has been used for hundreds of studies in a wide number of fields including healthcare, education and business. Wonder why they call it "Delphi?" Although some people cringe at its mystical connotation, the term "Delphi" was originally derived from the "Delphic Oracle" because of the tool's ability to look into the future and forecast technological developments.

2

When You'd Use the Delphi

A variety of research methods are available to you -- evaluation research, policy analysis, and surveys -- just to name a few. Each approach has its strength and weakness, and most research questions are best answered with one method than by another method. Here are three types of research techniques, a description of each, and an example of a research question related to each technique. Notice how the Delphi differs from the other techniques.

Type of Research Evaluation

Policy Analysis

Survey

Description

Determines whether an intervention had the intended result for a program. It examines the effectiveness or ineffectiveness of an action. Compares alternative solutions to a policy issue using a set of criteria to judge the outcomes.

Gathers original data from individual respondents about perceptions, attitudes or beliefs. Survey data may be gathered through written questionnaires, telephone, internetbased surveys or by talking with people face-to-face.

Example

? How effective is the Los Angeles Fire Department's counterterrorism program?

? What is the best solution for protecting emergency responders from chemical, biological, and physical hazards that exist after large building collapses?

? How well prepared are US public schools for prevention of and response to a masscasualty event?

Delphi is a specific type of survey that ? What are the emerging

queries experts and uses multiple

issues facing homeland

rounds of questions.

security officials in the area

of policing protection?

As you'll notice in the table, the Delphi method is different from evaluation research and policy analysis. It is also different than a standard survey that generally queries a larger, representative sample with an aim of determining statistical significance. Unlike the Delphi that includes feedback and follow-up questions for the respondents, the standard surveys is generally distributed in a single administration. The Delphi, a type of survey research, is most useful when:

q Limited information is available about a problem or issue q The problem or issue lends itself to subjective expert judgment q Controlled interaction among a group of experts might yield superior results

3

As you now know, the Delphi technique began as a forecasting tool and is still often used for that purpose. Besides forecasting, researchers and practitioners have expanded the use of this tool over the past 50 years, and now you'll see it being used for a wide variety of other purposes. Here are a few examples:

? Examining the significance of historical events ? Evaluating possible budget allocations ? Exploring planning options ? Delineating pros and cons associated with potential policy options ? Developing causal relationships in complex economic or social phenomena

Questions Best Suited for the Delphi Method

Here are some more questions that might be best suited to this method:

? What are the emerging issues facing homeland security professionals in the areas of: Critical infrastructure protection? Security and privacy? Border security? Public health preparedness planning? Agriculture emergencies? Interagency collaboration?

? How might federal funding be best spent to improve drinking water security? ? When will terrorists strike again? Where? ? How might law enforcement counter terrorism capabilities be improved through

consolidation?

Characteristics of the Delphi Study

Now, let's look at what makes this method unique. In its most simplest terms, it uses a multi-round survey process to structure anonymous group communication around a complex problem. The Delphi method was developed to elicit opinions from leading experts. Early on it was all done with paper and pencil through the postal system. Today, most Delphi's are conducted via the Internet. Such a process allows you to gather expert opinion without requiring people to travel to a single location. Given the schedules of these experts, convening such a meeting would be difficult, if not impossible, not to mention the cost involved. In addition to access, its feature of anonymity allows participants to express their beliefs fully and openly. In fact, the process was designed specifically to circumvent some of the negative aspects of heated discussions in face-to-face meetings.

In most of the documented accounts of this method, you'll notice these three aspects of the Delphi method:

4

? Anonymous response and interaction from experts. Delphi studies are often conducted on the computer avoiding direct confrontation among the experts. Experts are free to state their opinions without the undue influence of their peers. Unlike faced-to-face discussion, this method does not allow a domineering member to take over the discussion. The Delphi method allows the researcher to control the interaction among the participants to foster consensus.

? Several rounds of data collection with feedback to participants. Most commonly three rounds are conducted. The technique allows the researcher to poll a panel of experts for their opinion on an issue and then provide them with feedback of the polled opinions. Subsequent pollings allow the experts to respond anonymously to the opinions of the other experts.

? Presentation of statistical group responses. The final report generally provides simple statistical data showing the level of agreement among the participants.

If you begin reading about this method and examining studies using this technique, you'll notice several variations on these characteristics, but this should give you the general idea.

5

The Process Delphi studies generally follow a basic iterative process.

Explore the issue

Choose experts

Distribute round of questions

Compose summary

Collect inputs from experts

Analyze input from experts

THE DELPHI PROCESS

Generate final report

Adapted from Contagion and Conflict: Health as a Global Security Challenge, January 2000.

1. The first step in the process is to explore the issue that you've chosen to research. If you've chosen to use the Delphi, you may not find much written about your topic. Nevertheless, it's a good idea to do some preliminary research before you begin to craft your first survey. You'll want to know some background information about the issue. If you're going to query a group of experts, you'll also want to be familiar with the language associated with your topic so that you're able to ask intelligent questions. After you've done some initial research, you'll be able to pose a few general research questions that will guide your study.

2. Once you've done some research and have a focus for your study, you'll have a better idea of the types of experts you'll want to invite to participate in your study. While a traditional survey might use a random sample to represent a larger population, you'll want to carefully choose some qualified experts who have a deep knowledge about your topic. These experts might be academics, practitioners, and/or government officials. Depending on your research questions, you might wish to have one panel or you might want to choose two or three panels that represent different viewpoints. In either case, the literature generally recommends that you choose about 10-18 people for a panel. If you'd like to see very specific detail about choosing your expects see the reference I've provided by Okoli and Pawlowski.

6

3. As you complete steps 1 and 2, you'll want to be thinking of the questions that you want to pose to your experts. In this first round, the questions are fairly broad. In some studies, researchers use a more open-ended approach on the first round. In other cases, the researcher has formulated some specific questions that the participants will respond to. In either case, you'll want to make sure that your questions are carefully crafted and piloted tested before sending them out to your experts. In today's electronic age, you can use email, fax, and/or a web-based survey depending on the media that is most convenient for the participant.

4. After a reasonable amount of time, you'll make sure that you have all of the questionnaires that you sent. Because most Delphi's are now connected electronically, you can generally expect to conduct a single round within a two to four-week period, depending on the experts and the topic you've chosen.

5. Once you've gathered the data from the participants, you'll analyze them. If the first round was an open-ended set of questions, your analysis will focus on finding similar categories of themes that emerge. Some questions, especially those in rounds two or three, will often ask the experts to rank order or provide some other numerical response. In this case simple statistics such as mean, median, mode, standard deviation, chi square for categorical data, or ANOVA for continuous data will be appropriate.

6. After you've summarized the data from a round, you'll create another questionnaire that builds on the responses form the previous questionnaire. This process continues until you feel you've reached consensus with your group or sufficient information has been exchanged among the experts.

7. The final step to document your process and write up the final results and recommendations.

An Example

Now let's look at an example to see how it works in practice. Remember, there are many variations of the Delphi so this is just one approach. we've chosen the following research question which was investigated by GAO in 2004: How might federal funding best be used to improve drinking water security?

Efforts to better protect drinking water infrastructure were accelerated dramatically after September 11. After that time, the EPA and the drinking water industry undertook several initiatives to identify water system vulnerabilities. This particular study was developed to examine these vulnerabilities and determine priorities for federal funding.

To obtain data for their analysis, GAO conducted a three-phase Web-based Delphi survey of 43 experts. Experts were chosen to achieve balance in terms of expertise--state and

7

local emergency response, engineering, epidemiology, public policy, security and defense, drinking water treatment, risk assessment and modeling, law enforcement, water infrastructure, resource economics, bioterrorism, public health, and emergency and crisis management.

To answer the research questions, the experts were asked the following questions: 1) to what extent should utilities' vulnerability and risk assessment information be considered in making allocation decisions? 2) What types of utilities should receive funding priorities? and 3) What are the most effective mechanisms for directing these funds to recipients? Additionally, the experts were asked to identify specific security-enhancing activities most deserving of federal support.

While we won't present all of the findings here, we'll give you an example of the answers to the last question. The experts' responses fell into three categories: 1) physical and technological upgrades to improve security and research to develop technologies to prevent, detect, or respond to an attack, 2) education and training to support, among other things, simulation exercises to provide responders with experience in carrying out emergency response plans, and specialized training of utility security staff; and 3) strengthening key relationships between water utilities and other agencies that may have key roles in emergency response, such as public health agencies, law enforcement agencies, and neighboring drinking water systems. It you'd like to see more details about this study, you'll find a citation at the end of this module.

So, in summary, the Delphi technique is best when:

q Limited information is available about a problem or issue q The problem or issue lends itself to subjective expert judgment q Controlled interaction among a group of experts might yield superior results

In the case of the water system vulnerability problem, the researchers were able to glean funding priorities by querying a group of experts who were located throughout the U.S. A controlled, web-based, iterative survey allowed the researchers to interact with these experts in a systematic way without the time and expense of travel and without the time required for face-to-face interaction.

The Policy Delphi ? A Variation of the Delphi

One last note which might be applicable to the types of problems you will research. While the original Delphi method was designed to obtain consensus among a group of experts, the Policy Delphi, a variation on the original Delphi, seeks opposing views on the potential resolution of a major policy issue. According to Turoff, the scholar who developed this method, "an informed group presents options and considerations that would be used for making a policy decision." For instance, the method was used to draw out the differing views of individual Kentucky legislators on the topic of tobacco policy.

The Policy Delphi generally has six stages:

8

 ................
................

In order to avoid copyright disputes, this page is only a partial summary.

Google Online Preview   Download