`;FF @@@ @@@@d:_@ FG EN DB GP     & . / M [ S7 `pIBOel9m Agarwal1999 Amelia1993g6Anderson1994Baldwin-Morgan1993. Bedford19729 Bolger1991 Brancheau1987 Brancheau1996 Bui1986 Carter19931 Chang2000  Chang2000, Chang2000- Chang2000&Chrysler1978 Chung20027 Churchill1995% Clayton1997+ Couger1988 Cule1998w Cule20012Czinkota1997w' Dalkey1963( Dalkey1970/ Dekleva1996) Delbecq1975  Dickson19840 Doke1995w$ Fink19951 Fischer19782 Fowles1975 Gable2000 Gable2000, Gable2000- Gable2000 Guimaraes1983) Gustafson19758 Gutierrez1989' Helmer19632 Hill1975w!Hiltz Hiltz1985 Holsapple2002 Hurdley2001 Janz1996 Jarke1986 Jones1992 Joshi2002 Keil1998w Keil20012 King2002 Lai2002  Leitheser1984"Linstone1975Lyytinen1998Lyytinen2001w McCoy2001 McCoy2002 McCubbrey1999  Mehr1970w3 Millar1984Nambisan1999w  Naumann1982 Nechis19848 Neumann1970 Olson1993 Olson1993 Palvia19822Peter V. Marks2002 Ronkainen1997 Roth1990w9 Rowe1991# Rowe19996Rungtusanatham19944 Sackman1974: Sackman1975 Saker2001Saunders1992w Schmidt1997 Schmidt1998 Schmidt20016 Schroeder1994 Smith2001 Smythe2000- Smythe2000 Storrsten1993; Story20010 Swanson1995Timbrell2000-Timbrell2000! Turoff< Turoff1972" Turoff19757 Turoff1985 Turoff1989 Turoff1991 Turoff2002* Van de Ven1974d) Van de Ven1975d Weber1988  Welty1972 Wetherbe19844Wetherbe19877Wetherbe19966 Wood1990w; Woudenberg1991g9 Wright1991# Wright1999/Zupancic1996/Zupancic1996Zupancic1996  Authorsb Journals  Keywords                                0Gs 4b Agarwal, RituAmelia, AnnetteAnderson, John CBaldwin-MorganBedford, Michael T Bolger, F.Brancheau, James C Bui, Tung X. Carter, Mark Chang, S.-I. Chang, S.I. Chang, She-IChrysler, EarlChung, WingyanChurchill, G. AClayton, Mark J. Couger, J. D. Cule, Paul Cule, Paul E.Czinkota, Michael RDalkey, Norman CDalkey, Rourke Dekleva, S. Delbecq, A LDickson, Gary W Doke, E.R. Fink, Dieter Fischer, R. Fowles, J Gable, G. Gable, Guy Gable, Guy G.Guimaraes, TorGustafson, D H Gutierrez, O Helmer, Olaf Hill, K.Q. Hiltz, StarrHiltz, Starr R.Holsapple, Clyde W.Hurdley, Louise Janz, Brian DJarke, MatthiasJones, Jack William Joshi, K. D. Keil, MarkKing, William R.Lai, Vincent S.Leitheser, Robert LLinstone, Harold A.Lyytinen, KalleMcCoy, Randall W. McCoy, ScottMcCubbrey, Donald J.Mehr, Robert I Millar, VNambisan, SatishNaumann, Justus D. Nechis, Mal Neumann, SeevOlson, Gary M.Olson, Judith S.Palvia, ShailendraPeter V. Marks, Jr.Ronkainen, Ilkka ARoth, Roberta M. Rowe, G. Rowe, GeneRungtusanatham, Manus Sackman, HSackman, Harold Saker, JamesSaunders, Carl StoakSaunders, Carol Stoak Schmidt, RoySchmidt, Roy C.Schroeder, Roger G Smith, Gareth Smythe, E. Smythe, ErrolStorrsten, MarianneStory, VictoriaStraub, Detmar W. Swanson, N.E Timbrell, G.Timbrell, GregTuroff, MurrayVan de Ven, Andrew H Weber, Ron Welty, GordonWetherbe, JamesWetherbe, James CWetherbe, James C.Wood, William C.Woudenberg, F. Wright, G.Wright, George Zupancic, J.   Academy of Management Journal Academy of Management Review4.ACM Transactions on Information Systems (TOIS)Communications of the ACMCommunications of the AISDecision SciencesEducational PsychologyInformation & Management Information and Management*^,)Information Strategy: Executive's Journal<9Information Technology, Learning, and Performance Journal($International Journal of Forecasting$!Journal of Academic Librarianship$Journal of Information Systems,)Journal of International Business Studies,)Journal of Management Information Systems(#Journal of Organizational Computing Journal of Risk & Insurance Journal of Systems Management<6Journal of Technological Forecasting and Social Change<7Journal of Technological Forescasting and Social ChangeManagement ScienceMarketing Review MIS QuarterlyTechnological Forecasting0+Technological forecasting and social change          COMPUTER programmersu$ COMPUTER software -- DevelopmentI`]Cooperative work, communication design, group decision making, negotiation, office automation<6Delphi Method, Forecasting, Marketing, Decision-Making|welectronic commerce, disintermediation, reintermediation, cyber-mediaries, air travel distribution, Delphi methodology.om ELECTRONIC data processingake<8ERP, ERP Lifecycle, Delphi Method, Key IS Issues Studies\YExpert systems, Auditing, Public accounting firms, Delphi, Information technology impactszIS function performance evaluation, measurement, assessment, delphi technique, IS strategic impact, top management support|vIS Innovation, organizational mechanisms, technology cognizance, technology exporation, propensity to innovate, DELPHIHBIS management, key issues, management priorities, delphi techniqueTQIS risk management, IS project risk management, Risk assessment, Delphi techniquel:b4#9D>Judith S. Olson Gary M. Olson Marianne Storrsten Mark Carter 1993jcGroupwork close up: a comparison of the group design process with and without a simple group editor54.ACM Transactions on Information Systems (TOIS)114321-348OctoberA simple collaborative tool, a shared text editor called ShrEdit, changed the way groups of designers performed their work, and changed it for the better. First, the designs produced by the 19 groups of three designers were of higher quality than those of the 19 groups who worked with conventional whiteboard, paper and pencil. The groups with the new tool reported liking their work process a little less, probably because they had to adapt their work style to a new tool. We expected, from the brainstorming literature and recent work on Group Support Systems, that the reason the designs were of better quality was that the supported groups generated more ideas. To our surprise, the groups working with ShrEdit generated fewer design ideas, but apparently better ones. It appears that the tool helped the supported groups keep more focused on the core issued in the emerging design, to waste less time on less important topics, and to capture what was said as they went. This suggests that small workgroups can capitalize on the free access they have to a shared workspace, without requiring a facilitator or a work process embedded in the software.A wide variety of methods have been offered to overcome the various process losses associated with group work [Forsythe 1990], such as brainstorming, Delphi methods, Nominal Group Technique (NGT), root cause analysis, and stake-holder analysis.&Roberta M. Roth William C. Woodr 1990PIA Delphi approach to acquiring knowledge from single and multiple expertsl^XProceedings of the 1990 ACM SIGBDP conference on Trends and Directions in Expert Systems &Orlando, Florida, United States $ACM Press New York, NY, USA301-324 $Group Vs Individual Delphi!!!DA major unresolved issue in the knowledge acquisition literature is the appropriateness of using several experts as knowledge sources. If multiple experts are to be utilized as knowledge sources, then knowledge acquisition techniques suitable for use with groups must be identified and employed. This paper reports on the results of a study that used the Delphi technique as a means of eliciting knowledge from an expert group in an actual expert system development project. A parallel series of questionnaires was also conducted with a single expert in order to determine the content areas where the judgments of multiple experts converge and diverge. A comparison of the group's questionnaires with those of the single expert identified a core set of common concepts and sequence of problem solving steps. A significant number of unique ideas were generated by the group, however, suggesting that the use of an expert group in this domain helped surface a more comprehensive list of important domain concepts and factors.(In order to provide an opportunity to assess the performance of an individual versus multiple experts in the knowledge acquisition process, the Delphi study was carried out with a single expert in parallel with an expert group. Since a true Delphi cannot be conducted with an individual, a three-round series of written questionnaires was administered based on the same objectives as those for the group Delphi process. The rounds of the individual's Delphi followed the same format as the group Delphi, except that the individual expert was provided feedback only on his own previous responses. These processes generated the individual expert's list of important domain concepts and factors in a manner that is as comparable to a Delphi process as is possible.$Rowe, G. Wright, G. Bolger, F. 19914.Delphi: A re-evaluation of Research and Theory Technological Forecastinge39235-251 Critique of Delphi MethodFGene Rowe George Wright 1999F?The Delphi Technique as a Forecasting Tool: Issues and Analysis*$International Journal of Forecasting15  H Sackman 1974piDelphi Assessment: Expert Opinion, Forecasting, and Group Process, The Rand Corporation Report #R-1283-PR} The Rand Corporation AprilaSackman, Harold 1975Delphi Critiqued  Lexington, MA\ Lexington Booksi.'Carol Stoak Saunders Jack William Jones 1992@9Measuring Performance of the Information Systems Function0)Journal of Management Information Systemse8t4n 63-82: Spring Uses Delphi methodzIS function performance evaluation, measurement, assessment, delphi technique, IS strategic impact, top management supportThis paper reports a Delphi study followed by senior executive interviews aimed at determining the important dimensions and their measures for evaluating information systems (IS) function performance. The findings highlight (1) IS function performance dimension rankings and measurement usage, and (2) apparent contradictions between the considerable emphasis placed upon strategic impact of information systems and the lack of executive involvement.:3Boring Paper..........................khrrrrrrrrrrr Mark Keil Paul E. Cule Kalle Lyytinen Roy C. Schmidt 199882A framework for identifying software pro$Clyde W. Holsapple K. D. Joshi 20022+A collaborative approach to ontology design Communications of the ACM452 42-47February Uses Delphi method \Ontological engineering is concerned with the principled design, modification, application, and evaluation of ontologies. The accompanying table outlines five approaches to ontological design: inspiration, induction, deduction, synthesis, and collaboration. These may be used in the initial design of an ontology or the modification of a design (for example, in reaction to feedback on its applications, evaluations of its features, or domain changes). Hybrids of the approaches are possible. ADelphi-oriented structuring of collaboration: There are pros and cons to the five approaches. The inspirational approach can be questioned as lacking theoretical underpinning and may be impractical. However, this approach may also yield unique, innovative ontologies. An ontology designed via the induction approach may fit a specific case, but may not be generalizable. The deductive approach presupposes existence and selection of an appropriate scheme of general characterizations from which an ontology for a specific case can be devised. The synthetic approach implicitly covers the first three approaches in that its base set could involve any or all of them, and the base set can be supplemented by constructs currently existing in the literature. However, this approach is interpretative in nature, relying heavily on developer synthesizing skills. None of the first four approaches has a built-in evaluation facility to assess quality/acceptability of the resultant ontology. In contrast, the collaborative approach relies on assessments from diverse vantage points and tends to build commitment by iteratively reducing participants' objections. This approach relies heavily on the nature of participants, the degree of their involvement/diligence, and developer skills in overseeing the collaborative process. Collaborative ontology design can be applied to produce ontologies for e-commerce, agent coordination, distance learning, virtual enterprises, and other phenomena. Here, we have provided guidelines and a sample for this kind of ontology development. Beyond this, experience gained from the KM-ontology design case offers lessons that should be considered when using the collaborative design approach. There can be panelist attrition across rounds. Even though every panelist from round one indicated willingness to participate in round two, some did not. The task of reviewing a substantial ontology and carefully answering questions is time-consuming. Longer, more flexible response turnaround may help minimize attrition. One panelist observed that the Delphi process can involve an anchoring effect. Indeed, the ontology devised in the second phase serves as a focal point anchoring the whole process. While this helps structure the collaborative effort, it may also restrict panelists' expression and imagination. Thus, we advocate the use of synthesis with a broad, inclusive base set when preparing the anchoring ontology. Some panelists volunteered that participation in the collaborative design process was itself very useful. The summaries of panelists' responses enabled them to gain new insights and exposed them to diverse perspectivesnSatish Nambisan Ritu Agarwal 1999XQOrganizational Mechanisms for Enhancing User Innovation in Information Technologyu MIS Qu4-James C Brancheau Brian D Janz James Wetherbe 1996NHKey issues in information systems management: 1994-95 SIM Delphi results MIS Quarterly202t June Uses Delphi method{Over the past 15 years, the Society for Information Management (SIM) has periodically surveyed its members to determine the most critical issues in IS management. Again in 1994-95, SIM institutional and board members were asked to consider what they felt were the most critical issues facing IS executives over the next three to five years. Signaling an evolutionary shift in IS management, this study shows that business relationship issues have declined in importance compared to technology infrastructure issues. For IS executives and general managers, the key issue framework suggests some general directions for emphasis and provides a coarse measure for benchmarking their own concerns against those of their peers. The results of this study also impact educational missions in teaching and research to the extent that they need to be sensitive to the views of practicing IS executivesThe Delphi method was retained for its value in surfacing new issues and moving study participants toward consensus (Delbecq, et al., 1975). Essentially, the Delphi method employs a series of linked questionnaires. Successive rounds of questionnaires summarize subjects' responses to the preceding questionnaire and ask respondents to re-evaluate their opinions based upon the summarized results. Questionnaire rounds are continued until a reasonable level of consensus is achieved. Strength is that the issues and their rationale have been created and revised almost exclusively by the respondents themselves. Unfortunately, the approach has weaknesses as well. The two most important are the lack of clarity of some issues and the conceptual overlap of other issues in the framework. @ 96.,- &7%+('/) 0$182" 3 #94:!* ; p>rw  Џv >>PXcYZ>Z>dW>WP4>PW @  Ŭ@s the research method for this study. We formed our panel of experts by recruiting project managers with many years of exp; ,*vVan de Ven, Andrew H 1974.'Group Decision making and Effectiveness\ "Kent State University Pressn Ron Weber  1988>8Computer Technology and Jobs: An Impact Assessment Model Communications of the ACM311s 68-77uJanuarynA model is proposed that associates the impact of computer technology on a job, with the set of underlying characteristics that describe the activities performed on the job. An empirical test of the model has been undertaken. One thousand and thirty-five experts assessed the impact of computer technology that they believed would occur on 306 jobs over a three-year period. Job characteristics data was obtained from prior analyses of the jobs, using the Position Analysis Questionnaire (PAQ). Six job dimensions derived from analysis PAQ data were significant predictors of the technological impact ratings provided by the experts: engaging in physical activities; being aware of the work environment; performing clerical-related functions; working in an unpleasant or hazardous environment; performing service-related activities; and performing supervising, directing, and estimating functions. Gordon Welty 197282Problems of Selecting Experts For Delphi Exercises$Academy of Management JournalE151e121-124i March Talks about method itselfDiscusses the problems associated with selecting experts for the Delphi exercise. Ways to solve expert selection problems; Information on the Delphi exercise; Relevance of expertise to forecasting values6Woudenberg, F. 1991An Evaluation of Delphi2+Technological Forecasting and Social Change 4020131-150 Critique of Delphi Method%|7v&2-, She-I Chang Guy G. Gable 2000@9Major Issues with SAP Financials in Queensland Government D=Proceedings of the Americas Conference on Information Systems} "Long Beach, California, USA972-976;>8ERP, ERP Lifecycle, Delphi Method, Key IS Issues StudiesJDThis short paper reports a research project that seeks to give improved understanding of client-centered ERP lifecycle support issues in order that research, management and educational resources can be allocated and implemented effectively. The paper presents (1) the study background; (2) the research context and object; (3) the research questions and aims; (4) previous literature on major IS issues employing the Delphi method; (5) the research strategy and design; (6) progress to date; (7) expected benefits and outcomes; and (8) limitations and future research suggestions.This Delphi approach is particularly appropriate for the reference study. First, questionnaires are used to solicit expert opinion. This means that the respondents can be geographically remote. Second, though respondents are known to the researchers, confidentiality and anonymity amongst respondents is ensured, thus dominant individuals are unable to overly influence the results. Third, because respondents are known to the study team, the researchers are able to group respondents based on various demographics and follow up ambiguous and missing details. Fourth, an email-out is inexpensive and can be completed in a short time. Fifth, an open-ended survey allows respondents to fully express and delineate their views. Sixth, a two-round questionnaire enables information feedback, thereby stimulating reflection, movement toward a certain level of consensus or difference, and a greater depth of insight emerges. Seventh, the Delphi quantitative scores allow statistical summarization and comparison to demonstrate group consensus or differentiation. Finally, because many previous studies investigating IS issues have employed Delphi, comparing and contrasting findings with those of earlier studies may be possible (note possible difficulties mentioned earlier), and contributes to the cumulative information systems discipline. Thus, Delphi is deemed an appropriate method, since a major goal of this study isto systematically identify, analyze and determine the relative importance of major ERP lifecycle issues.KP4:*1CGS.I. Chang G. Gable 2000NHA Critique of the Delphi Method in the Context of IS Key Issues StudiesF@Proceedings of the Pacific Asia Conference on Information System  Hong Kong<0)S.I. Chang G. Gable E. Smythe G. Timbrell\ 2000D=Methods for Distilling IS Key Issues Using a Delphi ApproachlProceedings of the ACIS  1-11Chrysler, Earl 1978<6Some determinants of computer programming productivity Communications of the ACM\2162472-484aVPCOMPUTER programmers ELECTRONIC data processing COMPUTER software -- DevelopmentBComputer-Mediated Communication Requirements for Group Support*#Journal of Organizational Computing1 85-113Delphi is a methodology often used for complex, unstructured problems, such as trying to predict potential breakthoroughs in biomedical research and their societal impacts. The objective of a Policy Delphi is to develop the strongest pro and con arguments for alternative resolutions to a policy issue. Often misunderstood, Delphi's basic objective is the design of the communication structures to coordinate information exchange in asynchronous problem solving groups. 6:4Mark Keil Paul E. Cule Kalle Lyytinen Roy C. Schmidt 199882A framework for identifying software project risks Communications of the ACM04111 76-83iNovember Uses Delphi MethodVA Delphi study in which experienced software project managers identified and ranked the most important risks. The study led not only to the identification of risk factors and their relative importance, but also to novel insights into why project managers might view certain risks as being more important than others. This study employed a variation on the traditional Delphi survey approach designed to elicit opinions from a panel of experts through iterative, controlled feedback. Three expert panels were formed by recruiting from among experienced project managers in each country. The Delphi survey consisted of three phases. In the first phase, a brainstorming round was conducted to elicit as many risk factors as possible from each of the panels. The output of all three panels was then consolidated to produce a list of 53 risk factors. The consolidation process ensured the three panels had access to a common list of factors with common definitions. During subsequent phases, the panels were allowed to operate independently, each one selecting (and later ranking and rating) the most important risk factors from this common pool of factors. The purpose of the second phase was to narrow the list of items to a manageable number that could be meaningfully ranked and rated. This was done by having each panelist select the 20 risks deemed to be most important. For each country panel, risk factors that were selected by 50% or more of the panelists in that country were retained for the next phase of the study. The risk factors retained included a set of 11 risk factors common to all three panels. The third and final phase of the study involved the actual ranking and rating of risk factors. Ranking and rating rounds were conducted until each panel reached an acceptable level of consensus. Kendall's Coefficient of Concordance was used to measure the degree of consensus among panelists for each country. Rounds of ranking were repeated until either the panelists reached strong consensus or consensus did not change from one round to the next.60William R. King Peter V. Marks, Jr. Scott McCoy 200281The most important issues in knowledge managemento Communications of the ACMg459d 93-97u September9 Uses Delphi methodWe sought to empirically identify and rank the most important issues facing KM system designers and corporate managers alike using an open-ended process in which the researchers' own ideas could not bias participant response. We therefore used the Delphi methodology because, as explained in [4, 11], its "free text" nature ensures no researcher biases are imposed on participants, new ideas can be considered as the study progresses, participants can rethink their initial importance ratings in light of the ratings given by others, and participants are motivated to document their ratings, looking to convince others who initially might not have shared their views. t J""Vincent S. Lai Wingyan Chung 20020*Managing international data communications Communications of the ACMc453 89-93t MarchlInternational data communication (IDC) activities are vital to multinational corporations (MNCs) in managing the complexity of information exchanges required for the control and implementation of worldwide business strategies. IDC activities are also critical because they allow MNCs to extract information and move it globally without incurring major time delays, or transportation, reproduction, or inventory costs. Compared with data communications that occur within national borders, IDC encounters more difficulties due to differences in the international technological infrastructure, network systems, culture, government regulations, and the level of technological and economic development. Surprisingly, there has been little empirical research to identify the issues MNCs encounter when performing IDC, and needless to say, no solutions have been widely reported to deal with these issues effectively. However, several empirical studies have been conducted on the issues of information systems management. One of the earliest studies (1982) surveyed members of the Society for Management Information Systems (SMIS) [1]. Four subsequent studies [2, 3, 6, 9], based upon each other, were performed on the SMIS membership since then in an attempt to build a cumulative research tradition of IS issues in the U.S. Similar studies have also been conducted in other countries (such as [4, 5, 7, 10, 11]) to essentially identify and rate the relative importance of domestic and international IS (IIS) issues as implied by current competitive priorities, major concerns, and technology advancements. Although these studies do not produce uniform findings, network-related concerns have consistently been rated as top IS issues (see Table 1). This article reports on a two-round Delphi study to prioritize the IDC issues confronting MNCs. A questionnaire was administered to IS executives at 300 Fortune 500 companies. Only U.S.-based MNCs were selected because we wanted to improve the study's response rate by avoiding international mailings.& Harold A. Linstone Murray Turoff 19754.The Delphi Method: Techniques and Applications (!Addison-Wesley Publishing CompanysRandall W. McCoy 2001NGComputer Competencies for the 21st Century Information Systems Educatori@9Information Technology, Learning, and Performance Journalo192r 21-35 Fall Uses Delphi MethodThis article reports the results of a Delphi study to determine the general need for the preparation of computer technology competencies of business education teachers for the 21 st century. The study consists of three rounds of a Delphi instrument using electronic mail as the primary means of communication. Twenty-three experts nominated by the National Association for Business Teacher Education (NABTE) served on the Delphi panel. The responses generated by the first round contributed to the development of an instrument; the second round involved rating the statements; and the third round determined the consensus on items. The findings of the study include 95 statements about computer competencies that should be included in business teacher education curricula. These competencies were grouped in five categories: computer hardware, software, computer programming, computer integration, and general computer knowledge.spThe Delphi technique, however, is most useful for forecasting content in emerging occupational areas and fulfills the need for group consensus. History of Delphi: The term Delphi comes from a reference to the oracle at Delphi, a place in which the ancient Greeks believed the gods answered questions and gave advice concerning the future. In research, the Delphi technique is an organized methodology for correlating views and information pertaining to an area of strategy which allows respondents an opportunity to react to and assess differing viewpoints. The technique was introduced in 1958 through Project DELPHI, sponsored by the United States Air Force and directed by the Rand Corporation, to obtain the most reliable consensus of a group of experts concerning predictions of alternative national defense futures (Dalkey & Helmer, 1963). Three characteristics distinguish the Delphi technique from other methods of group interaction: (a) confidentiality, (b) iteration with controlled feedback, and (c) statistical group response. Because the originator of an input is not identified in the study, the opinion is not associated with a particular person. With confidentiality assured, the panelists have more freedom to alter opinions and are not swayed by the credentials of fellow participants. The number of rounds of review of responses depends upon a consensus of the panel.Donald J. McCubbrey 1999jcDisintermediation and reintermediation in the U.S. air travel distribution industry: a Delphi studya Communications of the AIS 1b5}~welectronic commerce, disintermediation, reintermediation, cyber-mediaries, air travel distribution, Delphi methodology.6/Observers of the Electronic Commerce (EC) landscape often comment on the prospects for disintermediation. Other observers note that the nature of EC will create new kinds of intermediaries, termed cybermediaries, who would occupy positions in Internet channels between producers and consumers. The word coined to describe this is reintermediation. In either case, traditional retailers would be threatened by new EC-enabled competition. This investigation was launched to predict the occurrence and impact of disintermediation and reintermediation in the US air travel distribution industry. A group of industry experts was assembled as a Delphi panel and asked to predict the effect that EC would have on the major channel players in each of five major market segments. The panel forecast that major disintermediation and reintermediation will occur and that there will be a sharp reduction in the number of traditional travel agents five and ten years in the future. The panel also identified a number of strategic threats and opportunities for the channel players.e Robert I Mehr Seev Neumann 1970 Delphi Forecasting Project"Journal of Risk & Insurance372f241-246\ June Talks about method itself Talks about advantages/possible weaknesses, problems associated with delphi method. Success with the use of Delphi technique is a function (skill with which experts are selected, the ability of pollsters to phase questions without guiding the respondents, the intelligence with which background information to be provided to the panel members is sifted, and the diligence with which procedures are followed)U! N<6Victoria Story Louise Hurdley Gareth Smith James Saker 2001voMethodological and Practical Implications of the Delphi Technique in Marketing Decision-Making: A Re-AssessmentMarketing Review14487-505= Summer Talks about method itself7<6Delphi Method, Forecasting, Marketing, Decision-MakingStandard operations-research techniques rely on the past as a predictor of the future. However, in conditions of high uncertainty, the past may have little bearing on the future. In contrast, judgmental forecasting techniques rely on opinion and are based on the aggregation of the collective knowledge and experience of a group, or panel of experts. The additional features of Delphi are that the process is anonymous and involves a number of rounds of data collection. This facilitates the formation of group judgment without introducing problems associated with group interaction. The globalization of business has also increased the dispersion of experts and what was once the remit of round table discussion between members of the same SBU is now a matter for cross-national debate. Delphi is a particularly useful method of establishing forecasts of future issues under conditions of little historical data, when experts are geographically dispersed, and when solving complex problems. However, as a technique it has received little attention in marketing. This paper re-assesses Delphi as a method of judgmental forecasting for marketing decisions and suggests a framework for conducting better Delphi studies in marketing.eTechnique has received little attention in marketing. Many of the criticisms made of Delphi relate to methodological considerations. However, when little historical data is available, using the opinions of experts is suggested to be a valid approach to decision-making. In using experts three problems are encountered; 1) conflict and disagreement between experts resulting in bias towards dominant characters; 2) difficulty in forming a working consensus due to geographical dispersion and a lack of round table discussion; and 3) problems are often complex, involving a wide range of topics and experts may lack knowledge in some areas. Marketing decision making and planning for the future environment needs to take account of a global scenario. The suggested procedure for conducting better Delphi studies outlined in this paper, attempts to incorporate good marketing research techniques into the Delphi structure to provide a more rigorous framework for undertaking Delphi research. It makes use of appropriate statistical techniques, such as cluster analysis, to segment views, accepting that experts may disagree, and provide more than one potential future that those with a vested interest can proactively aim to achieve. By using scenarios, the multi method and mutli sample comparisons, the results of a Delphi can be treated with more confidence. Future research should look to examining the types of Delphi studies already present in the literature and investigate the appropriateness of different alternatives in different situations and tasks. Murray Turoff Starr Hiltz&Computer Based Delphi Processesrf`Gazing Into the Oracle: The Delphi Method and Its Application to Social Policy and Public Health London Kingsley Publishers Murray Turoffa 1972F?Delphi Conferencing: Computer-Based Conferencing with Anonymitys<6Journal of Technological Forecasting and Social Change3159-204This paper is report on an on-line computer-automated conference that was conducted for 13 weeks in the Spring of 1970. The conference involved 20 individuals throughout the country who were able to engage in the conference discussion at their convenience by going to a computer terminal at theor time of choice. The computer maintained the real-time accouting of the dicussion items entered by the various respondents and of how the group had voted on these items with respect to the scales of desirability and feasibility(for proposals), or importance and confidence (for comments) The purpose of the conference was to evaluate potential applications of this type of conference system and to critique the author's design of the particular system in use. During the first part of the conference, no conferee was aware of the identity of the other conference members. In addition, the computer did not provide to the conferees the identification of the originators of the items or of the individual votes. Because of this anonymity, such systems may be referred to as "Delphi Conferencing Systems". The conference group was deliberately chosen by the author to be mixture of the Delphi designers, computer pecialists and operations research analysts. The group was also evenly divided as to affliation with government, industry, nonprofit organizations, and universities. About 1/3rd of the group had no experience with the computers or terminals. Another third had only minimal contact as users of computer services provided by others. Murray TuroffM 1989^WThe Anatomy of a Computer Application Innovation:Computer Mediated Communications (CMC)/<6Journal of Technological Forecasting and Social Change36107-122 Computerized Delphi9The fundamental idea of Computerized Conferencing was the ability of the computer to provide both communication structures and tools tailored to specific groups, specific applications, and specific communications objective. It was based upon the concept of Delphi method as the art of structuring human communications about complex problems. Computerized Conferencing really grew out of the idea of putting Delphi communications structures onto the computer.2+http://eies.njit.edu/~turoff/Papers/CMC.pdf  )/(' + J. D. Couger 1988lfKey Human Resource Issues in IS in the 1990s: Views of IS Executives versus Human Resource ExecutivesInformation & Management14161-174*$Michael R Czinkota Ilkka A Ronkainen 1997TMInternational business and trade in the next decade: Report from Delphi Studys0)Journal of International Business Studieso284;827-845 4th Quarter< Uses Delphi methodThe globalization of business causes major changes in the business environment and in business practices. Forecasting such changes is crucial for the preparation of responses to them. This article reports on a Delphi study conducted with a global set of experts drawn from the policy, business and academic communities. Over three rounds of mailings, these experts analyzed and debated the likelihood of changes in the international business environment over the next decade and the impact of these changes on corporate practices. The findings identify key geographic areas of business growth, major sectoral transformations, shifts in the trade framework and its institution, and strategies for corporate adjustmentVarious approaches can establish how pertinent constituents view the future. A broad-based content analysis of the current literature can examine trends [Naisbitt 1990; Wheeler 1988]. However, this approach is very resource intensive and, due to language limitations, possibly biased by the perceptions and interpretations of U.S.-based analysts. In addition, such an approach does not benefit from any interaction between policy makers, business leaders and academics. A second approach is the interview method, which allows for in-depth questioning. By convening a group of experts at one location and facilitating their interaction over a prolonged period, an in-depth assessment could be created. However, in order to be meaningful, such an approach requires the invitation of carefully identified and stratified experts, reflecting different types of insights and different parts of the world rather than the views of a convenience sample. Financial constraints made this approach unfeasible. This study uses the Delphi technique, which integrates the judgement of a number of experts who cannot come together physically, but also facilitates feedback, debate and comment. The overall objective of this technique is to achieve consensus among a diverse group of participants. Past studies using such an approach have typically used thirty experts based on the finding that larger groups create few additional ideas and limit the in-depth exploration of the ones generated [Delbeq, Van De Ven and Gustafson 1975]. For the success of such a study, it is critical to secure the participation of the right kinds of experts, who understand the issues, have a vision, and represent a substantial variety of viewpoints"Dalkey, Rourke Helmer, Olaf 1963LFAn experimental application of the delphi method to the use of expertsManagement Science9s3e458-467 Dalkey, Norman C 19704-Use of Self-Rating to Improve Group Estimates0>7Journal of Technological Forescasting and Social Changeo1b3o March 1970S. Dekleva J. Zupancic 1996NGKey Issue in Information Systems Management: A Delphi Study in Slovenia  Information and Management311 1-1160Delbecq, A L Van de Ven, Andrew H Gustafson, D H 1975ZTGroup Techniques for Program Planning: A Guide to Nominal Group and Delphi Processes Scott-Foresman and CompanyF?Gary W Dickson Robert L Leitheser James C Wetherbe Mal Nechisu 19844-Key Information Systems Issues for the 1980's2 MIS Quarterlym8\3 135-148} September Uses Delphi method.(Knowledge of the most important issues in the information systems field would help focus research and educational efforts. A Delphi study, using leading information systems (IS) professsionals, was used to identify and rank ten key IS management issues for the 1980's. Measures were also taken for the amount of agreement achieved on these issues and rankings. The number one issue was improved IS planning, followed by facilitation and management of end user computing. This articles describes the research approach involved and discusses the results..J6@9Anderson, John C Rungtusanatham, Manus Schroeder, Roger Gi 1994LEA theory of quality managment underlying the Deming management methodp"Academy of Management Review193M 472 &Uses Delphi Method - Managementc$Baldwin-Morgan Annette Ameliad 1993hbThe impact of expert system audit tools on auditing firms in the year 2001: A Delphi investigation$Journal of Information Systems71 16-34 Spring Uses Delphi method`YExpert systems, Auditing, Public accounting firms, Delphi, Information technology impacts ]Throughout the 1980s, large public accounting firms worked to develop expert systems for auditing tasks. This development was driven by increasing competition and specialization, decreasing audit fees, and pressure to boost the efficiency of audits without lessening effectiveness (Messier 1986). Little research has examined how these expert systems impact the firms using them. A Delphi investigation was undertaken to determine the most and least likely impacts on auditing firms of using expert systems as audit tools. The Delphi method is particularly useful under two circumstances. First, the study of the problem does not lend itself to precise analytical techniques but benefits from subjective collective judgments. Second, the individuals who need to interact cannot be brought together in a face-to-face exchange because of time or cost constraints (Linstone 1985). These consensus forecasts are more accurate than 95 percent of individual forecasts, and iteration reveals more reflective opinions than single surveys (Parente et al. 1984). The Delphi panel, which was composed of those actively researching, developing, implementing and using expert systems, suggests expert systems are very likely to have an impact on auditing firms in the next decade. Specific impacts identified as very likely include better documentation references, enhanced distribution of expertise, increased ability to handle complex analyses, and improved decision consistency and quality. The panel also indicated that use of expert systems in auditing is very likely to result in changes in the education of auditors, particularly in-house training. The least likely impact identified was loss of prestige for firms using expert systems. In addition, the Delphi panel concluded that efficiency is only somewhat likely to be improved by expert systems. This study provides new evidence for the types of expert systems impacts that are likely to occur in the next decade. It also changes some of the popular conceptions of audit expert systems impacts on efficiency that have been widely suggested in the literature, but for which little evidence has been gathered. For example, in the literature one of the most often mentioned types of impact is efficiency impacts. However, this study concludes that effectiveness, expertise, and education impacts are more likely to occur than efficiency impacts. LThe most suitable methods for identifying possible future technological events and trends are a review of the relevant literature and interviews with experts. Experts of special interest are those directly involved in research environments in which future implementation of technology is already being investigated and those involved in actually implementing the new technology. The Delphi survey method (Linstone 1985) is particularly useful when one's understanding the problem benefits from subjective judgments on a collective basis. A conventional conference tends to be dominated by particularly strong personalities or to give rise to an undesirable bandwagon effect (Linstone 1985). The Delphi technique is particularly suited for assessing the likelihood of future events and trends (Holstrum et al. 1986) and has been suggested as an appropriate technological forecasting tool for predicting the effect of technological Changes on auditing (Garsombke and Cerrulo 1984). Three rounds have proved sufficient to attain response stability, and excessive repetition is generally unacceptable to participants (Lin-stone and Turoff 1975). Linstone (1985) states that, in most Delphis, consensus is assumed to have been achieved when a certain percentage of responses fall within a prescribed range such as when the inter-quartile range is no larger than 20 percent of the scale. The minimum Delphi panel size is seven. With smaller size panels, accuracy deteriorates quickly, while extremely large panels experience slow improvements in accuracy. A survey of published Delphi studies shows that typical panels have from 15 to 40 participants, although some have hundreds of respondents (Dalkey and Helmer 1963).nx Michael T Bedford 1972ztThe Value of Competing Panels of Experts and the Impact of 'Drop-outs' on Delphi Results: The Bell Canada Experience  Bell CanadalOctober("James C Brancheau James C Wetherbe 19872,Key Issues in Information Systems Management MIS Quarterlyl111 22-45a March4 Uses Delphi methodHBIS management, key issues, management priorities, delphi techniqueThe results from five-part Delphi study of chief IS executives and corporate general managers indicate the most critical information management issues and consensus on their importance. The research project is second in a series of such studies conducted SIM and MIS Research Center at University of Minnesota. The research confirmed the expected in some areas and revealed surprises in other areas. While strategic planning continued to top all issues in importance, many changes have occured since 1983. Three new issues have joined the top 10 issues in importance. Also the rank order of several issues in the top 10 has shifted. Survey results are discussed in terms of differing views of IS executives and corporate general managers. A review of how these views have changed over time is also presented. A number of conclusion are drawn about management information systems and about the changing nature of IS executive's job.c  3xV Millar 1984The Delphi Technique0)Information Strategy: Executive's Journalr1  32-36 "Satish Nambisan Ritu Agarwal 1999XQOrganizational Mechanisms for Enhancing User Innovation in Information Technologyu MIS Quarterly\233e365-395} September Uses Delphi method|vIS Innovation, organizational mechanisms, technology cognizance, technology exporation, propensity to innovate, DELPHIFostering information technology innovation has assumed primacy in discussions of information systems management. Changes in the nature of available information technologies and their potential applications underscore the importance of creating new knowledge for deploying a technology within an organization rather than transferring such knowledge from external sources. Technology users remain a largely untapped source for such knowledge creation. This paper argues that deliberate organizational design actions in the form of mechanisms can enhance technology users' propensity to innovate in information technology. Specifically, a taxonomy of organizational mechanisms is developed based on the ability of various mechanisms to facilitate knowledge acquisition and knowledge conversion. The conceptual taxonomy is populated with specific design actions described in the literature utilizing a Delphi study. The effects of various classes of mechanisms on three key antecedents of user propensity to innovate in IT--technology cognizance, ability to explore a technology, and intention to explore a technology--are tested using a field study. Results provide support for the conceptual taxonomy. Implications for theory and practice are offered.ztA Delphi study was conducted to classify mechanisms into the conceptual taxonomy. The Delphi technique is deemed appropriate when judgmental information is indispensable. The participants of the Delphi study were a carefully selected set of practicing senior managers from diverse industries to guarantee a wider knowledge base. One of the researchers met with each participant separately and explained the research objective and the Delphi study process. The weighted averages of the points given to each mechanism by the six participants were determined to prepare the classification chart (group opinion). This information was fed back to the participants in the second round. The participants were asked to indicate whether they concurred with the group opinion, and if not, the reason for differing (the reasons presented were also fed back to all participants in each round).*#Justus D. Naumann Shailendra Palvia 19826/A Selection Model for Systems Development Tools} MIS Quarterlyf6}1 39-48 March Uses Delphi methodxSelecting from the many currently available systems development methodologies (SDMs) and development techniques is a difficult problem with economic, technical, and behavioral implications. A quantitative approach to the selection problem is presented. The selection model begins with a definition of a superset of functions expected of a systems development tool. Functions are then weighted, using a Delphi approach to achieve acceptable valuations among system managers. Next, each approach under consideration is evaluated with respect to each function desired. After scores are computed for each methodology, economic and qualitative aspects such as training availability and cost can be used to differentiate the highest ranked alternatives. A four-person MBA project team from the Graduate School of Management at the University of Minnesota, with the guidance from the authors, applied the model to a methodology selection problem. In addition to producing a quantitative ranking of competing methodologies, the approach described furthered understanding of the functions to be performed by the methodologies being considered. It also gained acceptance, admittedly reluctant, of the recommended methodology from managers who strongly advocated their own favorites.m6Uses Delphi little differently as compared to other papers. It was applied to choose between four different candidate SDMs. (The focus of the problem was on program-design phase i.e. communication from the analyst to the programmer). !5 different functions that usually are part of "program specifications" document were identified with literature review and interviews with data processing professionals. Researchers initially ranked these functions. Then each function was weighted by the managers and analysts involved. Once important functions were identified then four candidate SDMs were checked against to see if they had them. Weighted functions developed using Delphi technique are used to evaluate the alternatives based on the scores for each function. Qualitative factors are applied after quantitative evaluation. Advantage: Impersonal and objective. Prevents confrontation between group members eliminating negative aspects of individual dominance.p;v ,*vVan de Ven, Andrew H 1974.'Group Decision making and Effectiveness\ "Kent State University Pressn Ron Weber  1988>8Computer Technology and Jobs: An Impact Assessment Model Communications of the ACM311s 68-77uJanuarynA model is proposed that associates the impact of computer technology on a job, with the set of underlying characteristics that describe the activities performed on the job. An empirical test of the model has been undertaken. One thousand and thirty-five experts assessed the impact of computer technology that they believed would occur on 306 jobs over a three-year period. Job characteristics data was obtained from prior analyses of the jobs, using the Position Analysis Questionnaire (PAQ). Six job dimensions derived from analysis PAQ data were significant predictors of the technological impact ratings provided by the experts: engaging in physical activities; being aware of the work environment; performing clerical-related functions; working in an unpleasant or hazardous environment; performing service-related activities; and performing supervising, directing, and estimating functions. Gordon Welty 197282Problems of Selecting Experts For Delphi Exercises$Academy of Management JournalE151e121-124i March Talks about method itselfDiscusses the problems associated with selecting experts for the Delphi exercise. Ways to solve expert selection problems; Information on the Delphi exercise; Relevance of expertise to forecasting values6Woudenberg, F. 1991An Evaluation of Delphi 2+Technological Forecasting and Social Change 4020131-150 Critique of Delphi MethodThe literature concerning quantitative applications of the Delphi method is reviewed. No evidence was found to support the view that Delphi is more accurate than other judgment methods or that consensus in a Delphi is achieved by dissemination of information to all participants. Existing data suggest that consensus is achieved mainly by group pressure to conformity, mediated by the statistical group response that is fed back to all participants.