ABSTRACT

Manuscript rejection rates for the top IS academic journals average 85-90%. An undesirable consequence of this level of rejection is that the IS community becomes discouraged and disaffected with the publication process. Part of the reason so many manuscripts are not ready for publication may lie in the lack of agreement and understanding among IS researchers on the key criteria for evaluating IS research. The purpose of this study is to report on a survey of the perceptions of published authors, reviewers, and editorial board members about the manuscript requirements for publication in IS. Knowledge gained from the study has the potential to: (1) improve the overall quality of future submissions by focusing the researcher's time and effort on key criteria and normative standards for publishing research, as differentiated by research method (2) reduce the number of revisions required before a manuscript gets published, and (3) suggest journal evaluation forms that more accurately reflect the standards of the IS community. The empirical results reported here provide an introspective analysis of the IS field, a set of normative standards for IS, and an action plan for IS journals.

ii

 

 The Problem of High Manuscript Rejection Rates

A high manuscript rejection rate by scientific journals is a two-edged sword. Although high rejection rates may indicate that reviewers are making fine discriminations that result in the publication of only the best work, they may also discourage and disaffect the scholarly community (Rackoff 1985). In many situations, the community perceives these rates to be unreasonable, setting unreachable thresholds for the bulk of the profession. Promising, but poorly written work may never be disseminated to the field because many researchers respond to journal rejections by returning the work to the file drawer (Rosenthal 1978) rather than reworking it (Gottfredson 1978). Novice researchers unfamiliar with how best to make their case and with how to deal with the substance and mechanics of the journal reviewing process are particularly vulnerable to this natural and very human reaction.

In IS, this is not merely hypothetical. As in other fields, manuscript rejection rates in the top IS journals are very high. According to Boyer and Carlson's (1989) analysis of IS journals, manuscript acceptance rates for the top academic journals in IS averages only 10%. Even acceptance rates for IS conference proceedings can be dishearteningly low. The International Conference on Information Systems (ICIS), for example, accepts only about 15% of submitted manuscripts.

What can be done about high rejection rates? One seeming solution is to create new outlets and to allocate more space in existing journals. But this achieves nothing unless the evaluative standards of reviewers and editors also change in the process. As it happens, IS journals newly-created within the last several years do not appear to have higher acceptance rates than the older top journals (Swanson 1990). The editors for these journals are most often prominent IS researchers who draw their Associate Editors and editorial boards from the same scholarly community whose evaluations have already resulted in the exceedingly high rejection rates experienced by the top journals.

It would seem that the real problem that must be addressed, hence, is that manuscripts are not of sufficient quality to pass the review process. Beyond elaborate prescriptions for more thorough education in research or nostrums and incitations to do better work, what can realistically be done to improve the quality of submitted manuscripts?

 

The Value of Explicit Standards for Total Quality Management

This paper contends that manuscript quality can be improved by making explicit, to authors and reviewers alike, the standards that are being used when manuscripts are rejected or accepted. The straightforward and simple argument is that scientific journals should adopt a total quality management perspective and that this is precisely their proper role. As King, Kilmann, and Sochats (1978) contend in their Management Science article on the journal review process:

In establishing...editorial policy and publication criteria, a scientific journal is defining its own role and importantly affecting the future of the field it represents (boldface and italics added; p. 775).

Moreover, there is theoretical justification behind the assertion that explicit, well understood, and accepted standards will raise the quality of work, including knowledge work such as scientific research. The well established theory of goal-setting (Locke et al. 1981) argues convincingly that knowledge workers will dramatically improve their performance when they have clear, mutually-agreed objectives upon which to act. Contrariwise, when objectives and performance standards are not clear, productivity is known to decline.

Does IS have a set of mutually agreed-upon, unambiguous objectives and professional standards for acceptable, high quality manuscripts? A careful look at the top journals and their practices suggests that we do not. Nowhere, perhaps, is this lack of common standards more evident than in the evaluation forms that the reviewers are required to send in with each manuscript reviewed, as shown in Table 1. A glance at these criteria, criteria which presumably should be used in evaluating manuscripts, shows how widely they vary from journal to journal.

 

MISQ/Data Base	ISR	CACM	Mgmt. Science
Relevance	Significance of contribution	Technical content	Importance of research
Objectives	Technical adequacy	Originality	Impact on discipline
Readability	Appropriateness to journal	Style and organization	Impact on practice
Organization	Clarity of presentation	Overall quality	Presentation
Literature review	& significance
Methodology
Quality of evidence
Contribution
Potential contribution

Table 1. Evaluation Standards for Some Top IS Journals

Editorial standards can sometimes be gleaned from a careful reading of the comments of incoming and outgoing senior editors (King 1985; McFarlan 1988; Emery 1989; Ives 1992), but these standards do not apply universally, nor is it clear that they form a basis for actual evaluation by reviewers. They may, in fact, only reflect the personal beliefs of the senior editor. Senior editors are not the sole gatekeepers for research publication since reviewers and associate editors also play significant roles in the process (Rousseau 1985).

Because of the value of identifying explicitly formulated standards for high quality research, this study sought to discover the criteria used by reviewers and IS editorial board members in accepting and rejecting manuscripts. Knowledge gained from the study has the potential to: (1) improve the overall quality of future manuscript submissions by focusing the researcher's time and effort on key criteria and normative standards for publishing research, as differentiated by research methodology, (2) reduce the number of revisions required before a manuscript is accepted, and (3) suggest journal evaluation forms that more accurately reflect the standards of the IS community. Empirical results reported here provide an introspective analysis of the IS field, a set of normative criteria for IS, and an action plan for IS journals.

Literature Review

Although the underlying dimensions for high quality IS research have not been enunciated for the IS scientific community, there have been numerous studies of publication standards in sociology, psychology, organization behavior and the physical sciences, as shown in Table 2. In one of the first studies on dimensional structures for evaluating research, Chase (1970) found marked differences in the relative importance of normative criteria between the physical and the social sciences. Physical sciences stress precise mathematical and technical criteria while social sciences emphasize logical rigor, and theoretical and applied significance. Wolff's (1970) survey of psychology journal editors enumerated requirements for publication of manuscripts in clinical and personality journals. Results showed consistent agreement on the relative importance of manuscript criteria. Editors rated contribution to knowledge as the most important criterion followed closely by sound research design and objectivity in reporting results. Findings also indicated that an author's reputation and institutional affiliation were least important in manuscript assessment.

Price's (1985) list of criteria is oriented toward the practitioner's perspective on organizational science. He studied how practicing line managers seek information and knowledge from published sources. Among other factors, Price contended that relevance is crucial. Relevance is defined as the ability of research to provide new insights into the organizational problems and relate findings to organizational dilemmas.

In the field of organizational behavior, Daft (1985) performed an introspective analysis of his reviews of Administrative Science Quarterly and Academy of Management Journal submissions. Of the 111 manuscripts he reviewed, lack of theory, poor construct validation, and poor research design were the most frequently occurring problems. Mitchell et al. (1985) gathered descriptive data about the publishing process in organizational behavior. Criteria for quality were drawn from interviews with editors or members of review boards of five organizational behavior journals. Results show that contribution to knowledge was rated highest while poor writing and presentation were rated as less important considerations. The low rating for good writing in Mitchell et al.'s study is interesting because this criterion is often emphasized by opinion leaders in the social sciences. For example, Campbell (1982), as outgoing editor of the Journal of Applied Psychology, wrote:

[My] biggest shock in the entire nine years [as an editor] was the discovery [that] many people cannot describe clearly and directly what they wanted to do, what they did, and what they found out. Clearly written manuscripts are in the minority (p. 693).

	Chase	Wolff	Price	Daft	Mitchell
Criteria	(1970)	(1970)	(1985)	(1985)	(1985)
1. Statistical/mathematical analysis	x	x			x
2. Theory	x	x	x	x
3. Coverage of significant literature	x	x	x
4. Professional style & tone		x		x	x
5. Logical rigor	x		x	x	x
6. Contribution to knowledge	x	x	x		x
7. Contribution to practice	x	x	x
8. Presentation level		x
9. Research design		x		x	x
10. Adherence to scientific ethics	x
11. Manuscript length		x
12. Reputation		x
13. Replicability of research	x	x
14. Suggestions for future research		x	x
15. Topic selection	x	x	x	x	x

Table 2. Criteria for High Quality Research

Research Questions for Present Study

Descriptive Criteria for IS Research

Taken together, these studies demonstrate that scientific disciplines do emphasize different criteria in judging research. But because no prior study has assessed these standards for IS, criteria for IS research are unknown at this time. It can be argued, of course, that high quality research should meet most of the 15 criteria listed in Table 2. But the counter argument is that the IS community, like other scientific communities, will inevitably value some criteria more than others and, therefore, stress a subset of these criteria as most critical for new ideas to gain acceptance. Then too, while many of these criteria have been emphasized in doctoral programs and research methodology treatises, greater stress can and should be placed on disseminating these critical success factors (CSFs) to the entire IS scientific community so that we can reach a higher level of agreement. This self-reflective and introspective understanding of CSF criteria might also lead to greater convergence on standards for papers submitted to IS journals.

Prior studies on publication standards in fields such as psychology, sociology, and the physical sciences do not differentiate criteria according to type of research methodology. However, we believe methodology clearly dictates the relative importance of some criteria over others. For example, one would expect research design to be regarded as an important criterion for laboratory experiments, but irrelevant in conceptual studies. Morgan (1985), in exploring the logic of research methodologies used in the social sciences, agrees that criteria between different methodologies should be considered:

Methodologies...attempt to accomplish quite different things and call upon different criteria for determining how well they have been conducted and what they have achieved (p. 67).

To differentiate criteria according to type of research, the first research question (RQ1) addresses possible variation in the order of importance of criteria across IS methodologies. The first research question, therefore, focuses on how IS researchers rank individual criteria within each methodology.

There were several additional response dimensions that we felt would help us understand the current views of the community. An ancillary research issue addresses the awareness within the community of the need to anchor research on theory. According to Dubin (1976), scientific fields of study are held together and driven by theories that explain phenomena of concern for the field. Theories guide further research and allow accumulation of knowledge about topics of interest (MacKenzie and House 1978). Without theories to guide research, a field of study remains a "theme" (Keen 1980, p. 8). Lack of theory also has the effect of producing the appearance of randomness to those gathering the facts (MacKenzie and House 1978). Given the centrality of theory to developing paradigms, progress in a field will be closely related to how extensively theories serve as conceptual references for research (Webster and Starbuck 1988).

Presently, theory does not appear to be an important desideratum for good research in IS. According to Banville and Landry (1989), lack of theory in most IS research results in fragmented research streams. This observation has been confirmed for IS research in general (Alavi et al. 1989) and for research in DSS in particular. In the Alavi et al. review of IS research, only 15 articles in the previous 20 years to the study were found to be theoretically oriented. Given that Alavi et al. gathered data on nearly a thousand IS studies, this rate suggests that very little work in the field had to that point been based on theory. Adams et al.'s investigation of the DSS literature (1989) also supports Banville and Landry's contention that IS lacks theory bases. Given the paucity of theory in IS, an ancillary research question examines the extent to which attitudes about the role of theory in IS research might be changing and how much consensus there is in the IS community that theory is vital for good research.

The ancillary research question (RQ1a), therefore, focuses on the extent to which theory is perceived by the entire IS scientific community to be an important criterion for judging IS research.

A Parsimonious Set of Normative Standards for IS Research

While there are interesting intellectual and empirical questions that can be answered by pairwise comparisons of the individual criterion ratings across the fifteen criteria, we felt that the field would profit from a more parsimonious set of criteria than the existing fifteen that have been espoused in prior literature. Given that humans are cognitive misers and limited in their ability to process information (Miller 1956), we believe that a smaller set of orthogonal factors derived from the fifteen criteria will provide integrative, logically consistent standards which cover all important aspects for assessing quality IS research. A parsimonious set of standards can focus the attention of IS researchers as well as reviewers on the elements that need to be addressed for publishable papers. IS journals can profit from having a small number of empirically derived standards for their journal evaluation forms.

There is another compelling reason for delimiting the community standards for quality work. As stated earlier, goal setting theory provides evidence that performance standards and feedback on the extent to which individuals meet performance standards result in demonstrably higher performance. In terms of the review process, therefore, we can expect a higher quality of submissions and a higher rate of acceptances if these standards are articulated, generally subscribed to, and reflected in higher quality manuscripts. The second research question (RQ2) addresses whether it is possible to derive a parsimonious set of meaningful standards for IS research.

Methodology

To answer the above research questions, it was determined that a survey of the perceptions of published IS authors and editors would be the most appropriate methodological choice. Accordingly, a questionnaire dealing with criteria for evaluating the quality of IS journal articles was developed. Fifteen criteria, shown in Table 2, were consolidated from studies conducted in other disciplines (Chase 1970; Wolff 1970; Price 1985; Daft 1985; Mitchell et al. 1985).

Sample

To obtain a representative sample of the IS scientific community, names and affiliations of authors and editorial board members were drawn from complete volumes of the Communications of the ACM, Management Science, MIS Quarterly, and Information & Management for the period from 1985 to 1989. The final sample included 523 IS professionals.

 

Pilot Testing of the Questionnaire

To assess content validity of the instrument, a pilot questionnaire was administered to 40 faculty and doctoral students in an IS program at a major midwestern research institution. Based on the process for validating content suggested by Straub (1989), the 15 criteria examined in the pilot test were deemed sufficiently content-valid for purposes of judging quality of IS research submissions. The pilot study also suggested the need for different ratings across research methodologies. Following Van Horn (1973) and Vogel and Wetherbe (1984), six methodologies were chosen. They were: (1) case studies, (2) field experiments, (3) field studies, (4) laboratory experiments, (5) conceptual studies, and (6) reviews/tutorials.

Respondents were asked to select two research methodologies they felt most comfortable reviewing. For each methodology, respondents rated 15 criteria on a 9-point scale ranging from "not important" to "critically important." A sample copy of a survey sent to one of the participants appears in Appendix A.

To control for order effects, criteria were uniquely ordered for each questionnaire (Muller et. al. 1982; Perrault 1975-76). Each respondent, therefore, received a unique ordering of the questions, generated randomly by a computer program.

Results

The research instrument was mailed to every published author and editorial board member who appeared in the selected journals over a five year period. Of 523 questionnaires sent out, 144 (27.5%) were returned. Because each respondent evaluated up to 30 criteria (15 criteria for each of two methodologies), the total number of ratings was 4215 and the N for factor analysis was 281 (less 7 missing data points). This sample formed the data bank for subsequent statistical analysis.

Respondent Characteristics

A profile of survey respondents by geographic location is provided in Figure 1. All parts of the world were represented in the returns, with respondents from North America, United States (69%) and Canada (10%) forming the bulk of the respondents. Although there are reasons to believe that respondents from outside of North America could be a biased sub-sample that might not represent the research heritage of their respective international communities, a separate research note published by the authors suggests that this is not the case (Evaristo, Ang, and Straub 1992).

Of the survey respondents who were published IS authors, 49% were members of editorial boards of IS journals and journals in related disciplines, such as psychology, computer science, and organizational studies. The overwhelming majority of the respondents were academics (94%) with IS practitioners constituting about 6% of the total returns. A profile of professional characteristics is shown in Figure 2.

Tests for Non-Response Bias

Two time-dated waves were used to test for non-response bias (Babbie 1973). First-wave returns were received within one month after the survey was sent out. Subsequent responses, which were coded as second-wave returns, served as surrogates for non-respondents.

To test for non-response bias, time-dated waves were compared on criterion ratings across each methodology. No T-tests were statistically significant at the .05 level. These results suggest that findings can be generalized to the entire IS scientific community, to the extent to which that community is adequately represented by authors who published in the selected journals within the last half of the 1980s.

RQ1: The Importance of Criteria by Method

To address the first research question, rank-ordered mean ranks of criterion ratings for all six methodologies appear in Appendix B. As shown in Appendix B, criteria vary in importance across methodologies. One criterion in particular surfaced as crucial. As might be expected, contribution to knowledge was ranked first in three of six methodologies. Emphasis on this criterion concurs with the results of Wolff (1970) in psychology and Mitchell et al. (1985) in organizational behavior.

Appearing in the top half of every list, respondents judged contribution to knowledge, logical rigor, and theory to be key criteria for all types of research. Closely following these rankings was coverage of significant literature, which was slotted in the first half of all but one list. These results suggests that respondents felt these criteria were important irrespective of the methodology employed in the research.

Among the lower ranked criteria, author's reputation / institutional affiliation was generally considered insignificant for judging journal submissions. Other criteria received intermediate rankings and showed more variation in their placement than criteria at either extreme. Given the interest in the "relevance versus rigor" controversy in IS (Grover and Sabherwal 1989; ICIS panel 1988), it is surprising that contribution to practice placed at a lesser priority position in all lists but one.

In order to verify the face validity of our analyses of these data, i.e., to verify that methodology made a significant difference in how criteria were ranked by respondents, a 15 x 6 one-way ANOVA of all rankings (N = 4235 ) was performed. Both main effects of methodology and criterion type were found to be significant at the .05 level, as shown in Table 3. This test suggests that reviewers do consider type of methodology in weighing the importance of certain criteria when judging manuscripts.

Dependent Variable: Criterion ranking
Source	d.f.	Sum of squares	Mean square	F-value	P-value
Method	5	68123916.9		9.4	.0001
Criterion	14	2569177014.5		126.5	.0001
Model	19	2637300931.5	138805312.1	95.74	.0001
Error	4216	7125936719.9	1449834.5
R-square =	.27

Table 3. ANOVA Results

RQ1a: The Importance of Theory as a Criterion

The ancillary research question to the overall question of how the community ranked the criteria was the place of theory in the evaluation of manuscripts. As noted above, the rankings suggest that theory is viewed as a key criterion across all types of research. To test the face validity of this interpretation, we compared the rank mean of theory versus the rank mean of all other criteria for each of the six methods. The results, summarized in Table 4, show that theory was ranked significantly higher than the mean of all other criteria for all methods. These results suggest that, in spite of the historically low use of theory in IS research, the community as a whole does value theory in judging the quality of journal submissions.

Rank Mean	Conceptual	Field Study	Lab Experiment	Reviews, Tutorials	Field Experiment	Case Study
Theory	10.65*	9.40*	9.36*	9.17*	9.12*	8.42*
All Other Criteria	7.81	7.89	7.90	7.91	7.94	7.96

* Significantly higher at the .05 level.

Table 4. Nonparametric Rank Mean Difference Tests

RQ2: Derivation of a Parsimonious Set of Standards

To determine if there was a more parsimonious set of standards that could be used to characterize IS research, a principal components factor analysis was run on 281 evaluations (7 data points were missing). Using orthogonal rotation, the rotated factor structure with the highest explained variance (90%) is shown below in Appendix C.

At a .3 cutoff level, five factors emerged, including a separate factor for reputation of author and/or institutional affiliation. Since it has been previously determined that this criterion should not be not highly valued in academic research (Wolff 1970), the remaining four factors can be said to be "standards" for IS research.

The first standard can be termed "Conduct of Research" since criteria such as use of appropriate statistical techniques, design of research, and replication of other work all deal with how the research was carried out. Conducting research in an ethical fashion ("Scientific ethics") is also covered under this heading.

A second standard, "Presentation," was revealed in the underlying data. Within this standard are such criteria as logical rigor and professional style. Since length of manuscript was ranked at a low level across all methods, this criterion was dropped from the normative standards. Overall, the emphasis in this standard is on how convincingly the author is able to present his or her ideas to the audience.

The third standard can be termed "Conceptual Significance." Falling under this category are criteria like contribution to knowledge, topic selection, and theory. Literature review and directions for future research are also included under this category. Given that many or most of these elements appear at the top of the ranked criteria lists (Appendix B), it is logical to consider this standard to head a prioritized listing of standards for journal review purposes.

The fourth and final standard is "Practical Significance," which covers both selecting topics appropriate for the audience as well as dealing with problems that apply to the real world. Although criteria in this category are not ranked very highly by the respondents, IS is an applied field and it would seem to be imperative to highlight this standard in reviewing manuscripts.

Discussion

Overall, our study showed that criteria varied in relative importance for judging IS journal submissions. Nevertheless, a few criteria were consistently more important than other criteria. Contribution to knowledge, coverage of significant literature, logical rigor, and use of theory were rated as important criteria regardless of methodology. Contribution to knowledge, in particular, was deemed essential under all circumstances. This result confirms the conclusion of Daft et al. (1987) that the ability to add to our knowledge about a topic is the single most important factor in differentiating significant from not-so-significant research.

As discussed above, contribution to knowledge is sine qua non for all good research. The best research will be based on a thorough and demonstrated knowledge of the literature that rests on theoretical foundations and carries the weight of its argument by showing how this research extends core knowledge in the field.

If the results of the rankings by method are examined from the point of view of qualitative versus quantitative research (Straub 1989; Kaplan and Duchon 1988), highly interesting patterns emerge. From this perspective, case studies, which rely on relatively little statistical analysis, fall into a category with conceptual studies and reviews/tutorials as qualitative research, whereas field studies, field experiments, and laboratory experiments, which place more reliance on statistical analysis, are remarkably similar in the patterns of criteria importance and may be thought of as quantitative research.

According to the data, the communication of substantive elements in qualitative research is particularly sensitive to how well the report is written. That is, case studies, conceptual studies, and reviews/tutorials must organize the substantive message in such a way that the audience can readily understand (presentational level and professional style & tone) the flow of ideas and the power of the arguments (logical rigor). It is interesting to note that presentation, professional style, and tone, important criteria for qualitative research, are not in the top half of the criteria for quantitative research.

For quantitative research, research design overshadows others in the top half of the criterion sets. This rating concurs with Jenkins' (1985) assertion that poor research design has plagued IS research and that the field needs to pay greater attention to the matching of method to problem. Other researchers (Hamilton and Ives 1982; Farhoomand 1987) have noted that IS researchers often indiscriminately apply a research method, especially the survey method, to IS problems. Researchers should consider asking questions such as whether the strengths of the survey methodology in gathering opinion data are applicable in areas where dependent variables need to be measured with high confidence (Ives and Olson 1984).

The emphasis on statistical analysis in quantitative research suggests that IS researchers should place high value on careful and precise use of statistical techniques in data analysis. For example, Baroudi and Orlikowski (1989) observe that important relationships among variables in empirical work often go undetected because of low statistical power. IS researchers are also urged to be sensitive to internal and external validity issues and to the need for instrument validation. This emphasis corroborates Straub's (1989) argument for greater attention to the use of statistical techniques in earlier stages of the research cycle than statistical conclusion validity (Cook and Campbell 1979). Testing relationships between hypothesized variables without ruling out effects of moderating or exogenous variables means that internal validity has not been completely addressed through statistical or experimental controls (Jarvenpaa et al. 1985).

Looking at the criteria for each of the six methodologies, we notice a unique feature of experiments. Scientific ethics was rated almost equally in field and laboratory experiments and as a more important criterion here than in the other methodologies. This seems consistent with the features of experimental designs where the issue of scientific ethics becomes more pronounced as the researcher manipulates and creates artificial environmental conditions.

The view that theory is a key criterion regardless of background characteristics of the respondent was verified in the analysis. Attitudes toward the importance of theory were consistent across editorial experience, professorial rank, and extent of reviewing. This finding could be interpreted to mean that IS, as a field, recognizes the central role of theory and theory development in IS. If that interpretation is accurate, we can expect journal submissions to be judged more critically for theoretical content and will see more efforts channeled towards theory building in the IS context.

Normative Standards for IS Research

Our study found an underlying parsimonious set of standards for assessing IS research. The norms include conceptual and practical significance as well as conduct and presentation of research.

In spite of the knowledge the study has provided of the evaluative opinions of reviewers and editors, the practical value of this study will only be felt when the IS community comes to a better understanding and a more general agreement on the importance of these now explicitly stated standards. Journal editors can be instrumental in this process by emphasizing these standards explicitly in the evaluation of manuscripts.

 

Suggestion for New Journal Evaluation Forms

Based on the results of the present study, we propose a straightforward enhancement to existing journal evaluation forms. Although journal editors could choose to use up to six forms, one for each of the six methodologies surveyed, submissions that do not fall neatly into one of the six categories will be difficult to classify if this procedure is adopted. Moreover, editors may not want to adopt an arbitrary cut-off point for selecting a certain number of criteria.

Four proposed standards for a single purpose journal evaluation form are shown in Appendix D. The form reflects the parsimonious set of standards derived through the factor analysis, prioritized both by existing values of the field and by a normative standard ("practical significance") that is highly appropriate for an applied field like IS. Standards that are particular to the mission of a journal can also be added by journal editors for a final form. Moreover, journal editors may wish to formulate additional criteria that are very specific to the activity of evaluating manuscripts. For example, a category on "Potential for publication after revision" could be extremely useful in determining the fate of a paper.

Presently, as demonstrated in Table 1, the top journals have little consistency in standards, but, what is more crucial, they do not always stress the standards empirically identified in this study. A singular omission from these journal evaluation forms is a standard which highlights the value of theory-based research.

We urge journal editors and the IS scientific community to seriously consider revising standards used to judge the quality of journal submissions along the lines evidenced in this study. Normative standards proposed here might also be disseminated to prospective IS authors to help demystify the assessment of quality in IS research. The order of priority of the standards can be varied by the chief editor to reflect the mission of the journal, but all standards should be included in one way or another. If the normative standards identified here are attended to, we believe that there is a potential to: (1) improve the overall quality of future submissions by focusing the researcher's time and effort on key criteria and normative standards for publishing research, (2) reduce the number of revisions required before a manuscript gets published, and (3) adopt journal evaluation forms that more accurately reflect the standards of the IS community.

References

Adams, C., M. Eierman, and F. Niederman. (1989). "The Development of DSS Research: A Descriptive and Prescriptive View," Proceedings of the Twentieth Annual Meeting of the Midwest Decision Sciences Institute, Cincinnati, OH, April 19-21, pp. 56-58.

Alavi, M., P. Carlson, and G. Brooke (1989). "The Ecology of MIS Research: A 20 Year Status Review," Proceedings of the Tenth International Conference on Information Systems, Boston, MA, pp. 363-376.

Babbie, Earl R. (1973). Survey Research Methods. Belmont, CA: Wadsworth.

Banville, C. and M. Landry (1989). "Can the Field of MIS Be Disciplined?" Communications of the ACM, Vol. 32, No. 1, pp. 48-61.

Baroudi, J. J. and W. J. Orlikowski (1989). "The Problem of Statistical Power in MIS Research," MIS Quarterly, Vol. 13, No. 1, pp. 87-106.

Boyer, Glen L. and Gary Carlson (1989). "Characteristics of Periodical Literature for the Potential Reader or Author in Information Management," MIS Quarterly, Vol. 13, No. 2 (June), 221-229.

Campbell, J. P. (1982). "Editorial: Some Remarks from the Outgoing Editor," Journal of Applied Psychology, Vol. 67, No. 6, pp. 691-700.

Carmines, Edward G. and Richard A. Zeller (1979). Reliability and Validity Assessment. Sage University Paper Series on Quantitative Applications in the Social Sciences, series no. 07-017. Newbury Park, CA: Sage Publications.

 

Chase, Janet M. (1970). "Normative Criteria for Scientific Publication," The American Sociologist, August, pp. 262-265.

Conover, W.J. and Ronald L. Inman (1981). "Rank Transformations as a Bridge between Parametric and Nonparametric Statistics," The American Statistician, Vol. 35, No. 3 (August), 123-129.

Cook, Thomas D. and Donald T. Campbell (1979). Quasi-Experimentation: Design and Analytical Issues for Field Settings. Chicago: Rand McNally.

Daft, R.L. (1985). "Why I Recommended that Your Manuscript be Rejected and What You Can Do about It" in Publishing in the Organizational Sciences, L.L. Cummings and P.J. Frost (eds.), Homewood, IL: Irwin, pp. 193-209.

Daft, R.L., R. W. Griffin, and V. Yates (1987). "Retrospective Accounts of Research Factors Associated with Not-So-Significant Research Outcomes," Academy of Management Journal, Vol. 30, No. 4, pp. 763-785.

Dubin, R. (1976). "Theory Building in Applied Areas" in Handbook of Industrial and Organizational Psychology, M.D. Dunnette (ed.), Chicago, IL: Rand McNally.

Emery, J.C. (1989). "Editor's Comments," MIS Quarterly, Vol. 13, No. 3 (September), pp. xi-xii.

Evaristo, Roberto, Soon Ang, and Detmar W. Straub (1992). "Criteria for High Quality Information Systems Research: A Comparison of the Views of North American and International Scholars," International Information Systems, Vol. 1, No. 4 (October), 99-105.

Evaristo, J. R. and Elena Karahanna (1989). "An Empirical Investigation of the Difference between North American and European MIS Research," Proceedings of the Organizational and Information Systems International Conference, Bled, Yugoslavia, September, pp. 786-792.

Farhoomand, Ali (1987). "Scientific Progress of Management Information Systems," Data Base, Summer, pp. 48-56.

Gottfredson, S. D. (1978). "Evaluating Psychological Research Reports: Dimensions, Reliability, and Correlates of Quality Judgments," American Psychologist, pp. 920-934.

Gouldner, A. W. (1957). "Theoretical Requirements of the Applied Social Sciences," American Sociological Review, Vol. 22, No. 1, 92-102.

Grover, Varun and Rajiv Sabherwal (1989). "An Analysis of Research of Information Systems from the IS Executive's Perspective," Information & Management, Vol. 16, No. 5, 233-246.

Hair, Joseph F., Jr., Rolph E. Anderson, Ronald L. Tatham, and Bernie J. Grabowsky (1979). Multivariate Data Analysis. Tulsa: PPC Books.

Hamilton, S. and B. Ives (1982). "MIS Research Strategies," Information & Management, Vol. 5, pp. 339-347.

ICIS Panel (1988). "The Future of Information Systems as an Academic Field: Your Fate in 1988," in Proceedings of the Ninth International Conference on Information Systems, Minneapolis, Minnesota, pp. 337-340.

Ives, B. (1992). "Editor's Comments: Bridging Research and Practice," MIS Quarterly, March, iii-vi.

Ives, Blake and M. Olson (1984). "User Involvement and MIS Success: A Review of Research," Management Science, Vol. 30, No. 5, 586-603.

Jarvenpaa, S. L., G. W. Dickson, and G. DeSanctis (1985). "Methodological Issues in Experimental IS Research: Experiences and Recommendations," MIS Quarterly, Vol. 9, No. 2 (June), pp. 141-156.

Jenkins, A. Milton (1985). "Research Methodologies and MIS Research," in Research Methods in Information Systems, E. Mumford, R. Hirschheim, G. Fitzgerald, and T. Wood-Harper (eds.), Amsterdam: Elsevier Science Publishers B.V., pp. 103-117.

Kaplan, Bonnie and Dennis Duchon (1988). "Combining Qualitative and Quantitative Methods in Information Systems Research," MIS Quarterly, Vol. 12, No. 4 (December), 571-587.

Keen, P.G.W. (1980). "MIS Research: Reference Disciplines and a Cumulative Tradition," Proceedings of the First International Conference on Information Systems, December 8-10, pp. 9-18.

King, William R. (1985). "Editor's Comments," MIS Quarterly, Vol. 9, No. 4 (December), pp. xiii.

King, William R., Ralph H. Kilmann, and Kenneth Sochats (1978). "Designing Scientific Journals: Issues and Survey Results," Management Science, Vol. 24, No. 7 (March), 774-785.

Kuhn, T. S. (1970). The Structure of Scientific Revolutions, 2nd Edition, University of Chicago Press, Chicago.

Locke, Edwin A., Karyll N. Shaw, Lise M. Saari, and Gary P. Latham (1981). "Goal Setting and Task Performance: 1969-1980," Psychological Bulletin, Vol. 90, No. 1, 125-152.

MacKenzie, K.D., and R. House (1978). "Paradigm Development in the Social Sciences: A Proposed Research Strategy," Academy of Management Review, January, pp. 7-23.

McFarlan, F.W. (1988). "Editor's Comments," MIS Quarterly, Vol. 12, No. 4 (December), pp. xvii-xviii.

Miller, George A. (1956). "The Magical Number Seven, Plus or Minus Two: Some Limits on our Capability for Processing Information," Psychological Review, Vol. 63, No. 2 (March), 81-97.

MISRC (1989). Directory of Management Information Systems Faculty, ed. Janice DeGross. New York: McGraw-Hill.

Mitchell, T.R., L.R. Beach, and K.G. Smith (1985). "Some Data on Publishing from the Authors' and Reviewers' Perspectives," in Publishing in the Organizational Sciences, pp. 248-264.

Morgan, G. (1985). "Journals and the Control of Knowledge: A Critical Perspective" in Publishing in the Organizational Sciences, L.L. Cummings and P.J. Frost (eds.), Homewood, IL: Irwin, pp. 63-75.

Muller, Ross M., P.S. Levy, C.S. Byre, and D. Matthews (1982). "Effects of Characteristics of the Survey Instrument to a Mail Survey of Community Hospitals," Public Health Reports, Vol. 97, No. 5, pp. 465-469.

Perrault, William D., Jr. (1975-76). "Controlling Order-Effect Bias," Public Opinion Quarterly, Vol. 39, No. 4, pp. 544-551.

Price, R.L. (1985). "A Customer's View of Organizational Literature" in Publishing in the Organizational Sciences, L.L. Cummings and P.J. Frost (eds.), Homewood, IL: Irwin, pp. 125-133.

Rackoff, Vivian M. (1985). "On Being Published: A Contemporary Preoccupation" in Publishing in the Organizational Sciences, L.L. Cummings and P.J. Frost (eds.), Homewood, IL: Irwin, pp. 299-306.

Rosenthal, Robert (1978). "The File Drawer Problem and Tolerance for Null Results," Psychological Bulletin, Vol. 85, 185-193.

Rousseau, Denise M. (1985). "Publishing from a Reviewer's Perspective" in Publishing in the Organizational Sciences, L.L. Cummings and P.J. Frost (eds.), Homewood, IL: Irwin, pp. 182-192.

Spector, Paul E. (1992). Summated Rating Scale Construction: An Introduction. Sage University Paper Series on Quantitative Applications in the Social Sciences, Series no. 07-082, Newbury Park, CA: Sage Publications.

Straub, D.W. (1989). "Validating Instruments in MIS Research," MIS Quarterly, Vol. 13, No. 2, pp. 147-170.

Swanson, Burton (1990). Memorandum about the review process at Information Systems Research.

Van Horn, R.L. (1973). "Empirical Studies of Management Information Systems," Data Base, Vol. 5, Nos. 2,3,4 (Winter), pp. 172-180.

Vogel, D.R. and J.C. Wetherbe (1984). "MIS Research: A Profile of Leading Journals and Universities," Data Base, Fall, pp. 3-14.

Webster, J. and W. H. Starbuck (1988). "Theory Building in Industrial and Organizational Psychology," International Review of Industrial and Organizational Psychology 1988, John Wiley & Sons Ltd., pp. 93-138.

Wolff, W. M. (1970). "A Study of Criteria for Journal Manuscripts," American Psychologist, Vol. 25, pp. 636-639.

 

Appendix A

 

KEY CRITERIA FOR PUBLISHABLE, HIGH QUALITY RESEARCH

Instructions:

Imagine you are reviewing a manuscript which adopts the research methodology you checked off. Please rate the following publication criteria on their relative importance when you assess the submitted manuscript. Indicate your views by rating each issue on the following scale:

SCALE:	Not Important	Moderately Important	Critically Important
	1 2 3	4 5 6	7 8 9

 

Rating Criteria and Definitions

1. Research design

Appropriateness of the method, subjects, and techniques; appropriate operationalization of theoretical concepts; internal and external validity.

2. Professional style and tone

Appropriate and correct writing style; grammar; clarity of figures and tables; conciseness.

3. Topic selection

High or current readership interest; interesting choice of paradigm or data analysis technique.

4. Contribution to practice

Link to current technological and organizational problems or challenges faced by MIS practitioners.

5. Manuscript length

Length of the manuscript within a range of pages considered acceptable for a given journal.

6. Reputation

Status and reputation of the author and author's institution.

7. Adherence to scientific ethics

Observing the code of ethics for the conduct of human subjects research to best contribute to science and human welfare.

8. Replicability of research

Feasibility of conducting the same study based on the information provided by the author.

 [Continued on next page]

 

KEY CRITERIA FOR PUBLISHABLE, HIGH QUALITY RESEARCH

Instructions:

Imagine you are reviewing a manuscript which adopts the research methodology you checked off. Please rate the following publication criteria on their relative importance when you assess the submitted manuscript. Indicate your views by rating each issue on the following scale:

SCALE:	Not Important	Moderately Important	Critically Important
	1 2 3	4 5 6	7 8 9

Rating Criteria and Definitions

9. Theory

Use of theories from MIS or reference disciplines to explain the relationships among variables used in this study.

10. Suggestion for future research

Directions for extending or improving the present research.

11. Coverage of significant literature

Discussion of relevant literature; explication of underlying assumptions.

12. Contribution to knowledge

Extending or challenging present beliefs and assumptions in the MIS knowledge base.

13. Presentation Level

Presented at a level of sophistication and economy of explanation appropriate to the readership of the journal.

14. Logical rigor

Tight, logical flow of ideas with clear ties between literature review and method, and clear links between method and results.

15. Statistical/Mathematical Analysis

Appropriateness of analytical techniques (e.g., statistics); appropriateness of interpretation of analytical results; magnitude of effects.

 

Appendix B

Rank Means of Criteria Ratings

CASE STUDY (N=47)	RankMean		CONCEPTUAL (N=80)	RankMean
Contribution to Knowledge	11 .9		Contribution to Knowledge	12.2
Presentational Level	10.8		Logical Rigor	11.7
Contribution to Practice	10.7		Theory	10.6
Logical Rigor	10.7		Coverage of Sig. Literature	10.6
Professional Style & Tone	9.6		Presentational Level	9.6
Topic Selection	8.8		Professional Style & Tone	9.0
Scientific Ethics	8.5		Contribution to Practice	7.5
Theory	8.4		Suggest future research	8.8
Coverage of Sig. Literature	8.1		Topic Selection	8.8
Research Design	7.9		Research Design	6.4
Suggest future research	7.5		Scientific Ethics	5.9
Manuscript length	4.9		Manuscript length	5.6
Statistical Analysis	4.8		Statistical Analysis	4.8
Replicability	4.3		Replicability	4.4
Reputation	2.5		Reputation	3.5

 

FIELD STUDY (N=71)	RankMean		LAB EXPERIMENT (N=45)	RankMean
Contribution to Knowledge	11.1		Research Design	13.0
Research Design	10.7		Logical Rigor	11.4
Logical Rigor	10.6		Statistical Analysis	10.9
Statistical Analysis	9.3		Contribution to Knowledge	10.2
Theory	9.4		Replicability	10.1
Coverage of Sig. Literature	8.9		Theory	9.3
Professional Style & Tone	8.8		Coverage of Sig. Literature	9.2
Presentational Level	8.6		Scientific Ethics	9.1
Contribution to Practice	8.0		Professional Style & Tone	7.4
Topic Selection	7.7		Presentational Level	7.1
Scientific Ethics	7.6		Topic Selection	6.1
Replicability	6.7		Contribution to Practice	5.4
Suggest future research	6.2		Suggest future research	5.3
Manuscript length	3.9		Manuscript length	3.5
Reputation	1.7		Reputation	1.3

  

REVIEWS, TUTORIALS (N=32)	RankMean		FIELD EXPERIMENT (N=54)	RankMean
Coverage of Sig. Literature	12.4		Research Design	12.3
Logical Rigor	11.4		Logical Rigor	10.9
Contribution to Knowledge	11.1		Statistical Analysis	10.7
Suggest future research	10.1		Contribution to Knowledge	10.3
Presentational Level	9.1		Theory	9.1
Theory	9.1		Coverage of Sig. Literature	8.7
Topic Selection	8.7		Scientific Ethics	8.7
Contribution to Practice	8.3		Presentational Level	8.4
Professional Style & Tone	8.2		Professional Style & Tone	8.2
Scientific Ethics	6.4		Replicability	8.0
Research Design	6.1		Contribution to Practice	6.8
Statistical Analysis	5.3		Topic Selection	6.3
Manuscript length	5.3		Suggest future research	5.8
Replicability	3.9		Manuscript length	3.3
Reputation	3.8		Reputation	1.5

APPENDIX C

Factor Structure for Criteria

Rotated (Orthogonal) Factor Pattern for Principal Components Factor Analysis

 

	Conduct of Research	Presentation	Conceptual Significance	Practical Significance	Reputation
	FACTOR 1	FACTOR 2	FACTOR 3	FACTOR 4	FACTOR 5
Replication	0.86534	0.08235	0.04762	-0.13994	-0.02588
Statistical/mathematical analysis	0.81693	-0.10070	-0.05386	-0.13368	-0.04339
Research design	0.81457	-0.09904	0.00660	-0.02483	-0.18575
Scientific ethics	0.64277	0.09970	-0.02763	0.16663	0.13636
Professional style	0.00067	0.82743	-0.03942	0.05837	-0.04849
Presentation level	-0.05027	0.76575	0.11720	0.23737	-0.04151
Length	0.07558	0.58341	0.11168	-0.04855	0.24259
Logical rigor	0.05124	0.37871	0.22158	-0.54235	-0.39940
Coverage of significant literature	-0.07621	0.04436	0.72657	-0.07645	0.13926
Theory	0.18016	0.02425	0.70759	-0.14605	-0.13520
Suggest future research	-0.21462	0.32965	0.54136	0.11252	0.15892
Contribution to knowledge	0.03360	-0.05320	0.47847	0.31624	-0.53195
Contribution to practice	0.03254	0.12025	-0.13348	0.79441	-0.10276
Topic	-0.15976	0.33715	0.13176	0.54768	0.03757
Reputation	-0.04846	0.08771	0.16970	0.06037	0.77486

 

Variance explained by each factor:

FACTOR 1 FACTOR 2 FACTOR 3 FACTOR 4 FACTOR 5

26.1% 20.4% 16.9% 14.9% 12.3%

Final community estimates: total = 90.89%

 

APPENDIX D

Suggested Normative Standards for Journal Evaluation Form

 

I. Conceptual Significance

The work represents an important contribution to knowledge. It extends or challenges present causal assumptions in the IS theory or knowledge base. It uses theories from IS or reference disciplines to explain the relationships among variables in the study. Ties to relevant literature are clear as is the thrust of the central argument. The work explicates underlying assumptions well and provides direction for extending or improving on the present research.

II. Practical Significance

The work contributes to our understanding of current technological and organizational problems or challenges faced by IS or other practitioners. In presenting an interesting paradigm or data analysis technique, it maintains readership interest.

III. Conduct of Research

Methods, subjects, and techniques are well suited to the exploration of the research questions. The work demonstrates appropriate operationalizations of theoretical constructs and an acceptable degree of internal and/or external validity. The choice of statistical and/or mathematical analysis is appropriate as is the interpretation of results. Study results are objective and in such a form that other researchers could replicate the work. The work adheres to generally accepted standards for scientific ethics.

IV. Presentation of Research

The work adopts a professional style and tone and is concise. It is grammatically correct and clear in its use of figures and tables. The flow of ideas in the paper is logical and there is a clear tie between literature review and method and a clear link between method and results. The work is presented at a level of sophistication and length appropriate to the readership of the journal.