Graduates
Dissertation Title: Toward Evaluating Organizational Effectiveness
Abstract: If organizations are to survive and achieve long-run viability, they must be effective. Yet organizational effectiveness evaluation has been limited by a variety of factors—especially the ambiguity surrounding the concept of organizational effectiveness itself. The long-established approaches used by organizational theorists to define organizational effectiveness have emphasized different perspectives with respect to the organization type and degrees of importance of the various constituency groups comprising the organization. Moreover, definitions of the organization and organizational effectiveness have focused primarily on the dissimilarities among organizations and their constituencies, conceding efforts to identify commonalities. This has led to increased fragmentation of the concept of organizational effectiveness and weakened its utility. Because of its central importance to both researchers and management practitioners, among many other organizational participants, the development of a practical tool that overcomes the deficiencies and challenges in current approaches to organizational effectiveness evaluation is of theoretical and practical value.
This dissertation addresses the lack of satisfactory approaches for assessing organizational effectiveness and offers an alternative framework using an evaluation checklist. The Organizational Effectiveness Evaluation Checklist (OEC) outlines a comprehensive process for evaluating organizational effectiveness that specifically addresses the values issues with its inclusion of universal criteria of merit. Two separate investigations were conducted to empirically validate the OEC. The first investigation sought input from subject matter experts and targeted users regarding the merit of the OEC. The second applied the revised OEC in a real-world evaluation to assess the effectiveness of a for-profit organization. Both investigations were keyed to a common set of criteria to evaluate and improve the checklist. The original OEC performed well with respect to its pertinence to organizational effectiveness evaluation, clarity, fairness, and sound theory. The most significant structural change made to the OEC was intended to facilitate the efficient use of the checklist and increase its parsimony. Implications for organizations and the professional practice of evaluation are discussed, along with opportunities for future exploration.
Dissertation Title: Evaluation of Sustainability for Sustainability: Development and Validation of an Evaluation Checklist
Abstract: Sustainability is a buzz word these days that permeates many levels of human activity. Interest in sustainability is grounded primarily in the sustainable development field which is concerned with the survival of humans on planet earth, and with the growing demands of meeting people’s long-term needs. In the North American evaluation literature, however, sustainability is primarily thought of in terms of continuing program activity beyond initial funding cycles via diversification of funding streams or institutionalization.
As such, two distinct perspectives for evaluating sustainability were identified. The first is concerned with micro-level issues, that is, evaluators are concerned about the continuation of programs, policies, and other types of evaluation objects (i.e., evaluands). Second, there is a macro level perspective concerned with sustaining human, social, and economic development under consideration of protecting the environment. Both concepts are interdependent. Hence, sustainability evaluation should incorporate both concerns; the continuation of human activity (i.e., projects, programs, policies) and the maintenance of means for mankind to exist on earth (i.e., human, social, economic, and environmental needs).
To address both issues requires evaluation (i.e., the determination of merit, worth, and significance) of sustainability (of evaluands) for sustainability (human survival on earth). While there are numerous evaluation objects within the larger sustainable development movement that can be linked to sustainability (e.g., products, programs, services, organizations), this dissertation assesses the value and usefulness of a sustainability evaluation checklist, specifically designed for programs and projects that address basic human needs and deal with the dynamics that exist between human, social, economic, and environmental dimensions of sustainability. The checklist was developed based on a review of the pertinent literature and was validated and refined based on feedback from experts and practitioners who provided critical reactions to the draft checklist, its potential, and usefulness for evaluating sustainability in diverse contexts.
Dissertation Title: A Metaevaluation of Lessons Learned Repositories: Evaluative Knowledge Management Practices in Project Management
Abstract: The big picture of this dissertation is focused on assessing lessons learned repositories, which involves good practices of managing evaluative knowledge in the context of project management. In the past, primarily because of the inaccessibility to computers, lessons learned were stored on paper and subject to loss and destruction. Today, lessons learned are being stored electronically in the form of digital data and are theoretically more secure. Research indicates the necessity to develop better conformance standards to support systematic processes for maintaining lessons learned repositories. Metaevaluation plays an important role in this process.
This dissertation sets forth to: (a) review the increasingly-large literature concerning the principles, procedures, practices, and processes used to evaluate lessons learned; (b) investigate records management within the field of information science to study factors related to the design and usage of lessons learned repositories; (c) identify good practices in project management as it pertains to deployment and administration of lessons learned; (d) understand how individuals and groups use organizational learning theories in a quest to manage evaluative knowledge; (e) compare and contrast pros and cons of common approaches to creating lessons learned repositories; (f) demonstrate how managing lessons learned can be improved through the process of metaevaluation; (g) discover practical suggestions to improving evaluative knowledge management through the implementation of a lessons learned system; and (h) explore resources such as checklists and templates which purport to help improve lessons learned.
Dissertation Title: Legislative Program Evaluation Conducted by State Legislatures in the United States
Abstract: This study examines how U.S. state legislative staffs conduct evaluations. The study addresses the ubiquity of state legislative program evaluation (LPE) units, the standards those units follow, the recommendations that LPE reports proffer, and the quality of the reports on several criteria. The study also addresses the feasibility of using metaevaluation to evaluate a large number of reports using solely the information contained in the reports. The study uses metaevaluation criteria developed by combining aspects of, primarily, the Generally Accepted Government Auditing Standards (GAGAS) for performance audits, the Joint Committee’s Program Evaluation Standards (PES), and, secondarily, Scriven’s Key Evaluation Checklist. In the process of developing the metaevaluation criteria the GAGAS and the PES are closely compared. The criteria were applied to a random sample of 100 of the 1,911 LPE reports published by state LPE units from 2001 through 2005. The study finds that state LPE units, and consequently the reports they produce, are overwhelmingly more connected to performance auditing and the GAGAS than to evaluation and evaluation standards. The metaevaluation criterion on which the LPE reports varied most was the comparisons criterion. Roughly a third of all LPE reports were were graded excellent or good, another third fair, and the final third poor—reflecting no mention of comparisons in the report. Evaluations were more likely to be graded excellent or good on this criterion than were performance audits. This study also seeks to test a methodological model—that of using metaevaluation to examine a large number of reports. The results from this attempt are mixed. Using metaevaluation in this way can determine the specific areas where evaluation reports are excelling or failing. However, accurately and fairly evaluating reports solely from the report itself presents some major problems. Among these problems are the inability to check both the accuracy of most data collected and the propriety of techniques used to collect data from human subjects. Nevertheless, we can formulate important conclusions including how well LPE reports use comparative studies when reaching their conclusions, how focused the reports are on goals and objectives, and how closely the reports follow established professional standards.
Dissertation Title: International aid evaluation: An analysis and policy proposals
Abstract: Evaluation has been intertwined with international aid work since its inception in the late 40's-early 50's, but it is still an area with considerable room for improvement. If, as is often alleged, evaluations of international development efforts are methodologically weak they are misleading international agencies about the real impact of the sizable amount of resources being spent. A recent study by Chianca, described in this thesis, with a sample of 50 US-based international non-profit organizations (INGOs) illustrates the serious situation of the structure and practice of evaluation in those agencies. A number of efforts to improve this situation have been put in place. Some of them have greater focus on methodological solutions and push for the development of more rigorous impact evaluations using experimental or quasi-experimental designs. Other efforts, while maintaining perspective on the importance of adopting more rigorous evaluation methods, have instead prioritized the establishment of principles and standards to guide and improve evaluation practice. Studies involving thorough analysis of the main efforts to improve international aid evaluation and of the most prominent evaluation standards proposed to the development field are scarce. This dissertation is a contribution to the field in several ways: (i) it provides a general synthesis of the current movements to improve aid evaluation; (ii) it describes and assesses some of the most prominent standards for aid evaluation; (iii) in particular, it presents a thorough assessment of the most widely adopted set of evaluation criteria worldwide, the five OECD/DAC evaluation criteria, with specific suggestions for improving them; (iv) it discusses results of a survey of INGOs on their evaluation principles and practice, and their feedback on the evaluation standards recently proposed by InterAction (the largest coalition of US-based INGOs); and (v) in the light of the preceding, it provides InterAction and other aid agencies with concrete suggestions to improve future revisions of their evaluation standards and guidelines.
Dissertation Title: Conceptual and Practical Analysis of Costs and Benefits: Developing a Cost-Analysis Tool for Practical Program Evaluation
Abstract: Cost-effectiveness, cost-feasibility and comparative superiority are important cost issues that need to be addressed in ex-ante, in medias res and ex-post evaluations. A review of the evaluation literature, however, indicates that cost analysis is not always done—and when it is done, it frequently is not done well. This partly may be a result of its complexity, coupled with the fact that many evaluators do not possess the requisite skills to conduct cost analysis. This research developed a new tool to simplify the procedures considerably for conducting serious cost analysis in program evaluation. In developing this tool, the conceptual and practical analysis of costs and benefits in evaluation were analyzed in detail. Scriven’s 1991 cost cube that was illustrated by E. Jane Davidson in 2003 was also redesigned to make it more applicable to the Cost Analysis Checklist tool. The modified model is referred to as the Costs Identification Model. A Benefits Identification Model was also developed to facilitate benefits identification. In developing the tool, special care was taken to ensure that the tool is user-friendly for non-economists. The targeted audience for this tool is novice evaluators and evaluation students.
Dissertation title: Evaluation of researchers and their research: Toward making the implicit explicit
Abstract: Due to its very nature, the evaluation of research permeates nearly every aspect of the work of researchers. They evaluate the work of others or have their own work evaluated. They evaluate hypotheses that come to mind, the previous literature, the quality of data, the explanatory power of theories, or the design of experiments or instruments. And this is not always casual evaluation. It is highly skilled evaluation, and becoming a first-rate or world-class researcher is a process of improving the quality of these evaluations. However, deciding when someone is or has become a first-rate or world-class researcher is an evaluation at a somewhat different level. It is a complex synthesis of judgments about how well the researcher does each of the constitutive types of evaluation, usually as evidenced in the work they are producing. In the last few decades the evaluation of research has become a high-stakes enterprise. With increasing political governance and federal budgets often in the billions, the livelihood of individual researchers, research groups, departments, programs, and entire institutions often swing in the balance. Simultaneously, it has been recognized that many of the longstanding principles and practices often lead to poor decisions about the actual or prospective merits of researchers and their research. The research in this dissertation describes, classifies, and comparatively evaluates the national models used to evaluate research and allocate research funding in sixteen countries. These models vary widely in terms of how research is evaluated and financed. However, nearly all share the common characteristic of relating funding to past performance. Each of these sixteen national models was rated on more than twenty-five quality indicators by independent, blinded panels of researchers and evaluators in two countries. The national models were then ranked in terms of their validity, credibility, utility, cost-effectiveness, and ethicality. The results of the rankings show that the clear leaders are nations using large-scale research assessment exercises of various hues. Bulk funding and indicator-driven models received substantially lower ratings. Implications for research evaluation practice and policy are considered and discussed.
To learn more about Dr. Coryn's dissertation, please visit his homepage.