Outcome and Impact Based Evaluation of Research Program Implementation: A Case of Indonesian Public Research Institute

Wati Hermawati; Ishelina Rosaira; Radot Manalu; Agus Santoso; Saut Siahaan

doi:10.14203/STIPM.2018.132

Outcome and Impact Based Evaluation of Research Program Implementation: A Case of Indonesian Public Research Institute

Wati Hermawati, Ishelina Rosaira, Radot Manalu, Agus Santoso, Saut Siahaan

Abstract

This paper relates to outcome and impact based evaluation (OIBE) of a research program implementation at an Indonesian public research institute (PRI) ‘A’. The major funding for PRIs in Indonesia comes from government. It is very essential therefore, for variuos parties including policy makers to be informed about meaningful and relevant evaluation of the outcome and impact of such PRIto the welfare of the people, to technology development and innovation, and to the policy improvements in significant ways. The mixed-method with good blend of qualitative and quantitative analysis is used in the case study of a competitive research program (CRP) of PRI ‘A’. The case study demonstrates that the outcomes and impacts of CRP are found in the form of economic, social, and environment improvements in the communities; managerial change and innovation of technology and process for SMEs as well as improved existing policies; and increased capabilities of researchers at PRI. Recommendations of this study is that PRI’A’ should improve their research management and policy, strengthen collaboration with firms, including more human resources mobility is required to facilitate research collaborations and co-production process, and to have clear guidelines and targets of outcomes and impacts for their research results.

Keywords: outcome, impact, Public Research Institute, evaluation, Indonesia

Full Text:

PDF

References

Befani.B. (2016). Causal frameworks for assessing the impact of development programmes.. UEA seminar, 9 March 2017. Retrieved on 27 December 2016 from https://www.uea.ac.uk/documents/425303/8654741/Befani+9th+March+2016+Seminar+FINAL.pdf/4b8957dc-78bb-4801-9a1c-cad956fbf4a9

Befani, B. (2016a). Choosing appropriate evaluation methods: A tool for assessment & selection. Bond, London. Retrieved on 5 January 2018 from https://www.bond.org.uk/resources/evaluation-methods-tool

Boaz, A., Fitzpatrick, S., & Shaw, B. (2008). Assessing the impact of research on policy: A review of the literature for a project on bridging research and policy through outcome evaluation. Final Report under Defra Research Contract WR0808. Retrieved on 15 January 2018 from: http://www.psi.org.uk/pdf/2008/bridgingproject_report.pdf

Borgonovi, E., Anessi-Pessina, E., & Bianchi, C. (2018). Outcome based performance management in the Public Sector. Springer International Publishing, AG. https://10.1007/978-3-319-57018-1

Bovaird, T. (2014). Attributing outcomes to social policy interventions—‘Gold standard’ or ‘fool’s gold’ in public policy and management? Social Policy and Administration, 48(1), 1–23. Crossref

Creswell, J.W. (2009). Research design: qualitative, quantitative, and mixed methods approaches. SAGE Publication, 3rd Ed.

Cohen, W.M., Nelson, R.R., and Walsh, J. (2002). Links and impacts: the influence of public research on industrial R&D. Management Science, 48 (2002), 1–23.

Dutrenit, G., De Fuentes, C., & Torres, A. (2010). Channels of interaction between public research organizations and industry and their benefits: Evidence from Mexico. Science and Public Policy, 37 (7), August: , 513-526.

Dodgson, M. and Bessant, J. (1996). Effective innovation policy: A new approach, London: International Thomson Business Press.

Douthwaitea, B., Kuby, T., van de Fliertc, E., Schulzd, S. (2003). Impact pathway evaluation: an approach for achieving and attributing impact in complex systems. Agricultural Systems 78 (2003) 243–265. Crossref

Gertler, P.J., Martinez, S., Premand, P., Rawlings, L.B., & Vermeersch,. (2011). Impact evaluation in practice. The World Bank. Washington

Guinea, J., Sela, E., Gomez-Nunez, A.J., Mangwende, T., Ambali, A., Ngum, H., Jaramillo, H., Gallego, J.M., Patino, A., Latorre, C., Srivanichakorn, S., & Thepthien, B. (2015). Impact oriented monitoring: A new methodology for monitoring and evaluation of international public health research projects. Research Evaluation, 24, 131-145. Crossref

Hearn, S. & Buffardi, A. (2016). What is impact? Methods Lab. London: Overseas Development Institute.

Intarakumnerd, P. & , Goto, A. (2016). Role of public research institutes in national innovation systems in industrialized countries: The cases of Fraunhofer, NIST, CSIRO, AIST, and ITRI. RIETI Discussion Paper Series 16-E-041. March. The Research Institute of Economy, Trade and Industry. Retrieved on 2 January 2018 from http://www.rieti.go.jp/en/

Lente, H,. Van-Hekkert, M., Smits, R. & Waveren, B. V. ( 2003). Roles of systemic intermediaries in transition processes. International Journal of Innovation Management, 7(3), 247-279.

Leeuw, F. & Vaessen, J. (2009). Impact evaluation and development. NONIE Guidance on impact evaluation. Washington. DC. Retrieved on 12 December 2016 from: www.worldbank.org/ieg/nonie/

Maass, G. (2003). Funding of public research and development: trends and changes. OECD Journal on Budgeting, 3 (4), 41-69. Retrieved on 5 January 2018 from https://www.oecd.org/gov/budgeting/43494478.pdf

Mackenzie, J. & Hearn, S. (2016). Impact evaluation for portfolio programmes on policy influence: Reflections on the Indonesian poverty reduction support facility. Methods Lab publication. London: Overseas Development Institute. Crossref

Martin, B. (2011). The research excellence framework and the ‘impact agenda’: are we creating a Frankenstein monster?. Research Evaluation, 20(3), September, 247-254. Crossref

Mazzoleni, R. & Nelson, R.R. (2007). Public Research Institutions and economic catch-up. Research Policy, 36(10), 1512-1528 Crossref

Mulyanto. (2014). Performance of Indonesia R&D Institutes: Influence of type of institutions and their funding source of R&D productivity. Technology in Society, 38: 148-160. Crossref

OECD-DAC. (2010). Glossary of Key Terms in Evaluation and Results Based Management. Paris: OECD-DAC. Retrieved on 2 January 2017 from: www.oecd.org/development/peer-reviews/2754804.pdf).

OECD. (2011). Public Research Institutions: Mapping Sector Trends. OECD Publishing. Retrieved on 15 January 2018 from https://www.oecd.org/sti/sci-tech/48795219.pdf

OECD. (2017). The role of public research institutions in the Finish innovation system (Chapter 4). In OECD Review of Innovation Policy: Finland 2017. OECD Publishing, pp. 89-126. Crossref

Piric, A. & Reeve, N. (1998). Evaluation of public investment in R&D: Towards a contingency analysis. OECD Conference on Policy Evaluation in Innovation and Technology: Towards Best Practices, 26–27 June 1997: 49–64. Paris: Organization for Economic Co-operationand Development (OECD). Retrieved on 14 January 2018 from: http://www.oecd.org/science/inno/1822593.pdf

Peersman, G., Rogers, P., Guijt, I., Hearn, S., Pasanen, T., & Buffardi, A. (2016). When and how to develop an impact-oriented monitoring and evaluation system. A Methods Lab publication. London: Overseas Development Institute.

Rogers, P. (2008). Using program theories to evaluate complicated and complex aspects of interventions. Evaluation, 14(1): 29-48.

Rogers, P. (2012). Introduction to impact evaluation. Impact Evaluation Notes No 1. Inter Action. Retrieved on 5 January 2018 from www.interaction.org/document/introduction-impact-evaluation

Rogers, P. (2014). Overview of impact evaluation, methodological briefs: impact evaluation 1, Florence: UNICEF Office of Research. Retrieved on 2 January 2017 from: http://devinfolive.info/impact_evaluation/ie/img/downloads/Overview_ENG.pdf

Shalock, R.L. (2002). Outcome-based evaluation, 2nd. Kluwer Academic Puslisher. N.Y.

Siahaan, S., Hermawati, W., Rosaira, I., Manalu, R., & Santoso, A. (2017). Analisis dampak Program Kompetitif LIPI, tahun 2010 -2014 [Impact analysis of LIPI’s Competitive Program 2010-2014]. Research Report. Pappiptek-LIPI. Jakarta.

Stern, E., Stame, N., Mayne, J., Forss, K., Davies, R., & Befani, B. (2012). Broadening the range of designs and methods for impact evaluations. Report of a study commissioned by the Department for International Development. DFID Working Paper 38. Retrieved on 20 December 2016 from: www.gov.uk/government/uploads/system/uploads/attachment_data/file/67427/design-method-impact-eval.pdf

Suzuki, J., Tsukada, N. & Goto, A. (2014). Innovation and Public Research Instiutes: Case of AIST, RIKEN, and JAXA. RIETI Discussion Paper Series 14-E-021.

DOI: http://dx.doi.org/10.14203/STIPM.2018.132