P23: Evaluating large-scale S&T initiatives: A case study on the complexity of capturing and disseminating meaningful outcome and impacts data 

Friday, November 22, 2013 - 1:30pm to 2:10pm
Genome Canada

Driven by a mission to connect ideas and people across public and private sectors to find new uses for genomics, Genome Canada is investing in large-scale science and technology (S&T) to fuel innovation and translate discoveries into applications to maximize impacts across all sectors. How do we know if we are successful in achieving this? And how can we learn from our efforts? This session uses Genome Canada as a case study to explore emerging issues on how large-scale S&T initiatives are tackling the complexity of capturing and disseminating meaningful outcomes and impacts. It will look at what innovative approaches can be used to tell the stories that speak of both the successes and the challenges with a perspective to drive learning. The session draws together three perspectives: Genome Canada’s institutional perspective, a practical perspective from Science Metrix, a S&T evaluation firm, and an academic/policy perspective from a scholar leading Valgen, a Genome Canada funded project that adds value with socio-economic studies. There is increasing impetus for S&T organisations to demonstrate success along the innovation continuum, from research discovery to end-user utilization and its consequent translational benefits. Such benefits include aspects such as commercialization, technology transfer and valorization which incorporate the value-added impacts to society beyond science. While the remit for evaluation is vast, the budgets tend to be low, methods are often traditional, and learning opportunities are often limited. The refrain is often ‘evaluation is important, but how to we actually do it in practice’? To meet the challenges of an ever changing environment, S&T evaluation needs to be responsive. The following themes will be elaborated upon and discussed with practical examples to better understand the complexity of the case and its relevance in a larger context. • Institutional challenges: Within a research funding environment, the practical collection and monitoring of performance measures often reduces analysis down to easy-to-capture output metrics that do not delve into the complexity and interactions at play in large scale transformative research. Data collection ends with final close of funding, leaving post-funding outcomes untracked. The use of a balanced approach to collecting performance data, over time, that is both quantitative and qualitative is needed to tackle this complexity and properly support decision-making. • Methodological challenges: Evaluations often resorts to safe methodologies such as surveys, interviews, case studies and conventional cost-effectiveness analysis because of ease and cost. This hampers a willingness to take risk with trialing new or emerging methodologies in an environment where innovation is needed to paint a more complex and contextually relevant picture of impacts that are longitudinal in nature. • Policy challenges: The translation of evaluative findings to inform real time decision-making is often the most neglected aspect of the evaluation cycle. Causal pathways between decision drivers and outcomes are often not transparent or attributable; policy and program goals are unclear; and policy intent is often inferred with post-analysis of outcomes rather than with ex ante analysis of consequences. The translation of findings to drive policy issues is paramount.


Director, Evaluation

Ms. Michelle Picard-Aitken has recently been appointed Director, Evaluation at Science-Metrix, where she has played an increasingly important role in developing the firm’s evaluation capacity and expertise since 2008. Ms. Picard-Aitken has participated in over 20 medium- to large-scale evaluation projects for federal departments and agencies, as well as other Canadian science-based organisations. Her expertise lies in the design and implementation of evaluations that examine research funding programs (e.g., 10th-Year Evaluation of the Canada Research Chairs Program), S&T development programs, and initiatives that use science-based evidence to inform policy or other forms of decision-making. She has also assisted in several bibliometric assessments and performance studies, including for the European Commission and other international clients.

Ms. Picard-Aitken obtained her B.Sc. (Honours) from Trent University, Canada (joint major in Environmental & Resource Science and English), followed by an M.Sc. from the INRS-Institut Armand-Frappier in Experimental Health Sciences, Canada. Most recently, she completed an M.Sc. (with distinction) from Imperial College London, UK, in Creative Non-Fiction Writing. Ms. Picard-Aitken is a member of the American Evaluation Society, and Canadian Evaluation Society, and the Société québécoise d’évaluation de programme. She has presented the results of her work each year at the Canadian Evaluation Society’s national conference since 2009.

Johnson-Shoyama Graduate School of Public Policy, University of Saskatchewan

Dr. Peter W.B. Phillips, Professor, Johnson-Shoyama Graduate School of Public Policy, University of Saskatchewan, earned his Ph.D. in International Political Economy at the LSE and practiced for 13 years as a professional economist in Canadian industry and government. At USask he has held the Van Vliet and NSERC-SSHRC MOT Chairs and was director of the vCollege of Biotechnology. He has had appointments at the University of Edinburgh, LSE, OECD, European University Institute and University of Western Australia, is editor of AgBioForum (on-line journal), was on the NAFTA Chapter 13 expert panel on GM maize in Mexico and was member of the Canadian Biotechnology Advisory Committee. He is vice-chairman of Ag West Bio Inc., which operates a venture capital fund. His research focuses on innovation policy, including regulation, intellectual property, industrial systems and trade. He is currently co-lead/PI of VALGEN ($5.4M Genome Canada project, 2009-14) and a SSHRC $725K In-Sight Grant on Open Innovation. He is author/editor of 15 books, >40 journal articles and >50 book chapters. His work focuses on bioscience decision-making and decision-systems. He is PI of the CFI-funded Experimental Decision Laboratory that examines cognitive limits of decision-making in the context of profound uncertainty and complexity.

Genome Canada

Dr. Evans is responsible for the development, implementation and promotion of Genome Canada’s performance management and evaluation function. This role includes the implementation of effective policies, guidelines and standards for the on-going analysis and evaluation of the performance of Genome Canada’s programs and activities and reporting on Genome Canada’s progress in meeting its mandate and strategic objectives. Dr Evans has eleven years of experience in a variety of applied research and evaluation projects. She has gained a strong research background with expertise in program, process and organizational research utilizing mixed methods. Dr Evans has participated in a number of evaluations, organizational and performance reviews, and related assignments in diverse fields such as, audit, bankruptcy, consultation, charities and regulation, science and technology, funding agencies, environment, forestry, fisheries, agriculture; energy. Dr. Evans is a member of the Canadian Evaluation Society and a Director on the Board of the NCC chapter of the CES. She has a BA (Anthropology and Business Management), and a PhD (Organisational Behaviour).