Center for Civic Partnerships

Quantitative and Qualitative Evaluation Methods

Top Tips

Evaluation methods and the data they produce are grouped into two basic categories – quantitative and qualitative. In general, quantitative methods produce “hard numbers” while qualitative methods capture more descriptive data. The method(s) you choose are determined by the purpose(s) of your evaluation and the resources you have to design and conduct it. In practice, most researchers and evaluators agree that combining quantitative and qualitative techniques (sometimes called “mixed method” evaluations) produces a richer and more comprehensive understanding of a project’s accomplishments and learnings.

How are Quantitative and Qualitative Data Different?

At the most basic level, data are considered quantitative if they are numbers and qualitative if they are words. Qualitative data may also include photos, videos, audio recordings and other non-text data. Those who favor quantitative data claim that their data are hard, rigorous, credible and scientific. Those in the qualitative camp counter that their data are sensitive, detailed, nuanced and contextual. Quantitative data best explain the why and how of your program, while qualitative data best explain the what, who and when.

Different techniques are used to collect and analyze quantitative and qualitative data:

Quantitative Techniques	Qualitative Techniques
Surveys/Questionnaires	Observations
Pre/post Tests	Interviews
Existing Databases	Focus Groups
Statistical Analysis	Non-statistical (methods vary)

In general, evaluators agree that qualitative and quantitative data and methods have different strengths, weaknesses, and requirements that affect decisions about which methodologies are appropriate for which purposes.

Quantitative Data and Evaluation Methods

What are Quantitative Data?

Pieces of information that can be counted mathematically

Usually gathered by surveys from large numbers of respondents selected randomly

Secondary data such as census data, government statistics, etc. often included in quantitative evaluations

Analyzed using statistical methods

Best used to answer what, when and who questions

Not well suited to how and why questions

Strengths	Limitations
Findings can be generalized, if selection process well-designed and sample is representative of study population	Related secondary data sometimes not available, or accessing available data is difficult/impossible
Relatively easy to analyze	Difficult to understand context of program activities
Data can be very consistent, precise, reliable	Data may not be robust enough to explain complex issues
Data collection is usually cost efficient

How do you collect Quantitative Data?
Surveys

Most common method
Self-administered or by someone else
Face-to-face, telephone, mail, web-based

Secondary Data

Often used in conjunction with survey data

Includes census data, knowledge/attitude/behavior (KABB) studies, criminal justice statistics, performance data, non-confidential client information, agency progress reports, etc.

Qualitative Data and Evaluation Methods

What are Qualitative Data?

Usually gathered by observations, interviews or focus groups

May also be gathered from written documents and through case studies

Less emphasis on counting numbers of people who think or behave in certain ways and more
emphasis on explaining why people think and behave in certain ways

Involves smaller numbers of respondents

Utilizes open-ended questionnaires or protocols

Best used to answer how and why questions

Not well suited to what, when and who questions

Strengths	Limitations
Complement and refine quantitative data	Findings usually can not be generalized to the study population or community
Provide more detailed information to explain complex issues	More difficult to analyze; don’t fit neatly in standard categories
Multiple methods for gathering data on sensitive subjects	Data collection is usually time consuming and costly

How do you collect Qualitative Data?

Observations

Looking at what is happening rather than directly questioning participants
Used to better understand behaviors, their social context and meanings attached to them
Useful for certain populations - children, infants
Can identify unanticipated outcomes

Interviews (in-depth, individual)

Usually provide rich data, details, insights from program participants and stakeholders about their experiences, behaviors and opinions
Particularly useful for complex or sensitive subjects
Use open-ended questions

Focus Groups

8-12 people selected by non-random method, share some characteristics or experience relevant to the evaluation, ideally do not know each other, respond to questions from group facilitator
Use group dynamics to generate data and insights
Useful for generating ideas and strategies, defining problems in project implementation, assist with interpreting quantitative findings
Open-ended questions or topics designed to stimulate discussion; topics usually broader than interview questions

These "Top Tips” were adapted primarily from: Evaluation for Learning: Basic Concepts and Practical Tools, LaFrance Associates, and User-Friendly Handbook for Mixed Method Evaluations, Division of Research, Evaluation and Communication of the National Science Foundation.

On-Line Tools, How-To Guides & Practical Resources

Research Methods Knowledge Base – Cornell University

www.socialresearchmethods.net/kb/index.php

This is a very comprehensive web-based textbook that addresses a wide variety of topics in social research methods. It includes information on qualitative and quantitative approaches, surveys, interviews, sampling, qualitative data and methods, analysis and many others. Use the detailed table of contents or search for specific topics.

User-Friendly Handbook for Mixed Method Evaluations – National Science Foundation
www.nsf.gov/publications/pub_summ.jsp?ods_key=nsf97153
The handbook includes information on how to conduct a mixed method evaluation that combines quantitative and qualitative techniques. It provides detailed information on specific data collection methods and analysis for each type of measure. It was produced by the Division of Research, Evaluation and Communication of the National Science Foundation.

Evaluation Resource Center - Innovation Network
www.innonet.org/pointk

The Evaluation Resource Center (part of the Innovation Network’s Point K Learning Center), provides links to multiple web-based resources for a wide variety of quantitative and qualitative evaluation methods. It has specific categories for surveys, focus groups and interviews. Free registration is required.

Planning and Evaluation Service – US Department of Education

www.ed.gov/offices/OUS/PES/eval_faq.html

This site has archived FAQ’s addressing focus groups, survey methodology (telephone, mail, fax) and questionnaire design.

Basics of Conducting Focus Groups - Free Management Library

www.managementhelp.org/evaluatn/focusgrp.htm

Brief “how-to” guide with specific instructions for conducting focus groups, including preparation, planning, question development, facilitation and follow-up.

Conducting Focus Group Interviews - USAID Center for Development Information and Evaluation

www.usaid.gov/pubs/usaid_eval/pdf_docs/pnaby233.pdf

Provides on overview of focus groups – what they are, advantages and limitations, when to use them. Also provides step-by-step advice for conducting high quality focus groups.

Learn More about Quantitative & Qualitative Evaluation Methods

Analyzing Quantitative Data - University of Wisconsin Extension

http://learningstore.uwex.edu/pdf/G3658-6.pdf
Provides an overview of how to use common mathematical techniques to make quantitative evaluation data more understandable for non-statisticians.

The Use of Qualitative Interviews in Evaluation – University of Arizona

http://ag.arizona.edu/fcs/cyfernet/cyfar/Intervu5.htm

Guide on how and when to use qualitative interviews. Includes information on different types of interviews and discusses the steps and issues in designing and developing qualitative interview studies.

Harnessing Technology for Evaluation - The Evaluation Exchange, Volume X, No. 3, Fall 2004
www.hfrp.org/evaluation/the-evaluation-exchange/issue-archive

This issue of The Evaluation Exchange explores the contribution of technology to evaluation practice in four key areas -- data collection and analysis, collaboration, knowledge mobilization, and evaluation capacity building. Several articles focus on web-based surveys and the role of technology in evaluating youth programs. The Evaluation Exchange is a free on-line evaluation periodical published by the Harvard Family Research Project.

Sage Publications

www.sagepub.com/home.nav

Has a wide variety of books available for purchase covering all aspects of quantitative and qualitative evaluation – focus groups, surveys, quantitative and qualitative data and methodology.

Additional Resources Available from The Center for Civic Partnerships

www.civicpartnerships.org; 916.646.8680

Organizational Learning and Evaluation Webpage

¨Conferences ¨Training ¨Presentations

Copies of presentations and handouts from prior conferences are available

Tip Sheet Series on Organizational Learning & Evaluation

¨Evaluation ¨Organizational Learning ¨Logic Models

¨Theory of Change ¨Quantitative & Qualitative Methods

Tip Sheets covering a wide range of community building and improving livability topics

are also available on the Center for Civic Partnerships’ website

www.civicpartnerships.org/toolsRes