Design Evaluation Questions Participatory Available Data Observation

Presentation on theme: "Design Evaluation Questions Participatory Available Data Observation"— Presentation transcript:

1 Chapter 8: Selecting and Constructing Data Collection Instruments 模块8: 选择和构建数据收集工具
Design Evaluation Questions Participatory Available Data Observation Survey Questionnaire Focus Groups Diaries, Journals, Self- reported Checklists Expert Judgment Delphi Technique Intervention or Policy Collect Approaches

2 Introduction 引言 Data Collection Strategies 数据收集策略
Data Collection General Rules 数据收集的一般规则 Key Issues about Measures 有关测量的关键问题 Quantitative and Qualitative Data 定量和定性数据 Introduction to The Toolkit 数据收集工具介绍 Techniques for Using Surveys 使用调查的技巧 Techniques for Using Focus Groups使用专题组的技巧

3 No One Best Way 不存在一个最好的方法
Decision Depends On: 方法选择取决于： What you need to know 你需要知道什么 numbers or stories 数字或故事 Where the data reside 数据在什么地方 environment, files, people 环境、文件、人 Resources and time available 可用的资源和时间 Complexity of the data to be collected 所要收集的数据的复杂性 Frequency of data collection 数据收集的频繁度

4 Structured Approach 结构化方法
All data collected in the same way 用相同的方法收集所有数据 Important for multi-site and cluster evaluations so you can compare 对多地点和分类评价很重要，这样你能够进行比较 Important when you need to make comparisons with alternate interventions 当你需要在备选的干预措施之间进行比较时，结构化方法很重要

5 When to Use Structured Approach: 什么时候要使用结构化的方法：
You wish to be very precise 你希望非常精确 You are working with a large sample or populations 你的研究样本或群体很大 You are sure of what you wish to measure 你确定了你希望衡量的事物 You want to show your results analytically 你想要结果具有分析性 You need to make comparisons across different sites or interventions 你需要在不同地点或干预措施之间进行对比

6 Semi-structured Approach 半结构化方法
Systematic and follow general procedures but data are not collected in exactly the same way every time 具有系统性，而且遵照通用的程序，但数据的收集并非每次都是用完全相同的方法 More open and fluid 更具开放性和灵活性 Does not follow a rigid script 没有按照严格的文本 may ask for more detail 可能问到更多的细节 people can tell what they want in their own way 人们可以用他们自己的方式来讲述自己所要讲的事情

7 Use Semi-structured Approach when Evaluator: 评估者在下列情况时使用半结构化的方法：
Is conducting exploratory work in a new development area 在新的发展领域内进行探索性的工作 Is seeking understanding, themes, and/or issues 在寻求理解、主题和/或问题 Want anecdotes, stories, or in-dept information 想要知道轶事、故事或深入的信息 Is not sure of what you wish to measure 不清楚你希望测量的是什么 Has no need to qualify 不需要进行量化

8 Data Collection General Rules 数据收集的一般规则
Use available data if they already exist 如果有现成信息，则利用这些信息 If using available data, be sure to find out how they are: 如果利用现有信息，确保知道他们如何： collect the data 收集信息 define the variables 界定变量 ensure accuracy of the data 确保数据的精确性

9 Data Collection General Rules (cont.) 数据收集一般规则 (续)
If you must collect original data: 如果你必须收集原始数据： establish procedures and follow them 制定数据收集程序并遵照执行 maintain accurate records of definitions and coding 保持定义和编码的精确记录 pre-test, pre-test, pre-test 反复进行预先测试 verify accuracy of coding, data input 核查编码和数据输入的准确性

10 Key Issues about Measures 有关方法的关键问题
Are the measures credible? 这些方法可信吗？ Are the measures valid? 这些方法有效吗？ Are the measures relevant? 这些方法是否相关？ Are the measures reliable? 这些方法可靠吗？ Are the measures precise? 这些方法精确吗？

11 Credibility 可信性 How trustworthy or believable your data collection is
你的数据收集的可信性如何 Are the data you are collecting giving you information about the actual situation? 你正在收集的数据是否能够反映真实情况？ Make sure data you are collecting are relevant and most important information 确保你正在收集的数据是相关的并且是最重要的信息

12 Validity 有效性 Does the measurement actually measure what it is supposed to? 所进行的测量是否是真正想要测量的？ Two kinds of validity 两种有效性 face validity contents of the test or procedure look like they are measuring what they are supposed to measure 表面有效性 测试或程序的内容看起来像是它们正在测量要求它们测量的一样 content validity content of the test or procedure represents all that is required for validity 内容有效性 测试或者程序的内容能够代表所有有效性所要求的

13 Relevancy 相关性 Does the measurement actually measure what they counts?
测试方法是否真的测试它们所述的东西？ The most important information 最重要的信息 Do not just measure what is needed or just trying to measure everything 不要只测试需要的信息或者试图测试所有的信息 Design matrix is a tool to guide relevance 设计矩阵是确保相关性的途径

14 Reliability 可靠性 Term to describe the stability of your measurement
是用于描述测量稳定性的术语 Measures the same thing, same way in repeated tests 在反复测验中使用相同的方法测量相同的事物 Examples: 示例： in sports, speed measured by stopwatch 在体育运动中，用秒表测量速度 birth weights of newborn infants 新生婴儿出生时的体重 attendance rates at schools 学校的出勤率

15 Precision 精确性 How the language used in the data collection matches the measure 数据收集使用的语言与测量的匹配度如何 Example: 示例： if the question is about countries, the measures must be at the national level 如果问题是关于全国的情况，那么就必须在国家层面上进行测量 if the question is about people, the measures must be on the individual level 如果问题是关于人，那么就必须在个人层面上进行测量

16 Quantitative Approach 定量方法
More structured 结构性更强 Attempts to provide precise measures 试图提供精确的测量结果 Reliable 可靠 Harder to develop 方法的开发比较难 Easier to analyze 分析起来比较容易

17 Qualitative Approach 定性方法
Less structured 结构性较差 Easier to develop 方法的开发比较容易 Can provide “rich data” — detailed and widely applicable 能够提供详细的广泛适用的“丰富的数据” Is challenging to analyze 分析起来具有挑战性 Is labor intensive to collect 收集数据要做很多工作 Usually generates longer reports 通常会产生比较长的报告

18 Which Data? 使用什么样的数据？ 定量数据 定性数据 If you: Then Use:
如果你： Then Use: 那么就使用： want to do statistical analysis 想要做一些统计分析 定量数据 - know exactly what you want to measure 准确地了解你所想要测量的是什么 want to cover a large group 想要涵盖一个大的群体 want anecdotes or in-depth information 想要了解没有记载的或深入的信息 定性数据 are not sure what you want to measure 还不确定你所要测量的是什么 do not need to quantify 不需要量化

19 Obtrusive vs. Unobtrusive Methods 介入式方法 vs. 非介入式方法
Obtrusive – observations are made of behavior with the participant’s knowledge 介入 – 观察是在参与者知情下的行为做出的 interviews, surveys, focus groups 面访、调查、专题组访谈 Unobtrusive – observations done without the knowledge of the participant 非介入 –在参与者不知情的情况下进行的观察 historical/document/archival data 历史数据/文件数据/档案数据 watching participants at a distance 远距离观察参与者

20 How to decide on data collection approach 如何决定数据收集的方法
Choice depends on the situation 方法选择视情况而定 Each technique is more appropriate in some situations than others 每项技术在某种情况下会比其它技术更合适 Caution: All techniques are subject to bias 注意：所有的技术都会有偏差

21 Combinations 各种方法的结合 Can use a variety of data collection approaches to answer different questions or for multiple sources 可以使用多种数据收集方法以回答不同的问题或针对多种数据来源 Triangulation: collect same information using different approaches 三元法：使用不同方法来收集相同的信息

22 Measurement Considerations 有关测量的几点考虑
If people know they are being studied, they may act differently 如果人们知道他们在被研究，他们的行为可能会不同 women may respond differently to a male interviewer or if their spouse is present 如果女士的配偶在场或者访谈者是男性的话，她们的态度会不同 Starting by specifying what you want to measure the choices become clearer about answering questions 如果一开始就明确你的测量需求，就能更清晰地选择问题 22

23 Data Collection 数据收集 Participatory Data Collection 参与式数据收集
Available Data 现有数据 Observation 观察 Surveys 调查 Focus Groups 专题组 Diaries, Journals, Self-reported Checklists 日记、日志、自报告核查清单 Expert Judgment 专家评判 Delphi Technique 德尔菲技术 Citizen Report Cards 公民报告卡

24 Toolkit 1: Participatory Data Collection 工具1：参与式数据收集
Data that are collected when interacting with people 在与人们的互动中收集数据 Examples: 示例： community meetings 社区会议 social mapping 社会映射 transect walks 横断面走访法

25 Community Meetings 社区会议
One of the most common participatory methods 最为普遍的一种参与方式 Must be well organized 必须组织好 agree on purpose 同意目的 establish ground rules 建立基本规则 who will speak 谁能说话 time allotted for speakers 每个发言人的时间分配 format for questions and answers 决定问题以及答案 25

26 Mapping 映射 Collect and plot information on the distribution, access, and use of resources within a community 收集并划分有关社区内资源的分布、获取和使用的信息 Useful tool to involve stakeholders 使利益相关者参与进来的有用工具 provides a way to work together 提供一种在一起工作的方法 increases understanding of the community 增加对社区的了解 generates discussions, verifies secondary sources of information, perceived changes 引起讨论，核实辅助来源的信息, 感知变化

27 Mapping Process 映射过程 Draws a picture of the community 描绘社区的整体情况
Places resources and assets in their locations 将资源和资产放在各自的安置地点 individual assets: skills, talent, networks, money, etc. 个人资产：技能、才干、网络、资金等 civic assets: faith associations, clubs, social groups, etc. 公民资产：宗教团体、俱乐部、社会团体等 institutional assets: businesses, schools, health services, public transportation, etc. 机构资产：企业、学校、健康服务机构、公共交通等 environmental assets: parks, roads, farmland, housing, clean air and water, etc. 环境资产：公园、公路、农田、住房、清洁的空气和水等

28 Transect Walks 横断面走访法 Evaluators walk around community observing people, surroundings, and resources 评价者在社区走动，观察人，周围以及资源 Need good observation skills 需要良好的观察技巧 Walk a transect line through a map of a community — line should go through all zones of the community 走在社区中的一条横截线上。这条线能穿过社区中的所有地方 28

29 Toolkit 2: Available Data 工具2：现有数据
Example sources: 数据来源举例： files/records 档案/记录 computer data bases 计算机数据库 government reports 政府报告 other reports or prior evaluations 其它报告或以前的评价 census data 普查数据 documents (budgets, organizational charts, policies and procedures, maps) 文件（预算、组织机构图、政策和程序、地图）

30 Using Existing Records 使用现有的记录
Most organizations have already collected the data from clients, community, and internal information systems 一些机构可能已经收集了你需要的数据 May also have summaries and/or reports, such as: 可能已经有了一些总结和/或报告，例如： internal management reports 内部管理报告 budget documents 预算文件 reports to the public or funding agencies 向公众或出资机构提交的报告 Key issues: 关键问题： validity, reliability, accuracy 有效性、可靠性和准确性

31 Advantage/Challenge: Available Data 现有数据的优点/挑战
Advantages 优点 Often less expensive and faster than collecting the original data yourself 与自己收集原始数据相比成本较低速度较快 Challenges 挑战 There may be coding errors or other problems. Data may not be exactly what is needed. You may have difficulty getting access. You have to verify validity and reliability of data可能存在编码错误或其它问题。数据可能不是需要的确切数据。你在获取数据方面可能有困难。你必须核实数据的有效性和可靠性。

32 Toolkit 3: Observation 工具3：观察
See what is happening 观察正在发生什么 traffic patterns 交通模式 land use patterns 土地使用模式 layout of city and rural areas 城市和农村地区规划 quality of housing 住宅的质量 condition of roads 公路状况 conditions of buildings 建筑状况 who goes to a health clinic 谁到卫生诊所就诊

33 Ways to Observe 观察方法 Unobtrusive非介入式 No one knows you are observing
没有人知道你正在进行观察 Participant 参与式 You actually participate in the activity 你实际上参与到活动当中去了 Obtrusive 介入式 The people being observed know you are there to observe them 正在被观察的人知道你在观察他们

34 Guidelines for Planning Observations 规划观察的指南
Develop a check list to rate your observations. 设计一个核查清单来给你的观察评级 Develop a rating scheme. 设计一个评级方案 Have more than one observer, if feasible. 如果可行的话，要有多个观察员 Train observers so they observe the same things. 对观察员进行培训，这样他们能够观察相同的事物 Pilot test the observation data collection instrument. 通过试点检验观察数据收集工具 For less formal approach, have a few key questions in mind when you arrive 如果采用不那么正式的方法，当你到达时，脑子里要准备一些问题

35 Advantages and Challenges: Observation 观察的优点和挑战
Collects data on actual vs. self- reported behavior or perceptions. It is real-time vs. retrospective 收集有关实际行为或理解的数据与自报告数据作比较，实时数据与回顾数据的对比 Challenges 挑战 Observer bias, potentially unreliable; interpretation and coding challenges; sampling can be a problem; can be labor intensive；low responses rate 观者者的偏差，可能不可靠；解释和编码方面的挑战；抽样可能是个问题；要花费大量的劳力；反应率低

36 Toolkit 4: Surveys 工具4：调查
Excellent for asking people about: perceptions, opinions, ideas 是一种用来询问人们的对事物的理解、观点、想法的很好的工具 Less accurate for measuring behavior 测量行为的精确度较差 Sample should be representative of the whole 样本应该对总体具有代表性 Big problem with response rates 回复率是个问题

37 Methods for Surveys 调查方法
In-person interviews 亲自访谈 Mail /phone/Internet interviews or surveys 邮件/电话/网络访谈或调查 Self-administered questionnaires 自填式问卷

38 Structures for Surveys 调查的结构性
Structured: 结构化的： Precisely worded with a range of pre-determined responses that the respondent can select 用一系列预先设计好的受访者能够选择的回答选项来精确地设计调查 Everyone is asked exactly the same questions in exactly the same way, given exactly the same choices 用严格相同的方法向每个人询问严格相同的问题，并给出严格相同的答案选项 Semi-structured 半结构化的 Asks the same general set of questions but may leave many, it not all, of the answers open-ended 询问一组相同的问题，但可能包含许多（不是全部）开放式问题

39 Structured vs. Semi-structured Surveys 结构化调查与半结构化调查之比较
harder to develop 设计较难 easier to complete 完成较容易 easier to analyze 分析较容易 more efficient when working with large numbers 当在处理大量调查对象时效率更高 Semi-structured半结构化 a little easier to develop 设计较为容易 labor intensive to conduct 实施起来耗费较多劳力 harder to analyze but provide a rich source of data 分析较为困难，但提供了丰富的数据 subject to bias in interpreting 在解释时会有偏差/受偏见的影响 burdensome for people to complete as a self-administrated questionnaire 给完成自填式问卷的人造成负担

40 Advantages and Challenges of Surveys 调查的优点与挑战
Best when you want to know what people think, believe, or perceive, only they can tell you that 当你想要知道人们在想什么、相信什么、感知什么的时候，这是最好的方法，只有这些方法能够告诉你 Challenges 挑战 People may not accurately recall their behavior or may be reluctant to reveal their behavior if it is illegal or stigmatized. What people think they do or say they do is not the same as measuring what they actually do. 人们可能无法准确地回忆起他们的行为，或者如果行为是违法的或受污蔑的，他们可能不愿意暴露这些行为。人们认为他们做了什么或说他们做了什么与测量他们实际上做了什么并不一样

41 In-person Interviews 亲自访谈
Useful for in-depth understanding of experiences, opinions, or descriptions 对于深入了解过程、观点或描述有用 Useful when other approaches do not work 当其它方法不起作用的时候有用 Should be conversational 应该是会话式的 Can be done individually or in groups 可以进行个人访谈或者小组访谈 Can be structured or semi-structured 可以是结构化的或半结构化的

42 Mail / Phone / Internet Interviews and Surveys 邮件/电话/网络访谈和调查
Literacy issues 读写能力问题 Consider accessibility 考虑可达性 reliability of postal service 邮寄服务的可靠性 turn-around time 回收时间 do respondents have telephone access? 受访者是否能够使用电话？ do they have Internet access? 受访者是否能使用互联网？

43 Self-administered Questionnaires 自填式问卷
Written surveys that the respondent completes 由受访者完成的书面调查 Can be structured, semi-structured, or a combination 可以是结构化的、半结构化的、或两者结合 Should be short (no more than 20 min.) 应该简短（不要超过20分钟） Usually include one or two open-ended questions 通常包括一个或两个开放式问题 help make respondents more comfortable 有助于使受访者觉得更舒适

44 Advantages of Interviews 访谈的优点
Can be structured, unstructured, or a combination 可以是结构化的、半结构化的、或两者结合 Can explore complex issues in depth 能够深入地探讨复杂的问题 Forgiving of mistakes: unclear questions can be clarified during the interview and changed for subsequent interviews 容许错误：不清楚的问题能够在访谈中得到澄清，并且在随后的访谈中进行改变 Can provide evaluators with an intuitive sense of the situation 能够向评价 人员提供对情况的直觉了解

45 Challenges of Interviews 访谈的挑战
Can be expensive, labor intensive, and time consuming 可能需要耗费较多的成本、劳力和时间 May not be able to explore why people have different viewpoints 可能无法探究为什么人们有不同的观点 Selective hearing on the part of the interviewer may miss information that does not conform to pre-existing beliefs 访谈者有选择的倾听有可能会漏掉与以前相信的东西不一致的信息 Cultural sensitivity: gender issues 文化敏感性：性别问题

46 Toolkit 5: Focus Groups 工具5：专题组
Type of qualitative research where small homogenous groups of people are brought together to informally discuss specific topics under the guidance of a moderator 一种定性研究方法，将一小群具有共同特征的人集中起来，在主持人的指导下对具体问题进行非正式地讨论 Purpose: to elicit reliable data, not just interesting information 目的：得出可靠的数据而不仅仅是有趣的信息

47 Uses of Focus Groups 专题组的使用
Help develop a survey questionnaire 帮助设计调查问卷 Clarify sample selection 阐明样本的选取 Contextualize survey data 把调查数据放在背景当中 Be used in tandem with surveys 与调查一起使用 Be used as a separate data collection tool 作为一种单独的数据收集工具来使用

48 Focus Group Process 专题组过程
Step Process 步骤 程序 1 Introduce the focus group meeting 介绍专题组会议 2 Have the participants introduce themselves 让参与者进行自我介绍 3 Present the first question, it should be easy, an ice-breaker 提出第一个问题，问题应该简单，起到“破冰者”的作用 4 Ask the main questions 提问主要问题 5 Ask the last (summary) questions 提问最后一个（总结性）问题 6 Ask if there are other comments or questions 问一下是否还有其它的说明和问题 7 Write-up impressions, major issues and points of discussion 整理对讨论的印象、讨论的主要议题和内容

49 Advantages and Challenges of Focus Groups 专题组的优点和挑战
Relatively quick and easy, may take less staff time than in-depth, in-person interviews; provides flexibility to make changes in process and question; ability to explore different perspectives; it can be fun 相对快速和容易，可能比深入的亲自访谈要节省工作人员的时间；在过程和问题中可以进行灵活的改变；能够探讨不同的看法；可能较为有趣 Challenges 挑战 Analysis is time consuming, participants might be different from rest of population; risk of bias in interpreting data; risk of group being influenced by moderator or dominant members 分析耗时，参与者可能与总体中的其他人不同；在解释说明数据时存在偏见的风险；存在小组会受到主持人或权威成员影响的风险

50 Toolkit 6: Diaries, Journals, Self-Reported Checklists 工具6：日记、日志、自报告核查清单
Use when you want to capture information about events in people’s daily lives 当你想要获取有关人们日常生活中发生的事件的信息时使用 Participants capture experiences in real-time not later in a questionnaire 参与者获得了实时经验，而不是之后从问卷中获得的经验 Used to supplement other data collection 作为其它数据收集的补充

51 Guidelines for Diaries or Journals 日记或日志方法使用指南
Step Process 步骤 过程 1 Recruit people face-to-face 面对面地招募参与者 • encourage participation, appeal to altruism, assure confidentiality, provide incentive 鼓励参与，引起对利他主义的兴趣，确保机密性，提出激励措施 2 Provide a booklet to each participant 向每人提供一本小册子 • cover page with clear instructions, definitions, example 封面要写明清晰的指示说明、定义和示例 • short memory-joggers, calendar 简短的备忘录, 日历 3 Consider the time-period for collecting data 考虑收集数据的时间周期 • if too long, may become burdensome 如果时间太长，可能成为负担 • if too short may miss the behavior or event 如果时间太短，可能漏掉一些行为或事件

52 Self-reported Checklists 自报告核查清单
Cross between a questionnaire and a diary 调查问卷和日记的交叉 The evaluator specifies a list of behaviors or events and asks the respondents to complete the checklist 评价 人员具体给出了一个行为或时间清单，并请受访者完成该清单 Done over a period of time to capture the event or behavior 在一个时期内进行以便获取事件或行为的信息 More quantitative approach than diary 是比日记更具有量化性的方法

53 Advantages and Challenges of Diaries, Journals … 日记、日志的优点和挑战
Rich data can capture the details that might be quickly forgotten over time 丰富的数据能够获知细节信息，而这些细节可能很快就会被忘掉 Good for information on how people use time 对于获得有关人们是如何使用时间的信息很有用 Helps in collecting sensitive information 有助于收集敏感信息 Supplements interviews是对访谈的一种补充 Challenges 挑战 Requires literacy 要求有读写能力 May change behavior 可能会改变行为 Data may be incomplete or inaccurate 数据可能不完整或不精确 Poor handwriting, difficult to understand phrases 字迹潦草，理解困难

54 Toolkit 7: Expert Judgment 工具7：专家评判
Interviews with experts, one-on-one or in a panel 与专家进行访谈，一对一或对一个专家组 Can be structured or unstructured 可以是结构化的或非结构化的 everyone answers a set of specific questions每个人都回答一组特定的问题 or free flowing (focus group model) 或者自由发言（专题组模式） or formal presentations 或者正式报告

55 Selecting Experts 选择专家
Establish criteria for selecting experts based on: 基于以下考虑因素制定选择专家的准则： recognized expert 公认的专家 areas of expertise 专业知识领域 diverse perspectives 多样化的观点 diverse political views 多样化的政治观 diverse technical expertise 多样化的技术专业知识

56 Advantages and Challenges of Expert Judgment 专家评判的优点和挑战
Fast, relatively inexpensive 快速、成本相对较低 Communication of final results can give credibility 交流最终结果能够表明可信性 Challenges 挑战 Weak for impact evaluation 对于影响评价 而言显得不足 May be based mostly on perceptions 可能大多以感知为基础 Worth of data is only as good as the perceived credibility of the experts 数据的价值仅仅与专家感觉的可信性一样好

57 Toolkit 8: Delphi Technique 工具8：德尔菲技术
Enables experts who live in different locations to engage in a dialogue 使在不同地方的专家参与到对话中来 Experts asked specific questions 专家被提问特定的问题 Answers are returned to a central source for the evaluator to summarize and feed it back to the experts for further comments 答案被回收集中起来，以便评价 人员进行总结并反馈给专家以获取进一步的意见 No one knows what other people have said, thus conflict avoided 没有人知道谁说了些什么，因此避免了冲突 Experts can agree or argue with others’ comments 专家可以同意其他人的意见或进行争辩

58 Advantages and Challenges of Delphi Technique 德尔菲技术的优点和挑战
Inexpensive 便宜 Conducive to independent thinking 促进独立思考 Allows sharing of information 信息共享 Challenges 挑战 Judgments of a selected group only 只有来自挑选过的团队的意见 Tendency to eliminate extreme positions 消灭极端位置的趋势 Time consuming and requires skill in communication 耗时且需要交流技巧 Requires adequate time and participant commitment 需要充足的时间以及参与者的努力 IPDET 58

59 Tool 9: Citizen Report (Score) Cards 工具9：公民报告（打分）卡
Used to collect citizen feedback on public services from actual users of a service 收集公民对于公共服务的反馈 Can assess the performance of individual service providers and/or compare performance across providers 能测评单个的服务供应商以及对比供应商的服务 Can generate a database of feedback on services that can then be placed in the public domain 能生成关于服务的数据库，之后应用于公共领域 59

60 Advantages and Challenges of Citizen Report Cards 公民报告卡的优点和挑战
Mixes focus groups and questionnaire data collection Increases response rates 综合专题组以及调查问卷的数据收集增加了回收率 Simple communication 简单的交流 Challenges 挑战 Local conditions must be conducive 本地环境必须是促进性的 Requires a large sample 需要大量的标本 Lack of predictability in how different players respond 缺乏参与者如何进行不同回应的预测 IPDET 60

61 Developing the Survey 制作调查
Define the purpose and objectives 定义调查的目的和目标 Decide the important issues to be covered 决定要包含的重要议题 Establish the relative weight of the different modules in the survey 确定调查中不同模块的相对权重 Identify important issues within sectors 确认各部分的重要议题 Question writers may need to learn more about how specific programs work 问题设计者可能须要了解更多有关特定计划是如何工作的知识 Once this background work is done, the actual writing of the survey may begin 一旦这个准备工作已经完成，真正的调查设计就可以开始了

62 Six Types of Questions 六种问题
Experience and behavior 经验与行为 Opinion and value 观点与价值 Feeling 情感 Knowledge 知识 Sensory 感知 Background/Demographic 背景与人口统计学 Each can be asked in present, past, or future 每一个都能问及现在、过去或未来 Many times, the step of communicating and consulting with policymakers is not given enough attention. Many policymakers are not familiar with surveys and may have difficulty reading complicated questionnaires. As well, they may not know how to imagine how the answers to the questions will be analyzed. For this reason, it is important to show policymakers and program managers examples of tables or other analyses that could be produced from the survey as well as the survey itself. Iterative Process According to the LSMS[1], the process of survey development is an iterative one. Once the initial version of the survey is drafted, the various interested parties should review the draft in detail. During their review, they should make notes and share their criticisms and suggestions. Revisions are then made to the original draft. This process may be repeated several times. [1] LSMS, A manual for planning and implementing the LSMS survey. pp 25 – 32. 62 62

63 Wording Questions 措辞问题
Use simple words 使用简单的词 Make questions and responses specific 使问题和回答都很具体 Avoid questions that use the words “and” and “or” 避免使用“和”和“或者” Avoid questions that assume knowledge (may need to provide information) 避免使用需要知识的问题（需要提供信息） Avoid double negatives 避免双重否定 (continued on next slide)待续 LSMS points out that it is crucial for the team of question writers to have extensive local expertise when designing the questions. They bring the knowledge of the country’s society and of existing programs. They also know what issues should be emphasized. In some cases, they may also be familiar with earlier local surveys about some of the topics covered which may help them design pre-coded questions. In addition, they will know the network of people and institutions that should be contacted during the survey design process. 63 63

64 Wording Questions 措辞的问题（续）
Make response options balanced 答案的选择应该平衡 Avoid objectionable, intrusive, and/or condescending 避免否定、介入和/或诱导性的答案 If possible use an existing questionnaire as a guide 如果可能，利用已有的调查问卷 Make the survey easy to complete 确保问卷容易完成

65 Sequencing Survey Questions 次序调查问题
Move from easy and interesting to difficult and uncomfortable 从简单有趣的问题逐渐到难的让人不舒服的问题 Move from general to specific 从一般的到具体的 Keep the flow logical (chronological or other order) 保证逻辑畅通（时间顺序或者其他顺序） Ask the most important questions by 2/3 through survey 在调查的2/3的地方问最重要的问题 65

66 Questionnaires Tips and Tricks 制作调查问卷的小提示和小技巧
Use simple, clear language, appropriate for audience 使用简单、清楚、恰当的语言 Ask only one question at a time 一次只问一个问题 Write your questions so that all feel their responses are acceptable 问题要让所有人都觉得他们的答案是可以接受的 When possible, write questions so that responses range from negative to positive 可能的话，按照从否定到肯定的回答秩序来设计问题 (continued)（续） 66

67 Questionnaires Tips and Tricks 制作调查问卷的小提示和小技巧（续）
Avoid “yes” or “no” responses 避免 “是”或“否”的回答 Avoid absolutes at either end of the scale (always, never), consider using scales 避免用词太绝对（总是、从不），考虑使用程度的词汇 Ask questions about the current situation 问当前的现状 Leave exits (no opinion) 留下退路（不是观点） Avoid using double negatives 避免使用双重否定 67

68 Field Testing 现场检验 Test with small number of subjects, diverse areas and socioeconomic groups 对少数几个题目、不同领域和社会经济组进行检验 Field test should look at three levels: 现场检验应该考察三个层次： as a whole 整体 each section 部分 individual questions 单个问题 Make revisions based on results of test 在测试基础上进行修改

69 Sequencing Interview Questions 对问题进行排序
Get the respondents involved in the interview as soon as possible 使受访者尽量参与到访谈中 Before asking about controversial matters, first ask about some facts 在提问有争议的问题前，先问一些事实 Intersperse fact-based questions throughout the interview 面向事实的问题要贯穿在整个访谈过程中 Ask questions about the present before questions about the past or future 先问有关当前情况的问题，再问有关过去或未来的问题 The last questions might be to allow respondents to provide any other information 最后一个问题应该允许受访者提供其它信息

70 General Guidelines for Conducting Surveys (1 of 3) 实施调查的一般指导原则 (3-1)
Keep it simple, clear, easy, short 调查要简单、清晰、容易、简短 Locate other people who have done the kind of evaluation you are interested in and locate surveys similar to what you think you want to do 找到其他实施了你感兴趣的评价 的人并发现与你想做的类似的调查 Make sure people know why you are asking them to participate 使人们清楚你为什么邀请他们接受调查 Ask questions that are easy to answer and do not frustrate respondent’s desire to be clear in their responses 提出的问题要易于回答，切记不要影响受访者给出清晰回答的意愿 (continued on next slide 转下页)

71 General Guidelines for Conducting Surveys (2 of 3) 实施调查的一般指导原则 (3-2)
Do not ask respondents for information that requires them to go to a file or other source. If you must do this, you need to let them know in advance so the material can be assembled prior to administrating the survey. 不要向受访者索要他们需要查找文件或其它信息来源才能获得的信息。如果你必须这样做，你需要事先通知他们，以便他们在接受调查之前就整理好有关资料。 Respect their privacy. Treat surveys confidentially and have procedures in place to assure privacy. Make sure you can insure confidentiality. Never promise confidentially unless it can be absolutely delivered. 尊重他们的隐私。做好调查的保密工作，并有保护隐私的程序。确信你能保密。除非能够做到绝对的保密，否则不要对此做出承诺。 (continued on next slide转下页)

72 General Guidelines for Conducting Surveys (3 of 3) 实施调查的一般指导原则 (3-3)
Respect respondents’ time and intelligence 尊重受访者的时间和智商 Tell them how they were selected and why their participation is important 让他们知道他们是如何被挑选出来的以及为什么他们的参与是重要的 Do no harm: keep responses confidential. For example, in your report, use aggregate responses; and assign an identification number to the data and destroy the link to the person’s name 不要有伤害行为：对回答结果进行保密。例如，在你的报告中，使用总结后的回答结果；对每个数据分配一个识别号码并使其他人无法据其联系到回答者的姓名

73 Developing an Interview 设计访谈
Step Procedure 步骤 程序 1 Define the purpose of the interview. Link your purpose to the evaluation objectives 明确访谈的目的。将你的目的与评价目标联系起来 2 Decide on open-ended or close-ended questions 决定用开放式问题或封闭式问题 3 Draft interview questions and sequence the questions 草拟访谈问题并对其进行排序 4 Prepare an Introduction and Closure for the interview 准备一份访谈介绍和结束语 5 Prepare to record responses 准备好记录问题的回答结果 6 Pre-test the instrument 对访谈中使用的工具进行预先测试

74 Interviewing 访谈 Use trained data collectors 使用经过培训的数据收集人员
Use as few data collectors as possible 数据收集人员的人数要尽可能地少 Establish a protocol for data collection 制定数据收集的方案 Do a “walk through” or trial run 进行试验性访谈 3-4 face-to-face interviews 3-4次面对面访谈 10 questionnaires 10份调查问卷

75 Conducting Interviews 实施访谈
Let the interviewees know: 使受访者知道: why they are being interviewed 为什么他们会接受访谈 how they were selected 他们是如何被挑选出来的 how the data will be used 将如何使用数据 whether it is confidential 访谈是否是保密的 how long the interview will take 访谈的时间有多长 whether you might want to talk to them again 你是否还想与他们进行再次访谈

76 Conducting Interviews: Set Up 实施访谈：准备
Try to pick a time a place that is quiet and free of distractions 设法选择一个安静的不受干扰的访谈时间和地点 Ideally, have a second person to help take notes 理想的情况是有另外一个人帮助记录 Tape recording might be a possibility; make sure you check with interviewee and get permission before recording 可能使用录音设备，但事先要征得受访者的同意 might want to ask when setting up the appointment 可以询问什么时候可以开始约好的访谈

77 Interviewing Skills 访谈技巧
Stick to your script 依照你的稿件 if asking close-ended questions, ask exactly the way written 如果问的是封闭式问题，则严格按照已写好的问题来问 have a script for clarifications 有一个解释说明 if asking open-ended questions, go with the flow rather than directing it 如果问的是开放式问题，则自由交谈而不要给出任何指示 Be aware of cultural norms: eye contact, direct questions, gender issues 要注意文化规范：眼睛交流、直截了当的问题、性别问题

78 More Interviewing Skills 更多的访谈技巧
Balance: if you ask about what they thing are the major supports, follow with what you think are the major barriers 平衡：如果你问的是他们认为是主要支持方面的问题，接下来就要问你认为是主要障碍方面的问题 Try to avoid asking “why” questions, some may find this aggressive or critical 尽量避免提问“为什么”的问题，有些人可能会觉得这样做有些挑剔或侵犯性 Accept whatever they say with empathy and without judgment 全神贯注记下他们所说的事情并不加任何评判

79 Conducting Interviews: Added Touches 实施访谈：增加接触
Share interview questions ahead of time 事先交流访谈问题 no surprises 这样不会对问题感到惊奇 Offer to share a summary of what you understand from the interview 提议共享你从访谈中了解的情况的总结 this might be especially useful to give the interviewee (especially if a high ranking official) a greater feeling of control 这可能对于使被访问者（尤其是高级官员）有更强的控制感方面特别有用 Thank you note afterwards 访谈之后的致谢

80 Interviewing: Taking Notes 访谈：做记录
Take good notes without distracting from the conversation 做好访谈记录，在对话过程中不分心 maintain eye contact while writing 在做笔记时保持目光接触 write key words or phrases, not verbatim 记下关键字或短语，不要逐字地记录 but — if someone is saying something you want to capture, it is OK to ask them to repeat it or to finish what you are writing before asking the next question 但是，如果某个人说了一些你想要知道的事情，可以请他们重复或等你记完后再问下一个问题 may want to ask “May I use your exact words?” 可以问“我可以使用你的原话吗？”

81 Writing-up the Interview 整理访谈记录
Every word and idea is valuable 每个字每个想法都是有价值的 Take time to write up your notes as carefully and in-depth as possible 花时间尽量仔细和深入地整理你的笔记 It is best to do at least a brief clean-up of notes immediately afterwards (leave an hour between interviews) 访谈之后（在两个访谈之间留出1个小时）最好立即对访谈记录进行整理，至少有一个简短的概述 Write up full notes within a day of the interview: memory decay sets in quickly 在访谈当天要整理出一个完成的访谈记录：记忆会很快遗忘的

82 Planning and Logistics 计划和后勤保障
Need several weeks to plan 需要几星期进行计划 Focus group sessions usually 1 to 2 hours 专题组会议通常1-2个小时 Some up to 3-5 hours with planned break (executives, in-house management, high-level participants) 大约3-5个小时的有计划的中断（行政主管、内部管理人员、高级别参与者） Have clear starting and stopping times 有明确的开始和结束的时间 Consider need for food and transportation 考虑到用餐和交通的需要

83 Facilities and Materials 设施和资料
Ideal – commercial facility designed specifically for focus groups 理想的情况——专门为专题组设计的商业便利 Neutral, accessible, hotel meeting room, school or church meeting area 中立的、能够达到的、宾馆会议室、学校或教堂会议地区 Table and chairs (comfortable setting) 桌子和椅子（舒适的摆放） Name tents 桌签 Laptop computer for taking notes (if possible) 用于记录的笔记本电脑（如果可能的话） Consider need for video or audio tape recordings or have additional people take notes 考虑用录像机和录音磁带进行记录的需要或请另外的人进行记录

84 Number of Focus Group Sessions 专题组会议的数量
No fixed rule 没有固定规则 Generally, do sessions until no new issues emerge or when common themes are consistent 一般是会议开到没有出现新问题的时候或当已经取得一致的共同主题的时候结束 usually 3 to 6 group sessions 通常3-6个专题组会议

85 Ground Rules 程序 “What is said here, stays here” “在此所说的话，就到此为止”
Everyone is encouraged to participate but not everyone has to answer every question 鼓励每个人参加，但不要求每个人都回答所有的问题 Respect different viewpoints 尊重不同的观点 There are no right or wrong answers 答案不分正确或错误 One person speaks at a time 每次只让一个人说

86 Typical Questions 典型的问题
What did you learn at the teachers college that is most helpful to you in teaching primary school? 你在师范大学学到的什么东西对于你在教小学的时候最有帮助？ What are your greatest challenges in teaching primary school students? 当你在教小学生的时候最大的挑战是什么？ What should the college teach so that graduates are prepared to meet the current challenges? 大学里应该教授什么使得毕业生能够做好应对当前挑战的准备？

87 Facilitator/Moderator Role 主持人的角色
Be familiar with the script, rather than reading it, so the session appears conversational 熟悉稿件，而不是照本宣科，这样会议才会显得像个对话 Make sure everyone is heard 确保能听到每个人的声音 Manage time, closing off discussion, moving to next topic 把握时间，结束讨论，进入下一个话题 Set ground rules 制定程序 Say as little as possible 尽量少说话 Keep personal views outside the room 排除个人观点 Use active listening 积极地倾听 Probe for elaboration (Tell me more) 询问详细的解释（告诉我更多）

88 Write-up 整理 Write-up your impressions immediately after each focus group 在每个专题组会议结束之后立即写下你的印象 major issues, major points of discussion 主要的议题和讨论内容 anything unusual that happened 已经发生的任何不寻常的事情 Compare with partner 与合作伙伴进行比较 If recorded, make arrangements to have it transcribed, or to carefully watch/listen to it later 如果记录下来，安排转录，或者之后仔细地看或听这些记录 Identify themes and emerging issues 确认主题和新出现的问题

89 Data Collection Summary 信息收集小结
You can choose more than one data collection technique, multiple tools often help you meet the evaluation needs 选取不只一种信息收集的方法，多种方法让你能达到评价的需要 Do not let the tool drive your work 不要让方法驱使你工作 Choose the correct tool to meet the needs of the evaluation 选择正确的方法来达到评价的需要 89