Tutorial 8: Resources
A. Understanding Your Data
Action Research is not a single research project; rather it is an ongoing iterative approach that takes place across cycles of innovation and reflection. It is a way of learning from and through systematic inquiry into one's practice. Central to this process is the collection and analysis of data. The image below (Rie1, 2014) uses color to represent the growing knowledge of the action researcher. After a number of cycles, there is a reporting out or sharing function. In the last tutorial, we discussed what data artifacts you could collect. Now we are going to talk about the analysis of these data artifacts. This is often the step that practitioners find most difficult because they have not been trained to analyze data.
B. Organizing your data -- What is your storyline?
You will need to report your "findings" that you discover from exploring your data. To develop findings, you have to engage in some serious "looking" to find meaning from what you collected. Often new action researchers will simply describe a survey and then paste the questions with the group responses in their reports. But this is not helping the reader to make sense of what you learned. You haven't processed the data to learn from it.
It is helpful to begin with your research questions or your near and medium outcomes on your logic model. What does the data you collected tell you that will help answer your research question? How does the data help you to understand if you see evidence of the outcomes that you predicted in your logic model? To answer these questions, take the responses to your survey and group them around what you are trying to learn. You need to find the storyline that you will be sharing with your audience.
AN EXAMPLE-- Katherine Korte Flips her Government Course
For example, consider an educator teaching a government class who flipped her classroom as her "action" and, for data collection, asked her students to provide some feedback on their learning in a survey.
RESEARCH QUESTION: If I flipped my classroom using video to send lessons home and ibook technology, how will this affect the quality of knowledge building dialogues in the classroom?
She was interested in what students learned, how they learned, and changes in their attitudes and engagement in learning. Therefore, she might take her ten survey questions and organize them under these three topics with one question not fitting this process.
(1) knowledge
....(three survey questions)
(2) process
....(four survey questions)
(3) attitudes,
....(two survey questions)
Now she would compute responses to these groups of questions and think about other data sources besides the survey -- what tests or quizzes did students take that would help answer the first question about knowledge? How might field notes shed light on what students were learning? What artifacts were produced that could be examined? For the second topic, there were student responses that addressed their work in discussion circles, and what they learned from each other and some of the problems from group work. How did students' comments match her notes? Were there more or fewer problems that students needed help with? Then she might organize the analysis into three sections:
Changes in Student Knowledge
....analysis of three survey questions
....classroom quizzes,
....end of term assessment,
....ibook notetaking,
....performances
Changes in the Way Students worked
....analysis of four survey questions
....field notes on how well the groups worked
....what problems they had,
....teacher assessment of how prepared they were for the classroom discussions,
....speed at which they accomplished their tasks
Student Reactions to the flipped Classroom
....analysis of two survey questions
....students blogs
....observations about the comfort of students contributing ideas or making presentations
This organization helps her to find a storyline that she will tell about the data.
Validity of Measures
You can see that now that multiple sources of data are being examined to see what they say about the outcomes that the educator cares about. The analysis should be an honest hunt to figure out what can be learned from what happened. You are looking for valid measures that will answer your question. For example, if a teacher listed grades as evidence of student engagement, one might challenge this measure as not a valid measure of engagement because a student could have high grades and not be engaged and another student could be engaged and yet have a low grade. This is why researchers consider the validity of measures.
Participatory Data Collection
Empowering the group to participate in data collection and analysis helps everyone to have a sense of ownership over the outcomes and the deep understanding of change. These guides and articles might help you think about who should be involved in the data collection/analysis work in your action research. Even if you collect and analyze data within a group, you still have your own reflections on what this process means to you as an action researcher.
-
Guijt, I.,(2014). Participatory Approaches: This guide, written by Irene Guijt for UNICEF, looks at the use of participatory approaches in impact evaluation. While not exactly the same as action research it is very close and this guide will be useful if you are involving stakeholders in your process of data collection and analysis
-
Increasing Participation in Evaluating – Bruner Foundation Guide - This guide discusses how Organization Staff, Evaluators and Funders are involved in Participatory Evaluation.
-
Campilan, D. (2000). Participatory Evaluation of Participatory Research. Forum on Evaluation of International Cooperation Projects: Centering on Development of Human Resources in the Field of Agriculture. Nagoya, Japan, International Potato Center. http://ir.nul.nagoya-u.ac.jp/jspui/bitstream/2237/8890/1/39-56.pdf
-
Chambers, R. (2009) Making the Poor Count: Using Participatory Options for Impact Evaluation in Chambers, R., Karlan, D., Ravallion, M. and Rogers, P. (Eds)Designing impact evaluations: different perspectives. New Delhi, India, International Initiative for Impact Evaluation.http://www.3ieimpact.org/admin/pdfs_papers/50.pdf
-
Guijt, I. and J. Gaventa (1998). Participatory Monitoring and Evaluation: Learning from Change. IDS Policy Briefing. Brighton, UK, University of Sussex- In participatory monitoring and evaluation, stakeholders work together to decide how progress should be measured and how to understand the outcomes of change.
-
Zukoski, A. and M. Luluquisen (2002). "Participatory Evaluation: What is it? Why do it? What are the challenges?" Policy & Practice(5).http://depts.washington.edu/ccph/pdf_files/Evaluation.pdf
C. Exploring your data -- What is your story?
Coding your Data
The Center for Evaluation Research at the University of Calif, Davis has a good and brief guide on coding as well as other tools for data analysis that will be helpful to read before you begin your coding process. If you are doing coding of data-- for example coding what students write in their blogs, it will be helpful to develop a codebook where you list the codes and examples to help you make decisions. This serves to create consistency or reliability in your coding. Good research practice involves creating a codebook for any qualitative analysis you do. This can be very simple or more complex depending on the nature of your data and you scale of analysis you plan to do. This is similar to create a rubric to assess student work. If you are writing an action research dissertation, the use of a codebook is highly recommended. If you want to see how researchers use codebooks and coding schemes, you will find a number of them have been collected by Kimberly Neuendorf as an accompaniment to her Content Analysis Guidebook. I enjoyed exploring the codebook for the study of female roles in James Bond 007 movies.
In this video, Kevin Meethan also describes the process of going from an interview to codes-- if you go to the youtube version, I think you can find a link to the texts that are given in the examples, but you can just listen to the examples and a get sense for the coding process.
Reliability
In any coding, an important question to ask is: how reliable is the coding process? In other words, if this teacher told us what theme or content to look for and we coded the data would it be the same as her coding? To achieve reliability, researchers often have two people code data and compare. If they agree on 85% of the cases, then we say the coding is reliable. If you cannot find another person to compare your coding, you can check your coding with yourself at different times. Code a few cases and then wait a day and recode. Do they match? The closer the match the more reliable your coding.
Teachers often engage in a very similar process to grade papers in a consistent or reliable way. They create a rubric and then use that rubric to assess the student work. The rubric is their codebook and the access is the coding process.
Content Analysis
Let's stay with the teacher who flipped her classroom and had students keep notes in their iBooks as they watched the videos at home. This teacher wants to analyze this textual data. How should she approach this? Depending on which category the action researcher chooses, there are lots of ways to do a qualitative analysis of data. Some are more time consuming than others and a large amount of data might be more than a teacher can examine. Our teacher might want to create a sample to analyze. She could take a random set of three entries for each student, or all students' last blog before the test, or she might decide to examine all of the blogs of six students. These six could be randomly selected from all of the students or from groups of students. For example, the teacher might select two students at random from those that received an A, B or C grade in the course, or in a previous course, or one student from each discussion group.
A content analysis helps us to know what was contained in these blogs. Suppose there are some central ideas or concepts and our teacher wants to see how often, if at all, they appear in student writing. To access this information, she might use a single blog, or response to a question as the "unit of analysis" and then mark yes or no for a number of ideas that she has decided on. She would take each entry and code it, say 1 or 0, for each of the concepts. This would help her see how frequently individuals and the class as a whole explored these concepts. A different approach to content analysis would involve coding all of the topics that were covered in each entry. Then she could examine the topics that students selected as important. This approach would help her to understand what topics students judged to be important.
Theme Analysis
Theme analysis is similar to content analysis but a bit more difficult. Here, you are not looking for concepts but for themes. Some process themes might be positive or negative attitudes about the different forms of technology used in the classroom. As in content analysis, these could be set up by the teacher or they could emerge from the data as when the action researcher discovers themes through repeated reading, grouping them, and constructing a final set of themes to use.
Developing a Critical Eye
Action research is not about proving how successful you were at some new attempt. It is about learning from your efforts. It is not so much about whether it worked or not, but a more thoughtful examination of how it worked, or for whom it worked, and why things turned out as they did. So be ready to hear that your plans did not work as you expected. Try not to be defensive. Instead, see this as a time to learn from your inquiry. What worked and why? What did not work and why? it is important to examine evidence that contradicts or opposes your research question. You might find the "What Else Test" developed by Jess Dart will be helpful as you develop your critical eye.
Advice for Data Analysis in Social Services
The Institute for Research and Innovation in Social Services (IRISS) is a charitable company with a mission to:…promote positive outcomes for the people who use Scotland’s social services by enhancing the capacity and capability of the social services workforce to access and make use of knowledge and research for service innovation and improvement.
They have a number of resources can help think about action research starting with this animated video:
Outcomes ToolboxA toolbox of resources relevant to an outcomes-focused approach in social services
Developed by IRISS in partnership with Coalition of Care and Support Providers in Scotland (CCPS), the Outcomes Toolbox brings together a range of resources and knowledge relevant to an outcomes-focused approach in the social services.
Understanding and Measuring Outcomes The role of Qualitative Data
Emma Miller and Ellen Daly
D. Displaying your Data -- How will you Tell a Compelling Research Story?
Think about what your data has told you and tell others what you learned. Use chart, graphs or tables to help people quickly see what you have found. With all of the choices you will find yourself thinking which type of chart should I use. While there are suggested guidelines about these choices, you might find it works just as well to try displaying the data in a number of different ways and look at it and see what you learn. Then test it with someone who does not know the data. What do they see from your chart? Consider the signal to noise ratio. The signal is your data and the noise is graphical elements around the data you might not need. You want to encourage the eye to quickly zero in on what is relevant.
This video shows some examples might help you think about how to represent your data without unnecessary distortion. While design can be subjective (and while I don't agree with all of the comments) the suggestion to minimize "chart junk" is important. Try to find ways to clearly communicate your data in an honest and pleasing image.
Presentation and Visualization of Data
A site that might be helpful for understanding some basic concepts in statistics and how to display data can be found at http://www.shodor.org/interactivate/lessons/HistogramsBarGraph/ This site is arranged to support for high school math teachers but there some interactives that might help you think about data displays. (And it is a fun way to play with math and statistic concepts).
Many eyes (http://www.many-eyes.com) is a free tool for creating a visualization of data sets.
for examples see...
http://www-958.ibm.com/software/data/cognos/manyeyes/visualizations?sort=rating
​