Nathan Grawe (Carleton University) visited Endicott College today and presented on Quantitative Reasoning. It was the same presentation that can be seen through the Quantitative Reasoning channel on Vimeo. He gave out two handouts. The first was version 6 of the Quirk rubric and the other were The 10 Foundational Quantitative Reasoning Questions. Functionally, what Dr. Grawe was proposing was Quantitative Reasoning across the Curriculum, where disciplines that did not use quantification as a central strategy would teach peripheral quantification in their classes, while classes (like Dr. Grawe’s) were centered on quantitative reasoning (QR, or quantitative literacy, or numeracy, or statistical literacy, etc.) would add authentic figures to their charts and authentic contexts to their problems. Those teaching peripheral strategies would have to ramp up their understanding of quantification and be sure that any QR was embedded in their work and not extraneous. Those teaching QR as a central strategy would need to make their calculations and data more authentic (i.e. more messy), which would make grading more time-consuming but hopefully much more rewarding. The presentation was entertaining and the proposal made sense. The rubric felt like it needed work though, and the sample QR assignments available through SERC did not seem as extensive as one might hope. There are Six Examples of QR across the Curriculum from the Numeracy Infusion Course for Higher Education (NICHE) program and a number of K16 teaching materials at another tab at the same site, but I would have preferred for these to have been indexed at MERLOT, since it would allow one-stop shopping for learning objects (and RLOs).
This news release is one I put together back in 2003. I couldn’t find a copy of it online, so I’m posting this in case anyone is interested. I found this panel enormously interesting because all of these classes were very small and each of the faculty members had a different approach to their instruction.
* * * * * * * * * * * * * * * *
A panel of IU faculty members gathered on November 8, 2003, to share individual approaches to some common challenges of teaching small classes. Although we often hear about innovative ways of teaching large classes, small specialized classes also require flexibility and creativity on the part of the teacher. The four panelists came from different departments and taught four different languages — Catalan, Swahili, Uzbek, and Russian. After the exchange, the panelists agreed that they learned as much as the audience, and all involved look forward to the next opportunity to share teaching strategies.
Professor Josep Sobrer, who teaches Catalan in the Department of Spanish and Portuguese, finds innovative course resources during his visits to Catalonia. Catalan is a language centered in Barcelona and Catalonia, with only eight million people in the world who speak the language. Though instructional materials can be scarce, Professor Sobrer supplements his classes with a Catalan language instruction program on CD-ROM and with Catalan versions of popular American movies. These resources allow his students to immerse themselves in the everyday applications of the language. Professor Sobrer teaches Catalan whenever a cohort of five to ten students is interested in the language.
Associate Professor Aliwaya Omar, who teaches Swahili and other languages in the Linguistics Department & the African Studies Program, supervises peer instructors in more specialized languages such as Luo, Tigrinya, and Moroccan Arabic. When a student wants to learn a language like these, usually for a research trip to Africa, she matches the student with a native speaker on campus and gives brief pedagogical instruction to the native speaker. The peer instructor then customizes the lessons to the contextual and research needs of the student. This intensive program is made possible by Title VI funding, primarily Foreign Language Area Studies (FLAS) fellowships received by the students. Associate Professor Omar also teaches Swahili, with enrollments of up to nineteen students, and supervises regular classes in Twi, Hausa, and Bambara.
Dr. Malik Hodjaev, a visiting lecturer teaching Uzbek for the Department of Central Eurasian Studies, must develop his own course materials for this little-taught language. Before coming to the U.S., Dr. Hodjaev visited colleges and universities in Uzbekistan, trying unsuccessfully to locate resources for teaching Uzbek to Americans. His search was made more difficult because Uzbekistan has changed alphabets five times in the last two hundred years. They used an Arabic script from the 14th century to 1927, when it was replaced by a Latin alphabet. The Latin alphabet was in turn replaced by Cyrillic in 1940. In 1992, the Latin alphabet began to make a comeback, and in 1993 an alphabet based on typographic symbols emerged. In 2005, the Latin alphabet will again be the official alphabet of Uzbekistan, but in the meantime, many teaching resources use only Cyrillic. As a result, Dr. Hodjaev is concurrently developing course materials in both Latin and Cyrillic alphabets for the three levels of Uzbek he teaches.
Dr. Jeffrey Holdeman is the Slavic Language Coordinator in the Department of Slavic Languages and Literatures and is currently teaching Russian. He is working to build student enrollments and retention rates in Slavic language classes by developing a multi-faceted strategy combining program development, analysis of trends in student demographics, extra-curricular activities, instructor-training, advertising, and outreach. Like some of his colleagues, he finds resource materials scarce for some languages, such as Polish and Serbian-Croatian. He is working with the AIs in these languages to compile bibliographies and materials and to present these to colleagues at national conventions. Dr. Holdeman also uses electronic discussion groups, such as SEELangs (http://home.attbi.com/~lists/seelangs), to ask questions, share resources with other instructors, and combat the isolation felt by instructors of less commonly taught languages.
These four faculty members agree that the importance of teacher-student relationships is magnified in small classes. Instructors often need to respond to the strengths and weaknesses of particular students in designing syllabi and pacing the content. For example, student needs largely shape the content of the course in the one-on-one African languages courses. Instructors tailor these syllabi to concentrate on the specialized research contexts in which the languages will be used (reading or speaking, specialized vocabulary, etc.).
Relationships among students and differences in their preparation are also magnified in small classes. For example, a single language class may enroll novice undergraduate learners, heritage learners (students raised in households where the studied language is spoken), and graduate students (who may have advanced learning skills but little content knowledge). Both heritage learners and graduate students will often demand greater content than the novice learners can handle. Dr. Holdeman provides these more advanced learners with further material on the cultural context of the language, allowing them to deepen their understanding of the language without disrupting the level of conversation within class.
There’s an excellent article on classroom observation protocols in the Chronicle of Higher Education (Feb. 14, 2014) on classroom observations. The method used by Brett Gilley (University of British Columbia) is Classroom Observation Protocol for Undergraduate STEM (COPUS), and is essentially a time audit for activities in the classroom. COPUS is also used in Smith et al (2013) The Classroom Observation Protocol for Undergraduate STEM (COPUS). Both that protocol and the Teaching Dimensions Observation Protocol (TDOP) are classroom observation systems designed for science classroom. TDOP is described in Hora, Oleson & Ferrare (2013) The TDOP User’s Guide and is most recently mentioned in Hora & Ferrare (2014) Remeasuring Postsecondary Teaching.
This is not a new thing. Most systems like this are based on Ned Flanders (1960), Interaction Analysis in the Classroom: A Manual for Observers. The Clinic to Improve University Teaching at the University of Massachusetts-Amherst used classroom observation in its model for faculty development in higher education in the early 1970s, and the Clinic supplied the model for most faculty development centers in the U.S. for the next twenty-five years or more.
The method that I use is most closely related to anthropological non-participant descriptive observation. I learned it at Indiana University’s Teaching Resource Center, which was largely based on the Clinic model, but I’m not sure exactly where it came from. Jan Nespor (1997) talks about this method in Tangled Up in School: Politics, Space, Bodies, and Signs in the Educational Process, so it is not uncommon. Take a look at Whitehead (2005) Basic Classical Ethnographic Research Methods to learn more.
Because I am a non-participant, my presence can change the classroom dynamics, so I have the instructor explain that I’m there at her request because she is interested in improving her teaching. I try to dress down for the observation, to try to blend in as best as possible. Once the students know I’m not checking up on the professor, they tend to relax and act more normally. It is descriptive observation because the aim is to write as much down as I can, so I have plenty of evidence to discuss with the instructor later. I try to write down everything the students do, so I can judge whether the instructor is going too fast or if he is talking while students are busy writing. I keep track of how many PowerPoint slides are being used, how they are designed, and how easy they are to read from the back of the room. After about twenty minutes, I change my method of observation and start concentrating on the interaction between teacher and students, and among the students themselves. I draw a map of the room showing the various interactions and circling the students who talk the most. I note where the teacher moves and with whom she is making eye-contact. I note the types of questions asked, and how they are followed up. I typically end up with 10-12 pages of notes for an hour of class, so I also make sure to tell the instructor this so he doesn’t worry and think I’m just writing down things for improvement.
What is interesting to me about Gilley’s work is that a “High” level of student engagement during a lecture and demonstration session. I might report that for a demonstration session, especially if it models the work the stduents will soon be doing, but the rest of the time they are just listening. All the research I’ve seen does not see that as high engagement. Lectures are best at improving lower cognitive goals (memorization, understanding) by students. Lectures are not particularly effective at improving student thinking, which is what I equate with student engagement. Behaviorists like Gagne think of entertainment as a form of engagement, but Eriz Mazur and others have pointed out that while students may be stimulated and entertained by a rousing lecture, they really don’t learn very much.
Today at a faculty retreat, we were encouraged to write haiku of wisdom for our graduate students. Having done the haiku effort before, I resisted and instead wrote a sijo, a Korean form of poetry that predated haiku. Here’s my sijo effort:
The trees of Keumgangsan bend in the wind, grow to the sun.
The scholar droops in the night, leans to the bed.
Bed warmth, sun warmth. Both must wait for dissertation’s end.
NPR just had hosted an article by Tania Lombrozo about Solving the Conundrum of Multiple Choice Tests. Lombrozo summarizes the research by Arnold L. Glass & Neha Simha, entitled, Multiple-Choice Questioning is an Efficient Methodology that may be widely Implemented in Academic Courses to Improve Exam Performance. Actually, this research is nothing new. Repetition of practice can lead to learning — we knew that already. MCQs can contribute to this repetition and lead to improved long-term retention. Right. But retention of what? That is the key question. Behaviorists taught us all about improving memory, about stimulating the improvement of memory, and even how long-term memory eventually fades after the removal of the stimulus. Nothing new. But Lombrozo states that, “Students who received multiple choice questions throughout a course performed better than those who didn’t when it came to final exams.” That’s true, but easily can be misconstrued. MCQs aren’t an irreplaceable part of an education. They are one tactic that can be used, and unfortunately that tactic is not very good at teaching the upper cognitive levels identified in Bloom’s Taxonomy. They work well for remembering, understanding and even applying heuristics (which means they are helpful for mathematics), but they are difficult to use to check the ability to analyze, evaluate and create texts (making them not very helpful for the humanities or other textual forms of knowledge). If a student practices with MCQ quizzes, they will obviously be more ready to answer MCQs on the tests, particularly if the questions are parallel or perhaps even the same on both quizzes or tests. Glass & Simha demonstrate that MCQ quizzes can also help with short answer responses on the tests, which is useful knowledge. But we already knew that repeated MCQ quizzes could help with retention of content knowledge. Our society is pretty test-happy already. Great. But what we also need is the ability to analyze and evaluate that information and to solve problems based upon it, or develop new lines of inquiry. MCQs don’t do that very well. We also know that people get better with practice. So why not give them practice at those higher cognitive functions with weekly written quizzes or journal entries that will give them the practice to do them? The answer is time. Most teachers know what is effective and what is not, but they generally choose the easier methods even if they are not the most effective, because their lives are already too busy and they have to have some time for their own families.
The earliest article cited by Glass & Simha is A.P. Yonelinas (2002), The nature of recollection and familiarity: A review of 30 years of research, Journal of Memory and Language, 46, pp. 441–517. Unfortunately, our library does not have that journal in its subscribed databases, nor does JSTOR, so I’ll have to interlibrary loan it. But I suspect I will find that it lists several articles that reach similar conclusions to Glass & Simha. I’ll let you know.
In the meantime, we can look at Douglas Courtney’s similar research in the 1940s at Boston University. The article is Courtney et al (Feb. 1946), Multiple Choice Recall versus Oral and Written Recall, The Journal of Educational Research, 39(6), pp. 458-461. Courtney noted that,
“It is often observed that some children who make high scores on reading tests fail to do commensurate work in the classroom. This disparity in performance may be due in part to the fact that the reading test demands a type of recall different from that demanded by classroom discussion or written tests.”
Indeed, Courtney’s research showed that “The scores on the multiple-choice test were markedly higher than on the written recall test.” Moreover,
“It is evident from this study that multiple-choice recall does not insure comparable written recall. A study of individual scores shows many pupils with high recall on multiple choice items but very low recall on written reproduction. The wider standard deviation of the written recall test is also of significance. These findings indicate the desirability of supplementing the usual reading test with measures of written recall in order to get a more accurate picture of the pupil’s ability to deal with materials he has read.”
All of this was before Bloom and the other Univ. of Chicago faculty identified different types of cognitive acts (Bloom’s Taxonomy). It is clear, however, that transfer of knowledge from the declarative knowledge tested by MCQs was not helpful for Bloom’s higher cognitive levels.
Now, I know most of you will ignore me, because you think MCQs will save you time, but actually it takes quite a while to design effective MCQs. Please at least read Thomas M. Haladyna et al’s (2002), A Review of Multiple-Choice Item-Writing Guidelines for Classroom Assessment, Applied Measurement in Education, 15(3), pp. 309-334.
This question came up on the POD Listserv, so I’m posting my response here. Here’s my tips on how to make grading papers easier.
1. Set the theme for the semester paper but give the students lots of choice on how to answer it. Pick a theme that other instructors don’t use, so there are not a lot of opportunities for plagiarism. Let the students choose the lens to examine the theme, the evidence to use, and the opinion to take. This way you’ll have a lot of different papers to read at the end of the semester, not a stack of blue books all trying to remember what you told them. With any luck, you’ll actually learn things from your students. Think of them as your apprentice researchers.
2. Train them how to write. I give undergrads a 1,000 word journal to turn in every week on the readings, due Sunday night before class. This is not only a check on their reading, but also allows me to us JiTT to customize the class period using my knowledge of their understanding. It takes 2-3 weeks to train them so they believe you actually want at least 1,000 words. Give individual feedback only when truly needed. The rest of the time, give group feedback in class on spelling, grammar, and content mastery. Be sure to praise insightful or meaningful observations in the student papers.
3. Teach them what academic dishonesty is, and uphold your standards. Really, most know what it is but they’ve been doing it since the 5th grade and getting away with it, so they’re going to be shocked that you actually checked and held them to it. Hopefully they will learn, because they are out of my course on the second time. In 25 years of teaching, only one of my students has been suspended (but that was for two years, because he had done it in other classes as well).
4. Structure the assignment with appropriate scaffolding (Baby Bear style — not too much, not little, just right). For me, that is a topic sentence assignment first, then after approval of that, a first draft (1/3 size) with comments from me. Then a second draft (2/3 or more done) with peer revision. Students come to class, swap papers and mark them according to a rubric I hand out. That trains them on the rubric as well. I grade the second draft as “done” or “not done” but I also copy the marked rubrics before returning them, so I can predict next year what section of the rubric they will understand least. The final draft is turned in at the end of the semester, one week before student evaluations are handed out. Final exams are replaced by a week of reading each other’s papers, synthesizing and generalizing the results, and summarizing our work over the semester.
5. Only mark the papers that students want returned. For those papers, ask for a self-addressed stamped envelope (9×12″) so you can ship the paper to the student after grading.
I hope these work for you. A lot of this comes from my old professor, Robert Ferrell. The rest is largely in there because of all the Writing Centers I’ve worked with over the years. Thanks folks!
P.S. Nick Carbone just posted 10 Tips for Grading Writing with Less Stress and Frustration. If you are looking for advice on dealing with the emotional frustration of grading, take a look. He has some great suggestions in there!
I just finished reading (and grading) my students’ second drafts of their semester papers. In my response to them, I sorted the types of papers I had received into four types. Here are those four types, and my advice to my students, with appropriate musical accompaniment.
1. Cut & Paste: The person cut and pasted large sections of text verbatim from one source to the paper. This is plagiarism, which received a zero on the assignment and a report to the Vice-President of Academic Affairs. Ouch. To avoid this happening to you, read through Laura Plummer’s Plagiarism: What It is and How to Recognize and Avoid It.
Sloppy by Devo
2. Modern Machine: The person wrote their paper as they read from Wikipedia, copying the sentence structure but changing a few words in the sentence. Surprise! This close paraphrasing is plagiarism too, because it uses someone else’s ideas and appropriates them as one’s own. A lot of students do this these days, because the web makes it so easy. Some students will only do a bit of this, because of sloppy writing habits (i.e. writing their papers while reading from the source). It is a short cut, and means that someone did not take the time to work on their own composition. Please be aware that I could give a zero for the assignment as with paper type #1, but this time I only dropped the assignment a letter grade because the folks who were doing it had not done it throughout the whole paper. They had at least done some of their own composition.
Modern Machine by the Crowd
I know my concern with close paraphrasing is often a mystery for students, so let me explain. Close paraphrasing allows one to write a “paper” very quickly and with very little thought. Cut and paste, change a few words, and voilá, a composition is born. Unfortunately, this cuts out what we call “education.” If I all I wanted was the data, I could just go to the original site. If I wanted to learn about a topic, I would go out and do the research myself, which would usually include reading anywhere from twenty to seventy-five articles for one of our class discussions. I do that every week. The reason that we (your teachers) assign papers is so that you have an opportunity to learn how to analyze sources and decide how much to trust them, how to communicate with others so that others understand you as best you can, and how to frame your arguments either through logic or emotion so other people will listen to you and perhaps even agree with you. These are valuable skills. You cannot learn them by cutting and pasting from Wikipedia. You have to practice them, hopefully with a coach to help you improve. That coach is sometimes called a teacher.
There is also an ethical lesson here also. You are in school and that is a safe time to practice those skills. Even if you did paper #1 above, it was a mistake you made at a safe time, when the only repercussion was a reduced grade (as long as you didn’t do it again). Once you are out of college, you can be sued for putting forth someone else’s work as your own, particularly if you are successful and have money or position. Of course, it’s a lot like our discussion in class this week, which was “when should one break the law?” Answer: If you are most concerned with society’s needs, you don’t. If you are most concerned with your needs, then whenever you can get away with it. The more successful you are, the more people will be gunning for your job or be resentful of your position. With more people watching, your odds for getting away with it drop. More importantly, your influence drops when you are perceived as taking advantage. A leader or boss actually has to be perceived as helping others in order to inspire and motivate them to do what he/she wants them to do. Often a person will mess up just when they think they’ve got it made. They do something stupid and get caught. Bill Clinton is an example of that. To avoid this in your own life, make it a point of personal honor to be a hero and not a zero. Stop yourself from making that stupid mistake by making a habit out of helping others and doing the work. You’ll be a lot more successful when all of your work and planning finally get you to the position you’ve always wanted.
Ok, lecture over. You can tell this stuff is important to me, so please don’t do it, at least in my class. As I’ve said, it takes time to document and I’d rather spend that time with my family. On to the next type of paper…
3. Short and Sweet: The person has done their research, then closed their notes and the web and written their own composition. So far, so good. It reads well, and more importantly, I can tell it is their own work. It sounds like them. But the person has only done half the work that the rest of the class has done, and the essay is half the length it should be. Can anyone guess which essays will be receiving the most praise? Not this one, even though it was good solid work. This is easy to fix. Next time, write a long enough essay. And if this is the sort of work that you enjoy, you might consider a field where technical writing is the preferred mode. You’ll be good at it.
4. Solid Potato Salad: The person has done their research, used sources that are reliable and as authoritative as possible, and has then closed their notes and the web and waited a few hours before writing their own composition, synthesizing their thoughts and creating a personal interpretation of the facts, presented in their own words. This took them longer, but along the way their neural networks had time to build new nodes, connect the new data to old, and increase the number of connections in their brain to form a new understanding and provide more nodes for future assignments and tasks to build on. Yay! The paper they write may vary in quality and may not be an “A” yet, but if the person keeps this up, it will be there someday. Certainly their teacher is a lot happier when grading their paper, and that’s got to help! More importantly, this person learned a lot, and even got better at learning this sort of thing. There may even be some positive transfer to other skills.
Solid Potato Salad by Ella Mae Morse