The IDEA Center Blog

Bill Cashin Receives McKeachie Award

Angela — Tue, 15 May 2012 00:42:33 +0000

On April 12, Dr. William E. (Bill) Cashin, former Director of the Center for Faculty Evaluation and Development at Kansas State University—which preceded The IDEA Center—received the W. J. McKeachie Award at the annual meeting of the American Educational Research Association (AERA) held in Vancouver, Canada. The AERA Special Interest Group (SIG) on Faculty Teaching, Evaluation, and Development bestowed the award, which recognizes outstanding contributions, over a career, to the fields of teaching, evaluation, and development.

Left to Right: Steve Benton, Marilla Svinicki, Bill Cashin, Bill Pallett

Those who use IDEA Student Ratings of Instruction are well aware of Bill Cashin’s influence on the field, as he has authored or co-authored 16 of the 52 IDEA Papers, which focus on topics related to improving teaching and faculty evaluation. In fact, Bill initiated the series in 1979. Each year thousands of readers download the papers, and hundreds of teaching and learning centers have links to this resource.

Bill’s passion for faculty development began in 1975 when he began working for the Center. He served as Director from 1985 to 1996. At the time of his retirement in June 1996, over 1,200 higher education institutions had used the Center’s resources. During his career, he has written over 60 articles, papers, and book chapters and made more than 300 presentations at two- and four-year colleges and universities.

This is not the first time Bill has been recognized for his achievements. In 1987, the American Psychological Association (APA) invited him to deliver the G. Stanley Hall Lecture at its annual convention. Six years later he became an APA Fellow in Division 2—Teaching of Psychology. In turn, AERA awarded him Fellow status in 2008. Across the years he was served on the editorial boards of Teaching of Psychology, Journal of Educational Psychology, and Innovative Higher Education.

We at The IDEA Center marvel at all Bill has accomplished during his career. President Bill Pallett says, “Whatever successes the Center currently enjoys in serving its mission are in large part due to Bill Cashin’s steadfast effort and commitment to doing high quality work.” Without Bill Cashin, the Center would likely not be in existence to serve its mission today. He leaves a lasting legacy. Congratulations, Bill!

Adjusted Scores: What are they?

Angela — Mon, 07 May 2012 22:22:41 +0000

By Shelley Chapman

This post is the first of a three-part series on “Adjusted Scores” and describes what adjusted scores are in the IDEA system. Future posts will describe why they are useful and when to use adjusted rather than raw scores.

Unique to IDEA
Adjusted scores are modifications of the raw scores based on contextual factors that are outside the instructor’s control. The purpose of adjusting scores is to level the playing field when comparisons need to be made between instructors teaching different kinds of classes (e.g. general education, first year, or upper level, for the major).

Variables
The two factors outside the control of the instructor that have the biggest impact on students’ ratings of the course and their learning are student motivation to take the course and students' typical work habits:

Item Number on Diagnostic Form	Item Number on Short Form	Wording of Item
39	15	"I really wanted to take this course regardless of who taught it."
43	13	"As a rule, I put forth more effort than other students on academic work."

Adjusted scores, therefore, really depend on the group of students in a particular class at any one time and are not something that the instructor can change or control.

When They Go Up and When They Go Down
Adjusted scores go up when the students in the class report lower than average motivation to take the course (regardless of who taught it) and/or lower ratings on their work habits than the average of classes for those items in the IDEA database.

Adjusted scores will go down when the students in the class report high motivation to take this course (regardless of who taught it) and/or high ratings on their work habits.

If you look at the bottom of page 2 of your reports, you will likely see that in the course where your scores were adjusted upward, students' motivation and/or work habits are likely "lower" than the IDEA average (if both are "lower" or "much lower," you will see more dramatic adjustments).

When your scores were adjusted downward, it is likely that on page 2 you will see that the motivation and work habits items are "higher" or "much higher."

Adjustments are more dramatic when both motivation and work habits are either "higher/much higher" or both are "lower/much lower."

More Information
You can learn more about adjusted scores by visiting the IDEA Help Community.
For information regarding the statistical process of adjusting scores, please see pages 36-43 in Technical Report 12.

It’s a Tough Job: Supporting Academic Department Chairs

Angela — Mon, 30 Apr 2012 22:15:28 +0000

By Amy Gross

One reason I am so committed to the work of The IDEA Center is that providing feedback to guide self-reflection and professional development is a common principle underlying all our products and services. We continually ask ourselves, “What can we do to support faculty and administrators to get better at what they do?” Through this, I truly believe the IDEA Center can have a positive impact on the landscape of higher education.

Recently we have focused our research and development efforts on supporting those that lead academic departments at colleges and universities. What have we done?

A Fresh Perspective: Updating the IDEA Feedback for Department Chairs System
Though the name has changed, we have offered the IDEA Feedback for Department Chairs system since 1977. From our research, experience, and feedback from clients, we recently gave that instrument a face lift. Our primary goal was to facilitate deeper personal reflection through building on strengths. The biggest change is the option of including the department chair’s self-rating. This additional piece of information allows the department to compare his or her self-perception with that of the faculty. This can lead to valuable insight about gaps that might need to be addressed to improve effective leadership. Check out the new faculty survey, Chair Information Form, and the updated report!

New Services: Department Chair Coaching and National Seminars
After a department chair receives feedback from their faculty, then what? We know from professional development research that having feedback alone helps us improve, but what can really make a difference is the opportunity to talk to someone about the feedback – and this has certainly been my own professional experience! With that in mind, we initiated the IDEA Department Chair Coaching Service. Because every department chair has unique needs, we have a number of different options – an individual session to talk through IDEA Feedback results, a year-long coaching option for individuals who want to develop and work on their own professional development plans with the support of a coach, and finally, up to six chairs can work as part of a learning community for a year to focus collaboratively on their professional development plans.

On a personal note: We currently have three coaches available to work with chairs—Dan Wheeler, Al Seagren, and Delivee Wright. They are experienced practitioners and authors in areas related to department chair and/or faculty development. I have personally learned so much from them as we worked together to develop the coaching service, that I can only imagine how much department chairs could benefit and grow from a year-long coaching experience with them.

Finally, we have started the IDEA Department Chair Seminar program which is offered twice a year – typically in June and November. Sometimes it is necessary to step away from our daily routine (and interruptions!) in order to focus on ourselves. The seminars provide this opportunity for department chairs. We are privileged to work with national leaders in their field who guide participants through interactive sessions addressing some of the most important (and challenging) aspects of the department chair’s role. We limit registrations to 100, so we are able to keep the learning environment relatively intimate and in-depth for 2 days.

Continuing Service: Academic Chairpersons Conference
I would be remiss if I did not also mention the Academic Chairpersons Conference which we have co-sponsored with the Division of Continuing Education at Kansas State University for nearly 30 years – a great opportunity to share and learn best practices from other department chairs!

The position of academic department chair is one of the most challenging there is. We hope that our work provides some support for their personal reflection and professional development.

It’s all in the name: “Student Ratings” versus “Course Evaluations”

Angela — Mon, 23 Apr 2012 22:03:44 +0000

By Steve Benton

Course evaluations - student evaluation of teaching - student satisfaction surveys - teaching evaluations. Faculty, administrators, students, and even experts in the field seem to use these terms interchangeably. However, The IDEA Center intentionally uses the term student ratings. Here’s why--

Evaluation connotes making judgments whereas ratings provide information or data that require interpretation. In a broader faculty evaluation process, students provide one source of information via their ratings.

To better understand the difference between student ratings and course/teaching evaluations, it helps to first distinguish between several concepts used in educational measurement. I’ll use the analogy of getting an annual physical examination to help clarify each. The purpose of an annual physical exam is to assess one’s overall wellness and to identify any health concerns. Assessment, then, includes all of the different ways of gathering information to be used in decision making. Similarly, in assessing the quality of a course, the instructor gathers information about overall teaching quality and student learning. During an annual physical exam, the physician takes several measures, both quantitative, such as blood pressure, and qualitative, such as asking how often we have been depressed during the past month. Such measurements involve the systematic assignment of numbers to some attribute. Physicians typically take multiple measures so that they can make an evaluation or judgment about our health. As with student ratings, those measures often include self-report data, such as patient reports of smoking and exercising behavior as well as the occurrences of various ailments.

In the same way that physicians collect multiple measures, instructors should provide multiple indicants of teaching effectiveness and student learning to help in evaluating their courses and their teaching. Besides student ratings, teaching portfolios might include peer ratings or observations, samples of student work, teaching awards, new course materials, grants to support or improve teaching, among other measures. After evaluating the evidence from these different measures, the instructor--as does the physician--engages in reflection and decision-making about what actions, if any, should be taken.

Evaluation, then, is a broader term than student ratings, because it includes all the ways that we might gather information about teaching effectiveness. Student ratings are but one measure. The IDEA Center highly recommends that other types of measures be collected and analyzed. However, student ratings should be a part of any course/teaching evaluation system. They are a necessary but insufficient part of the assessment process. We would certainly question the legitimacy of an annual physical exam if the physician neglected to take our blood pressure. On the other hand, we would be concerned if that were the only measure taken. In the same way, we should question the validity of course evaluations that do not include student ratings as well as those that rely solely on them for the decision-making process (we actually recommend student ratings should count no more than 30-50% of the overall evaluation of teaching effectiveness).

Granted, extreme scores on any single measure can raise concern—be it elevated blood pressure or low student ratings. Nonetheless, the wise doctor and instructor will examine more deeply beyond the single measure to uncover possible underlying problems.

Resources
Using IDEA for Administrative Decisions

References
Benton, S. L., & Cashin, W. E. (2012). IDEA Paper No. 50: Student ratings of teaching: A summary of research and literature. Manhattan, KS: The IDEA Center.

Hanna, G. S., & Dettmer, P. A. (2004). Assessment for effective teaching. Boston, MA: Pearson.

Seldin, P. (2006). Evaluating faculty performance: A practical guide to assessing teaching, research, and service. Bolton, MA: Anker Publishing Co., Inc.

What are IDEA Papers?

Angela — Mon, 16 Apr 2012 22:38:12 +0000

By Bill Pallett

One of the ways The IDEA Center serves its mission and the higher education community is by providing IDEA Papers. The series was initiated by the former director of the Center, William Cashin, in 1979 and continues today. The papers focus on topics related to improving teaching and faculty evaluation. Authors are IDEA Center staff members and leaders in the areas of faculty development and evaluation. The papers receive frequent praise for their high quality content, succinctness and practical value. Recent titles include: Promoting Deep Learning, Strategies to Improve Student Writing, Effective Classroom Discussion, and Student Ratings of Teaching: A Summary of Research and Literature.

Hundreds of teaching and learning centers have links to these resources, and the papers receive thousands of “hits” annually. The papers are free and we encourage you to make use of this valuable resource. We invite you to browse the titles of the IDEA Papers today and see which ones might be particularly helpful for you and your colleagues.

Happy reading!

How do I get better ratings? Think about student learning.

Angela — Tue, 10 Apr 2012 04:38:31 +0000

Dr. Amy Gross joined The IDEA Center in 1997 and has worked to provide exceptional service and rich resources for a continually growing list of clients nationwide. Now Vice President for Research and Development, Dr. Gross is responsible for overseeing the organization’s research and development initiatives to support its mission, vision, and strategic plan.

After working with many campuses and individual faculty over the last 15 years, a single question is consistently asked: How do I get better ratings? With my personal passion for personal reflection and professional development, I can get disheartened by the “bottom line” question. But, I also understand our need to do well and wanting to see positive results for the “bottom line” questions related to excellence of teacher or excellence of course.

A few years ago I gave a presentation at the Professional and Organizational Development Network in Higher Education (POD) titled, “Getting Better Ratings by Learning to be Better Teachers.” I have to admit, it is one of my favorites because the research is so compelling, exciting - and by thinking about “getting better scores” we can actually improve student learning!

The Basics
The presentation was based on research using the IDEA Student Ratings of Instruction instrument which includes 20 teaching method items, 12 learning objectives, 2 summary items (excellence of course and excellence of teacher), and a number of student and course characteristic items. The details of the research are available in IDEA Research Report #4: Teaching “Styles” and Learning Outcomes and IDEA Research Note #1: The Excellent Teacher Item.

One of the most interesting findings to me is what we learned about predicting student ratings of overall excellence of course and overall excellence of the teacher – the two summary items that seem to appear in some form on nearly all student ratings forms.

The Results
In a nutshell (but I encourage you to the read the details), the research (using nearly 45,000 classes) found that when predicting student ratings of the course, the 5 teaching approaches (comprised of the 20 teaching methods – or what the instructor does) accounted for 73% of the variance. Teaching approaches accounted for 85% of the variance when predicting teacher excellence. Even though these studies are restricted to the questions we ask, those percentages are quite respectable. If there was no meaningful relationships, then the percentages would be much lower.

The most important Teaching Approaches for Excellence of Teacher – all of equal importance - were:

Stimulating Student Interest (i.e., demonstrating importance of subject matter, introducing stimulating ideas about the subject, inspiring students to set and achieve challenging goals, stimulating intellectual effort)
Structuring the Classroom Experience (i.e., clear communication, scheduling coursework to encourage students to stay up to date on their work, making it clear how topics fit into the course)
Establishing Rapport (i.e., displaying a personal interest in students and their learning, finding ways to help students answer their own questions)

The most important Teaching Approaches for Excellence of Course were:

Stimulating Student Interest (primary importance)
Structuring the Classroom Experience (secondary)

Of additional interest was that if there was too much emphasis on Establishing Rapport, and the other two approaches were neglected, ratings were somewhat lower. So, if instructors are too focused on only their personal relationships with no substance, students are less likely to view the course as a good experience.

So…How to get better ratings?
How does this get back to getting better ratings by being better teachers? The five teaching approaches also predict student learning on the 12 IDEA objectives. So, if faculty members are motivated to improve their summary scores on student ratings and they focus on their teaching methods (what they do in the classroom) – they will not only improve their “scores,” but will likely improve student learning – what a bonus!

My personal preference is to focus on improving student learning – and the overall ratings should follow. But, we are each motivated differently.

As an aside, an important Teaching Approach for ALL 12 learning objectives – and THE most important approach for 8 learning objectives was Stimulating Student Interest. If we do nothing else – get students engaged!

Resources for Stimulating Student Interest
IDEA Item # 4: Demonstrated the importance and significance of the subject matter
IDEA Item #8: Stimulated students to intellectual effort beyond that required by most courses
IDEA Item #13: Introduced stimulating ideas about the subject
IDEA Item #15: Inspired students to set and achieve goals which really challenged them

Can students rate their own learning?

Angela — Tue, 03 Apr 2012 13:42:02 +0000

Dr. Steve Benton joined The IDEA Center in 2008 after a prestigious 25-year career as professor and department chair at Kansas State University. He is a Fellow in the American Psychological Association and American Educational Research Association, as well as an Emeritus Professor of Special Education, Counseling, and Student Affairs at Kansas State University. Dr. Benton is known for his passion and dedication to careful and meaningful research.

One of the most frequently asked questions about the IDEA Student Ratings of Instruction system is “Can students really rate their own learning?” This is an important question because one of the underlying assumptions of the IDEA system is that student learning is a key indicator of teaching effectiveness - and the IDEA system asks students to rate their progress (or amount of learning) on 12 learning objectives. But...this is a difficult question to research because student ratings are anonymous, making it impossible to compare their self-ratings to actual measures of learning.

However, with the help of a professor and his students at a large university in the Southern U.S. we were able to address this question. The research results (Benton, Duchon, and Pallett, 2011) were recently published in the journal Assessment and Evaluation in Higher Education. Specifically, we examined the relationship between how students rate their own learning progress and their performance on exams.

The professor taught three sections of the same course and used five common exams. At the end of the course, after students had completed all of the exams, they rated their progress (from 1 = no apparent progress to 5 = exceptional progress) on the 12 IDEA learning objectives and identified themselves on the survey form so we could match their self-ratings to their exam performance. Independent of the students, the instructor identified 2 of the 12 objectives that were were addressed in the course.

Research Results

On the two IDEA learning objectives the professor emphasized in the course, student ratings correlated positively with 4 out of 5 exams and the course total (r = 32). In contrast, self-ratings on learning objectives not emphasized in the course did not correlate with exam scores. Moreover, those students who rated their progress as either exceptional or substantial generally performed better on course examinations than those who rated their progress as moderate or less. These findings are comparable to those reported by Cohen (1981) and Feldman (1989) who reported average correlations of .47 and .46, respectively, between student ratings and final exam performance.

Summary

Although the correlations reported above aren’t perfect (see the study for a detailed explanation), the evidence cited here supports the notion that students can assess their own learning with some degree of accuracy and that student ratings can provide one measure of teaching effectiveness—how much students have actually learned in a course.

References

Benton, S. L., Duchon, D., & Pallett, W. H. (2011): Validity of student self-reported ratings of learning, Assessment & Evaluation in Higher Education, DOI: 10.1080/02602938.2011.636799.

Cohen, P. A. (1981). Student ratings of instruction and student achievement: A meta-analysis of multisection validity studies. Review of Educational Research, 51, 281-309.

Feldman, K. A. (1989b). The association between student ratings of specific instructional dimensions and student achievement: Refining and extending the synthesis of data from multisection validity studies. Research in Higher Education, 30, 583-645.

Q&A with Dr. Shelley Chapman on the updated IDEA Center Website

kyle — Fri, 23 Mar 2012 14:58:38 +0000

The IDEA Center has a newly updated website. The first blog post published to our new site was an introduction to the blog by Dr. Bill Pallett, President of The IDEA Center. This post captures a Question and Answer session with Dr. Shelley Chapman, Vice President of Client Services. Dr. Chapman led the effort to refresh The IDEA Center's home on the web.

Why did you decide to refresh the IDEA website?

First, we really wanted to help clients access information more quickly. The most common feedback we heard about our previous site is that some items were hard to find. The IDEA Center has lots of rich information and it can be challenging to organize it all in an accessible way. Our new site uses three main areas (Services, Support, Research and Papers) to provide a framework for content organization and has a very prominent site search.

We also were looking for ways to enable clients to communicate and share information with each other. The re-organized Support section of our website pulls topics from a new Help Community we have created with Get Satisfaction. In addition, we are hoping to tap into the power of social media in appropriate ways to engage more openly with clients.

What changes are you most excited about?

I am most excited about the new look and feel. It is much cleaner and organized in a more dynamic way. I'm also excited about the content organization and section cross-linking. Each page has a sidebar with links to related content. I'm hopeful that the new Support section will provide another way for clients to get help and a new way to engage with each other. Overall I think it really is better looking and easier to use.

What changes have you made to make it easier to get help for IDEA products and services?

A significant feature we have introduced with the new website is something we are calling IDEA Help Community. The feature enables clients to find answers, ask questions, or share feedback with an open community of IDEA clients. We hope this will complement our high level of existing personal touch of which we are very proud. It will open another way for clients to get help.

How does the website help me stay up-to-date on IDEA?

There are at least two new ways we hope to make it easier for clients to stay up-to-date. One is through Twitter. We invite you to follow us on Twitter where we will be sharing new information about IDEA, upcoming seminars, product announcements, new research, and new ways of using IDEA. We look forward to engaging with you.

Another way to stay up to date is to subscribe to our blog. Our new home page shows the most recent posts and enables subscription via RSS or Email. If you subscribe by email, you will get an email each time a new post is created. Our current plan is to post about once a week.

Also note that we will be featuring the latest IDEA Paper on our home page. We want to make IDEA Papers easy to discover and easy to be recognized since we know they are a highly valued client resource.

IDEA has a reputation for providing valuable research and papers. Are these still available and easy to access?

Yes! As I mentioned in the previous section, our new home page features the latest IDEA Papers and we have a specific area in the Research and Papers section of the site reserved for IDEA Papers.

We are also very excited to be working on digitizing our research papers. Currently they are available for download. In the future we plan to make them easier to read on the web, search, and share with others, yet still be easily printable.

If I'm interested in IDEA, how can I find out what others think of your products and services?

There are a few specific ways we are hoping to share an authentic view of client perspectives. One is through our new Help Community where clients can openly ask questions and share feedback. Another is by reading the conversations on our blog posts or following our Twitter updates. Last, we are always willing to connect you with existing clients to discuss their experience with The IDEA Center.

Any other thoughts?

I do want to encourage our clients to participate in the new IDEA Help Community. We really hope it will grow organically over time via client participation and become a rich source of help topics for IDEA products and services. Please feel free to join us and ask questions. We are monitoring the community and are committed to providing timely responses.

Welcome to the new IDEA Center Blog

kyle — Thu, 22 Mar 2012 17:42:54 +0000

Dr. Bill Pallett, President of The IDEA Center, provides an introduction to the Center’s new Blog, describing its purpose. Dr. Pallett has served as Director, and later as President of The IDEA Center for the past 15 years. Today, over 400 campuses of all sizes and types avail themselves of the Center’s Student Ratings services.

Welcome to the new IDEA Center Blog. We believe the Center is in an ideal position to contribute to and encourage conversations about teaching and learning, student ratings, faculty evaluation, faculty development and administrator feedback tools. Not only does the Center conduct significant research in all of the above areas, it also has contact with faculty and staff from 400 colleges and universities who use the IDEA Student Ratings system. These Center clients serve as an outstanding resource by providing many useful recommendations and helpful stories—stories we believe merit being shared. However, this blog is not intended to be a one-way dialogue, but rather a fruitful conversation. We look forward to your helpful comments and suggestions in response to the blogs we post.

Finally, I would be remiss if I did not mention what this blog is not. This is not a place to engage in lengthy debates about the value of student ratings or administrator tools. Rather, it is a place to learn how to make best use of well-designed feedback tools to support our improvement efforts.

Some of our upcoming blog posts will include topics such as the following:

Getting better ratings by being better teachers
Feedback for Department Chairs—Why have a Chair Coaching System?
Misconceptions about Student Ratings
What Questions should you NOT ask on a Student Ratings Form
Effectively using student comments

We look forward to your active participation.

- Bill Pallett