Educational Advancement Foundation



01 8-2-16 Opening Remarks (Douglas Bressoud Hodge)Dr. DouglasGood afternoon. I think we should get started. I'm going to start out by saying a little bit about the origins of this meeting, and then we will talk a bit about the logistics and how it's going to be conducted.I've been teaching mathematics for more than 50 years and the way one teaches mathematics and the possibilities and the way the fads have come and gone, and one of the rubrics we have been using that we did not invent, is the notion of active learning.I've been involved directly in one part of it, one kind of it, called inquiry based learning, for the last ten or 15 years. About five or ten years ago the Education Advancement Foundation decided to fund a little bit of assessment and evaluation, a little more in depth than is often done, and that was Sandra Laursen's group at University of Colorado that did this, and the information we got back was what we had hoped for of course, but we also got some surprising information, or what seemed to be surprising information to us at the time, and there was a meeting in Washington so I took the opportunity to meet with Joan Ferrini-Mundy, the head of NHR and NSF and I told her about these findings or these results or what we thought were findings and she was enthusiastic but she thought what would be good at this point would be if one could encourage more people to do assessment and evaluation of active learning and not ask the somewhat tired question of "is this better than lecturing" or "if we teach calculus this way or that way, who will get the higher grades on the calculus exam?" The idea is that in fact there's a lot of agreement that active learning approaches are at least as good and in many cases the demonstration and results are better than simply the lecture method. Let's assume that, let's not spend all our time worried about that. Rather, what we should spend our time with is what works, what's best, perhaps what makes active learning a more powerful method of learning for certain groups of students. Some of the findings which one found were, for example, for women the findings we found out were if you look at men lecture versus IBL it turned out it didn't seem to make much difference, the results were the same. But for women the results were quite contrary. In other words, the results using active learning or inquiry based learning were much better than the ones by lectures. And the same thing was true for a bunch of other things. So in fact, following her advice or following her suggestion I, over the last couple of years, have tried to organize a group where we bring together people who were involved in the mathematics community, teachers if you like, people who were interested in teaching, and people that were in the assessment and evaluation community, and try to get them to compare notes as to what we know, what kinds of questions one should ask, and what answers one should expect, what the obstacles are to introducing active learning in other places, in other words just the whole general circle of questions. And that's what we are here for now. The Sloan Foundation, the National Science Foundation, and the Educational Advancement Foundation have provided modest funding in order to bring you here together. The plan of action is the following. There will be a panel today where we will be focused on what active learning is, what the obstacles are, some of the bread and butter issues about that. There will be a panel tomorrow morning that will focus on assessment and evaluation, what we know both in terms in mathematics and perhaps related areas, what questions we need to ask, in other words, not so much a review of what is known, but what questions we should be asking and trying to get answers to. And that's why you are here and I welcome you here.We've got the two panels we start today, and then two tomorrow. After the presentations the plan is to have interactive back and forth questions and answers. There's a microphone in the middle of the room for people to ask questions and we'll do this to some extent as we go along. but we'll take the opportunity at the end to try to pull things together and try to focus on what we need to know, what knowledge if one could obtain it, could be useful in advancing how mathematics is taught in all levels. The focus for much, of course, it is the collegiate level, but all things start back at the K-12 so some of it will be on one area and some of it will be on the other. But I encourage those in the audience who aren't on the panel to ask questions. If you are not an expert then of course you should have a lot of questions, if you are an expert then, in fact, you can disagree, there's no complete agreement on lots of these questions and that's part of the purpose of doing this.Before we get started I'm going to ask Angie Hodge to bring up a little about logistics of the meeting.HodgeHello, I have just a few reminders. One, we will end the day today at 5:45 PM. To the panelists, just a reminder that there is a 6:30 PM meeting. If you have forgotten where, ask myself, Ron, or Norma and we can remind you on that. And then tomorrow morning breakfast will be in this room from 7:30 AM to 8:30 AM and all are welcome to attend that. We hope you enjoy your couple of days and hope to get to know you during some of the breaks if we don't know you already.DouglasIf you have questions about anything you can ask me, you can ask Norma who you met coming in, or in fact Angie or some of the other people. But let's get started now, in other words, David Bressoud, who will be the leader for the first panel, and I want to acknowledge thanks to him and Doris Zahner who will chair the panel for tomorrow morning, who have assisted me in putting this program together, so David.BressoudThank you, Ron, and welcome to everybody on what I hope will be an informative two half-day sessions. We have left lots of time for discussion and questions and I hope we will have lots of lively discussions going on within this session. So we are looking at active learning. The first part of this afternoon's sessions will look at what is active learning, and one of the resources I recommend to everybody if you haven't already seen it is the Common Vision report. This is a report that was put together by five of the major postsecondary societies: the American Mathematical Association of Two Year Colleges; the American Mathematical Society; the American Statistical Association; the Mathematical Association of America; and the Society for Industrial and Applied Mathematics, and it summarizes what each of these societies has been doing, the kinds of reports, the kinds of recommendations that they have come out with. I am pleased also to report that these societies as well as the other professional societies in the mathematical sciences under the umbrella of the Conference Board of the Mathematical Sciences are now working on a common statement in support of active learning procedures that all of the societies, their presidents, now have this statement which Ben Braun has worked hard with a large team of people to draft and we're in the process of collecting their signatures on this common statement of support for active learning.Increasing recognition of the importance of active learning. I've been running studies of what is going on in the precalculus through calculus sequence in the United States, beginning with the survey the AMS, or the MAA rather, conducted back in 2010. We did another survey of universities with graduate programs a little over a year ago, a survey that was filled out by chairs of the departments. We had earlier identified eight characteristics of successful programs at the first year level, things like attention to placement procedures, work on graduate teaching assistant preparation, the student support services, challenging courses, monitoring local data. One of the pieces of information that came out of this that I think was very encouraging was when we asked departments of universities, these are research universities offering a PhD in mathematics, how important was active learning to a successful program and over 40% of the research universities said that the use of active learning was very important. Now you may say that's less than half of them think it's very important but I can assure you the percentage was nowhere near 40% five or ten or 15 years ago. I see this as a real trend that is extremely encouraging.We also asked them how successful they considered their departments to be in actually implementing active learning and only 10% of the research universities felt they were very successful in implementing active learning and I see that as an enormous opportunity. We've got almost a third of the research universities, so these are the R1, R2 universities out there, who understand active learning to be very important but they are not yet very successful at this and so this I see as a very opportune moment for really pushing active learning, understanding what it is, how we can make it effective, and of course the first step in making it effective is really understanding what we mean by "active learning". When I've presented this slide in other venues that's usually one of the first questions that comes up, "what do you mean by active learning?" Our first group of panelists this afternoon, will be digging into just that question, what do we mean by active learning. Our three panelists for the first session are Ben Braun from the University of Kentucky, who I have mentioned has worked as the lead author on drafting the common statement the professional society presidents are now signing, Angie Hodge from the University of Nebraska-Omaha who has been working with inquiry based learning and other active learning approaches very intensively and has been a real national leader in this area, and finally Michael Starbird of the University of Texas at Austin, who is well recognized for his work in inquiry based learning and active learning more generally.So with that I will ask the three panelists to come up and then we'll start with Ben.02 08-02-16 Benjamin BraunThanks everybody for coming. I want to thank the organizers for putting this symposium together, and I want to thank David for asking me to speak because this is really exciting. First, I want to ask you all a few questions just so I can get a sense of who I am talking to. How many of you, and you can answer yes to any of these all or none, how many of you are interested in active learning primary as somebody coming who's really interested in inquiry based learning, the IBL community? Raise your hand if that's true. Alright, how many of you are coming because you are really interested in active learning in large lecture environments? Raise your hands. Alright, how many of you are really interested in active learning in small classes but not necessarily inquiry-based, but small classes? Alright, and how many of you are here because you are interested in conducting research about active learning? Alright, cool. That's pretty broad. That will work.The first thing I want to do is ignore David's question. I don't want to ask "what is active learning", and I will explain why. Instead, I would rather ask "what does active learning do?" and so let me tell you why I think this is the right question.We are going to start with some definitions. Some various definitions of active learning exist in the literature and I'm going to read two of them. I just grabbed these from a couple of research papers. One says, "Active learning is generally defined as any instructional method that engages students in the learning process; in short, active learning requires students to do meaningful learning activities and think about what they are doing." And another research paper says, "Active learning engages students in the process of learning through activities and/or discussion in class as opposed to listening passively to an expert, it emphasizes higher order thinking and often involves groupwork." And so a really important thing to realize here, and I think everyone is probably aware of this, but if you are not, and I'm going to assume not everyone here is an expert in active learning so if you are, bear with me. There's not a unique definition of active learning: not in the research literature, not used in the popular use in newspapers or anything like that, and, in my opinion, all the existing definitions are inherently vague. So I find these to be very vague statements. And everything I've ever read, including things I've written, have been quite vague. So why is that? Because it is really hard to define active learning, so there's no simple definition of active learning that can simultaneously and effectively address the range of active learning techniques used across a diverse classroom environments, of institutional expectations for faculty in diverse employment contexts, and for, of course, student learning outcomes across different institutions and departments. And so when I say that what I mean is we have people either using or trying to use active learning in large enrollment lectures with recitations, in service learning courses, small enrollment lecture courses, inquiry based learning courses, and online courses I mean, that's a huge range, right, with lots of different constraints. And the people who teach those courses are in lots of different employment contexts. We got people who are long term contract faculty, at the two year college level especially but even at regional and big state schools we have a lot of adjunct and part-time faculty, we have tenure stream faculty, there are people teaching courses who aren't technically faculty but they're administrative staff at a lot of places and all of these different employment contexts and employment lines have different institutional reward systems, they have different institutional expectations and so we need to make sure that what we say active learning is is something people can actually do and be rewarded for. If we shoot too far away then nobody's going to buy in at the moment. And then finally, in the courses people are teaching in all of these environments, the courses themselves have a lot of different purposes. There are courses that are just to satisfy some gen-ed requirement, there are ones that satisfy a more well-thought-out quantitative literacy requirement, courses for non-STEM majors that need some quantitative stuff, STEM majors, math majors, right? So for me...I'm at the University of Kentucky and when I teach I will teach in a given year a PhD level graduate seminar with seven students, a junior-senior level course for math majors and minors with 25-30 students, and a freshman engineering calculus course with 180 students where I have TA's running recitations. And I do active learning in all these courses but I've learned from experience that the same methods that work for my undergrads do not work for my graduate PhD courses and the methods that work for my PhD courses are literally not implementable with 180 students, okay? Period.Another consequence of this is, of the vague definitions, right? So we got vague definitions and they're necessary, but a consequence of those is that faculty, administrators, public policy makers, student advocates, other stakeholders in postsecondary mathematics often talk past each other when we're talking active learning. I can't tell you how many conversations I've been part of where I'm saying active learning, somebody else is saying active learning. Whether they are a proponent or skeptic or whatever, we are just thinking different things and we don't have the right language. And it causes a lot of miscommunication. I think that better conversations can occur if we define active learning not by trying to say what it is but what it does in a specific context. And this is inherently more complicated but that is because active learning is complicated. So I'm going to tell you what I see as the way I like to talk about active learning at this point in time, and in six months I'll change my mind because I'm always changing my mind about these things. I think that the right thing to do before we talk about active learning first we have to talk about what it means to be mathematically proficient. And then we have to talk about the course we're teaching and what the student learning outcomes are, and those conversations can be used to define what an active learning technique is. The rest of the talk is going to be figuring out what I mean by this. So let's talk about mathematical proficiencies and SLO's. At the K-12 level a lot of people have thought about what it means to be mathematically proficient and a very well-known, highly regarded framework for proficiency was given about 15 years ago in the National Research Council report "Adding it Up" and they emphasized a five strand model. This was primarily focused at K-8 and was informed by math education research, cognitive psychology, theories of learning, expert practitioners, all sorts of people. And the five strands are: conceptual understanding, procedural fluency, strategic competence, adaptive reasoning, and productive disposition. And I don't want to go into what all these things mean, but I will say that that chapter in the report is well worth a read by everybody, including postsecondary math educators. And when I read that chapter, when I read these things, I think to myself "that's a pretty good description of what I want my engineering calculus students or my linear algebra students or my number theory students to do in a given context with mathematical content." And I think this framework works really well for high school and at least the first two years of postsecondary math ed and it can be a base to build on for higher up. And I think one of the reasons it works so well as a framework is that these five strands don't only talk about mathematical content but rather they reflect what we know from modern psychology about how the human brain works and the human psyche works. So modern psychology provides a basic framework for the human psyche with three domains: the intellectual functioning domain, the behavioral domain, and the emotional functioning domain. So the intellectual domain is what you know, like your content knowledge and your understanding of it, the behavioral domain is what you do with how you act on that, and your emotional function is how you feel about what you know, what you do, and what you feel, it's kind of self-referential. And this five strand model of proficiency, if you read through it you'll see it really does cross all these domains, and this is a really big simplification but when I give this sort of talk in front of psychologists nobody screams so I think it's okay. And just as an aside, I put this on the slide because I didn't want to forget. Many math courses that I have taken, that I have taught early on my career, that I've seen people teach and observed, focused primarily on the intellectual aspects of student learning, it's about content knowledge and understanding. And most of the active learning techniques that people are using engage students across two or three of these domains. And under the hood of active learning, what really makes it work and one of the things that make it really hard is that it makes teaching being about the intellectual domain and it opens up this can of worms that you get with behavior and emotion. And that's a really hard gap to get past for a lot of faculty and students. And I think that's one of the things that is a barrier, and we can talk about barriers later, but I think that's a big deal.These two, the "Adding It Up" report and the psych literature, aren't the only places where this kind of thing shows up. The Common Core States Standards for mathematics has a broad description of mathematical standards including behavior and content. Also the 2015 MAA curriculum guide to majors in mathematical sciences has four cognitive goals and nine content goals that span multiple domains of these psychological domains. So you can anchor yourself in multiple places, but any robust framework for proficiency is going to span these multiple psychological domains.So once we've had a conversation with our peers about what we want our students to do beyond just learn this one technical thing in math, then at the course level we should make sure that every course should have clear student outcomes that are representative of all these psychological domains and all the components of some robust framework for mathematical proficiency. To show you an example of this, at University of Kentucky a few years ago we decided to make number theory a prereq for a number of upper division courses, and number theory serves as our intro to proof course, and along the way the department had a conversation and after a lot of robust discussion we recommended the following SLO's for the number theory course. The first line is "students will deepen their understanding on the following topics and improve with regard to the following practices". There are ten items. The first five are about mathematical content. The first four are topics in number theory and the last is methods of proof. The last five are behavioral and emotional aspects, things like being persistent, working through perceived failure, productive collaboration, developing communicating proofs, things like this. So whenever somebody teaches number theory, we give them this and say this is what the department has recommended. Not everyone has used exactly these, but it has certainly made an impact on people's choice of teaching style pedagogical techniques.Once we have had these discussions at the local level, then I think we are ready to actually talk a little more consistently about active learning. I, at this point, am always talking about course specific definitions of active learning. In a given course, an active learning method is a classroom teaching technique in which students complete a task or activity directly supporting development in one or more student learning outcomes, one or more domains of mathematical proficiency, and one or more psychological domains. The two key things are that the students complete the task or activity, and that it happened in class. A lot of people say students should be doing this outside of class, that is something they should know how to do. But an active learning technique is something that happens in class, and the students do something. In my opinion, our goal for each course is to incorporate multiple active learning techniques that collectively support development across all the SLO's, domains of mathematical proficiency, and psychological domains. So it's not that we are looking for one technique or one way of teaching, but rather that in a course with a set of outcomes we have a set of techniques that cover those outcomes in class. I've got three examples. The first one would be procedural computation in large lecture Calc 1. Say you are someone who has not used active learning before and you are just trying to get an inexperienced person that is willing to try active learning but that just don't know what to do. What's the first thing you get them to do? To support the intellectual domain, the procedural fluency and the ability to use derivatives, when you are working a simple example in class take one minute to have the students compute the derivative of a polynomial independently. This can be done with clickers or without. I don't use clickers myself. You could make it more sophisticated if you want, but for a lot of faculty who have never done this and they have 150 students in front of them, this in of itself is a toe in the water, even though to me it's like you aren't doing much, but you are doing a lot if there are other things going on that cover the other psychological domains and the other proficiency domains. A slightly more ramped up example, and the next two examples are both from number theory because I showed you our SLO's and I chose number theory. Suppose you have number theory that's taught in a small class but by somebody who really does like to lecture but they want to use some active learning. They could use a think-pair-share. If you don't know what that is, ignore it. It means have the students talk for a bit. So to support the behavior and emotional domains, conceptual understanding, productive disposition, and productive collaboration with others, you could early on when you prove the infinitude of the primes ask the students to use Euclid's proof of the infinitude primes to produce as many new prime numbers as possible starting with only the prime 3, and you give the students three minutes to work independently on that, and there's some choices along the way so hopefully different students will get different things. So then they get three minutes to talk, so they think, they pair, and they share. They get three minutes to talk to one or two of their neighbors in the class about what they got. And that's a six minute active learning technique, and again, this is something that I think does benefit students a lot and doesn't require Example 3, which is an IBL-style small group activity in number theory. This is like full-on inquiry based learning, supporting everything all at once, and some of you may have experience with this, some of you may not. You can assign students to small groups, give each group a theorem with a 15-line proof where each line is separately cut out, you mix them all together, and the proof has one fixable error. And then the first must collectively reorder everything to reconstruct the proof and then they have to identify and fix the error. Now that's high cognitive demand and requires a lot of productive conversation and collaboration. That's the kind of thing that's really effective if you have a class of 25 and you've got the students bought into this kind of thing. I think a handful of people can do this with a bigger class but I've been unable to do stuff like this with my 180 student classes.So the last thing I have to say which is an important question because everything I've told you is like my giant active learning fantasy and so here's the reality that punctures it and brings it back down to the ground. The literature has many papers studying the aggregate impact of an active learning technique, how do we determine whether or not if a specific teaching technique in a specific classroom environment supports a specific SLO proficiency domain or psychological domain? And the answer is I don't know. And it probably isn't possible because human research is impossible, as many of you in the audience may know. To my knowledge at this time many faculty using active learning techniques make their choices about techniques based on their experience, their intuition, educated guesses that have been informed by reading the research in math education and psychological, and it would be interesting to know if we can build up the next generation of active learning techniques. I don't even know if any studies have been attempted, let alone published, to tease out this level of fine detail in terms of the efficacy or impact in terms of given techniques in given environments. We certainly know a lot about aggregate stuff. That's something I'd be interested in. So I will stop there. Thank you very much. (applause)BressoudWe will save the discussion for later but if there are any clarifying questions for Ben I'm sure he would be glad to answer them.(inaudible audience)BraunThe question was, “Do I give my student learning outcomes to the students?” Yes and no. In any courses where it is not a coordinated machine, yes and I listen very explicitly. I usually have eight to ten student learning outcomes and I usually tell my students that they are content and practices. And usually along the way I will give a written assignment. I use a lot of reflective essay assignments in my courses with less than 35 students and I will have them write some kind of reflective essay mid-semester about their progress on the SLO's in the syllabus so they have to revisit the SLO's. In the large courses we have a coordinator for our large lecture course and to date we have never had good SLO's but for better or worse I'm coordinating calculus this fall and I can tell you that we will have some better SLO's but what we won't have is any explicit tasks built in to online homework or recitations that deal with that because we have multiple faculty teaching lectures, we have 15 to 20 grad students. We tend to put our new grad students as TA's in calculus, like half of them. So there's a lot of indoctrination and training going on. I've had a lot of difficulty figuring out how to do this kind of stuff at the big level, at scale.BressoudSo let's thank Ben again. (applause)03 08-02-16 Ange HodgeI'm going to start out a little bit talking about myself, even though I have "who are we" on here, and I'll explain that in a second too. For those of you who don't know me, and I know probably half the audience, so the other half of the audience. I'm an associate professor of mathematics at the University of Nebraska Omaha. I also have a Haddix Community Chair position there which means it's my job to recruit people into going into math, going into math education, getting the community excited about mathematics and that's one of the reasons why I decided to start doing IBL at the calculus level. I had done IBL since 2007 when I became a Project NEXT fellow and right away in my first class at North Dakota State I started using inquiry based learning and I haven't turned back since, although it took me a lot longer to do IBL in my lower level classes such as calculus, differential equations. I started out more doing active lecture in those and then about four, five years ago I started in the calculus sequence and so what you are going to see what I do in my calculus class took a while to get there but I think it's, well there, we always want to make it better but I think it's at a good spot right now.So that's how I heard about inquiry based learning was being a Project NEXT fellow. In 2007 one of the workshops I went to at the beginning was an IBL workshop and I also went to the Moore Conference in 2008 and that got me started as well. And then right now I'm the Special Projects Coordinator for the Academy of Inquiry Based Learning so if you have any questions about getting interested in it. You are here probably because you do some IBL but if you don't already and you want a mentor, have any questions, feel free to ask me and I can help you be paired with that as well. And in my free time, for those of you who don't know, I run ultramarathons. My math and my running are the two things that keep me going.Actually before that, the "Who Are We". I did IBL at my other higher level classes, in geometry classes for teachers, in history of math, in proof space classes but it took me a long time to be brave enough to try it in calculus until I got to the University of Nebraska Omaha and met a colleague of mine, Janice Rech. Stan Yoshinobu, Dana Ernst and I brought an IBL workshop to Omaha off of a small grant that we had and she was all in. She was like "I want to try IBL and I want to try active learning in calculus and I will meet with you weekly, I will help create notes, get this going." So it's not just me that does this, it's “we”, and the “we” has grown since then but that was what really helped me be brave enough to try this in calculus.So, why calculus? I think the answer to this question is probably pretty familiar to all of us that's here because it is a gateway course for the STEM disciplines. We see a lot of people drop out of these courses. The drop rates are very, very high usually. It's usually, "Am I going to go into a STEM field or am I going to drop and switch my major to something else" in terms of taking the calculus sequence. And many of those who aren't successful in Calc 1 then continue to struggle in Calc 2 and it continues to spiral down if they even get that far, if they don't drop. And the third one is related to my own position and trying to increase the number of people going into mathematics and in particular mathematics education. I had to have some sort of draw in going to Omaha, what was I going to do to reach students, and while I was at North Dakota State one of my colleagues taught Calc 1 large lecture and was able to bring the math club at North Dakota State from something that was nothing to huge. She had a crowd of about 60 people at 4pm on a Friday night all recruiting from Calculus 1 so I thought, "Why not go into there, show them what mathematics is and hopefully more people will go into math and math ed,” and they have.So what does IBL or active learning calculus mean to me or in my classes or at Omaha or the people that I have mentored and have been engaged with? Students are actively engaged and I would argue not only inside of class but outside of class. They are actually doing the math. It's not just me telling them what to do, they are the ones actively engaged in the material and really thinking like mathematicians as they are doing calculus and problem solving in their head. I give few traditional lectures. Every once in a while there's a five to ten minute mini-lecture if everyone is just completely stuck and that's usually in terms of creating the materials. If we thought a worksheet was good, but there was a part that just bombed and I needed to fix so I fixed that so hopefully the next year that's better. And or sometimes there's a five minute mini-lecture on a proof that might be a little bit beyond them and I just want them to sit and listen for a minute but very few the lecturing. I would say in a 65 minute class I'm up at the front five minutes total. If it gets to be ten minutes my students are wiggling and they're like, "What are you doing? Go sit down and let us do some work here." Activities are selected to guide students to develop their understanding. We very carefully been revising and writing materials to go with Calculus 1 and Calculus 2 for the last four years. They are really coming along but it's taken awhile. So what does it look like? This is just one way to do active learning. When Ben was up here he talked about how active learning looks different in each of the classes he taught. I always teach graduate courses for practicing teachers ranging from elementary school teachers to high school teachers and it looks different in those classes too so this is just in a calculus class and would be similar, I would say, in a precalculus, differential equations, anytime you have a larger group of students. I have 40 students in my classes. We were able to bring the cap from 50 down to 40 in order to do the inquiry based learning. What's most important I think is the structure of the room. If you can, I know that you can't always do this and we've had people do IBL calculus at UNO with desks and they have been able to do it successfully but it's harder than when they have tables. If the students walk in and they see ten tables with four chairs at each table there's a different atmosphere, there's a different feeling when they walk into that classroom. There's white boards all around the room and those things are two things that I think are really important. If you can, convince your university or walk across campus. We had to do that for a while to get a room that actually looks like we are going to work together and talk to each other. And like I said before, a little introduction. The students would get a little worksheet or an activity that's a tactile activity and that is on their desk when they get there and they're working right away. They are using working before class starts because they know what to do. Or if there is a little bit of mini-lecture that has to happen I usually write that on the board so they know that they have to wait a second to get going but they're usually ready to go right when they get started.Since I am teaching a calculus class and I am at a university where some of our professors do active learning but some of them don't we have a standard university textbook. There's not much that I can do about that. I'm on the committee but still they want a standard textbook. It doesn't matter what textbook you are using. We use it as a problem bank. Our students, if they want to watch a lecture after class they can because there's, you know, just like Hahn Academy there's a series of lectures that go with that if they feel they still want that afterwards. But for the most part that is a problem bank for them and a place for them to read if there's something they don't understand. So don't let that be a barrier if you are somewhere where you have to follow a certain textbook or a certain amount of chapters. We have to and it works. It doesn't hinder us. I'll just give you a few things about my class and you can ask some questions if you want to afterwards. Group structure, I think everyone will answer this question differently. I put it up there because everyone always asks, "How do you structure your groups in your classes?" And my answer is different than other people. I've been on a panel with four or five other people that all do active learning in their classes and we all answer differently. My answer is I let my students self-select. I use the first two weeks and especially in Calc 1 I have to teach them how to do this, to shop around. I tell them they are not allowed to sit with the same people at all the first week. They have to get up, move with somebody else, and then the second week they can settle into a group that they like. I let them kind of pick and choose until they find a group that they are comfortable with. And I try to break them up after tests. I have another instructor who does active learning at UNO in calculus and she mixes them up after every test and she's successful with that. I'm too much of a softy. I'll mix them up and then the next day they will be sitting back in their old groups kind of giggling at me. "Ha ha, we've snuck back in our old groups" and it doesn't matter to me as long as they are learning. By self-selecting I found something very interesting. I have always two tables in my room that fill up with what I call my slacker tables and especially one table. I don't know why every single year for four years this table has been, it's in the corner, it's the farthest away. It's been the students that's the weakest students and at first I was really worried about that but over the years I've learned that slacker table, eventually someone's got to do something if they want to pass the class. They come to class every day and this year was probably the most remarkable. I had four boys sitting at my slacker table this year and about a month left to class they're like, "We really got to learn this if we are going to pass this class" and one of them just got in there and every day that last maybe five weeks of class he would get his groupmates working, he would get them going, he would make study groups after class. They all passed and they were not passing for a long time. I was really worried, wondering why they were still coming to class, they hadn't dropped. But eventually they became friends. They came every day and they realized they were a team and if they were going to pass then they had to do this. And they actually all came back for Calc 2 and they passed Calc 2 as well and they actually did pretty well actually in Calc 2. So I used to be scared about that table but now I know that eventually that they will grow together if you let them bond. (audience) Do you tell them it's the slacker table? (laughter)HodgeNo! But at the end of the year, at the end of Calc 2 or even at the end of Calc 1 I do kind of use that word. But I was like, "You guys were all being kind being a little lazy and now you are getting it all together" and they were like, "Yeah, we decided we wanted to work". OK. (laughter). So at that point of time you get comfortable with them. So not in those words but kind of. And I think the biggest things with group is as long as you are consistent. If you're going to switch every day, switch every day. If you're going to switch after every test, switch after every test. If you don't care then let them go. Something else I see with groups is that I often have one student who doesn't fit in with her or his group and then I ask them if they want to join a different group and that helps as well. So you have to watch a little bit and can move things around but for the most part I let them work together.What do I do otherwise that's important for active learning? Making them do work every day in class every day, out of class. If you take my class you are going to work a lot of hours. You're going to be doing mathematics. Even the slacker table doesn't slack. They still have to do so. So every day they turn in homework, graded for completion, 0, 1 or 2. Did you do it? Did you not do it? I'm pretty lenient on that. Just did you do it? And you can work with other people. Every week they have a take home quiz and my students say it's one of the most valuable things. We do that to save time. I don't give quizzes in class. They take that home. We give it on a Thursday and they turn it in on Monday. They can work with any humans that are in the class. They can't bring it to our learning center or anything like that and that is really where I think their learning occurs. They really have to talk to each other and really have to think about what it means to do mathematics with each other. Two part tests: one in-class individual. There still needs to be some individual accountability, I believe. So that's a standard, looks like anyone else's class. Very procedural knowledge. And then the take-home is more conceptual and that part again is given Thursday, due Monday. They can work with other people in class. And the tests are significantly harder before than when I didn't do IBL. Now I'm able to give two full tests, double. And ask a lot more questions. I find it harder to the final exam because that's only in class than when I do have this time to ask them more questions.The other part that people always ask about is participation. Every day I take attendance. They need to present. We do student presentations besides the groupwork. 5% of their grade is participation. It doesn't sound like a lot but it's enough to for my students to get them motivated. If you have a smaller class you might be able to do a higher percentage. All they do is, in Calc 1 did they go to the board or not. If you have students in an intro to proof or real analysis you might be able to do more with that than with Calc 1 where you mostly have engineers and computer science majors. They're scared to death to even go to the board so for some of them it takes them a lot just to be able to do that. So it's learning these other skills, just like Ben talked about, not just intellectual but some of these social skills as well and being able to communicate through mathematics. It's not up here, but I also have them participate in outside math activities. They have to attend, we call them cool math talks, you might call them colloquiums, seminars. They have to show they're doing mathematics outside of class as well. Group activities, I talked a little bit about this but I'll say it one more time, what are they doing in those groups. Our activities started being based off of AP calculus materials. My other colleague does a lot of AP calculus workshops and in fact that's why she got excited about it. She said, "I'm already doing IBL and I didn't even know I was doing active learning with the AP calculus." So that's where things started and now they have grown into our own notes. They started as notes people probably wouldn't be able to recognize, handwritten, copied and pasted from different things. We did the IBL this year, we tex'ed everything up, and getting everything in order to hopefully submit everything to JBLM in in the next year or two. In 2014 we added another component to the worksheets and that's tactile activities called "tactivities". With the University of Colorado-Boulder I went there to give a talk and I had one activity that was already a tactile learning activity and they wanted to meet and create more so now we have 20, 30 tactile learning activities for Calc 1 and Calc 2 and hopefully are going to create some more of them as well. So these are the things our students do in class. Here's a quick sample tactivity. If I had more time I'd have you actually do one but I'll show you the website where you can see them. Things like, they have function, the derivative and maybe if you want to include a statement about that function and derivative. And they have to match these ups. So a lot of our activities are little of moving pieces where they have to match things, group things, put things in a certain order. We actually have an activity for definition of derivative that's similar to what Ben talked about where the steps are cut up for them and they have to put them in order. I like his idea of putting an error in there too. We don't have any with an error. But that's what some of the students are doing. An activity like this often takes a full class period, especially when you bring it back together and talk about that conclusion, what mathematics did you learn in there. Integral dominoes are one of the students' favorites. This activity is one where the students match up the integrals and the anti-derivatives. Some of them are very easy, like integral of E to the X dX and other ones they have to get out the pencil and paper, work together, talk to each other. They're harder. They have to review their trig identities or simplify it algebraically. This one would come at the end before a test where they're actually reviewing the things instead of just giving them problems or doing the problems at the board for them. We have some activities where they are actually engaged with the mathematics. And the other part of these is that they are all self-checking. This one makes a circle, for instance, so students can tell when they're done. I still walk around and check, but it's a way for them to say, "I've probably done this one correctly". If it doesn't match up then they have to find their error and that's something that takes a little bit of time.And then I mentioned the outside of class activity where they have to go to math club but here's a few other things that make this active learning. Instead of having office hours I have the hour before class designated to myself being in a room with tables and white boards and I help the students. I guide the students to answer their questions. I'm not up there doing them for them. If I have to miss for a meeting the students are helping each other with the homework questions so I don't spend the first 20 minutes of class anymore like I used to doing homework problems. I spend no time doing homework problems at the beginning of class. It's done before class and if there's a few that a lot of people had troubles with, the students will present those when they get to class so they have taken ownership of this as well. And this is really something about the help room, where the students bond and they form what a mathematical community is like, what it means to work together. They really benefit from this quite a bit.I talked about the group work on take home quizzes. Did IBL work in our calculus classes? We are working with our statistician right now to analyze some of our actual data to give you the numbers, statistical significance, things like that. But it's hard, as Ben said, especially as you are creating these materials to find time to do the research on it. At first they were slow to accept this but now they just know. Now our sections fill up before other sections. We have IBL calculus listed on our IBL sections and those are the very first ones to be full. They are begging to get in there. The success rate has increased. Our success rates are way higher than any of the other non-IBL classes and our IBL sections are all taught a little bit differently. Our ideas of what does it mean for active learning is different from class to class but at the same time the active learning components are similar. And just as a side note, only one student failed my Calc 2 class and he just disappeared. I think he came there the first two weeks so I don't know if that counts, if you're not in class. But everyone who was in class passed the class which I was very happy about. There was no one that was there that did not pass. And it's similar for the other IBL sections. Class attendance is very high. Students know they can't just read the book because we are doing something that is completely beyond the book. If you look at our worksheets you might not have a clue what to do unless you are actually in class and get some guidance on how and what you are supposed to gain from them so they learn that coming to class helps them learn not just sit and listen to someone talk like they could pull up a video on the internet. And a number of students have become math and math ed majors. Our number of majors has increased significantly since we started doing this in Calc 1 and Calc 2. And I want to add one more thing to that that I thought that I put up there and I didn't. University of Nebraska Lincoln and University of Nebraska Omaha have an engineering program that is paired. Our UNO students get their engineering degrees from Lincoln. Lincoln has noticed that people taking active learning Calc 1 and Calc 2 have been more successful and stayed in their engineering programs and they have now reserved half of our seats full before anyone else gets them. It will be the first year we are doing this but half of our seats, Lincoln said, "We want those for our engineering students", so they are claiming them as well so we think that that really says something about what they are seeing from our students. Not only just mathematics but their other skills, their persistence, their problem solving skills, their ability to work with others. Things like that were getting noticed on that piece of it.So what worked, finding a team? If you don't have anyone else at your university who wants to work with you you've got a team of people here. There's always people who want to work together and that's really helped in developing these materials and finding good materials and good assessments and that's something that takes time but it's definitely worth it to create those materials and being able to work together and then come up with more. For instance, the tactivities I would not have thought of other than the one or two that we had if I hadn't went to Boulder and started working and created that connection with them. And in the future we hope to grow the connection, especially between Omaha and Lincoln. If you get a bigger area in your state or in your region to work together. Lincoln has started to do precalculus and then now our instructors are starting to do IBL precalculus in the fall as well so it's like they heard us doing calculus so they started doing something and we're doing more. It's almost like it's become a little competitive but in a good way in terms of who's doing more IBL. And then refining of the materials, making them more publicly available. I have a website up here and I think it's kind of hard to see, but if you just google "active learning calculus" it shows up but I will warn you, this was September 2015 when we last updated it and we have now tex'ed everything at Omaha and now we have worksheets for everyday and they are not up on the website. There's some change of staff at Boulder and I haven't figured out how we are going to get that updated so if you want the most up to date, if you are teaching Calc 1 or Calc 2 send me an email and my student worker will get you on a list to get you added to our Dropbox for now for that as well. And that's all.(applause)BressoudAre there any clarifying questions for Angie?(inaudible audience question)HodgeWe've had instructors at UNO do it in 50 minutes. I do it in 65 but 65 there's always a little extra time at the end so we wrap things up, maybe start homework but 50 minutes they can definitely be done in. At Boulder they do them in 50 minutes. In fact, they have large lecture calculus sections and in their recitations is when they do their tactivities and they have their graduate students trained to do those in 50 minutes as well. Some of them are even shorter but that's the most they should have to take. (audience) ...homework reading...HodgeBefore?(audience) Yes.HodgeI don't because I actually want them to come to class to learn. Now half the class has taken calculus before and that doesn't mean anything in terms of most of them. They realize doing active learning that they never understood and didn't know why. They were following rules. So personally I don't have them do anything before class. Maybe one day a semester they might have to do something. For the most part it's in class. And their homework is working on stuff they learned that day.Yes?(audience question inaudible)I'll start with the second one first, in terms of time. Our students get to know that if you take one of the IBL sections and we even have a note on there that if possible to leave the hour before open. There are some students who can't make it so we have alternate times or have them go to the math science learning center but for the most part the students know the reputation and clear that hour before hand. We have been really lucky. The whole university has been really supportive of active learning and some of the classes for the majors they have not put in those times before as well. In terms of homogeneous groups, yes and I really don't know the answer to fix that problem or if we should fix that problem. Students do, especially our international students, tend to group together but that's who they work with outside of class too. I definitely try that first week to mix up and meet new people and some of them mix up a little bit but some of them just like, I let them be with people they are comfortable with because I really push them outside of their comfort zone in that class. I give them hard problems. So as long as they are with people they are comfortable with. But that's just my style.BressoudYes? In the back?(audience) inaudibleHodgeIt's harder. I've taught it as a night class using active learning before. We met twice a week and I've had other instructors do the active learning at night as well. It's definitely a little more challenging because you don't see them as often but it's possible. You just take a break. We'll do one activity. We'll take a five or ten minute break, go out in the hall and come back. It's more intense. I do have the help hours more days a week then and I also have a room reserved for them if they want to get together and meet outside of class so they still are doing something more often. It's definitely more ideal the more days you can have a week. It's still possible to do but it's not ideal. And I don't know how many more questions you want to do, when you want to stop.BressoudLet's just take one more.(audience) inaudibleHodgeUh, precalculus. One of my colleagues, Michael Matthews, is actually working on designing some precalculus materials and I know he's working with Nathan Wakefield at University of Nebraska Lincoln and I think Gary Olsen has some college algebra tactivities that he's done as well. And I know there will be more time for questions after our next speaker as well.BressoudLet's thank Angie again.(applause)04 08-02-16 Michael StarbirdI found both these two talks really great and several things struck me as very thought provoking including the concept of the slacker table and I was looking at watching you and I think there's a couple of slacker tables out there! (laughter) You know yourself but I don't know. Check out, slacker tables, but I think that was really terrific.But another thing that struck me in Angie's introduction. I often worry about how well people understand things, particularly math. It's a real problem when they don't understand it and she started out in her introduction with something I didn't understand, which was ultramarathon runner. (laughter) I think I know the definition of that, is that right? How far do you run?HodgeActually I sometimes purposefully don't do that so people can ask me a question afterward, even my students. Ultramarathon is anything over 26.2. 50 milers are my favorite and I'm training for the Leadville 100 in August. I do that every August usually.StarbirdIt wasn't that I don't understand the definition! (laughter) It's much more basic than that. WHY?!?! WHY?!!? First of all, how is it possible? And why? This actually brought me to a serious point. Just last week I was giving a workshop for high school and middle school teachers and any of you that have done this know one of the features that you see is the level of the mathematical knowledge that teachers frequently expose. There's a problem there. The level of the understanding is not what we would like it to be. But if you don't understand something, it's not obvious what it means to "No, you don't understand it". Those of us who have a lot of experience in mathematics we understand that in the sense that you don't know a certain branch of abstract algebra, you know what that means, that you don't understand it in some way. But if your whole level of understanding mathematics or anything is restricted, what does it mean to really understand that gap, which is the way I feel about ultramarathon running! The reality of that is not part of my mental frame. I think that one of the things inquiry based learning does, or at least attempts to do, is to raise the standard for what we mean by understanding. For anybody, for our students to actually come to a stage that is higher than they previously had had. I don't know about you, whether you have a consciousness in your own experience of when you took steps toward higher levels of precision and depths of understanding. Me, I happen to have an actual experience which was in an IBL class in graduate school. When I went to college I was a math major but I never really understood anything extremely well. I mean, it was regular lecture classes, you learn things, you did the test and that was it. It wasn't until graduate school when there was an inquiry based learning class where I was actually proving theorems on my own that I realized it was possible to understand things at an entirely different level from what I had ever experienced before. At the end of that year I could prove every theorem from the year with a blank piece of paper and all the statements and the proofs and so on. That was a completely unexpected reality for me. When I think about, by the way, the title of my subtalk had to do with this session is "What Does IBL Do For People", and it was the idea of trying to raise that standard of understanding and give people an experience of what deep understanding of anything is, what does it entail. I wanted to say a couple of things further about both Ben's and Angie's talks before I go back to what I was going to originally say. Ben's talk, one of the things that he talked about was the human psychology, the human psyche where you divide it into three parts, the intellectual part, the behavioral part, and the emotional part. That really struck me, because to me that is one of the fundamental differences between the actual experience a person is likely to have in a more active kind of engagement with a course compared to a lecture kind of experience. In a lecture experience it’s very clear what the role of the student is compared to the role of the instructor. They sit there and try to remember and understand what is being said whereas in a more active, engagement kind of setting they are doing things that are different. They are constructing the knowledge by actually figuring things out, they are explaining it to one another, they are developing a community, they are having emotional responses that I think are incredibly important to acknowledge and engineer. I want my students, as one of the goals in my classes, to have the experience of making mistakes on a regular basis and learning from them and for the purpose of changing their internal reaction to mistakes. Ordinarily, people have their entire life history, making mistakes is viewed as bad. And certainly in their math classes, they get points taken away, they get derided for it one way or another, it's all bad. But the reality is for any of us who have ever done any mathematics that is creative, you know what's it's built on. It's built on mistakes and then another mistake and another mistake. You learn from the mistake in order to direct your next attempt and your next mistake. I want the experience of students to engineer the emotional reaction to be altered. That is, their emotional reaction to things like mistakes. I want them to come to the stage where by the end of the experience they understand that when they make a mistake they just don't feel that cringing feeling. They don't feel that they should hide their heads and pretend that it didn't happen or hope that it didn't happen. That they view it in a more realistic sense that that is a step in the direction of success. And I think that we can do that. In my classes basically every single day students make mistakes in public. That is part of the everyday experience of the class. At first, the first couple of days of class they may be reluctant and feel bad about it, but very soon it's just the norm of the class. You can create an environment in which those things are supported and celebrated. So I always have people, for example, present wrong proofs. Even if a person has done a correct proof if there is some important canonical error that has not been exposed I will ask somebody who I've maybe walked around and seen make that error say, "would you please present that mistake," so you can actually learn from that mistake and see the value of it. And I loved Ben's cutting up the proof. First of all, that in and of itself, I've never done that but I might try that, cutting up the proof but then including mistakes in so they can not only reconstruct the proof but find the mistakes in it. I thought that was very clever. I wanted to also reinforce one other thing Angie said about what she wanted to do before class and after class. The flipped class model. I think there's a lot of value in that except for the problem that if you ask people to do things before class then the reality is that some of them will have done that and some of them will not have done that. Then you are dealing with a mixed group of students. So what I do in my own classes, and even using, for example, in my liberal arts mathematics course where I wrote the textbook that we are using, my instruction I give them is that you may not read that section before class. And 100% of the students do that assignment! (laughter) It's great! So then I know where everybody stands, it's not different. And the idea is then you can give them this experience of novelty. You can give them that sense of exploration that you've engineered. You can have them discover these step by step illuminations of ideas. And I certainly can't resist the final thing when Angie was responding to the question if that they have calculus before and she said, "Well, many of them have but it really doesn't matter." That's a little bit of an indictment to our education system, right? The idea that all of us have had exactly the same kind of experience. Oh yeah, they had it before but it doesn't make that much difference, it's all new to them. Well, that shows how deeply we tolerate their level of understanding.Let me just say a couple more things about this depth of understanding. When I think about education, I think the most important thing we are doing is actually changing students' inner life permanently and in all levels of their life, not just their math. To me when a student has changed attitudes where they have come to, they have had an experience they've never had before of creating ideas and of actually grappling with something challenging and overcoming it through their own efforts, that's an important experience. We can engineer that and have that be a daily experience so they get to the realization after the course of a semester or more than one semester they come to the realization that that is who they are. You are actually changing the inner concept of a human being. Things like being able to make mistakes and being willing to make mistakes is part of the process that they entertain in their everyday life. That's one of the features but other kinds of features are persistence. The idea that if we somehow engineer it that they learn more effort actually leads them someplace, that somehow we get them the idea that they can discover things more deeply regularly and apply it to everything they do. That's the goal. The goal is not the math. The goal is can we actually envision changing people. I think when mathematics is taught focusing entirely on the intellectual side the change that we can expect is much more limited than when we actually embrace this concept of community building, of emotional challenge, this experience of overcoming obstacles as part of what we do. In my own classes, in my inquiry based leaning classes, the way I conduct a class is that I've given my students some challenges and presented some new challenges in the class and everyone is at the board every day. So if it's a 24-person class I have them in groups of three and I just say, "You go do number 12, you do number 13, you do number 14", and they are talking together, working out their proofs and presented them on the board. Their discussions with each other are always things that they point out as the most important part of the day and the semester because they're explaining their own ideas to one another. And I walk around and be somewhat helpful but the important thing is that they are talking to each other. And after a while they sit down and I ask them to present their work to people. But when I think about what is really making the difference, I think it's subtle and I don't think we thoroughly understand it. It includes all these social parts, the community building, the talking to one another, the common experience of error and success. I have to say every time I hear these things, I wonder are we really looking at the right variables. For example, one of the things that I'm trying to do is develop some online materials. It's not that I think that that's the best way to instruct people and it's not obvious it's even possible to give them the kind of experience of inquiry that I value so much, can you do this in a really large scale way? It's not obvious but I'm doing it because we need to do the experience. And part of the reason we need to do the experiment is because when I go to these workshops, and particularly looking at teachers in the high schools and junior high schools, there's so much problem with the accuracy of the mathematics that I worry about that. It would be ideal if every single teacher in elementary school loved mathematics and knew it well and conveyed that love and enthusiasm for students. It would be a revolution of reasoning if we could get the best teaching if we could get the best possible teaching from kindergarten through the rest of their lives. Absolutely, a revolution. People would think clearly and be used to the idea of thinking for themselves. But we don't have that so I'm trying to figure out is it possible to convey some of the spirit of inquiry via some of the new mechanisms that we have for conveying new ideas. I think the answer is we don't yet know and I think we as a community owe it to the world to not only turn our minds to our classes and our own domains but to also think big about whether or not we can find ways to identify what the salient experiences really are and then can that be delivered on a scale globally. So I'll stop there.(applause)BressoudQuestions for Mike?(audience) It looks like Angie and you do not expect your students to read at all. Is that true? I hope it's not. I believe one of the objectives of any math class is to teach them how to read mathematical literature. And reading mathematical literature and novels are absolutely two different things. We know this. My question is where and how do you teach them how to read?StarbirdIn my classes what I do is I have them, say, in my liberal arts math class where there's a textbook that explains things, I first introduce them to ideas in the class where they are discovering things. I pose a question and then I pose another question and they work on it, and so on, to develop theme. And then I ask them afterwards to read the textbook which is a coherent explanation of the thing but there's usually more details in there and some extensions that weren't actually treated in the class, and that's one way. Another way is that I have some sections of the book which I ask them to do as reading assignments that will appear on the test and then I don't treat it in the class at all. So that's an example of where I just ask them to actually realize that they can actually learn from reading. I think building in some reading assignments as part of the experience that they get in class, that is a very empowering thing. If you can teach a person how to read and actually learn from reading, you're giving them an incredible power that they didn't have before. But generally in my experience, mathematics is very difficult to read for all of us and certainly for students and most mathematics textbooks are not written to be intended to read, as far as I can tell. I think they are written for the purpose of that person who knows it already to say, "Oh yes, all the details are there" and the purpose Angie mentioned which is a repository for questions at the ends of the chapters. It's come to be that that's the norm for the writing of textbooks. Edward Burger and I wrote "The Heart of Mathematics: An invitation to effective thinking" textbook and in the preface we said one of the novelties of this book is that we wrote it to be read by students. (laughter) And it was a novelty.05 8-2-16 general discussionBressoudMaybe we can move it into a general discussion now and open it up to your questions, concerns, problems.HodgeLet me first add that I, the same as Mike, think calculus, that's one of the ways you say, how do you cover everything and so IBL, these activities take time. There are some sections that are only take-home and the students have to read the textbook, answer the questions, and they have to ask questions about them to myself or my TA before they're given the test because it will be on the take-home test and they have to be able understand that from the reading. And then I use his book in one of my classes as well and make the students read that. So it depends upon the class and whether the textbook is useful or not.BraunSo my background, when I was an undergrad I double majored in English and math and many of the things that I brought to my own teaching as an early career faculty member were actually things that I experienced in the humanities courses that I thought were really effective that I wanted to pour it over. And an interesting split that happens in the humanities that doesn't happen in math is that they separate both in programs and courses most of the time between the creative act and the analytic act. For example, if you get a degree in English, right, that typically is going to focus on literary analysis which is different than getting trained to be a creative writer for which you often get an MFA, right, a masters in fine arts. And in mathematics we don't have that split and depending on who you talk to in the mathematic community, some will view the creative act, the act of figuring out how to cleverly solve the problem or figuring out how to do the model and solve it for an applied thing as the fundamental act of mathematics, whereas other people in the community will value much more the analytic aspect of really delving into the legacy of Euler or whatever. In our conversations about teaching we don't do a very good job about discussing how we are distinguishing might be wrong but how we are simultaneously training students to be critical in their analysis of existing mathematics and also be effective and efficient in their creation of new mathematics or at least mathematics that is new to them. The typical class, the typical assignment you are going to have is that you are going to give the students a problem and the harder the problem is usually the more creative they have to be to solve it. But very rarely do we give the students a text and ask them to deconstruct that text in some meaningful way.For me, when I teach IBL courses, I've taught college geometry, history of math, methodical problem solving and stuff like that, does not look at all like the IBL courses that they are talking about in that I purposefully go to the pop lit shelf and get books that were written to be read. Barnes and Noble's got tons of math books, right? And I use those as much as possible as anchors that the students are actually interested in reading and typically don't even know exist. They read ahead of time and then come in and the work of the groupwork in the bulk of my classes is the students conversing to deconstruct the text and make sense of it, which I then compliment with the creative side mostly on the homework. When we are talking about all these things, this is again one of these issues where it's really easy to talk past each other where we often times don't clearly articulate what we want the outcome of a particular class or program to be. For example, the type of number theory course that Mike's book is sort of designed to enculturate would be a wonderful introduction to proof if it were then programatically followed up by some course that developed stronger analytic techniques but that requires a lot of cooperation and teamwork, as Angie mentioned, at the department level to have coherence in the curriculum and I think most of us know it is really hard to work with people.StarbirdLet me chime in on that. One of the things I said I'm trying to do is this online stuff and one of the exercises that I ask them to do at the end of each section I call it the "tell the story of the concept creation exercise". What they are asked to do after the march through, and by the way the way I did it is I would sit with a couple of students and then with a camera going I'd pose questions to the students and they'd struggle and work trying to prove things and all with the idea that the camera was this fourth person at the table and they are supposed to participate in this discovery mode. So the experience that this external learner is doing is they are seeing these little videos and participating by responding as my human students are at the same time, except that happened much earlier. But then after the end of a section, for example, the proof that there are infinitely many primes, you know, the idea of introducing the concept of primes and prove there are infinitely many primes, is that I ask them to tell the story of the creation of this idea and I give them outlines of what we have seen and then they are to write an essay that includes examples and motivations, why did I think of multiplying numbers together and then adding one, what brought that to mind. They are trying to tell the story as if they were the creator of the idea so that's something in the spirit of creating a narrative.BraunWhich is the expository aspect which is a whole other issue...StarbirdYeah, that's sort of, that's not the creative part. They've done that as they go. But then putting it together and rethinking it as we always do every time we teach something, by the way, helpful.BressoudYes?(audience question inaudible)Angie could you repeat the question?HodgeSo Susan asked "How do we frame stuff so our students don't say on the evaluations 'You're not teaching me anything, you're not teaching' because the students think of teaching as lecturing?" It takes time, especially more time in Calc 1 than Calc 2. Half the students have had me for Calc 1 usually and they sell it for you. If a student is complaining that I'm not up there teaching them, they'll say "Oh yeah, you are going to learn. This is how you are going to learn." Calc 1 though, we have that discussion at the beginning of the day that I'll probably have Mike talk about, his "what do we want you to remember in 20 years" talk. We do that very often, you know, why are you taking this class, what are you supposed to get from your career, and we revisit that in Calc 1 probably every couple of weeks, why are we doing it this way, what am I doing behind the scenes, what are you doing when you go home, what does it mean to teach and what does it mean to learn. We spend a lot of time in Calc 1 framing it. I still probably get two people that say, "I wish she would have taught us more, I had to think too hard in this class", things like that but they are always paired with, "You made me think too hard and I had to do too much work on my own" and I'm okay with those couple, but it's better than in the beginning when I started this and wasn't selling it well and I'd get quite a few. For the most part they're all saying, "Now I know how to learn and I know how to think and I like math" and things like that. You'll always get a couple, I think, that complain but really carefully selling it and framing it, more than just the first day and revisiting that again and again in Calc 1 especially. Mike, do you want to talk, anything to add, especially the 20 years thing because I do that all the time and my students love it.StarbirdThe only thing I want to add is to your comment about the reception of the class. Once a class is known to be successful, you don't have to sell it anymore. You don't need to explain because kids talk to each other. So the next time you have a new group of students they're not coming in with the expectation that you are going to stand up there lecturing, because they know. Because they've talked to their friends and they're friends have told them, "Oh, this is really cool." So this becomes the norm of the new civilization. In a way I think that's what we're doing, we're creating a new culture. When the norm becomes, "Well, when you come into a math class this is what you expect, you expect to actually think, you expect to work, you expect to come up with new ideas on your own", well, then this is the expectation. Everybody comes in with that expectation. So that's why it's really a snowball effect that improves not only one individual class and one individual person but trying to have the whole culture of the community be altered.HodgeStudents will start to go to the department chair and ask, "Why aren't other people doing it this way?”BraunAnd so for me, I've actually outsourced this to the cognitive psychology literature. What I found was whenever I was trying to make the point of saying, "This is how math really is", I would often say, "You're going to graduate in three years and then you're going to go get a job and you're not going to have a professor there, so how are you going to do it?" None of the things that I would say really would overcome the immediacy of the now. The students were like, "Yeah, yeah, yeah, I'll figure that out then. I need to pass this class now and you're not teaching me." And I still get a lot of pushback. We have some incredibly strong math majors, so for like the 200-400 level courses that have a lot of math majors and minors, we have a lot of super stellar math majors who are doing lots of research and then we have students who failed out of engineering school and their advisors said, "Well, you're almost there on the math so you should major in that even though you've never gotten better than a C in any class", so we have this really weird student spectrum sometimes. So when I say I've outsourced this what I mean is, the first two days of the semester I don't worry about the math at all. I completely devote it to an explicit discussion about the psychological issues related to mathematics. So I shouldn't say I abandon the math. And I don't know how this would work in a calculus course, but in my upper division courses the first thing I do is have the students work on an unsolved problem and I don't tell them it's unsolved. And then after a few minutes I ask the question "How long would you work on this" and have everybody talk about it and think about it. Usually they'll say somewhere between two more minutes to a week more and then that's all. I'll say, "Well, this has been open for 60 years. What do you think about that?" And then they're like, "Oh my god, why would anybody think about that?" and then the discussion turns to, "What's the point", and I simultaneously have them read some kind of survey article about either mindset research or stereotype threat or something in that litereature. Carol Dweck's got several really good articles about mindset and I have a course policy that I put in my syllabus about using constructive language so students are banned from saying "I'm an idiot" or "I can't do that" and then I enforce that in the small group work. So I do very similar things in my large lecture calculus courses, I don't know if this is relevant to you but I don't do the unsolved problem stuff. Instead I have the students on the first day of class watch a 10 minute video about mindset, there's a TED talk that's really good and then I do some of what's called "values attribution intervention" or something like that, and on the way, that tends to help a lot with getting the students to buy in. I'm not making it about me, I'm not making it about math, I'm making it about, "Hey, look at this shiny thing that is out there in Scientific American, there's this cool article and let's read it and talk about how it's going to affect our class," and it really sort of defuses that situation a lot. So that's been the most effective thing I've ever done as a math professor, has nothing to do with math.HodgeI will add something I started doing in the last couple of years. Math autobiography, listening to them at the beginning of Calc 1. I let them tell me what is their learning style. Almost all of them say they hate group work, they don't like to talk, they like to work alone, and they like to listen. I do it before and after and that really changes by the end of the first semester. The first couple of days we are doing a lot of math. I have a lot of hands-on, more tactile activities those first couple of days so they see what it is, so we can talk about what are they doing, what is their role, what is my role, and they understand that because it is very different than what they've had in high school for the most part, for all of them. StarbirdI guess I should do a little advertising while I have the chance. In my classes I always ask them to read the book that Ed Berger and I wrote, The Five Elements of Effective Thinking, which is not about math. But it really is what I view as the goals of education, of what you want them to do the rest of their lives.BressoudYes?(audience) Ben when you said in your frame and a lot of what you referred to with the behavioral, the emotional and the intellectual, that's really true for the instructor as well. I want to point out the tremendous amount of skill that all three of you have demonstrated, we're presuming you know the math, but you are really demonstrating a lot of skill in the emotional aspect as well as. I think with the behavioral we think of the pedagogy but we don't talk about what it means. What you do with the slacker table, what do you do with the resistance, those are huge issues. Also Michael, you talked about the skill. K-12 isn't the only issue. I'm looking at this partly as a department chair now and we have a default of inquiry based, but we still have to hire people as adjuncts. The level of math is an issue for teaching college. You have people with a business degree teaching math classes at colleges. I think some of this rolls into other things we're talking about. I think it's really important for us to frame this in the level of skill that we're talking about that's required because at some level, some of it is that we really need support for how important it is and how creative it is to do the teaching like this. There's no short cut from what you're talking about to the adjunct who's teaching six classes and three different schools who doesn't know the background. You don't have an extra hour, you don't have all these other things. I don't know that this is really a question, if there's anything in this that you want to comment on, but I think the framework that we’re talking about for students is also right there for the instructors too. And we have people that have problems in these, people who aren't comfortable with the psychology, people who don't write well in English, all these other things for the instructors have become issues.HodgeOne thing we do have for instructors is we have four more years on our grant for the IBL Summer Workshops so if you're interested in that talk to me, Stan Yoshinobu, and they fill up very quickly but they are very successful. Sandra Laursen has been doing research on them for years and we have a nice model for those workshops so if you have faculty who are interested and want some mentoring there are four day very intense workshops with yearlong mentoring after that I think have been very successful in helping instructors make that leap.BressoudIf I can jump in on the calculus studies that we did. We found that of the large public universities those that were most effective in their undergraduate teaching and their calculus teaching were putting a lot of thought and effort into what do you do with adjunct faculty, what do you do with postdocs, new faculty. How do you prepare them, get them to understand what active learning is about, getting them paired up with mentors who will really help them get through this process. That's critically important if you really want to do this more than having just a few experienced individuals who are implementing the active learning or the inquiry based learning in their classrooms. To get it spread more broadly you really have to think about systemic changes within the institution that support everybody in that. I think in the second session this afternoon we will get a more indication of that.HodgeRon, do you have a question?(audience inaudible)StarbirdRon asked to compare the active learning in math to the other STEM disciplines but I wanted to make one further comment on the previous topic. You know the question of how to get mathematicians to be more sensitive and successful at the human side of teaching, it's something we don't talk about. It's kind of a taboo subject. It has the potential to really be an awakening experience for many mathematicians. There's a lot of joy to be had in becoming more aware of these sides of the profession, namely the human interactions, and the emotional side, and the actual skills associated with the humanity of it that would be a huge benefit to people well beyond the math and well beyond the teaching. I think people would actually be open to it and enjoy it. It’s something that we as a community have not embraced, and it was just a thought that came from your comment. I'm not proposing anything but it struck me.BraunI'll think about Ron's comment for a minute. The first thing that came to mind when Ron asked the question comparing active learning in mathematics to the other sciences is how faculty respond. At a workshop about a year ago Tara Holm and I were having a conversation with Sandra Laursen who has done a lot of research in IBL and Tara asked her, "You study a lot of stuff about active learning and IBL in math, you've also studied other sciences. Is it different talking with mathematicians?" And Sandra immediately said "Oh yeah! The main difference is when I gave my main talk to mathematicians and I show them my data they have no idea what it means." When she's talking to chemists or physicists or biologists or whatever and she will put up the statistical analysis and they'll say, "Oh!" and have a reasonable conversation but the mathematicians are just completely adrift and have no sense how to handle data. I think that this is an interesting thing. We are in a proof based discipline. We are unique in the sciences in that the tradition of pure mathematics is not so empirical, though that's changing in a lot of ways with the advances in computer technology and the increasing use of modeling in applied disciplines, we're just not that used to data. It's hard to talk about efficacy amongst ourselves if A.) we don't know what we all want to get for the students out of their learning experience and B.) we don't really have a good grasp of what we are talking about. This has come to the forefront for me recently because I serve on a number of PhD committees in our College of Education and the STEM Education department. I've never had a probability or statistics course in my life. I'm a combinatorialist. So all the statistics I know is discrete probability I've probably learned secretly. And it's kind of embarrassing frankly when they are talking about "she's a T test, no you need to look at the power index, and the effects size" and I'm like oh my God! They're looking at me because I'm the content expert but I know nothing about measurement and evaluation. That's one thing, I think people in the sciences are much more used to talking about messy data. The second thing I think is the difference between being a proof based discipline and being an experimental discipline. Many of the things that happen in the first year for other sciences is that they do observations of experiments and try to make sense of what happens and that is much more amenable to active learning in a lot of ways than the traditional definition-theorem-proof development. I've taught a lot of history of math and what I know is that the way we see mathematics now is not representative of the historical development of mathematics. Our understanding of mathematics is a social construct. As a community we can choose to decide that mathematics has a much more empirical flavor than we've been taught in our upbringing for the last 50 to 70 years. And I actually think that's a really worthwhile thing to do. I like the idea of tactile stuff but I'd go one step further and have the tactile stuff be a start of a mathematical observation in calculus or studying something in linear algebra that's modeled linearly but that the students can actually look at. I do think that's an aspect of active learning that we're missing a little bit. I use a lot of open unsolved problems in number theory as discussion problems for this. The students can get their hands dirty with it, the students can use the computer and they can make observations and they are free from the burden of finding the correct answer. That's not the only way to do it. StarbirdI have an empirical question. I'd like to take a poll, since you recommended it. I was so taken with your comments about the professor. How many of you would attend if offered a two hour workshop in making human connections given by, say, a theatre professor? OK, that's only half here. I'm just curious. I think it would be interesting.(audience) I'm curious about the issue of coherence of materials, something that I've been wondering about for a while. Something I'm curious about, in your way of doing things is that something you pay attention to or is that something that these other attributes that students pick up fix by happenstance? I'm thinking here in terms of a student taking Calc 1, Calc 2, if you would ask them "what were the big themes they take out of Calc 1, Calc 2", what do they say? What would a student who's been through your courses be able to say? I see my students think of these things as completely disconnected and I'm wondering whether this way of doing things fixes that.HodgeI've never really asked them that directly and it might be a good end project. We do a lot of writing projects at the end. But I see when we do topics they make the comments of, "Oh that's how that fits with that" and, "Oh I see this together" and they'll make those comments all the time. You can definitely tell who's had an active learning Calc 1 and Calc 2 and who hasn't because they see the things that fit together from Calc 1 and how it goes together with Calc 2 and how they see it all going together. I haven't directly asked them that question, I don't know. Those are little comments that they make while things are going on in class. Has anyone ever done that activity?BraunI have to distinguish between my large lecture and small courses. In our large lecture courses I'm not sure and I also do much different types of active learning. It's much less small group inquiry and a lot more like six minutes of think-pair-share interspersed throughout a lecture. In any class that I've taught with less than 35 students in the last six or seven years the final assignment of the semester is graded by completion only and it's an assignment in which I ask the students to discuss six things that they learned from the course that they will take away into the future that will last beyond the course and I say they have to discuss a combination of mathematical ideas and cultural, social, whatever, historical ideas. It's always really fascinating to read these, in part because of the fact you really get a sense of what the students value and the thing I've noticed is students seem to value a huge range of things. One of my favorite quotes, it was a history of math class and the student said "I learned a lot of things in the class but I don't think any of them where the things I was supposed to learn". It was a lovely essay. I would say that the active learning techniques alone have not helped the overall mathematical learning. What has made a big difference in terms of the quality of the mathematical discussion in the end of the semester essays has been changing my assessment techniques. A couple of years ago I switched from doing two or three midterms to giving, at least, two out of every three weeks I give a 15 minute mini-exam and I actually don't do any big exams now. Everything's cumulative. So every week they have a small exam and it's cumulative, and I give two problems. It's the same amount of testing but it's spread out. This is based on research in the Ambrose book, How Learning Works, or something. There's a lot of research saying that frequent low stakes promotes deeper learning at a much greater level than infrequent high stakes assessment. For me, that seems to have made a tremendous difference in how much my students learn. As a department there's generally buy-in about that, that we should be testing more frequently but it's very hard to do when we have 900 students in calculus spread across multiple sections. We do have weekly quizzes and the recitations but it's still not the same. We haven't quite figured it out in a large lecture setting.HodgeI would say those weekly quizzes that we do, the take-home ones, are really similar. My students always say it's the most valuable part of the course, giving them some problems where they get really good feedback on, they're essentially like take-home tests for them. We take them very seriously.(audience) I want to frame my question as a conference. I am Project NEXT as of last year and I've been teaching a lot of IBL or IBL-style courses. As I gather from talking to my colleagues, I'm on the prolific end of content creation so I make a lot of worksheets and I find that in talking to my students very useful and I'm growing as an instructor. I find this kind of conference, meeting very useful as well for getting new ideas. One thing I have not yet been able to feel successful in, like one dimension of growing as an instructor, is looking at other instructors' materials. University of Michigan have made freely available tons of IBL and IBL-style worksheets and Angie was talking about how all their materials are online and so on. I pull up those materials and I know they are all tried and true and I feel like it's a monumental task to somehow go through that and come away being a better instructor. If I just took them verbatim and used them I'm sure they would work very well but using them as a tool to become a better instructor for myself? That's something I confess that I haven't been able to figure out how to do and I'm wondering if anyone else here has had that experience or if anyone would like to offer some suggestions for growing in that dimension.HodgeI would say my materials should eventually go along with a workshop because if you just look at them they could just be easily used to lecture and fill in the blanks. They could easily just be used to give to your students and not do anything with them. I think there's something that goes with that dialogue on how do you use these materials which would take a sabbatical to write these guides and who's actually going to read these guides. For mine at least, they definitely should eventually be paired with a workshop. While you're interacting and thinking about how do I use them and can we get ideas from other instructors on how to use them better. That's just my opinion on mine. BraunFor me, I read a lot of stuff and I've gotten a lot of ideas from a book by Ken Bane called What do Best College Teachers Do which is not math specific but has a lot of wonderful ideas. More than anything, I've been fortunate that I have Carl Lee at the University of Kentucky who's a Haim Award winning teacher and a well-known researcher in algebra and combinatorics. He was assigned as my mentor when I arrived and so I had someone who was a dedicated, serous, and incredibly thoughtful teacher to talk to regularly. It was only later on that I realized he enjoyed talking to me too! (laughter) In hindsight you were probably excited to have someone to talk about this stuff with. There's a lot of value to be had from having somebody watch your class and give you feedback but for me even better has been a few people, he's not the only one but he was a real influential person, but somebody to sit down once every two or four weeks, have a cup of coffee, and say, "I did this in class and this sucked. It was bad. Mega fail." And then they can say, "Oh yeah, whatever", and vice-versa. I think that is really the right way to go. I know Charles Henderson is probably going to talk about professional learning communities, I'm not totally sure. I know his work has studied you can be more formal. I have friends who honestly don't have someone like that at their own institution and who have gotten a lot out of Twitter and social media and gotten a lot that way. StarbirdI visited at Yale this spring and they had something called Bulldog Days, I think that's their mascot. During a week they had professors volunteer to have their classes viewed and any other professor could sign up for seeing those classes. So during that week people would go and visit any class that they signed up for. The follow up was less formal but the idea was that you would see other people doing things and you would learn from them and you would also have people to come in and learn from you. It was a way to make people become aware of one another's strategies. And we are going to do that at University of Texas on September 8 at the College of Natural Sciences just to see if it would be helpful as well. Seeing one another's teaching is great because if you see something that someone is doing is good you just adopt. And when you see something crummy that's even more instructive! (laughs)HodgeThis will be our fifth year doing the IBL calculus. There's been several colleagues, but there's one colleague I talk too more. Every single day we talk about our teaching and it doesn't matter if we taught it last year. We still afterwards, and I'm the one that goes first so I'm usually the one that finds out if there's any kinks, but we talk every day. And I talk with my TA also and I email people if I'm not sure. I never stop reaching out reaching out with people and talking about whatever class it is you're teaching. You can always learn. (audience) Greg Foley, Ohio University. The question was do you look at other people's notes and learn from their notes? The response was all about watching it live and interactive. And I wonder, show of hands, how many people have taken someone else's notes and not work through them with students but read them to try to get ideas from someone else's IBL notes? It would be interesting to know how you use those.(audience) I think the question two ago was great, and this question that just came up was great. We do have professional training and one of the things I learned in grad school from an excellent professor he once said you could tell who was doomed as a researcher because they would look at results and never think, "My God, that's the wrong way to prove that result!" I think we carry that over to the written materials that other people produce and the textbooks which I won't comment more than Mike's comments about some of the textbooks but we have a certain mindset. HodgeTo answer that question about do we look at other people's notes. In creating the materials we are creating now, I don't know how many things of Google that I looked through. I read notes, I take pieces you like here, pieces you don't like as much. It's constant process and even now there's still some spots where we could do better and pulling out textbooks, pulling out notes online, it's a continual process I think to look at what's been done and how you can improve on that.(audience) I want to join you in that confession. I really struggled at looking at other people’s notes. But then I joined this TIMES program, which is Teaching Inquiry Mathematics Extending Supports. The way they have you go through their notes is a lesson study. So this is an easy way to get, if you grab some notes and grab some colleagues or some other Project NEXT fellows. You can meet once a week, pick one of the lessons, and work through it, do all the math in it, think of how the students are going to react to it as a group. That's really helpful and, if you can, record yourselves teaching it so that in a later meeting you can watch each other teach remotely and that made me a much better teacher. Justin Dunmar from Frostberg University in western Maryland. (audience) My name's Gary Richter and I teach at Southwestern University. This is not a question but a comment on all the discussions of taking material from other places. I just noticed on the schedule that at the banquet Tom Lehrer is going to be present and perhaps singing some of his songs. I don't know if you know, he was a Harvard mathematician decades ago and he became famous as a satirist in the songs that he wrote. One of them he wrote that is mathematically related is called "Lobachevsky" and the main theme of "Lobachevsky" is that the way to be successful at mathematics is to plagiarize. So just a note. BraunI'll add one thing. It's just interesting. I haven't really thought about this question or issue. There's two things that you can look at. One type of thing is other people's actual mathematics notes, and the other thing you can look at are things they do in how to prepare for a course. What I find is that a lot of people who are successful at teaching spend a lot of time thinking about course design before the semester starts. I have found it very useful to read people who really write a very thorough and thoughtful syllabus. I find that very useful. And also some textbook writers who write good textbooks as opposed to the kind Mike was talking about, who put together a book that makes sense. When I'm using a good book it's clear to me how one should structure one's activities to compliment it. Maybe that's just me. (audience) I just wanted to comment also that I didn't know Tom Lehrer was still alive. (audience) It's a Tom Lehrer singalong, Art Benjamin will be singing.(audience) Yeah, I thought he was dead. About the notes. What I found most useful honestly were ones from Journal of Inquiry Based Math, Ted's journal, because I go more method style and I learn so much from his notes, his dad's notes, and I just taught real analysis with his dad's notes, and it took me back 30 years to the original because my notes had become so unintelligible over the last 30 years and went back and saw how it was done it was so much richer for my students. They absolutely adored it. But also as you said, books. I collect old books and one of them is Calculus by Riddle that he wrote and he dedicated to R. L. Moore and it's an excellent book. I found some of the old out of print books much more useful in terms of lining things up. Last but not least, with my friends, the syllabi, seeing how they develop and what they do. I learn a lot from that before I start teaching a class, especially one I haven't taught in five years. All of that really comes together and helps me understand and to reflect back. And if you happen to have your old notes from when you were a student, I use those. I go through those, for example, with Mary Francis Naff or Coke Reed or whoever I took a class with and I go through my notes and I see how much I didn't understand and it helps me empathize with the students who have trouble. It also keeps my mind open to they could know things that I don't know and they could look at things differently. So all those things together. It takes a little while but I think it's all worthwhile.(audience) I'm Stephan Pollard, a teacher at UC-Santa Barbara. Again, I have a question but also responding to your comment. One resource I've found to be maybe a model for successful notes that can be followed well in addition to the Journal for Inquiry Based Learning in Mathematics is specific to linear algebra but the Inquiry Oriented Linear Algebra website from some folks at Virginia Tech. Not only do they have very detailed lesson plans they also say what sorts of responses to expect. They have examples, even videos, of student work and I found that enormously valuable. They go through probably more detail than I've seen in one of these programs before and I think that would be a good model to follow in the future as more of these materials get posted, seeing more how they've been implemented in classes and seeing how students responded and what to look for. Angie said, "Who's going to look at it". I think people will look at it, especially if it is helpful for the teaching. I also had a question, comment about a comment that was made earlier that, yes, that active learning can mean a lot of different things in different contexts, in teaching contexts and instructor situations. One area that has been studied in depth, we were talking about what parts of active learning have what effect, what is valuable and why. One thing that has been studied a lot has been the use of clickers in large lectures. There are lots of papers on this from Eric Mazure and others. I'm wondering what lessons we can learn from that. There seems to be a lot of empirical data in that area. Part of the reason maybe why it's easier to collect that data about that is that we are talking about large lectures so we have a large capitol N here. But are there other lessons we can learn when we're talking about assessment in specific practices and in specific situations.BraunI think the answer is yes, but I also think the answer is that you have to be very careful. One thing the physicists have that we don't have is that they have a force concept inventory and that helps them a lot. While we do have the calculus concept inventory that Epstein and other collaborators developed it's not at the same level and there are some issues with it. There are also concept inventories for precalculus and college algebra that Marilyn Carlson and other people have developed. I do think there's some really horrible ways to use clickers that just are terrible and useless. I think that the issues comes down to implementation. I've talked to people that said, "I tried to use clickers but it didn't do any better" and I said, "What did you do with them?" They were like, ”Well, I took attendance at the beginning of class with that and then a half hour into it I had them answer a multiple choice question and then at the end of class I had them answer another multiple choice question." I was like, "Well, did they discuss anything?" and they were like, "No, they just answered the question right away". No discussion, they didn't do the multiple choice question and then let the students do it again if they didn't get it. It was like, "Oh, you didn't get it so I'll explain it again", and then move on. So this is a tricky thing. Yes, we certainly know clickers can be good but the question at this point the question is how do we effectively give professional development to people and create local communities in which people are using these things effectively according to what we know about the research. I think that is the issue with that particular technique.HodgeI think that's one of the reasons why we are today. Ron will correct me if I'm wrong, but there's a lot of good stuff going on with active learning and inquiry based learning and what research questions do we need to ask, what needs to be out there, what research needs to be to be done to contribute to the literature, to help convince people this is working. Yes, we see this working in our classrooms, we see the changes, but what actual research can we do on this to share with others.(audience) This is really more of a comment. Bear with me, sorry panelists. Everybody stand up real fast. I'm going to count to three and while I'm counting to three you all have to dance. OK, ready? Go. One-Mississippi, two Mississippi, three Mississippi. OK, sit back down. Now my point here is, and I'm sorry to anyone who's seen me do this. Why did you do it? Why did anyone do it? You wanted to know why? Some didn't dance. Sometimes people don't do it. Sometimes people do it because they trust me that I have a point. These are all the things students are thinking when we suddenly throw them in a course and tell them to go to the board and find math. Was anybody scared? What was your reaction? It's fun? Some people will think math is fun at the board. Some people do it because the person next to them is doing it. Some people do it because I said to. For a lot of us, we are good at math and we like talking about it and it's fun for us. So the concept that we are throwing students into a totally new active environment and they have to actually go show stuff in front of people they're scared of is terrifying. I think that has helped me think about the emotional side for students. I compare it to my imposter syndrome. I sat next to Mike Starbird on the plane last night and he said, "I have a question for you" and I thought is that what my students feel like? (laughter) I want to crawl under the chair! What if he asks me something I don't know? At that point I should have not revealed myself as a math person, I should have played it cool for a while.StarbirdAnd you still haven't answered that question!(audience) I know! It was a really hard one. So I think that thinking about what our students are feeling and thinking about what is really scary for you to do in front of a public scene can you help you figure out how to motivate them.(audience) I have one question. A square one question. It looks like the definition of active learning in mathematics instruction was in the title of the session. However, at least two out of three presenters talked about IBL. I believe partially Ben also talked about inquiry based instruction. At the same time, analyzing literature I also came across some statements that active learning and inquiry based instruction are the same. My question to the panelists, do you believe there's some life in active learning outside of inquiry based learning?BressoudI think I can answer that. I think most of us would agree that inquiry based learning is a subset of active learning. Not in any sense are they synonymous. BraunBut I also think active learning is fundamentally inquiry based. It's guided inquiry, whatever you want to call it. At K-12 level it's called discovery learning, guided inquiry, problem based learning. There's a hundred ways to say it. They're all slightly different. It's really important to distinguish in the mathematical community at this time there is a reasonably well understood meaning of what IBL means but it's not totally concretely written down. At least among research mathematicians, like when I go to AMS sectional meetings and give talks and organize special sessions at such, IBL is a source of contentious discussion. That's my experience. IBL is too but with IBL it's basically like it is what it is. It's hardcore small group work with very little instructor lecture. The instructor does a lot of work but it's in very careful scaffolding appropriate activities for students that developmentally brings them to a certain point. The instructor work goes into the scaffolding rather than being in front and taking charge of the class. One of things that the phrase active learning can do is distinguish between that level of small group work, high level of students doing scaffolded activities in class as opposed to activities in which students have chances during class to do something active, to do a task or activity. And the reason I think that's important because the way that the mathematical community currently understands IBL is not broadly applicable to all instructional contexts. I think whatever we are going to use to talk about getting students actively engaged in their classes, we have to have a language, we have to an understanding that in the big picture it's got to be broadly implementable. And that's my take. That might just be because I teach at an institution that has large lectures and I come out of my environment, maybe that doesn't matter as much to someone teaching at a small liberal arts college. For us, at least for me in my institutional context and many of my peers in the mathematics research community that's a really important distinction.HodgeI think part of it is even political. I only used inquiry based learning until I started working with Jim Lois on the Math Teacher Education Partnership and he said, "IBL has some contentions, how about we call it active learning?" And then that's what we've been calling it since then, about five years ago when we started doing the MTEP stuff. Really, it depends upon the audience for me. I think Ben's points were spot-on in terms of do you want parts of it. People are more okay with this as an active learning technique and they see IBL as a full, encompassing idea with some strict rules which in my mind it isn't and doesn't have to be. But it can help you communicate with people and the words are depending upon your audience for me.BressoudI'd like to thank you. It's time we took a break. There'll be some refreshments at the back and we will convene at a quarter til 4. Thank you. (applause)06 08-01-16 Afternoon Remarks (Bressoud)BressoudFor the first half of the afternoon we heard some remarks on what active learning is, what it can be, how it can be implemented. I'm sure you are all convinced that it's the right way to teach. The question is why isn't everybody doing it? We did our survey of research universities a year ago. I've got the results up on the slide. 63% of the universities said that for their calculus classes the primary means of instruction was purely lecture. 18% were doing some active learning which means most of it was lecture but there might be some things, such as the use of clickers, in their calculus classes. 3% were actually teaching their calculus classes mostly in active learning mode which means that there was very little lecture within the teaching of it. Another 13% said there was too much variation so some people using active learning procedures, some people who were not. In all it was just a little over a third of the departments, 35%, said there was at least some active learning going in at least some of the sections at their universities. What we are going to look at the second half of the afternoon is what does it take to bring about the kinds of changes that we'd like to see. We've got three really good people who will be addressing this issue. The first is Dennis DeTurck, who is Dean of Arts and Sciences at the University of Pennsylvania who will give us a dean's perspective on some of these issues. The second speaker is Charles Henderson from Western Michigan University. I have snuck a physicist into our midst. Charles is someone who has really been exploring this whole question of what does it take to get better practices adopted. Why is it that people see evidence that active learning works better but they don't do it or they don't stick with it? He will address some of those issues. We will finish up with Tara Holm from Cornell University who will talk mostly about the efforts of people at the research universities transforming post-secondary education in mathematics and what's going on there to change the way we approach our undergraduate instruction. With that I will turn it over to Dennis to start.07 08-02-16 Dennis DeTurckDeTurckThanks David, and thanks to all of you ahead of time for listening to me for a few minutes. As David said, I am the Dean of the College of Arts and Sciences at Penn, but I'm also a mathematician so I have a couple of perspectives here. Before I get started I want to put succinctly what Michael was trying to say about ultramarathon running. He understands the concepts but he can't do the problems! (laughter)What I'm about here is to talk about how I've tried to encourage and support active learning practices in mathematics, but also across the sciences, chemistry, physics, biology, and even beyond the sciences and mathematics within the School of Arts and Sciences. I do want to make a word from our sponsor in a way. Something that helped us get things rolling with our faculty was a grant from the American Association of Universities, the AAU. They had this undergraduate STEM initiative where they fundedhad eight institutions where they hadwith help from the Leona Helmsley Foundation. It wasn't the moneany that helped, actually, because we spent a lot more money than we got from the foundation but it was the momentum, the impetus that that provided both with the faculty looking in that direction but also with the provost and the president of the institution so we could actually do some of the changes that we wanted to do. I'm going to skip past the next slideat one.Penn's AAU STEM Initiative, as it's called, its goal is to improve stuff because everybody has this feeling that it's not perfect. Whatever it is, it can get better. We're not quite sure how, but we think it can get better. We've all heard of this thing called “active learning” or “inquiry- based learning” or “problem solving learning” or pick a label. We've heard a lot about it. The first thing we did, and I wish I could claim it was my idea. We have the Wharton School at Penn, as Donald Trump will constantly remind you, and we figured out that we had to brand it. Whatever it was we were going to do we had to brand it. And we weren't going to use one of these words, we're going to have our own brand, partly because we found ourselves getting into these discussions of what it is, what it isn't, and “you're not doing it right”. We wanted to get past that. We want the faculty to be intentional and think hard about what they are doing in the classroom. These are research faculty who are a little resistant to change for a few reasons. One is that, they are successful faculty so they think they are doing it right, a lot of them. To convince them they should change their ways when they've been awfully successful is hard in large numbers because here we are at an Ivy League institution and we think we are distinctive and in some ways Penn is quite distinctive but in some ways it's not. No matter how much a group of faculty members or administrators or other people would present evidence to our faculty member, these research faculty members who are in some cases chemists, physicists, and biologists who deal with empirical data and try to draw conclusions from it, their response was always, "Penn's not like that, that couldn't possibly apply here." And I think that's probably a common response not just at Penn but everywhere. That's just about those institutions and we are different. You have to find some way of motivating folks I think is really important and get people to talk to one another.You can see that the elements of SAILLE as we call it, “Sstructured Aactive Iin-class Llearning”, are the things you think it ought to be. The students come to class and are engaged. Exactly what happens and exactly how that works will vary from instructor to instructor and section to section. Some of it is like don't read or prepare in advance and we are going to do something conceptually new in class. Others provide very explicit instructions on how to read the book before you come to class. Still others will provide video material for students to look at before the come or after they've had class. The technique varies. We've learned some interesting things. Neil's questions about using someone else's notes. How that rattled around in my head because it was something we tried. One professor would try to use the video another professor had prepared and it was a disaster because “it's not my professor”. We did this experiment with voiceover that actually improved the students' acceptance of this thing. It wasn't like the person was standing there, because we have a faculty member, I'll show ure you some of his artwork in a little, while named Rob Ghrisreist who does these incredible animations of calculus concepts. He draws them himself and it's unbelievable how slick they look. When he's the one talking over them someone else's students won't like it. So we had somebody transcribed what he said and then had the other professor recite as the video was going and the students said, "This is really cool!" (laughter) It's the oddest thing. This is the business slide. How did I convince, and my counterpart in engineering , convince the faculty that we needed to do something. This was data about Penn and it was data about persistence in science and engineering. Students come in and say they want to do science and do they persist in it. A lot of students come in to Penn thinking they are going to be pre-med, or thinking they are going to be engineers or computer scientists, or mathematicians, physicists or chemists. One thing we learned the first time we did it, because someone read something about Swarthmore and we said, "Hmm, I wonder if that's true here". At Penn students don't come in having declared a major in arts and sciences. In engineering they do, they have to pick a part of engineering. But in arts and sciences we don't allow them to declare a major until they are sophomores. All we had was, "Are you interested in science?" coming in the door was a box they checked in their admission application. We learned that was a really poor marker for what they were interested in when they came in. AWe discovered, after reading about a similar experience at Swarthmore, we discovered that 30% who said they were interested in science on their application were already not interested in science on their first day of school, first day of freshman year. It wasn't anything that we did! That wasn't the right thing so what we ended up usinged as a marker for interest in sciences was whether, you took general chemistry. Not everyone interested in science takes general chem but probably 80-90% of students who are interested in science take gen chem. The math majors don't but there are way more bio majors, chemistry majors an engineers who have to take chemistry in their freshman year. So we ask the question. How many students who take Chemistry 101 end up as science engineering math majors? Nationally that number is about a quarter. They start out thinking they are going to do science and three out of four don't. At Penn we fareir better, it's around 75% which we think isthat's great, and we feel good about ourselves. We looked a little bit deeper and we learned that among underrepresented minorities and first generation students that number was under 60%, was in the mid 50s. Huh, here's a differential outcome and it must be based on something we're doing. What are we doing wrong. We collected more data and learned that in fact if you looked at underrepresented minorities in our science and math gateway courses, calculus, chemistry, physics, and tried to normalize inputs, like SAT scores, did you have AP courses in high school things like that. Comparable minority students averaged a third of a GPA point lower in those classes. If this majority kid got a B+, the minority kid only got a B. That was disturbing. We didn't say they come in different, they are coming in the same, but the outcome was different. Those facts are what convinced people that we really had to act. We got a big group of people from across the disciplines to talk about it. I think that talking about it was probably the most effective thing. Probably half now of Calc 1 and a little fewer of Calc 2 and we have this course that's called Introduction to Calculus that you could think of as Calc 0, though we don't call it that, those are done by SAILLE now. It doesn't always mean the same thing but it sort of means what you expect. There's lots of experiments with active learning approaches. Some examples of this as I said are these directed viewings of videos, directing reading of textbooks or the various course materials. Robin PeP. Mantle started a course called Math 110 precisely for Wharton students because our business students have to take calculus and they found being in there with the engineers, they didn't like that for any number of reasons. This emphasis is on what Robin himself calls a "meta- curriculum". , Wwhat is it? It's kind of like the calculus concepts inventory but we've developed our own. So here are some pictures and the pictures in the northeast and southwest are some of the stuff that Rob Ghrisreist draws. You can imagine a 10 minute video full of these drawings, and they're moving, and it's kind of psychedelic almost. The students really seem to get stuff out of this when they're narrated by their own instructor. It's just one of those odd outcomes.Here's some numbers. We started counting these things in 2012 when there were just a couple of people, one in math and one in physics, actually doing what we now call SAIL., Wwe didn't call it SAILLE at that time but starting in 2013 we did. And you can see how it's grown. Also in the upper blue boxes, you can see that it's spread elsewhere. So we have economics professors doing SAILLE, we have political science professors doing SAILLE which is one of the more interesting adaptations. The question for them is here's this thing about doing problems, learning how to do problems on worksheets in class and how do you transform that into a political science perspective? We've had some very interesting conversations that revolve around that and getting information back from the political scientists to inform what our practice is in science and math. Probably the second biggest expense in terms of the dean has been lunch. I buy a lot of lunch! (laughter) The discussions involve sStudents too, but far moreor faculty members, TA's, within departments and , across departments. We have a terrific Ccenter for Tteaching and Llearning at Penn that helps facilitate these conversations. These people are really good at getting people who don't naturally talk to each other to talk to each other and listen to each other. "Gee, it might be a good idea to visit each other classes" and, no, the young professors shouldn't feel threatened by this because we are not evaluating you at this moment, this is not an evaluation, because that is always the first place the junior faculty member's head goes because that's where their life is. We have to be careful about that and careful where the conversation goes afterward. On the flipside, the number of students has continued to grow. We are expecting quite a few this next academic year. Just over 2000 students will be in SAILLE classes this year, which is about halfway to where we'd like to be.There's a lot of words on this slide. I'll post them on my website or maybe we'll have a website, I'm not sure. A lot of this is about how are we going to go about demonstrating that this works. There's a bunch of questions that the faculty ask. Again, it's the sort of nature of who they are to put an enormous amount of effort into things as long as they can expect a return. A lot of faculty members, there are skeptics to be sure, are willing to invest quite significant effort into transforming their teaching if they believe two things: 1.) That it works, at least as well as the old way and hopefully better; and 2.) once I've invested the effort I can reap returns oin the investment. I'm not going to have to be investing that effort all the time. I've perfected this set lectures and I give this set of lectures every fall semester to a different cast of students and that's how it works. My teaching life is in this box and that box can be, it's not a tiny box, it's a significant box, but it's in a box because I have these other boxes. That's important and some people have to believe that too. The most expensive part of this is space. Space is the hardest thing on any campus for any purpose. A campus in the city where there's not land to build new buildings on, it's really hard. We've been steadily developing new active-learning spaces but as it says on this slide, the picture on the upper right is from the Philadelphia IEnquirer on a slow news day when the math graduate students demonstrated because we were going to co-opt about 25% of the math/physics library to build a SAILLE classroom. It must've been a slow news day because the reporter came out, took their picture, the whole bit. They were above the fold in the local section of the newspaper, about how this terrible thing was going to happen. They were going to lose the library. Of course, this is the part of the library that nobody uses. It's the part where all the bound journals are. When's the last time you opened one of thoese? (laughter) There's the design of classroom space, I mean, how do you find places? There's some pictures, as you can see. Building new classrooms has been our biggest challenge. We have eight of them now, two of them are in the math/physics building. We did just squeeze a new biology/psychology building in and I did make sure there was a new SAILLE classroom in there. In fact, it's the biggest one we have so far. And then there is assessment. Unfortunately, when you say “Iis it better?, Ddoes it work better? , Ddo the students get better scores on the final exam?”. t The answer is, “Hhmmm.” The one year they seemed to, the next year when we did this sort of head-to-head comparison, the difference between instructors because we had some SAILLE classes running and some traditional lectures running. Intragroup differences swamped the SAILLE versus non-SAILLE effect so we couldn't tell. Here are's some statistics. Three of the four are significant in the official sense, in the .05 sense, and at least two of them indicateare, that students persist. The students in the SAILLE sections persisted into more advanced math classes in greater numbers. That's controlled for what they thought they were going to do coming in. You're taking this calculus class now, do you expect to take another calculus class? The percentages were essentially identical coming in, but they weren't coming out. There's some indication that whatever SAILLE is doing, it's helping to address the problem we started with which was URMN persistence in science and math and that's where we wanted to go. Thanks! And now I'm really done.(applause)BressoudAny clarification questions for Dennis?audienceYour persistence numbers at the end, that was for the total class population or for the underrepresented minorities?DeTurckThose numbers were for the total class population but there are subscores where we broke it down demographically. The minority students in the SAILLE classes persisted roughly in the same numbers as the whole group. That was one of the goals we had going in. We had no idea whether we were going to achieve it. There are other programs at Penn that target minority and first year students to support them in more or less a general social and academic way which is always the problem with social science. Nothing is ever "ceteris paribus", as the economists say. (inaudible question)Ron's question is do we have a conjecture for why there was this differential in performance and in persistence to begin with. I have a lot of them. Some of them are who do these students see in front of their classrooms because changing the demographic makeup of the faculty is slow.. We make concerted efforts and we've moved the needle, well, epsilon doesn't begin to describe it. Less slow in terms of women than in terms of minorities in science. Percentage-wise, and in absolute numbers too, there are more women on the faculty in science and math than there were nine or ten years ago. But minority faculty has been a tough nut to crack. There are more than there were, but again, who are the students seeing, what examples do they have. There are examples of stereotype threat. The phenomena of stereotype threat is real. Unconscious bias. We have had faculty, most of our faculty, again based on the data, have undergone voluntarily threat stereotype training, unconscious bias training. They almost uniformly come out of those sessions saying, "I had no idea, I didn't know I did that" because it's real. A place like Penn where there are a lot of sort of urbane people, there are people who feel like they do not have a place. And that's a real phenomenon that we work like hell to address. That's a problem with first gen students in a big way and it's definitely also a problem with students in various underrepresented groups. I think that that phenomenon was real but it also existed apparently because of what we did in our classrooms because when we did something else, the numbers changed.(inaudible question)We have a sort of university classroom committed that is mostly comprised of facilities people, people who know how to build things, and faculty. We actually went to places. We went to Michigan, we went to UMBC where people were doing this successfully and we wanted to know what are the elements. And what do the faculty want to do in these classrooms because for some of the classes, like the physics classes, it was essential that there be certain kinds of power, there be certain kinds of experimental possibilities so that the students could deal with phenomena so the physics class wouldn't just be worksheets. The physics professor, our highest is 4.0, he could probably get 4.0's and stood in front of the students and read the telephone directory because he is just a cool guy! (laughter) The students all want to be in his class and he's a terrific guy, a terrific teacher, and he has his students doing all kinds of really interesting things. BressoudLet's thank Dennis again. (applause)08 08-02-16 Charles HendersonHendersonGreat, thanks! It's nice to be here. As David mentioned earlier, I am a professor in physics so I apologize for that. I do like to point out to audiences like this that my undergraduate degree is in math so I have a little something going for me.What I want to do is talk about the research that I do which is on the problem of we know a lot of good stuff about teaching and learning. How can get it implemented into more classrooms. And the way that I like to frame that is in terms of the knowledge-practice gap. Let me tell you what I mean by that, and I'll do it by way of a couple of exams. One example is that we know that from research there's about 100,000 U.S. hospital deaths linked to poor hand hygiene and the reason that this happens is not that medical professionals don't know about germs and don't know that you need to wash your hands. They know about it but yet they don't do it. That knowledge isn't sufficient. We also know that about 700,000 U.S. deaths are linked to smoking or obesity. Again, it's not that we lack knowledge of the harmful effects of smoking or that any of us are unsure about whether we should eat the apple or the donut. We know these things but yet we don't always act on that knowledge. A knowledge-practice gap exists when we know what the problem is, we know what the solution is, but the solution isn't implemented. When we go to undergraduate STEM education, we have about 400,000 new STEM students that enter degree programs each year with the intent to get a STEM degree. About half of them get a STEM degree. This has been happening for quite a while. One of the main causes, not the only cause but one of the important causes, is the poor teaching practices in undergraduate STEM classes. Again, we have known about this for quite a long time. In case we forget about it, all you have to do is open your email inbox and there's probably a national report in there that says the same thing, that there's poor practices in undergraduate STEM instruction and we need to do something about it. So we know about this problem. We also know about solutions. As many of the speakers have already talked about, there's this thing called active learning. We might not know exactly how to define it but we kind of know what it is. We kind of know we have really strong evidence that it's better in a number of metrics than traditional instruction. So this is a meta analysis that I'm sure many people have seen shows that active learning increases student performance across a wide variety of disciplines and it decreases failure rates across a wide variety of disciplines. So we know what the solutions are. We also know that the solutions aren't widely implemented. This is data from a national survey, a special analysis on Harry's survey of college faculty in a number of disciplines and you can see that especially in the STEM disciplines but in many disciplines professors report a use of traditional lecturer and they report a low use of things like clickers that might be indicators of use of active learning strategies. Again, active learning is not widely implemented, I'm arguing. The main argument I want to make as change agents we should be thinking about working smarter and not harder. What I mean by that is typically in undergraduate STEM education we are focusing on changing individual instructors, going around person to person with the hope that we can reach enough to make a difference. My main argument is that we need to think on a structural level. If we go back to the diet and nutrition analogy, focusing on providing information to individuals is like putting product information labels on the ice cream. If we want people to eat less ice cream we're going to tell them, "Do you know ice cream has a lot of calories and fat in it and you probably shouldn't eat too much" and it turns out that it doesn't make a lot of difference. There have been some studies that have shown that if you give people a smaller bowl and a smaller service spoon and serve themselves, they can serve as much as they want to but they tend to serve themselves smaller amounts of ice cream than if you give them a larger bowl and a larger serving spoon. Changing the environment can have a very powerful effect.What does this have to do with undergraduate STEM instruction? I want to frame this in four categories of change strategies. These are categories that I developed through a literature review with two of my colleagues, Andrea Beech and Noah Finkelstein. We looked at 191 journal articles that had to do with change and undergraduate STEM education. We developed two categorization criteria. One was what was the focus on the change initiative. Was it a prescribed outcome or an emergent outcome so we do know what the outcome is when we go into the situation? And two, is it focused on changing individuals or is it focused on changing environments and structures? When you put those together you get the four types of change strategies: disseminating curriculum and pedagogy; developing reflective teachers; developing reflective policy; and developing shared vision. I'll show you some images to give you an idea of how these work. In the prescribed final condition, the way that they work is that you've got an expert coming on, or someone who thinks they're an expert at least, and in the focus on individuals this expert is telling the individuals what they should be doing. It's someone going around saying, "You should be using active learning, let me tell you about it, I've got some evidence you may be interested in." The prescribed environmental side, often you have an organizational hierarchy where the expert is pushing information through often through a policy change, for example. Like you might have a dean who wants you to do something. On the emergent side you can think of these as inquiry cycles that either an individual goes through, so an individual will try something new in a classroom, reflect on how it went, revise it and try it again, and so forth and probably get better and better at it. Or it can happen in a group. A group can co-develop some knowledge, refine it, continue that development. Some people call it organizational learning, for example. These are the four change strategies. I'm going to talk about each one of them individually and I'll use these little icons to keep track of where we are. I'll talk about each one of them very briefly just to give you a feel for what the state of the knowledge is for each of these. The first one, disseminating curriculum pedagogy, this is the one that STEM change agents naturally gravitate to. I'll argue that it seems to work pretty well for getting information out to people and motivating people to try but it does not change the environment. This leads people to use strategies inappropriately and discontinue. The data that I've got here is from a survey of physics faculty that I was involved with a number of years ago. It was a pretty good representative sample of physics professors across the U.S. We asked them about use of a set of 24 research instructional strategies and you can see on the list that they know about quite a few of these. Their use lags their knowledge. That was across the board. On average they know about seven times more of these things than they use. This isn't terribly surprising. You can't use all these things together. The more interesting result was this, that when we took all 24 of these research based instructional strategies and put them together, what we found was that almost everybody knows about at least one of these active learning, research based instructional strategies. More interesting, surprising to me and to most people who see this for the first time, was that about three quarters of the faculty had tried one of these research based instructional strategies. In fact, the biggest loss was this discontinuation. Of the people who tried these things, about a third of them ended up not using that strategy. By discontinuation I mean they don't use that strategy and they report not using any of these active learning based strategies. I don't have time to show you this data today, but the other thing that we find for people who do indicate on the survey that they use these strategies most of them aren't using them as recommended by the developer and probably in ways that degrades the usefulness of the strategy.The reason we think this happens is because of the situational factors that these instructors find themselves in. This model that Melissa Dancey and I developed awhile back, we call it a toy model, it's a physics term. The idea is that it's a simplified view of reality. People make decisions based on their individual characteristics, beliefs, knowledge, values, prior experience and so forth, and the situation that they're in. What we found was that most instructors are actually somewhere over here. You sit in the back of their classroom and you see somebody lecturing, you might infer from that this instructor is a strong believer, they can translate information to a student via lecturing and that this is the most effective way of teaching. When you talk to them this tends to not be the case. I know few lecturers that really believe that strongly. What we find is that most instructors are in some sort of mixed stage. They have some semi-alternative ideas about teaching and they have some traditional ideas about teaching and learning. Almost all of them work in these very traditional environments. If you're a change agent you have to think, "If I'm going to focus on changing instructor beliefs it's only going to get me so far. If I really want to make some change then I have to focus on changing the situation. If I can just make the instructional situation neutral then I have much better likelihood of alternative teaching modes happening."The last thing I want to say on this focus on development and dissemination. I'm sure many of you have seen this before, to remind you that in diffusion and innovation theory that the first people to adopt a new thing like active learning are different than anybody else. And by the first people to adopt it I mean you. You are different than your colleagues because you're willing to try these things. The things that resonate with you and the other early adopters are probably not the same things that resonate with your colleagues and the non-early adopters. Just because something is working for you or because you were able to convince one of your colleagues to try something and it was effective. Don't assume you can scale that to the rest of your colleagues, it probably doesn't work that way.I'm going to talk very briefly about developing reflective teachers. This was the promised faculty learning community talk. And I'm not going to say a lot about it because a lot of it has been done for me. These developing reflective teaching strategies, an individual who's going through self-reflection cycle often with the support of other individuals and this can be very good because this means that the instructor is developing something for themselves that fits with their local situation which is often not the case. On the other hand, these inquiry cycles often don't build on or add to the knowledge base so each person is reinventing things for themselves, so it's a very inefficient process. If you are interested in it, Milt Cox at Miami University of Ohio has a great website, lots of resources. He's definitely the expert on faculty learning communities. I'd highly recommend you go to his website and look at it for more information. Let's talk about the developing policy strategy. This is the prescribed strategy that is focused on environments and structures. The most common large scale model for this is John Kotter's eight stage leadership process. This is a process, a sort of typical top-down process where a vision is created, there are strategies that are put in place to implement that vision, strategies to bring people on board, and strategies to institutionalize the vision. He's got a great book, I highly recommend it. It's advice that's built on his experience as a consultant and organizational change expert over a large number of years. It's a very readable approach. Very few change agents make use of his advice. I want to give you an example from the STEM world where we did an analysis based on the Kotter categories and this is an instructional strategy called scale-up which is very similar to the SAILLE strategy we just heard about. It was developed by Bob Beichner in physics a number of years ago. It spread to a number of institutions and departments across the country. It involves radical changes in both the department structure and pedagogy, again very much like SAILLE. Incidentally, the results are similar to the results we just heard about for SAILLE, that it has much better, higher retention rates and disproportionally positive effects for underrepresented minority students so it's a very reproducible effect.I want to give you two examples of how, various short examples. You’ve heard one example of how a SAILLE-like thing was set up. I want to give you two additional examples. We pulled these examples from a set of case studies that we did from about 20 setups of successful implementations of scale-up around the country. One was what we called Midwest University. This was a top-down change effort. It came from the upper administration. There was an opportunity the administration had. There was a flood, they had to rebuild. They had some money. The provost wanted to set up these scale-up rooms and that's what happened. They started with two of these classrooms. The important thing here was that they involved the center for teaching and learning early on and there was mandatory training for people who used these classrooms. They were centrally scheduled rather than being owned by a particular department so this helped to circulate use. Some of the department chairs were on board and helped to further this idea with their faculty. The success of these rooms and the scale-up pedagogy led this to spread to other departments. So now it's used by a lot of faculty, a lot of departments, a lot of students at this university.Another example is a bottom-up change. This is what we call Southern U. It was led by two faculty members in two different departments. They saw an opportunity. They said "our administration is concerned about these low pass rates in these gatekeeper courses and the scale-up room can help with this problem. Can you fund the development of this?" And so they did. These results were successful. The department heads were brought on board. They instituted a procedure where faculty can visit other faculty while they're teaching in these special rooms and then this spread to other disciplines. It's now used on ten classrooms in ten departments. To sum up these policy changes. Contrary to popular belief, it can happen top-down or bottom-up. History is important. In all of these case studies we found that actually groups of people who were working together prior to the beginning of the scale-up initiative. Another thing that the structural change that having this redesigned classroom helped with was that it added visibility to the effort. When a tour group came on campus they would go past the scale-up room. The scale-up room would be featured on the glossy university magazine that got sent to the alumni. Donors would be brought by the room and so forth. It was a focal point.Finally, shared vision. This is the strategy that involves environments and emergent change. This strategy involves groups and I want to contrast that to the groups that are involved in the faculty learning community. In the faculty learning community the unit of analysis or the unit of change is the individual faculty member. In what we often call instructional development teams in this shared vision category, the unit of change is the group. That is an important distinction and we find that people misuse the term sometimes. You have to think about what is it that's really changing and what is it that you want to change. One of the conceptual lenses that we use to look at this is complexity leadership theory. This is cyclic change model where there's a disruption phase, some new ideas are developed because of this disruption, and the good parts of those ideas are then amplified and fed back into the cycle. I'll give you one example of that. This is the Carl Wieman course transformation model. Carl Wieman, who was a physicist at Colorado and at University of British Columbia, now at Stanford, he started this initiative a number of years ago. The example I'm going to give you was from the upper level physics courses. He'd done this in many departments but this was one example. What he did was he started with an upper level electricity and magnetism course which is a course taken by juniors in the physics department, a standard course taught everywhere. He had 13 instructors, a large percentage of that department at that point. They met seven times. They set up the goals for this new course. There was a lot of work by a postdoc who was involved in this change initiative and the task that this group was set up with was to develop an assessment instrument. The way that they encouraged novelty was by using a core question. What's junior level electricity and magnetism all about? How's it different from introductory level electricity and magnetism? This is, of course, a more interesting question for faculty to talk about than let's write out a set of measurable goals and objectives which faculty will go running if you start talking that way. They got engaged in talking about this. What is junior level electricity and magnetism all about? There was significant behind the scenes work by the postdoc and other people involved in this project so they would talk to faculty individually, make sure that things were moving and these course level changes led to broader changes within the department. What happened was they took this example of the electricity and magnetism course, they developed a set of big picture learning goals for that course and then extrapolated that up to more generalized learning objectives for the upper level courses, very similar to some of things we heard earlier with respect to mathematics learning goals. Those were fed back into other upper level physics courses. This really developed a new shared language for the department to talk about what they were doing in their new teaching and learning processes. Just to sum up the key features of this instructional development team approach from complexity leadership perspective. There's a disruption phase that requires some support of these teams. If you just let faculty go they aren't going to do anything by themselves probably, and some interdependence. There needs to be a reason for this group to work together, a shared product that they're working on. For novelty there needs to be some diversity but not too much. There's a sweet spot between too little and too much. This idea of a simple rule was very powerful. Another example that we use is that's been successful at another university is that students should do science in the first two years. It was a simple rule that people can get behind and judge their ideas against. In the interpretation phase, facilitation is important both within the teams, both within the group setting and individually one on one outside of the group setting and then communication is important outside of the teams as well, so taking these ideas and making sure that they get put into the broader landscape. So I will end there with these three takeaway messages that I will let you read.Maybe I should say thank you? (audience)BressoudAre there clarification questions for Charles?HodgeAre your articles available on your website?HendersonYes.HodgeThanks.09 08-02-16 Tara HolmHolmThank you for making it this far. I think we'll maybe at the end of my talk have another dance session so that we can activate our brains again.I want to tell you a little bit about the efforts of a group who call themselves TPSE Math: Transforming Post Secondary Education in Mathematics. I think we all agree, or maybe we don't agree, but the landscape of higher education has changed dramatically in the last century, or longer than that. That's a picture of the School of Athens, the part that is mathematics. The Socratic teaching method, you know, active learning isn't new. In any case, the landscape of higher education has changed. I think higher education is essential to economic mobility in this country, it's an essential aspect of the American dream. Colleges and universities at the same time and students are under severe financial pressure. Tuition increases can't continue the way they have. And as we are gathered here today to talk about new pedagogies we have to figure out how best to implement those and incorporate the ones that work and figure out why they work and which ones do not. Another key part of TPSE Math's message is that mathematicians have to play a central part in this transformation. If we don't, other people will and we risk losing not only the substantial role that math departments play in undergraduate education but also the health of the U.S. math research enterprise. There's some concern that TPSE Math is very R1 focused, but I would argue that we are all stewards of the U.S. math research enterprise because we train students who go on to become mathematicians. We train students who go on to become ambassadors of mathematics, who go on to become fans of mathematics and make the policy decisions that affect our funding, and so on and so forth.Before I go on I will introduce the TPSE leadership. We are funded by Carnegie Corporation and the Sloan Foundation. Carnegie Corporation of New York has funded a lot of work in K-12, pre-K-12 math education and they saw this as an opportune moment to talk about undergraduate education. They gathered together a bunch of mathematicians to talk about how change could happen. Phillip Griffiths and Uri Treisman were both there and David and others. One of the things coming out of this was this group, TPSE Math. The leadership includes, it does sort of go diagonal. It started with Phillip and Ernie, and then Mark and Eric joined. And then Jim Gates and I and Karen joined, and we are the board of governors. The group has been incorporated at the University of Maryland with the executive director Brit Kirwan who is a mathematician but who has most recently been chancellor of the University of Maryland system. He's retired from that, he's now the executive director of TPSE as well as the chair of the conference board of the mathematical sciences.The vision of TPSE, the simple message of TPSE is that postsecondary education in mathematics should enable any student regardless of his or her chosen program of study to develop the mathematical knowledge necessary for productive engagement in society and the workshop. And I think is something we all share. So maybe we don't have a very strict definition of active learning or share a strict definition of axioms about beliefs about mathematical talent. Talent isn't a good word but I have to call it something. My axioms would be that mathematical talent is uniformly distributed among the population and nurturing it and harvesting it is the smart thing to do. It's crazy for us to leave any group out. And I would also argue that everyone, and I think that this is a large part of the active learning philosophy, any student can have an authentic and meaningful mathematical experience and to foster that for all students. As I said, we need not just research mathematicians, not just college math teachers, we need ambassadors and users and fans who will support us. And so the TPSE Math vision incorporates this. We believe this is a national discussion. We need community wide action and TPSE hopes to foster community wide national level discussions and action. We hope to narrow the gap between how mathematics happens today and what mathematics students see in the classroom. The National Research Council Report Mathematical Sciences 2025 gives a beautiful description of the wide variety of ways that mathematics arises in the world and many of our students leave mathematics thinking that it's just college algebra or calculus and what a shame.This is by chance the same picture that is on the front of your program. If you google "big lecture image" this is the first picture you will find. Of course this is what we hope to change. One might hope for small active classrooms like this, this is what Dennis was telling us about. One might also hope and figure out how to adapt some of the techniques for big lectures like that and everything in between. Some surprising statistics that motivate us and some surprising statistics that I've learned about since TPSE. The first set of statistics is about of course our leaky pipeline. This first question comes from David's calculus study. How much more likely are men than women to choose not to continue beyond Calc 1 even if Calc 2 is required for their major? About twice as likely. Bachelors degrees in math, 40% of them are earned by women. 32% of PhD's in math and stats are earned by women. That includes stats and biostats. If you look at pure math it's down to 20% or less. Postdocs in math are about 25% women, tenured faculty in doctoral granting math departments 14%, top 50 departments 4%. These are the models that our students see so this is a challenge. Some other surprising statistics are from two-year colleges and questions of transfer. Out of curiosity, people in this room, when you were undergraduates did anyone take any courses at more than one institution? As an undergraduate did you get college credit from more than one institution? It looks like somewhere between 10 and 20%. What percentage of undergraduate students attend two year college? This is not students getting four year degrees which I assume you all have, but students who are just attending college. 42%. The next question is the one I asked you, 46% of students who finish a four year college have enrolled in a two year college. You may say maybe they took one course at a community college in the summer, what does that 46% really mean but of that 46% 10-20% took one semester or less but nearly half of that 46% took five semesters or more of their coursework at a two year college. So this is very different from how it was a hundred years ago. What percentage, this gets to the difference between lower income, higher income. 44% of lower income attend two year colleges while only 15% of higher income, and there's also a question about URM's. 39% of white students attend a two year college compared to about 54% of the population, compared to 56% of Hispanics who make up 21% of the population. So the differences affect different groups in different ways. Another fact about two year colleges. The percentage of students attending two year colleges who take math courses who are not math credit bearing so this is pre college algebra, 60% and of those, 70% never complete a math course and that stops you from getting a degree. So math is a barrier for over 40% of students in two year colleges which remember is nearly half of our undergraduate students.The strategic priorities for TPSE Math include coherent pathways which is transitioning between different institutions. There's been a lot of work done already. Uri Treisman was very involved, the Dana Center at University of Texas has been very involved in, for example, here in Ohio all of the math chairs from all over the public institutions got together and created various articulation agreements about how credit would transfer and so on and so forth, and this is happening now in a lot of other states. Enhanced alternative pathways, a focus on the upper division and this is building of course on work of CUPM from MIA and many others. This is I think an exciting time. There's so much change in the use of mathematics. What we see in the mathematics curriculum hasn't changed drastically in the last 50 years. The last drastic change was Sputnik era, push calculus down, make AP calculus part of the high school. I hope now we have the opportunity to change some things. New teaching students, of course, that is what we are here to talk about today. I'm very excited to read some of your references to about how we can lead some of this change through both sparking national conversations and supporting chairs. And of course there are questions about graduate education, not changing how we produce research mathematicians but changing how we do all of the extra training in graduate school to prepare them for successful careers.Some of our key activities. We've had meetings, strategic planning, we've met with funders, we've incorporated at the University of Maryland. The exciting development is that we are having a chairs plus one meeting. We see the chairs as the drivers of change within departments and we are having a meeting with 150 department chairs plus an extra leader of their choice from their department to discuss this and we are looking for a broad representation of people from different institutions geographically sparse across the country and so forth.And the last thing I want to mention, and this is come up a number of times, growth mindset versus fixed mindset. The underlying question is can mathematical success be achieved through hard work or only with innate talent? This is an insidious question. The smart label helps reinforce the fixed mindset. There's the reference that Ben gave about Carol Dweck's article in Scientific American that I agree is good reading and good discussion for conversations with students. But labels matter and there is this interesting statistic from Harper's Index about the factor by which white public school students are more likely to be labeled gifted than black students by a large margin. I think the other thing that is interesting about growth versus fixed mindset is a very important discussion for us to have because I think that you can replace the question can good teaching be achieved through hard work or innate talent. Certainly I have many colleagues who would say, "Oh, you're a good teacher so that's natural for you, how would I do that?" I don't know how we have this conversation but it's an important one to have.The question of efficient sharing of materials and resources that has come up. Not just successful materials and resources but how they can fail as well and identifying the salient features are and which aspects can be thrown away. This is a big open question. I was chair of the AMS Education Committee for a while and I wanted to know what innovations were out there because I was convinced that pretty much everywhere there is something interesting going on and I got about eight, maybe a dozen departments to send me a paragraph about what they were doing. There are 2000 institutions in this country and I don't know how to solve it and I would love feedback. Public relations, I think it's getting worse. When you talk to people on an airplane and you tell them that you're a mathematician, oh my gosh. Sometimes they tell you, "Wow, you must be smart" so fixed mindset blah. Sometimes you get some interesting questions like what do you do. More than half the time I get the answer of "my 4th grade teacher" and "calculus" and we are stewards of the math community and we need know how to address with this.Connections with the policy world. The PCAST report has come up a couple of times. For you that don't know, this is the President's Council of Advisors on Science and Technology. They wrote a report called "Engaged To Excel" in which they made several recommendations about mathematics. They identified mathematics as a barrier to STEM completion. They suggested for example that there be new curricula designed by faculty from math intensive disciplines other than mathematics, and this got a lot of rise out of the math community. I had a discussion with my dean who was one of the lead writers on the PCAST report. He's a physicist. I said, "You didn’t really consult any mathematicians". They talked to two mathematicians. They don't have any mathematicians on PCAST. They looked at the AMS and said, "Well, there are no publications.” APS, the American Physical Society, has publications on undergraduate physics education. AMS does not so they said, "Well, there must not be any." They didn't know that we were doing anything. We need to connect better with the policy world. It's an open question how you bring about a cultural shift in the value placed on teaching. Leadership matters and the value a department places on teaching matters.Let's work together. One of the things you can do is send me a postcard. I put self-addressed stamped postcards on the table so if you have any comments or feedback or you just want more information about TPSE math you can go to our website or send me a postcard.(applause)BressoudClarification questions. Yes?(inaudible question)HolmI had a brilliant idea for a publicity stunt which would be to produce. You know, if you have a Windows machine it might have a sticker that says Intel inside. I think we should have a math inside sticker that should go on just about anything and everybody.(audience) With just a slight dig at the physicists. One of the things happening as I look at it a lot of the sciences want a lot more math. Except our math curriculum was designed to help physicists which means you have to get through three semesters of calculus to do anything. Can you think of ways or are you working on ways to come up with other, my description is traditional upper level courses that don't really conceptually rely on calculus to be taught without calculus prerequisites because there are upper level courses so we require three semesters of calculus to do combinatorics or graph theory or all kinds of things the biologists and the computer scientists want and the economists, and structurally it probably needs to come from R1 institutions and places that have enough prestige and clout that when they do it the lower level institutions can say, "We have to keep up with Penn and Harvard and Princeton." Can you figure out how to change that?HolmThat's a very good question. As part of our enhanced and alternative pathways that's very much on our minds and is also on our minds at Cornell. We have the econ, I said econ because the econ faculty were in our department the other day saying "this is absurd, we don't have students who can do physics, we want something very different. Can you help us?" Certainly one of our messages is that we need to talk to partner disciplines and we need to be good partners to those disciplines so yes. Thank you and please write to me.BressoudLet's thank Tara and then we'll open it up for general discussions.HolmDo I need you all to stand up and dance while I say one-Mississippi, two-Mississippi, three-Mississippi, thank you! (laughter)10 08-02-16 general discussion 2StarbirdI think I just missed something. You said that changing the culture at research institutions to give more value in teaching. What I didn't hear was how to do that.HolmI don't know. The chairs are the, and the deans, and the personnel committees.DeTurckI think that's going to happen very slowly. But it is going to happen. It's already happening. Some of things we are doing may seem symbolic but they matter. Faculty every year write an activity report, a summary for the dean and the department chair. We've added more detail than "What did you teach" and "how many students were there and what rating did you get?" That was the sum total of it a few years ago. We don't know how to conceptualize what you taught this year. Have you participated in SAILLE? We make it explicit that these are things that are being considered when we do raises at the end of the year, when we do promotions and reappointments so when we advance people up the ladder we actually include things like that and we insist on it. What we can't do is change the mindset of people all at once. The people that matter in that world are external referees, external reviewers of work when we send out for letters for someone to get tenure. That's hard because they often don't have direct exposure to how somebody teaches. They may have heard them give lectures at conferences but they've never observed that person's teaching directly. They'll write, "I've never seen Professor X teach but he gives kick- ass talks at the society meeting so he must be a good teacher." That is something, but there need to be other ingredients in the dossier that do address that, whether they are observations that do matter for evaluation purposes or something more holistic. Part of the problem is that it's hard to evaluate teaching. We ask this questions all the time. What is a consistent, reliable, efficacious way of evaluating the effectiveness of somebody's teaching? The sheets that the students fill out are some indication of how much they liked the teacher but not necessarily an indication of how much they learned. We can ask better questions and sometimes we can get better answers but not uniformly. Alas. That's one of the problems.StarbirdLet me phrase it differently. Do you have any sense of what would be a better proportion for decisions such as raises, what percentage would you view as an appropriate balance in the teaching/research/service spectrum you think would be appropriate at a research institution for teaching?HolmAt Cornell raises are decided in a back door meeting by some people. (laughter) It's a partly private institution though.StarbirdI was just wondering if TPSE or other people who are thoughtful on this have any sense of a quantitative suggestion. By the way, I don't have an opinion myself so I'm not arguing for anything but it would be interesting to hear somebody actually say, "Well, I think anybody who's in this business, 30% of decisions about raises or things, or 10%, or it should vary from person to person.” That seems sensible to me, different people do different things in different amounts. Maybe there should be a cultural appreciation of that variety. That seems more sensible to me. But maybe an articulation of some matters may be useful. I don't know.BressoudLet me jump in with some of the things that we observed from the calculus study we did in 2010 and the case study follow ups done in 2012. Some of the most successful departments in terms of undergraduate teaching got there from a real shock to their system. Their calculus was failing seriously and not just their dean or provost but the board of trustees was concerned with how bad things were getting in the department. What they did, and I'm thinking particularly of Michigan but we also saw other examples like this. You had a really effective chair in place but that chair was able to mobilize senior faculty but that was a quarter of a century ago and they have managed to keep senior faculty involved in this process. Not that every senior faculty member is concerned about what's going on in the first couple of years in the undergraduate program but they have a strong core of highly respected senior faculty who are really monitoring what's going on, they are mentoring junior faculty, they are making sure that the process continues and is strengthened. Michigan did get a very strong shock that started this. They were able then to continue with the change. One of the things that we've seen broadly is the importance of monitoring what is going on in your programs. One of the most striking things that we found was that almost every department said that placement is the key. Getting the students into the right courses is one of the most important things that we need to do to have a successful program. Then we ask them if they are actually looking at data about how well your placement program is working and only 40% of the programs were actually looking at any data to its effectiveness. If something's important you need to monitor it. When you come up with problems and there's lots of problems out there if you'll look them, then that will give you the momentum you need to make some sort of change. Monitoring data I think is one of the most critical factors departments need to do and that's relatively easy to convince them. You think your placement is working, well, look and see if it really is. It's not as if you really need to make a huge dramatic change. You need to set in place this process of looking at what you are doing and thinking about how you might do it better and that's very much in line with the lower right hand corner of change that you talked about, Charles.HendersonWhile we're talking about culture change let me talk about a few things. Culture change is a very ambitious thing, it's not just a tweak in the system. You really have to investigate what does the entire system look like and what is out of alignment. Typically what people want to do is they want to change one small aspect of the system and that won't result in a cultural change. That change will get partitioned off and get pushed to the side and spit out quickly. It's really important to think of how the culture is currently not aligned towards caring much about teaching, how much of that is embedded in the language and the practices that are used by universities. Probably you won't have the power to change all those things but you need to think about them all and strategically act on more than one of them at a time. If you push for only one at a time you will get rejected and spit out.HolmI will say I have hope. The median age of my department is somewhere in the 60s. I see a difference between the younger half and the older half in the commitment to undergraduate education and I see that in other departments.(question) This is something I think about a lot but I have no answer. I'm directing at this at everybody, but especially to Charles. What is an ambitious but reasonable set of short term, let's say three to five to ten year goals that we could actually achieve and what are the ways we go about achieving that. There are certain things professional societies can do, there are certain other things that can be done by TPSE, but essentially everything's got to happen at the local level, institution by institution, as we've heard. Collectively, talking at the national level, what is a reasonable goal to set or a collection of reasonable goals?HendersonI don't really have an answer for that because it's a really hard question. You can point to examples of places where there have been successful changes. One example is University of Colorado at Boulder who has done some very significant changes in the way that they have treated teaching and learning in the STEM departments. Yes, they had a Nobel Laureate and they had multiple millions of dollars in grant programs and they had all of these resources to use in that process but they went about the process systematically from focusing on individuals but changing the structural features of the university down to changing the way teaching is evaluated. That's critically important. Also, the language that people use. Teaching had become a part of a typical departmental discussion at Boulder and that was a key piece. In my mind we spend a lot of time talking about incremental rewards. If we give this person a couple percent increase that's going to make a difference but I don't think that's where the differences happen. The differences happen in terms of what is it that people talk about on a day to day basis that's going to add to the respect that you feel. If people are always talking about teaching and how to improve teaching . This is where many people are going to put their energy and time working on improving that. That's where cultural changes come from. That didn't exactly answer your questions. It can happen, it's not easy. I don't know how you get other places to emulate the Boulder model or something like that.(question) I have a comment and a big question. One of the things I think is not easily fixed but everyone can work on. I went to a large R1 school, I wanted to teach, and I was denigrated by most of my advisors and committee because I was throwing away a career. I went to the AWM meetings to try to find support there and because my question wasn't "I have a kid, how do I balance life" or "I want to go to an R1 school, how do I deal with sexism?" I was basically told we don't have anything for you. Not in so many words, but I definitely got the sense that because I wanted to teach I was setting women back 50 years and I should've even go into math. I came from a large R1 school where most of the grad students don't go onto teach at R1 schools because there's not enough space. There were some teaching assistant faculty who were fantastic and some full professors who were fantastic and supportive but there were a lot that were loud and weren't. There were a lot that denigrated even the teaching assistant professors because they went into teaching only because they weren't good enough at research. I have gotten emails since I've left from students and students at other schools who've said, "I'm really good at research, I've got lots of publications and I could go R1 but I don't really want to and my advisor is mad at me now. Where can I look for help, where can I find faculty mentors without pissing off my main advisor who I need to keep happy because I've got to graduate." At bigger schools, having some faculty members who stick up and say it's not a failure if you want to teach. Everybody has different things that you are interested in, different things you are good at. I may not be the next Fields medalist but maybe I will teach him somewhere. Everybody who likes math can generally point to a teacher somewhere and that person was good at math. I think that's one thing that would help change the culture a lot.Along those lines, you guys are all at big schools. I'm at a small liberal arts school. There are six math professors. We only have 1500 students. I don't have a problem incorporating active learning. In fact, during my three year review I was asked if maybe there was too much active learning in the math department and maybe some of us should think about lecturing! (laughter) So that there's a difference. I think Ted once said if everyone did IBL you'd lecture just to have a difference. It would be really nice for me if I could have the big schools start doing IBL then I could point to them and say "We are doing what the Joneses are doing" and they'll be like "Great! good job!" How can I help more than sending in a tweet? My chair is unlikely to be at the chair plus one meeting.HolmWe need people, when I said all institutions I mean four year colleges too.(audience) Can I talk to you afterwards then? So how can we little ants help? My school's not going to pay for me to go to conferences but I want to help so what can we do?DeTurckSome of that is really working from within. We have to overcome this notion, and I face it at Penn in a different way, is that all that stuff doesn't apply to us. The big schools aren't doing it, so why are you doing it here? It's just the flip side of "Penn is different, it's not like all the schools they did the research at." It's that notion that our situation is singular and that this data doesn't apply to me. Helping people to see the absurdity of that position is important. Right at home, helping a dean to understand that, or the provost or department chair, or another colleague in another department. Why are you mathematicians doing that? That's not how we do it chemistry. We do it the right way here, w. Whatever the configuration is. If there's a small liberal arts college or a consortium of small liberal arts colleges because there are a lots of regional consortia of small liberal arts colleges that collectively have a lot of clout, itthat may be that the way for someone at a small place to have that kind of effect is to work with colleagues at other institutions as well as across departments. One of the most powerful things for me has been getting faculty together across departments. We have discussions about what does the economist want from Calc 2 and it's not Stokes' Theorem. (laughter) There is something else they want. What does the chemical engineer want? They do want Stokes' Theorem. Also having discussions where, what is it that the students don't seem to be able to do when they get there? We all have the experience of teaching something on the third of the month and we come back to it in some way on the 17th to use it and the students say, "You've never taught us that, we've never heard of that before. Derivative?" That effect gets magnified when they leave our classroom and go to somebody else's. Having some way to communicating back and forth is essential and is helpful for getting support in pedagogical experimentation I think.HolmI want to respond to two things. First, your comment about going to a teaching job seen as failing. Think about your colleagues who left academia. This is another cultural problem we have. People who leave academia entirely are considered the worst of the worst. That's absurd. I acknowledge that it's a problem and in my small way in my department we are trying to fix it. I hear you.In terms of how can you be involved? TPSE has many models that we look at. One of the models we looked at is division and change report. Biology went through a similar phase. Biology had PULSE fellows, we thought it would be funny to have TPSE fellows! (laughter) We also view Project NEXT as a natural model. Project NEXT has been incredibly successful for what it's achieved in four year colleges. Yes, not everything transfers exactly because we have large lectures and we have to figure out how to adapt things but your experiences certainly inform our priorities.(question) I didn't say my name last time. I'm Debbie Borkovitz from Wheelock College. I really want to follow up on both of the things that Susan said. I went to MIT. My advisor told me that being interested in teaching was like being interested in golf. It was nice to have a hobby but it was irrelevant. I went to Wheelock College after MIT. I thought I was kind of dropping out of the math community because I was interested in working with preservice teachers. This was in 1994 and pretty much mathematicians weren't doing that then. I learned from math ed people, I learned from K-12 teachers, I learned from everyone that I could. I could do that because I was at a small school with a different reward structure. We have to have excellence in teaching to get tenure. Scholarship is less of a thing. I don't really like your framing, Susan, of how can we help. You are doing the real work. I don't say this to be arrogant but I've been doing inquiry based teaching for a very long time and I'm not on any of these panels because I'm at a small school and it's a lot of hard work. I do a lot of work keeping the program going and I decided to prioritize that. I think we might know stuff that you don't know because we don't have the obstacles that you do. We're smaller and more nimble in that way. It's not just how can we help you, but what are you missing from not having us up there? How can you help us? How can you have resources to help study us? We have no traditional program at our school. We did not come from a calculus sequence. We come from the kindergarten movement. My school's been doing active learning since 1888. There may be something in here and it doesn't just automatically scale, it's more complicated than that. HolmPlease, send me a postcard.(audience) I will.HolmI have to add. Your comment reminded me of one thing. As rosy as I am about I hope that change can happen now there are obstacles. I was at this big funders meeting I referred vaguely to and a dean from a top ten university said to me, "Focus on that enhanced alternative pathways because with pedagogy, forget it. You'll never get there." And he's a dean at a top ten university.(audience) Just to respond to the last two comments. I just want to ask if any of you are familiar with the CIRTL network? It's an NSF center, I think it stands for the Center for the Integration of Research Teaching and Learning and what they do they are using graduate education as the leverage point for training faculty who understand effective teaching practices and are committed to effective learning. The great thing in their meetings a lot of what folks discuss is these exact issues of being a graduate student and not being encouraged to teach, that's not the best use of your time. What the network has done has created a network of 60 institutions who are trying to instill CIRTL's principles. One of the things they're very good at they've done a lot of research on what it means to be in the network and what does it mean to produce CIRTL work and promoting those principles. The research shows that graduate students who go through the CIRTL process are much more likely to attain faculty positions than those who don't. There's a lot of growing research around this and that is something that should be looked at.(audience) To piggyback on what Susan said. I'm at a department of two. I'm now the most senior faculty member of the department which is an odd position since I've only been there two years. My administration has already accepted inquiry based learning. They think that it's great. I've gotten grants before for my teaching. One of the things that came up is that this is a resource problem. You have to have classrooms that work. All the classrooms at my school face forward and are built on the lecture model. How do you pay for this stuff if there's no institutional money for it? Just wondering if there's resources out there.HolmMy vision for TPSE is that I hope there will be resources. They're not there now but I hope there will be. When you talk to administrators a lot of the discussion is that at the end of the day it can't cost more but there has to be start up costs.DeTurckThe same way with changing pedagogy. Individuals changing. There has to be an investment in learning but it has to attenuate. HolmAlso Penn has not only the fancy SAILLE classrooms but they have the standard desk, the little chair with the desk attached, but they have wheels and they were super easy to move around and they were awesome. If your desks get replaced get them.(audience) Thinking about structures of inertia against change. One of those is the students, because what the students want is the easiest method that requires the least out of them, by and large, which means they will be quite resistant to something like inquiry based learning. I live with students, I talk with them, and they say they never read before coming to class because the most efficient thing to do is wait until the instructor explains it in class. It's a waste of time to come to class prepared. Given that and what shows up in various processes often tends to be student evaluations, what they like. Any idea or research on what's the critical mass you need to go into with a chance of success and not get beaten back? If you are one of two teachers at a place, change is not as hard because you do it and then convince the other person to do it. If you are one of 30 and your section is one of 15 then it gets harder. SAILLE, you made the whole institution do it.DeTurckIf I could only make that statement with conviction!(audience)Do you have an idea or research on what's the critical mass you need to get involved to for it have a chance to be sustainable?HendersonCan I commit on this one? There's a common misperception that students resist active learning. It's not to say it doesn't happen because it clearly happens. What often happens is that if you have a class of 100 students there will be two or three students who really don't like it and they're going to let you know. And they might even let your department chair know. If you were to survey the class it's two or three out of 100 and there might be 20 students in the class that really like the active learning strategies. The data set that we have. We have this thing similar to Project NEXT in the physics world called the New Faculty Workshop. We bring in new faculty and they're usually at the beginning of their second year of teaching and we tell them all about this great active learning stuff and we encourage them to use it. One of the follow up surveys that we've done with the New Faculty Workshop participants and this is roughly 500 people responding that had been at the New Faculty Workshop over a period of years. What we ask them is to tell us how their student evaluations went because we know this is a common perception that people have. What they told us was that about 40% of them said their student evaluations went up, about 40% said their student evaluations stayed the same, and 20% said their student evaluations went down. More likely than not, your student evaluations will go up or stay the same. We also asked them what percentage of time do you spend lecturing. When that number dropped below 20% that was where you started to see them reporting that their student evaluations went down, when the percentage of lecturing dropped below 20%. That's sort of the parameters that I see working.The other thing that we know is that there are good ways and bad ways to introduce students to active learning. There's lots of advice out there about how to prepare the class, how to structure the activities, how to help the students reflect on the value of their learning. It happens. I don't think it has to do with the number of students in the class or the number of faculty doing it. Ben was telling me that he was the only person in his department using active learning. He felt students were gravitating towards his class and away from the traditional lecture classes because he was using active learning strategies and they wanted to get away from those horrible professors just standing there talking.DeTurckThere's another threshold that we've observed strongly at Penn. This phenomenon varies depending on the type of students. The students at Penn tend to be pre-professionally oriented. Medicine, finance, law or whatever it is. They are very sensitive to any change that they feel could have a negative impact to their GPA. If you change anything the initial reaction is always negative unless you tell them they're all going to give them all A's. You could feed them cookies and they'll say last year's students didn't have to eat cookies. The first year invariably active learning or SAILLE is introduced in any course or any department, ratings and such went down and complaints about it went up. Students are just like us. It's not that they are lazy or that they don't want to learn things. It's that they want to be successful. They know how to work the system as it was because they hear about it from their predecessors. "Here's what you have to do in DeTurck's class. You have to study this or this or this or this but you can ignore that because he never tests on it." They've decided they've got a bead on what I do and that's fine. But if something changes if it was good then the year after the ratings will go back up but the complaints will go back down because the students will adapt to the change. If the change is efficacious and it helps them learn better, great! Even the students who went through SAILLE the first time in their freshman year but are now seniors, a lot of them will say they learned a lot in that class but it was awful. The juniors will say, "Yeah, that class was pretty good."(audience) This discussion about the difficulty of change and that change is resisted by the older faculty and administrators and people in power. It just reminded me of something I read about the radical changes in physics in the early 20th century. You had relativity, quantum theory, and some of the older physicists resisted the new theories, the new physics. There was a physicist back then who commented, "Science makes progress one funeral at a time." (laughter)(audience) I didn't say who I was last time. I'm Douglas Mupasiri and I'm from the University of Northern Iowa. I was tantalized by a couple of comments Dennis you made, and Charles, about what seems to be a disproportionate gain in URM's when folks were using these active learning techniques. I wondered if there are any attempts to tease out what might be responsible for those gains? Might this have anything to do with the fact that in this somewhat unfamiliar environment for everybody, the URM's didn't have the sense they were at any disadvantage...I mean, are there any guesses out there, any theories about what might be responsible for the disproportionate gains?HendersonI don't know of any good research on this. My best guess is that it's a feeling of a sense of belonging to the group. We do know students that feel they have some connection with other people and some support system tend to persist at college better. That's a fairly well known result. I suspect that's what's happening in these interactive sessions. One of the other things that we know from research is that students who are from the majority population that have done well in previous schooling tend to naturally know they need to form study groups whereas often people who look different and talk to different tend to feel isolated and don't join these study groups and they miss out on the opportunity to study with other people and learn things. I think it's that feeling of connection that is a key part of how active learning helps. That's some conjecture on my part.HolmThere's been a cultural shift in mathematics from the lone mathematician working in the attic on his or her problem to a more collaborative venture and maybe that is part of the explanation for why the younger generation sees collaboration as more effective.(audience) I just wanted to pick up on something Dennis mentioned in passing. This comes up repeatedly, this question of teaching evaluation and what's good teaching. My department in part in a crisis based on a stupid one number summary of a complicated thing ended up doing exit interviews of all seniors as part of our teaching evaluation process and that turns out to be incredibly useful and I encourage everyone to do it. I'm Bob Sax and I'm from George Mason University.(audience) I wanted to push back on something that was said earlier. It was probably a mischaracterization of what you meant about certain groups of students not necessarily knowing that they should be in study groups. I want to push back on this because I think it could be the experience of the students if they try to join these productive study groups that there's hindrance and they're not comfortable with it. If you're not comfortable with it then why would you continue to do it? I want to push back on this notion that less successful students they don't know the appropriate strategies of more successful students. I think there's more history going on there in their own experience that they see with their own eyes and not that they don't know what to do.(audience) I want to say that Tara your optimism is infectious. I'm really excited to hear about what you are doing with TPSE. I'm wondering if part of your vision you have something more specific to say to someone like me working at a midsize state school serving a particular region of a state, how I might feel the ripples of the work that TPSE is doing. One specific thing is this chair plus one thing. If my institution sends some representative, are there other specific things in the next five years you feel are going to have an impact for people at institutions like mine?HolmThe answer is yes. On many days I'm optimistic and on many days I feel like I'm flying by the seat of my pants. Change is slow but this is happening kind of fast. This chairs plus one meeting is getting some funding as we speak, so it's very mutable what's going on. We've had a lot of ideas sent to us recently, "I'm coming to the chairs plus one but I'm wondering if you can help me organize a Big Ten chairs plus one meeting because I think that would help us a group figure out how to push the right levers and move on." I hope that we will also see regional meetings of chairs and that's where you institution might see some ripple effects. There are some states that are trying to understand student transfer issues among students within the state and that work is ongoing. Please send me a postcard. I will help connect you to the people I think might be useful.BraunI wanted to mention a reference. The question about what might be driving some of this improvement in unrepresented students. One thing that is worth thinking about is all the literature on the emerging scholars program that Uri Treisman has done. There's a lot there. Another good source of information comes out of the psychology literature about stereotype threat and mindset. Anyone who is interested in that stuff, there's a lot there. People in psych have explicitly thought about math at the college level so there's papers in psych journals about this. The other thing that I wanted to mention is that there was a beautiful paper by a sociologist named Leona Burton that was published in 99-2001 entitled "The Practices of Mathematicians". In it she interviewed 70 mathematicians in the United Kingdom and asked them how did you come to know mathematics. They discussed their own styles, their preference for collaboration, things like this. In the end of the paper she explicitly makes the point that all the reason the researchers gave for why they want to work collaboratively and how they want to do their research matches up exactly with the literature regarding active learning and group work. Then she points out that there's this massive cognitive dissonance between what they do and value in their research work and what happens in the classroom. This is not a new conversation but there's a lot of insight to be gained thinking about students doing mathematics like we do mathematics. We don't talk about that a lot. Anyway, Burton's paper is really beautiful and I just wanted to mention that.BressoudLet's thank the panelists once again. (applause)11 08-03-16 Opening RemarksZahnerGood morning everyone. We are going to be talking about assessment and assessment, from what I learned from yesterday from active learning, is a broad term. We are going to be looking at different types of assessment at the individual classroom level. Assessment of the actual programs, the efficacy of the programs and also assessment at the university level. The sessions will be broken up into three groups. The first panel will be Ted Mahavier and Nancy Ritze. Then there's going to be a break. These are hour long sessions. They are a little shorter from yesterday. We will have a coffee break. And then the second session will be with Mikhael Bouniaev and Jerzy Mogilevski and they will be talking about formative assessment in the classroom from the University of Texas at Rio Grande. And then finally we have the last panel. Representatives from Columbia University's Center for Teaching and Learning, Zach Kornhauser and Suzanna Klaf. Before we start I just wanted to share with you. People have asked, what is Council for Aid To Education? I don't come from a university or college. I represent a non-profit organization. Some of you may be familiar with CAE. We have the Collegiate Learning Assessment, which is the assessment of critical thinking, written communication, higher order skills that are used at the institutional level to measure learning gains. If any of you are familiar with Richard Arum's work. His work Academically Adrift used the CLA as part of his analyses. Currently we have improved our instrument. There were issues with the CLA including student motivation and the seniors testing and not showing learning gains as high because they weren't motivated to do well. So we made the CLA+, which is the individual students with stakes attached. The students are using these performance task assessments that the students have to actively participate in. It's not collaborative in nature but they have to fill out a performance task and they are given an authentic situation and put in a role and they are asked to make a decision. I want to share with you some brief analyses that I'm doing for a paper I'm trying to put together and it's about student effort and engagement. Of course, related to the assessment itself. Large sample, right? 21,000 students from the 2014-2015 academic year and there were mostly females with 80% of students having English as a primary language at home. This is a national set of students and they are pretty representative of what students across the nation are. The institutions themselves, there were 173 colleges and universities who participated in this particular academic year. These ratings here are based on the Barron's Competitive or Selectivity Index. The students at the end are given a survey about how much effort and how engaging did they find the performance task. The performance task is the piece where the students have to actually answer the item. It's 60 minutes long, they have to be very engaged but there's also a short section of selective response items which are basically multiple choice. Here's what the students said. It's a scale of 1-5 with 5 being the highest and 1 being no effort at all and this how much effort the students said they put into performance task. Compare that to the multiple choice items where they've put in significantly less effort and I've done the T tests for these at the difference is significant. And then I asked them how engaging they found the tasks. This was the distribution for the performance task. I guess most of them found it to be okay but not awesome. They were taking a test so how engaged could they possibly be. And then finally comparing that to the selective response item. We are seeing that the students are even in a test where they are not happy about taking a test they are finding that it's more engaging and they are putting more effort into it. The second piece of research we do at CAE related to the CLA is measuring student outcomes. I look specifically at the predictive validity of the instrument and how this affects post-college outcomes. The same college students we looked at, the 21,000, I only looked at the seniors who took the CLA in 2015 and we surveyed them three months, six months, and twelve months following graduation. This is approximately 13,000 students of which 1585 agreed to participate and almost 1000 persisted. And there's a larger paper, if anyone is interested I can share. But this is a summary of the results that I found where students' critical thinking skills were predictive of positive post-college outcomes for all of these things: salaries, employment, full-time employment or enrollment in graduation school. So the things that you are doing in your classrooms. Yes, it's mathematics but it's bigger than that. You are getting them to think critically. You're getting them to do problem solving. These are the skills that then transfer on to their careers. The students who have these skills in analytic reasoning and problem solving, critical thinking are more likely to have more successful outcomes one year out of graduation than those who do not have the skills. So with that I'm going to pass this on to Ted so he can speak about his project.12 08-03-16 Ted MahavierMahavierWe seem to have a tradition here, we're all kind of nappy from last night. Everybody stand up. Come on. There's a theme to every conference, right? And dance for me for three seconds. One, two, three. OK, there was a reason I wanted to do that, and it wasn't because I wanted to see Mike Starbird dance. Where are you, Mike? I missed him. Oh, he's at the slacker table. I knew he'd wind up at the slacker table. Why doesn't that surprise me. It's because I wanted to put this up. That was the active learning portion of the talk and we're now going to move to inquiry based learning. Inquiry based learning refers to any pedagogy that utilizes presentations or other student-centered activities to develop in students the confidence and ability to do mathematics on their own. Instructors providing notes to facilitate this, etc. etc. I have a definition that I have used for 25 years for inquiry based learning. It is a subset of active learning. It is not synonymous with active learning. What I came to talk to you about today was a workshop we conducted in December 2015. The workshop was titled "Inquiry Based Learning in Real Analysis". The idea was to take from the wonderful work that Sandra Laursen had done which had established certain principles without question that I've believed for decades were true, that IBL serves women, minorities better than lecture. Through my own experience I saw this as a transitioned from lecture to active learning or inquiry based learning. There were some things that Sandra Laursen's study said that didn't tell the whole story. For example, her study showed that the best students did no worse under this schema. Well, I believe that is a true statement. I also believe they do a whole lot better in things that were not measured in the study. So we organized this session, Paul Dawkins who's not the Paul Dawkins at Lamar University where I work. This one is at Northern Illinois University. Myself and Mike Workman, who is at Oklahoma State. They submitted a proposal to the American Institute of Mathematics. If you are not with this entity, boy, you should be. What a wonderful conference they put together. I first have to thank John Fry for the original funding and the National Science Foundation and the three people that guided us through our conference there. We spent five days there working on trying to formulate questions that we felt should be assessed and had not been addressed to date. The workshop was trying to identify and frame future collaborations to refine research based studies and develop a research agenda responsive to existing needs regrading IBL practice. We had a very specific goal. We wanted to address IBL and real analysis. We wanted to develop research questions we thought could be studied, were realistic, accomplishable goals. Many a little bit should be said about AIM workshops. How many people know how an AIM workshop works? A small percentage. They have a very controlled method. You come in and they tell you, "This is what you will do for the first two of the hours of the first morning, this is what you will do for the next three or four hours." It's extremely controlled and it's very good. I don't think any of us thought so when we started but we certainly all thought so by the end. They have a system that works for addressing research problems. In the beginning you start developing ideas and you do some framework discussions. And then they break you into groups and you can self-select into the groups. You vote yourself into groups. So I wasn't in every group that developed every one of these areas of inquiry but you could move between groups so I had my finger in each of these pies at one time or another. In these afternoon sessions four areas of inquiry that the majority of these people wanted to know about came to the surface. What I'm trying to make the point, the way the AIM workshop worked so well is that these were not preconceived, we did not walk in saying these are the things we think we should study. This group of 25 people, this is what bubbled up during the process.Instructor choices. What choices do instructors make when you are teaching an inquiry based course. Originally, this was set in real analysis but most of this supersedes real analysis. This could all be studied in other areas. Persistence in identity, this was one of the things we thought was brought out in other literature but we thought was really important to IBL and real analysis, that it develops in students the ability to persist and to get a sense of their own mathematical identity. The development of problem sequences and learning trajectories, what does that mean? There are certain things you can do when you develop am IBL problem sequence that will facilitate development of your students, facilitate the speed at which you work, facilitate the success of the classroom environment, and promote the learning trajectories that you want to see with the students. And then proof. That's really far too short to just say what it means. Ultimately in a real analysis class you want your students to be able to prove theorems. You want them to be able to prove hard theorems, not follow your nose theorems, not here's the definition of limit point, is this a limit point or prove this is a limit point, but theorems that pull in continuity and differentiability and several other theorems to prove a more powerful theorem. Ultimately this is what has to happen for success in a real analysis class. So what facilitates this development in students?So we were trying to surpass and skip individual implementations and ask, I could answer that for question myself. I give them a sequence of problems and ask them to work individually. I don't allow them to collaborate. I want you to know at the end of my course that when you proved that theorem, you proved it and that goes into that persistence and identity. At the end of the semester I don't want you to think that if I work with Nancy then I can do mathematics. I want you to know, James, that you can do mathematics because I want everyone in my class to know that they are mathematicians when they graduate, that they can do mathematics independently. I'm not opposed to collaboration but I believe that sense of identity that's in that second item comes from independently doing something.I want to talk briefly about each of these four areas because one bullet isn't enough to tell you about what questions we came up with. Establishing a classroom culture. The instructor choices, one of the choices they make is how to establish a classroom culture. We asked questions. What strategies do IBL instructors use to create and sustain an appropriate classroom culture in IBL and real analysis? What are the essential aspects of appropriate classroom culture that faculty make explicit moves to establish and maintain regardless of the classroom structure or strategy for doing so? Under each of these bullets we raised questions and developed a research strategy to study it. I'm not going to put the research strategy up there, that's the full report. I have copies of the full report that I'll happily share with anybody that is how we plan on studying these things. The first assessment studies that we're doing starts in three weeks. We have a lot of assessment ideas and a lot of people interested in implementing them and the first ones are going to start this fall. If you are interested in seeing what those are, I can share those with you at a later date. Learning through proof presentations was a big question that came up. Why do we thing students learn better this way and how can we document that they do? Students presenting at the board, does that give them an innate better understanding other than the student that's presenting it? What about the students in the audience? I'm a professor, I can put up a nice polished proof. Is it better to have a student put a proof up that's not polished? It sounds irrational. Have a student put a bad proof up, will the class understand it better than if a professor puts a polished proof up? Is that true? There were several hypotheses here. One was that students learn more from proof presentations in IBL courses because they have already attempted the proof pass because I give this list of the problems out and it's a relatively clear list and you're only working a few problems in advance. Everyone has worked on what's going on the board tomorrow. So perhaps because they have all worked on it then when a student does present it then they understand it better than if they just knew "I'm coming to class, the professor will put up a polished proof, and I'll figure it out after he does so". So the question was how do students having attempted a proof influence their learning from the presentation? Does them having worked on it before class affect their understanding? I thought it was interesting Mike Starbird says don't read the book before class because he doesn't want them to do a thing and I do the exact opposite. I want everyone to have worked on this. Now he said that was very effective because they follow the assignment and that's not true. I read the chapters before going to class and I asked the questions in such a way to make me look intelligent as if I had just come up with the ideas when in fact I was manipulating the system. I was gaming the system because I did read it in advance and I did figure out how the answers but I wanted him to think I didn't because that was the game he wanted to play. You've got to watch out, there are some smart ones in your room that are going to game your system. Hypothesis two was the students learn more from proof presentation in an IBL course because they pay closer attention to find and identify mistakes because I'm not going to make a mistake when I put it on the board. Let me give you a little tip. I keep all of my students awake at 8 in the morning when I'm teaching calculus precisely because I make mistake after mistake after mistake. My graduate student said to me, he was one of my mentees, I think it's amazing how you exactly intersperse mistakes when you are exactly afraid they are not paying attention! Yeah, that's it!Question 2. How do students' ongoing experience in IBL or lecture influence their learning from proof production and observation of a proof presentation? How do students learn through other student presentations? We want to learn the answers. What is it that makes it effective, if it's effective? You will notice these are digging deeper than previous studies have done. This is what we are trying to do. We believe that the other studies have clearly established that IBL is effective. Now let's dig deeper. Is there something about how I develop my notes that makes it particularly effective so that we can drill down into best practices for implementation. That's my goal.Persistence and identity. Exploratory of student development, i.e. what are the developmental categories in a Moore method classroom? Can we refine and explain these categories? Ben listed a bunch of categories that are common in mathematics education. Our categories are more basic. You're a basic student, meaning you have never successfully proved a theorem and presented it and defended it. You're a novice, meaning you presented one problem that's a follow your nose. Here's a tiny definition and then something that follows immediately from it. You're an advance student. You've used multiple definitions or multiple theorems to prove something more significant. You're an expert, meaning you are raising your own conjectures at this point. You have said, "Ok, I've proved this theorem but I really don't think we need this hypothesis. I wonder if this hypothesis can be removed". You're now an expert. You are no longer just proving the problems put in front of you, it's your mathematical explorations.This I mentioned earlier, the benefits of the modified Moore method. The Moore method, IBL, whatever you want to call it, over lecture for strong students. We strongly believe that it's much more effective for strong students just like it's more effective for weak students. We want to see that established. What are the long term effects on the strong students and how does that compare with those of more traditional students? How can we further describe the student development within the strong and big achieving category? In what ways to Moore method courses support this development specifically for the strong students? Enough about the strong students. I want to know whether the Moore method affects them positively and, if so, how. The third year problem sequences and learning trajectories. This one was a room I was heavily involved with. When I build theorem sequences, there are two things that I try to build into absolutely every one of them. One is problems that are very low level and closely spaced to facilitate success in students who aren't having success and problems that are meant to be roadblocks that should stump the best students in the class, that are hard and challenging. The second thing I do is try to intertwine multiple topics because if I'm only working on about limit points right now and Julie doesn't understand limit topics then Julie's not going to have success tomorrow. But if I'm intertwining limit points and a tiny thing about continuity and a tiny thing about sequences and there are all independent but woven together in the thread then if Jerzy's not having success with limit points then maybe he's having success with convergence and the same tools will eventually lead him to success with the other. Making sure there are multiple threads interwoven in the problems so they are not like a book is, broken into these categories where you master this category and if you didn't understand it then you didn't understand the homework. There are multiple areas of exploration in each thing. So the question is can problem sequences be developed in such a way to facilitate efficient transition to the material and maximize learning? I believe the answer is yes, but only through my experience, not from validation.Strategic walls. That's what I said, these hard problems. In my theorem sequences after a while of show, show, show, there's a question. Is this true? The whole point is to demonstrate how much harder research is if you know this is true, then you just try to prove it? But what if it's a question? Is this true? Now you have to decide if you are going to disprove or prove it. That's a lot harder. I'm still trying to prove that the Hilbert cube is not homogenous. I don't think I'm ever going to be able to prove it. Zonal approximal development, a long standing thing in mathematics education research, how far apart the steps are and making sure the steps are appropriate distances apart for each student. I don't think I'll say more about that. The fourth area of proof, how does an IBL class affect a student's understanding of proof? A lot of questions here. In what way does IBL impact a student's proof schemes? Affect student's conceptual understanding of meaning of the mathematics? What do we mean? Do they know what axioms, conjectures, definitions, and theorems are and what is their role in mathematics? Developing confidence in proving things and what features of IBL create this impact. I'm going to conclude. I'll answer any questions you have before we move onto the next speaker. (applause)(audience) With this discussion of student presentations this is a good question. Are you thinking at all about situations where you have, perhaps, more than just one student at a board presenting work to a class? Perhaps some sort of smaller scale thing where you have groups of students presenting work to each other?MahavierWe were trying to bypass implementation. I can tell you my personal biases. I prefer to have one student presenting. I prefer to have all of us focused on this one problem. As I've said I don't allow collaboration in my classes because I really want you to feel that at the end of the semester that you know what you have accomplished. The active and collaborative learning occurs when you start writing that problem on the board and you write that first line and you say "I believe from this it follows" and you make the next statement and someone says "Why does that follow?" Now that may be because it doesn't follow or they can't see why that follows or maybe you wrote something you didn't intend to write and have a typographical error on the board that's confusing them, a notational error. The class conversation that begins there will often raise other questions and that's where the collaborative and active learning happens in my classroom. I don't answer my questions. They look at me when I first put a line on the board. And I look over and say, "Teresa, do you follow that line? Is that correct?" And she says yes. I say "Can you put it in your own words, can you say it differently"? I do something to bring Teresa in because she hasn't said anything today. I bring out the active and collaborative learning in the room but I want you to have worked independently outside of the room. One of my biases for doing that is when we are working altogether in the group if you sat next to me you would discover I am one of the slackers. I'm not actually a slacker, I'm just slow. I can't process things at the same speed as 75 or 80% of people in this room can. If you let me work at home then when I put a problem up on the board people go "Wow, how did you think of that"? I don't know, six hours? That's how I thought of. As G. H. Hardy wrote, "the harder I work, the luckier I get". So the collaboration happens in the class and people are worried about putting certain populations at disadvantage. I'm one of those populations. At age 13 I weighed 48 pounds. I was sensitive about my size and insecure. I learned if you prepare well, you can perform on par with anybody. If you work hard you can perform on par. So allowing them to work independently outside of class and present in class I find really develops my students. But that is a really different perspective from many people that like to see the collaborative and group work. I'm not opposed to that, I'm just saying what works for me.13 08-03-16 Nancy RitzeRitzeI'm Nancy Ritze and I'm here to talk about some of the most underprepared students and how important active learning is to help them be successful. Bronx Community College is part of the university system in New York. We serve probably one of the most diverse and challenged student populations. They are ethnically diverse. The vast majority are minority students. Many are non-native Americans. They are very academically underprepared. The only entrance requirement is a high school equivalency or diploma. About 90% require remediation in reading, writing, and/or math and I would say 90% require remediation in math. That includes basic arithmetic and elementary algebra. In addition to being academically underprepared, students are poor. Not quite half have a median household income of less than $15,000. A third are supporting children. Half are employed and many are non-native English speakers. So we have a real challenge here. Somebody yesterday said something about mathematics and other intelligence being distributed equally among populations and we find that to be true at Bronx Community College. There's a lot of untapped intelligence and that's a really important challenge that we face. What we have seen over the years is that students who get the right experiences in the classroom and the support can be very successful but it takes a lot of effort. I'm going to talk about the challenges we faced and some of the efforts we've made to bring active learning to scale at the college. We have been successful in some areas but not so much in others and I'm going to seek some advice from you to help us. As you can imagine we have three fundamental problems that we are challenged with every day. Student retention, academic progress, and graduation. We have recently been recognized for doubling our three year graduation rate from 8% to 16% and that's way too low. We've had some special programs that provide a variety of very targeted support for students that has resulted in 61% of those students graduating and they are basically the same students. We have a lot of commitment and interest in trying to figure out how we can maximize the student success at our college. About six years ago we engaged in an institutional self-study to try to identify in addition to all the barriers and challenges students faced what were the basic reasons for the lack of progress at the college. As you might expect, student disposition was one of them and that includes all of the factors in terms of their past poor education, their lack of self-confidence, not being aware of college expectations, and so on. Also, you might think that faculty who come to teach at community colleges, are there any people from community colleges here? Ok, a couple. Some people who are not part of community colleges really think that the faculty that come to teach at community colleges are trained to teach community college students but in fact that is not the case. In the City University of New York, community college faculty are expected to prove the same kind of, maybe not quite as much scholarship as the colleagues in the senior colleges and graduate school and so they have, you know, in a lot of graduate programs they are not well prepared to teach students. And the third problem is insufficient academic support, but I'm not going to talk too much about that.What did we do when we discovered this? We realized maybe we need to have some kind of high impact initiative to help students immediately in their first semester prepare to be effective college students. That was one effort. The second effort is that we had for many years, maybe 15 years or more, we had a center for teaching excellence, as many of you do, where faculty would self-select to participate in faculty development efforts in support of active learning and effective teaching and learning. What we found over the years was that the same faculty would sign up over and over again but that we were not reaching all of the faculty and that typically after faculty attempted to teach some difficult or challenging classes they retreated back to how they were taught. So we developed a new faculty seminar, which is a year-long effort, which required participation along with reassign time and we found that to be quite effective. I will talk a little more about that. And finally to engage in assessment and evaluation efforts that would facilitate improvements in teaching and learning.First I'm going to talk about the first year seminar for incoming students. These are theme based seminars which integrate academic content and skill development. There is an emphasis on student-centered pedagogy. There are embedded academic advisors and pure mentors. There's rigorous tracking and reporting of student progress for intervention in actions. And there's a team approach to student success. One of the things that we thought would be a really effective thing was to engage the faculty teaching these seminars, and the goal was to have faculty from across the curriculum, full-time tenured faculty, participate in development in active learning strategies including e-portfolio, group work, flip classrooms, and so on. Our hope was that in engaging faculty and teaching first semester freshmen that it would impact how they would teach in other classes. In fact, as we have studied this faculty have indicated it has changed how they teach in their disciplines. We hoped that it would become a sight for experimentation, an exploration of good pedagogical practice. We purposefully kept this outside of the promotion and tenure determination so that faculty would feel comfortable doing something that was outside of their comfort zone and learn more about teaching and learning for students. We hoped that this would be an incubator for best practices, creating strong student-centered learning environments. This is happened in some cases but not across the board, as you might imagine. We continue to study this effort. It's only been in effect since 2012. We have recently collaborated on a formative and summative evaluation of the effort with the Community College Research Center, which is located at Teachers College Columbia University. And we also engage faculty in assessing the student learning outcomes across each of these seminars. We've done a number of studies comparing the success of students who participate in the seminar versus a matched sample. We are at about 50% of our freshmen students participating which is about 1000 students. We keep increasing each semester to include all students. We have done several statistical analyses that demonstrate that students who participate do better in their first semester and subsequent semesters and we are almost ready to see if this i having an impact on graduation rates as well. In addition, with the CCRC, we did what's called a fidelity of implementation analysis. We created a rubric of the major components of the first year seminar class. We tried to articulate what it meant to be a low implementer with active learning in the classroom, a moderate implementer or a high implementer. The researchers from CCRC sat in on classes, observed classes, and wrote a paper about their findings. One of the most interesting findings was that the same way that we find our ill-prepared students make a decision early on in their first or second semester about whether they think they're really college material, I think it's the same way with college faculty with active learning strategies. Sometimes they will try it and if they are not getting the feedback and the results from students they drop out. We've seen that over and over again. One of the things we've tried to do is to continue having faculty observe each other in their teaching of the first year seminar using the rubric modeled after the rubric we used in the study of fidelity of implementation to see to what extent are people really teaching in an active learning environment because if we do a survey faculty all think they are conducting active learning classes, they all think they are covering critical thinking and reasoning and analysis. When you look more objectively at it you can see that they're not all doing that.What we've determined in our ongoing assessment of this class is that yes there is a positive impact for students that participate. There is considerable variation in the fidelity of the program design and classroom implementation and that we are seeing a stronger impact for students where the course is being implemented more to the design. So what are our challenges? We think we have developed a theoretically strong program which will function as an incubator for active teaching but we didn't have enough faculty participating. In addition, among those that were participating not all of them were implementing the course as we had expected. And finally, as we had said before, faculty who struggle with this approach drop out and then go on to criticize the program and advocate against it. Here are some of the challenges that we found with the self-selection in professional development. One of the reasons for faculty not opting to participate in these active learning initiatives and efforts and there's significant literature that very particularly community colleges and underserved populations really benefit from this, and this is where new faculty seminar comes in, required of all new faculty. We have about 300 full-time faculty. We hire about 20 new ones each year. It may drop off a little bit going forward but that's quite a bit. There's a real opportunity to change the culture in engaging faculty in this effort. In addition, we have a relatively new provost and very new president and they have articulated clearly to the academic departments that they are expected to engage faculty in these efforts. And finally we have collaborated with the Council for Aid To Education, providing performance task workshops for faculty in a variety of areas. Most recently, after the provost had indicated to the department chairs that she expected one or two faculty from each of the academic departments to be trained to teach FYS, first year seminar. We engaged with the CAE to work with the faculty to develop performance tasks where students would be engaging in the kind of tasks Doris spoke about earlier for graduates but in their first semester. They would be engaging in a task that would focus on critical thinking, reasoning and analysis. Zach, who you will hear from in a little while, conducting the first of those workshops on our campus. It was extremely well-received and we are hoping with the 20 new faculty who have been trained in this, when we see the implementation of this next semester we will see higher caliber work done by the faculty and better results from the students.In addition, as I said earlier, we have developed a new faculty seminar. We had always had a normal orientation program for faculty. This is a year-long effort with substantial reassign time attached to it. There are three major topics that are covered and there are three very strong faculty mentors who deliver this seminar. The first component is on pedagogy, and they are introduced to active pedagogies and are asked to do some assignments in their classrooms. The second is on assessment of student learning outcomes, which is particularly important for all of us these days. The final one is relating to their own professional development at the college. What do they need to do to get tenure, what do they need to do to get promotion, how can they publish in their field, or in scholarship in teaching and learning. In addition, the faculty who participate in this must create teaching portfolios. They do a presentation at a day that's focused on teaching and learning at the college. The faculty are usually pretty excited to continue to engage in other interdisciplinary activities on campus and they are encouraged to participate in things like FYS, e-portfolio, and some of the other active learning strategies that we have on campus.I want to say a word about performance task workshops that we engage these faculty in. We had two other sessions for a variety of faculty across campus. Each upcoming FYS section will include a performance task and faculty will participate in the semester in assessing student work across the seminars using a common rubric. We are really excited to see where that goes. We are proud of some of the efforts we have made to systemically advance active learning but we still are not anywhere near where we need to be. Most but not all academic departments are participating. Faculty in some disciplines are more open to and more effective at active learning. A question I'd like to raise to this group is how to incentive participation from all departments and disciplines.A second challenge we have and this is particular to us but I think there's a lot of guidance we can get from this room, success in our math sequence is the single biggest barrier to student academic success and progress and the college. Here is our full disclosure. Our math faculty are the most resistant group on campus to engage with active learning strategy which may not be surprising to some of you in this room. We are really desperate to identify what strategies we can engage in to engage them in active learning. We just had a transition in the leadership of our math department which I am really encouraged by because every time I have mentioned anything to do with active learning, problem based learning, the previous math chair would roll his eyes right across the table from me, and that was rather difficult. And our final challenge is that promotion and tenure at the college does require faculty scholarship. Faculty in some disciplines already engage in scholarship of teaching and learning. The English as a second language faculty, the digital design faculty, the English, history, but it's really a problem and challenge for our science faculty. We don't have the kinds of science facilities for them to do research although some do, so we are really interested in trying to support in a systematic way more faculty scholarship of pedagogy.14 08-13-16 general discussion(audience) Nancy, are you doing any of the active learning professional development with part-time faculty or adjuncts? How are they fitting into this program?RitzeIn fact, faculty who are adjuncts are eligible to participate in this. When we first started the first year seminar full time faculty were not terribly interested in participating so we really attracted some part-timers. I do think that we will be developing concrete faculty development programs for part-time faculty and I think that's one of the things that we really want to do with our part-time faculty.(audience) Are they compensated?RitzeWhen they participate, yes.(audience) I am actually a former co-director of the first year experience program at Worcester State which is a comprehensive regional university in central Mass. I just wanted to answer some of your questions. Some of the things that we found to be very effective at our institution were collaborative learning communities among the faculty in the first year experience program. We had some training a colleagues from Holyoke Community College where they have an incredible collaborative teaching program where they focus on team teaching. We couldn't implement it at the level that they have because of financial constraints. So we instead we opted for faculty to get together to implement some common theme in the seminars and some common activities during the semester. Having to meet with each other and talk about how things are going in their classrooms, that was very effective. Also, in terms of assessment, we used the critical thinking and the written communication rubric that the AACU has put out and we actually had some positive results. In our institution every first year student has to take a seminar. Compared to transfer students who haven't been through this program, the students perform much better on what we call a signature assignment which is basically a big writing assignment which incorporates all the features you would like a first year seminar to have. I just wanted to encourage my mathematics colleagues to jump on this bandwagon. If you have a first year seminar program please think about offering a quantitative literacy seminar. Quantitative literacy is really an orphan child and we as mathematicians have the responsibility to step in and really make a difference. In our institution I have encouraged some of our younger colleagues and we have some examples of seminars that can work. This fall I am teaching one for some of the honors students on what the numbers say and we are going to be focusing on the presidential election. A seminar doesn't have any math requirements so this kind of quantitative literacy seminar can really affect students who are in remedial math and it can show them the power of numeracy. Thank you.(audience) For scholarship of teaching and learning, you might like at Curtis Bennett and Jackie Dewar. They do mini-courses on SOTL often at MathFest and sometimes at the JMM. They have a book that's through the MAA that goes through examples of how to do SOTL and that kind of stuff. That's helpful if you don't have a ton of resources. It sets you up with how to get a question that's measurable, some ways of how to actually measure things, and where are places to publish. I found that very helpful. As far as selling active learning or IBL to math people, and I was trying to think of what sold me on it. It was probably that it didn't click in my head until my junior year of college what math actually was. I realized it was much easier to understand the proof than to just memorize it. I don't know why it took to junior year (laughter) but I'm not the fastest person. Somebody said yesterday that framing it to mathematicians is that it's doing math the way we do research is giving students a more authentic feel for what math is. I've had some students come up to me afterward. I had one student who I thought hated my course so I was really confused. He did a proof for his parents and he said his mom was like "Why would you ever use this?" and he said "It doesn't matter if you use it or not, it's beautiful! It makes sense". I think framing it in the sense of, you get the chance to tell these students who probably hate math to see some the beauty in math and they are actually using it all the time and they just don't know it. They like problem solving.As far as workload, I find it's a similar workload. Granted, if you have your notes that you've been using since 1976 and you haven't changed your homework, it's not the same but it's similar but the payoff is better so I don't mind doing the work because I actually see the students improve. That was a comment but I also have a question for Ted. I was at the workshop. What's starting in three weeks? Can you tell us which group is starting and if they have a plan?MahavierMichael and Paul contacted me. We have the spreadsheet that says who's teaching what when. I don't think this is any credit to me, except that I was the first one to come up teaching analysis. Because I'm teaching analysis in the fall they want to do case studies of the students. They want me to keep a diary. They want to get to some of the details of the categories of students, beginner or novice things, and try to document when they change. There's this conversation about sometimes a student will make this transition late in the semester and you can't tell figure out when they will develop. So they are going to be doing some case studies and some interviews with some of my students starting in a few weeks. That's no credit to me. I rolled around first on that spreadsheet of who's teaching analysis next. (audience) I'm Teresa Shelton from Southwestern University. I had never taken any IBL or active learning classes. Some of my colleagues were very versed in this. I taught with lecture method for quite a while and I've become a recent convert to these methods. At Southwestern University we had an institutional impetus with an HHI grant. I would say the pedagogian jelly workshops that we had together where the faculty would get together and say what works and what doesn't. I was a time scholar like Justin in differential equations and the online working groups that we had with that were extremely effective with me. I know that, Ted, you have done IBL for a long time. Is that across Lamar University or just mostly in your math?MahavierWe have a center for teaching and learning which promotes it across the campus which I'm peripherally associated with. I try to give them guidance but try to make the young people do it because that's where the impetus needs to come from. We do not have a top-down pressure to do IBL. IBL sprung in the math department before I was there. There was a long list of people who had done IBL in the math department. When I came in I certainly raised the percentage of IBL courses taught. It was not across the board but just as the evidence is showing that this is effective teaching. Just like all the evidence is showing that calculus across the country, I can't tell you how many requests I've had because I teach IBL calculus to travel across the country and talk to people because we are not alone in watching our pass rates in calculus. David Bressoud's website is a wealth of information on the calculus, the documenting of the failure rates across the country, the DMW rates are going up and so lots of places across the country want to see that done and we've been doing it for a long time at Lamar.(audience) Hi, my name is Tracy Saltwick at Bergen Community College. I was wondering, Nancy, if you had any type of training for first year seminar teachers to transfer those strategies into their discipline, into the math or into the science.RitzeWe have not in particular. I think we may try to build on it. We have had these 20 some-odd faculty who did the performance task workshop. We are thinking about where else besides the first year seminar we can incorporate similar kinds of training. We are encouraging all of our departments to up their games in terms of student program outcome assessment. In some cases they use e-portfolio or they have a capstone course. We may try to think of systematic ways to engage faculty in continued effort. We have got the new faculty coming in, the FYS faculty, we've got some technology initiatives but I think if we can get larger categories. And I do think faculty enjoy interdisciplinary work but I do think that in our case in particular the math faculty are more likely to listen to other math faculty. They find some of the professional development a little too touchy feely. (audience) Greg Foley, Ohio University. Nancy, I believe the Borough of Manhattan Community Colleges did some innovative work at the beginning college level with quantitative literacy courses. I'm wondering if there's any cross-fertilization with what you are doing and what they are doing and if you are familiar with their work if you could tell us a bit about it.RitzeI believe they are engaged with the Statway and Quantway initiatives. They had a provost who was a mathematician for many years. My sense is that they are doing more experimentation not only with what students are learning but how they are learning it. I don't think we have had as much cross-fertilization as we could. Some of the colleges are more progressive in certain areas and some of us are less so. I do understand that they are engaged in some pretty interesting work that they've had some good success as have a couple of the other colleges as well. (audience) I have a very short anecdote that I just recalled from my own institution. I had barely started, I'd been there for a week and one math colleague of mine remarked a very genuine question. "All these people talk about student-centered education and I just never got the point." (laughter) He really wanted me to try to explain to him what the purpose was genuinely.For our institution we are an R1 research university and we have some of the same issues with resistance from some faculty. The biggest thing that we have done, and I had nothing to do with it but it was great for the department, our dean shelled out money to bring in three speakers per year specifically for the math department. People could propose "oh, I'm interested in clicker systems" or "I'm interested in calculus for the life sciences", things like that. There are lots of reports available from common vision and other organizations, bringing someone physically in front of the faculty was definitely the most high impact thing for us. Part of it was that it makes the conversation not about us but an external source so we can stop fighting about our own opinions and start talking about someone else's opinions. It's still a fight but at least we're not yelling directly at each other, we're yelling at that person instead.StarbirdI had a couple of comments on this last theme. I think a lot of the problem is people who are not experienced either through their own education or through practice may never have seen what really goes on in an active learning or inquiry based learning classroom. It's really a mystery. Visiting other people's classes who do it well would be great. Having people come and talk and do a workshop that's then followed up with actual discussions I think is effective. I'm wondering as a group whether we need other forces in the line. For example, I know there are some efforts to make video presentations of models where people can actually see what happens. Some people who are not accustomed to it would say what's the point if I'm not teaching them, if I'm not explaining things to them, how are they learning. It is a mystery until you see that it's not just a random oh you figure it out. It's very directed and methodical and designed. And, Ted, your description of exactly the goals of the sequence of getting students to actually develop the new strategies both of thinking and of math in a particular order with design, that's the kind of thing people don't appreciate.MahavierAnd when people attempt to start it on my own they have two hurdles that are too great. It is like deciding to cross the Atlantic. I will build the boat and learn how to sail it. Maybe not! All evidence to the contrary based on history. When you decide to teach the course and create all the materials you have created two hurdles that are almost assuredly going to fail. Whereas if you borrow, steal, plagiarize from other materials which someone has put long and timely effort in you can see later see that you could do it better at your university by modifying it. But now you only have to fight the pedagogical hurdle the first time. There are two huge hurdles there and your point about starting with someone else's notes is why the journal was founded.StarbirdI have two other things. One is self-defense. In my inquiry based learning classes that I teach, upper division classes, I do them just as you describe. Namely, I have the students work in advance on theorems then presenting them. I think it's clear that having worked on them makes them much more receptive and learn far more when they hear a presentation. I'm looking forward to getting data that I can present to people I'm trying to convince of the efficacy of this method of teaching. I'm delighted that you and everybody are doing this. For me, the challenge is I know what I'm trying to teach students. I'm trying to teach them an attitude that they can personally develop knowledge on their own, that they have skills of understanding more deeply than usual, that they can make mistakes and learn from them, they can raise questions and all these things but I've never seen an assessment instrument that I'm persuasively convinced is actually measuring the things I actually care about. The changing their lives for the whole life. And I liked your first presentation, looking beyond school, what effect things are having in school. I'd love to see an assessment that I could say that is what I'd like to measure in what they are actually taking from this experience.MahavierI'll give you a copy of the report of what we were trying to do because that was the goal, to get mathematicians. I always felt the same way, Mike, that I wanted to have my classes assessed and I wanted to document these things but when the people contacted and this is the assessment that they wanted to do but I didn't care about any of that. I'm glad it was being done but I didn't want to put my time into it. I want these specific things assessed so we put a group of mathematicians and math educators together who are attempting to accomplish that and, of course, you'd be a great candidate. And you'd better watch out because I understand there's a plant headed your way to validate exactly what you do in your classrooms (laughter).(audience) Mike May, St. Louis U. Thinking of some of the things that have been mentioned. I start with if I think of who the best teachers I've experienced are, one did active learning back when I was an undergraduate longer ago than I wish to admit and the other was the best classic lecturer I'd ever seen. You could take his notes and turn them into a book. Part of the resistance is I think part of how all of us teach is think of who has done a good job that we have experienced and do that. But I've also been thinking about you mentioned math faculty are more resistant about some other things. I think part of it is we have more constraints on our class than anyone else. My calculus class has to keep ten other disciplines happy and respond to faculty in other disciplines who want their favorite technique covered and part of why we can't just cut things out is I have 23 courses at my school in math that are specifically required by somebody else and I don't think that's unusual. What I think would be really useful as I think about it is what was done on coming up with worksheets that others have done is good to steal but let me be more specific. Packets that you could use with a standard textbook so that when you walk through the exhibit I want something for Stewart, I want something for Hughes Howlett, I want something for Anton that would let someone pick three to five activities in a semester with a standard textbook from college algebra to Calc 3. Those being available for people to start with, I need to start with the faculty who are new faculty or old faculty who are overworked and don't have enough time and that makes a manageable thing to start with. Do three days of worksheets during the semester and someone has made the worksheets going along with the book you are already using.MahavierI'll comment on that. When I started the journal in 2007 we published only full course notes because that was where we decided to start and we wanted to see if there was a market. There is a market. We have consistently published the only refereed IBL materials out there that are freely downloadable. However, we are moving towards adding two categories very soon. One, we are going to have a user review section so that, like at Amazon, how do you decide which toaster to buy, you read the reviews and pick a toaster. We plan on having user reviews. We just brought on Matt Miller at University of Colorado to be our user reviews editor and we are in the process and will very shortly announce a modules editor. Those modules will be precisely what you mentioned. We feel like the journal has established itself as successful, now it's time to expand into other things. We recognized five, ten years ago the need for small modules. I want to teach integration by parts and I want that to be an active learning portion of my course or an inquiry based learning exercise or I'm a teacher that wants to put a toe in the water and try this. It would also be nice if you are teaching real analysis to say, Mahavier does kochi sequences right but Starbird does limit points right and Teresa Shelton does the extreme value theorem right. I'd like to steal, plagiarize and pull from all of those which my discrete mathematics notes are exactly that. I stole from four or five sources and unified the language. We are very aware and ultimately that is what we'd like to be, a refereed repository of materials that you could say I want to teach four IBL exercises during the semester but your colleagues want 15 weeks of IBL and has a course constraint that these topics must be covered and no one set of notes covers them. No problem, you pull from here, you pull from here, you pull from here. Will it happen all immediately? I don't know. If you'd like to fund me a million dollars you'd get it done within the year. Right now, I don't have that funding.ZahnerCan we thank the panelists? (applause) We are going to take a short little break and resume at ten o'clock.15 08-03-16 Mikhael Bouniaev Jerzy MogilskiBouniaevGood morning. First of all I believe that it's time on behalf of you to say thank you to our organizers, Doris, David, and Ron. Let's give a round of applause. (applause) I think it's a timely event. We are going to take about one hour of your time. I would like to introduce ourselves, our team. Dr. Mogilski and Mikhael Bouniaev, University of Texas Rio Grande Valley. This is a U.S. university in the United States, just one year old. It was combined University of Texas Brownsville and University of Texas Pan America. Both Dr. Mogilski and myself consider ourselves as active mathematicians. We publish papers in theoretical math, so at the same time we love teaching, training teachers, and also doing research in the training of teachers. Also, Dr. Mogilski happened to be my supervisor. Before we got promoted last year he was my department chair. Right now he is associate chair of mathematical and statistical sciences and I am a very faculty member in this department.I'm going to talk about some theory and our own practical experience because as a poet said, "The theory is gray but the golden life of trees evergreen." We are going to split our presentation. Mostly I'm going to talk about theory because mostly I have a postdoctorate degree in psychology of math education so this is my second degree after PhD. At the end of the presentation I will talk a little about our experience and Dr. Mogilski will mostly talk about some projects. And here is a list of projects that we have based our observations on. First of all, development of major oriented calculus lab as a part of active learning and active learning assessment. Then the big project of curriculum alignment and college readiness and this project was supported by a huge grant from the Department of Education and Bill Gates Foundation community grant, about $3 million dollars combined. As a part of this project we developed two gradient classes for math teachers. The first math curriculum alignment and college readiness, I don't know about your area but in Rio Grande Valley one of the major problems teaching calculus or college algebra is they are not great. And we are trying to address this issue. On top of this, we also developed elective gradient class of collegiate teaching. Right now we have been working on developing a PhD program in mathematics with emphasis, and one of the emphasis would be collegiate teaching. Then integration of challenge based instruction into STEM curriculum. This is a very old project, so 1992/1999. Teaching lenient algebra with technology called ATLAS. It was supported by NSF foundation and a lot of materials for active learning and assessment had been developed in the framework of this project. Science component of competency based biomed program. We had a new chancellor in our system and one of his priorities is to develop competency programs. One of these programs is the biomed program and by nature it is supposed to be not only competency based but active learning, and I have the privilege to supervise the science component and develop the math component by myself. And finally fleet classroom, college classroom, contemporary math, and pre-calculus, statistics, and so forth.This slide we are talking about theory. I was not sure if your attention had been drawn to this publication. I was wrong because Charles yesterday briefly mentioned this publication. First of all, this is megastudy. It studied about 200 published papers and about 100 unpublished projects that relate to active learning. A very interesting publication, it's very easy to find on the website. I'm not going to interpret these graphs, but the bottom line is that active learning. At the same time, I also want to draw your attention to one more thing. According to some studies and I've read four or five papers about this, faculty members are really excited about active learning especially those who have already been doing active learning. But there is no formal study that supports the claim that students are also excited about this. I believe Ben yesterday in his presentation mentioned something like this. It was anecdotal but at the same time a formal study that shows students' attitude is not as clear as we think they are.In this slide I would like to talk about formative assessment and summative assessment. As a mathematician I gave two definitions of formative assessment, one from the point of view of students, left definition information communicated to learner that is intended to modify his or her thinking or behavior with the intention of improving learning and the right side is from the point of view of instruction or instructor. Formative assessment is usually defined as assessment for the purpose of instruction or interpreted for the purpose of modify instruction. At the same time, I would like to compare formative assessment and summative assessment. Most data shows that about 75% of our assessment is actually summative assessment. So what's wrong with summative assessment? First of all, some studies show that very often students don't care to look at this kind of assessment for two reasons, because this assessment normally the primary purpose is to give grade. If grade is good, then fine. Why should they care. If the grade is bad, they create a kind of protective mechanism. And because of this protective mechanism, I better not take a look at this. Believe it or not, sometimes this happens with faculty members. Just about a couple of weeks ago, I talked to my friend from the school of business and he's a world class expert in information systems and all of a sudden he said to me, "I never looked at my faculty relation from my department chair." Why? He had the same reason. If it's good, what is the point? I know what I am good at. If it is not good, then I don't want to put him in an awkward position and start arguing. No, this is not accurate, you are wrong here, it was a very interesting conversation. So 75% of assessments we are doing happen to be summative assessment and this is not very good.Over here, on the left column and right column of the slide, and left column and right column have two different reasons. First of all, some definition or some methods of formative assessment of gathering data observation, test, but later on I also find in the literature exam, and clinical interview. First time I read it I didn't think it was true. It was very narrow. But then I started thinking about it and the combination of this three is probably a very accurate observation that we have very limited methods to collect data. The teaching is to learn about performance, thinking, learning potential, and affect motivation. And any assessment must be a part of the process. The first publication that I found about this, 1949. Curriculum development objectives, right now are called learning outcomes, content, teaching strategy, and assessment. And talking about this biomedical program that I mentioned. We are trying to develop this program following this recipe. The reason I list these quotes over here. What I would like you to do. The left side says active learning demands active assessment. Instead of physical exercises like dancing or singing, and taking into account that we have a workshop or symposium in active learning what I would like you to do is spend seven minutes talking about this hypothesis and I've suggested same equations to address. However, if you find a different perspective in your discussion that is also fine. But my suggestion is do you agree with this statement. Of course, some substance would help. Second, what is active assessment compared to assessment that is not active? The third one, examples if you believe there is some active assessment and not active assessment. I would really appreciate if you would give some examples, active, not active? And finally, which of these three components objectives, content, and methods affect assessment strategy the most? According to the theory, curriculums are supposed to build objectives, content, methods and assessment. Assessment is the last one, therefore naturally the first three components should affect the last one. My question is what affects the most assessment? Let's do seven minutes after we finish presentation I will claim your responses. Please do it. But if you prefer to dance, then you can stand up and dance!Table leaders, who are the table leaders? My apologies, time flies, seven minutes is gone really fast. As I said, what we are going to do I am going to finish my part than Jerzy is going to talk about these projects then after we will have about ten minutes to listen to your input.Some challenges and questions. The first question, what data to collect. Whatever is listed here is not necessarily my own perception. I was trying to summarize theoretical studies in this field. And because most of these recommendations are from educational psychologists, sometimes we interpret terms differently. So please be aware of this. Performance, what you can see on the surface, how your students perform. Thinking knowledge that address cognitive process, so you need to find out how your students perform this or that way. Next one, learning potentials. Actually two issues. The first issue, are they really ready to learn college material and are they really ready to move forward? And finally, affect motivation and I wrote there it is commonly not assessed. (audience unintelligble) I agree with you 100% and probably in one of the (audience unintelligible). The suggestion was to include also student self-assessment, also called metacognition. From my point of view that is what I am saying. Next question. When and how often to collect data? Yesterday I heard a wonderful thing Angie said, she collects data every week. Wonderful, right? The recommendation is immediately after the learning takes place. What learning means, what does it mean that learning took place. Next recommendation from my point of you, give some guidance. Formative assessment. Motive, students need it. Second, students and willing and able to use it. Next one, opportunity. Students receive time to use it, therefore if you look at summative assessment it absolutely lacks this very important feature. Next question, how to adjust students activities and your own teaching. Here's some questions. Here I will talk about some theory because all of you know about but recently there is two dimensional autonomy. There's some verbs and nouns and I really like this combination. Actual knowledge, conceptual knowledge, procedural knowledge, and metacognition knowledge. Metacognition means what do students think about their own learning process. And I listed over there the whole concept of some different dimensions. I want to draw your attention to, and this did not come from math study, this came from huge geology class. McDonald was trying to implement active learning in his huge geology general education class. We are talking 200, 300 people. This is his recommendation what to use. The end diagram and concept maps. Therefore he throws in to his assessment some physical objects. From my point of view and the points of view of constructivists and other theories, it's very important. Angie was talking yesterday about dominoes yesterday. A physical object to work with. I don't know if she was working with this intuitively or if she read about something but this has very deep meaning. So I can talk a couple of hours about this, but I will briefly mention about this.This is some criticism about constructivism. The theory says that all the assessment efforts is just as good as the theory you follow, the theoretical background. I absolutely agree with it because sometimes we don't have a deep enough model of our teaching to make decisions and to communicate our dimensions. Here are listed two theories. Constructivism and the quote from Piaget, what reflective obstruction is, and probably everyone knows about constructivism. And the stage by stage development of mental action theory. About 15 years ago Mike Connell and I have published several papers because philosophically these two theories are not alike. We are not philosophers though. We are practitioners. From the practical point of view we came out with the conclusion, though we came from a different philosophical background, but practical recommendations would be about the same. Briefly about stage by stage development of mental action theory and preparation. We are talking about arranging activity of our students, activity or action. Every action is considered as preparations. Then there are different groups of actions, called orientation. Orientation means you start thinking about what you are going to do, therefore developing some plan. Then execution, then some kind of self-assessment. Have I done what I was planning to or not. If you look at composition of any action you can clearly find all these three groups. There are three different forms of action: material, speech, and mental. Three independent characteristics I've mentioned you can assess. There are five stages of mental action development and all actions fall into groups, specific action and general logic action. Here's an example. I will talk because of time only about linear algebra. Think about first a linear algebra course. What do students do to solve a problem? They have a problem and they have to create mathematical model of the problem. What model is it? 90% of the time the model is a matrix and I can list about 20 different problems and the model will be the same matrix. The first step is how to link your problem with the matrix. This is called general logic action. Then you take the matrix and you find your own use of that matrix. This is specific action. It's specific only for mathematics. And it's a piece of cake, right? Say you don't want to do it by hand, you can use a calculator or use technology. Finally, you find the matrix in the usual form and you need to interpret your result. Therefore it happened, and this is general logic action. You are thinking thoroughly. You can do the same analysis with calculus but linear algebra is a little bit easier because linear algebra is the easiest of the gradient courses.Very often we stop teaching something new. If we are talking about action, it must be the object of action. But object of action does not exist in the students mind. This is a new concept. The trickiest way is, okay, this is your object. And this is where the concept of representation came from, constructivism, the concept object of something to work with as introduced by Mike Connell and myself. This is one example of this called a curriculum alignment card. Basically, this is a concept map. When we studied our curriculum math project we had a group of math teachers from the local school district. What we are going to do is prepare you students better for college. Agree? Agree. We are going to align school curriculum, Texas standards and our college algebra teaching. Agree? Agree. That's how we developed this concept of this curriculum learning card that really ties together college algebra material, school material, and a Texas system called requirements for graduation from high school. And after this we started developing the information on these cards. At first it was very raw material. By the end of the class they developed wonderful products. So then we gave these products to teachers for a series of workshops and they aligned these cards with their own experience. This is an example of active learning with object of some concept that did not exist in their minds. Of course, the card is not alignment. This is just methodology, how to develop this extra concept of alignment. With this I am going to move the table to Dr. Mogilski.MogilskiI will be talking about some projects that I did with Dr. Bouniaev. At that time he was my boss. We switched recently. I was working under him. The first project that we had was the part of the grant from the Department of Education was this specialized calculus lapse problems for students. The whole idea was to include nine majors in our college. At this time engineering, biology, chemistry, computer science. And the way that we worked was that we paired faculty from mathematics with faculty from a specific department. So there was a pair of engineering and mathematicians working and the same for other areas. Typically the cycle to develop the set of problems for a specific area was one semester and the following semester it was implementation and we were doing assessment for how it worked. If you asked the colleague in engineering what is important in calculus for engineering, what part would you like to cover, you won't get a very good answer. So it takes some time to discuss and to come to the same platform and be ready to choose the good problems for the lab. The whole idea was to reverse teaching because mainly what we do is teach calculus, the content of calculus to students and then we illustrate the thing with applications or they take the next course in physics or engineering and they see these applications but the typical complaint from the faculty is that they didn't learn anything in calculus, they don't remember what to do, etc. Here it is a little bit different because the applications come first and then the content of calculus. I will talk more about that in the example I have. The other was to cohort calculus students, so we have the cohort of engineers, the cohort of biology students, and so on. They were going to labs together, and then they work on the problems in the labs, and then they come back for the lecture in regular class. We hope that students in this reverse order students get motivated to study calculus. First of all, they see calculus is about something that they are going to use in their favorite topic of study. Second thing is that this time gap between seeing applications in their specific area and learning calculus, we make it shorter. Typically if it's too long students forget about everything. And to help us improve retention. Here are examples of this specialized lab problem, related to biology. That is how the chance of heart attack depends on arterial pleasure. We ask about the rate of change. They can use the book, the notes, but they are introduced to the concept of rate of change. So they can do that formally calculated the derivative or they can use one of the meanings of rate of change and see how it works. You can make it more complicated when you include the sectional area and show this pleasure invested proportional and try to find the rate of change of well. This is a taste of what we are doing in this lab.The other topic that Dr. Bouniaev already mentioned, teaching students how to use this already constructed curriculum alignment cards. WE did this in a series of workshops that were well attended. First of all, we had some faculty from the school that helped us with the cards. They tell us what they do at school, what kind of mathematics they covered and what typical problems do they give to students and present on tests. Based on it, we are adding our college level problems and we are trying to find a connection. The faculty members who work with us become leaders of the workshop because they help us conduct this workshop. We have this group of teachers attending this workshop and we try to discuss the cards with them. The cards were challenging for most of the teachers. They didn't make much sense to them, they didn't see how to use them, so we decided to do the following thing. We first introduced the cards a little bit, then we divide the teachers into groups and give them some cards that have some blank spaces, and they help us to fill the blank spaces, adding some problem, classifying that this one is a terminology problem, this one is a procedure problem, and so on. After that, the groups present discussion on the cards, and they can change their opinions and their remarks. And after that, the next step is to give them completely blank cards and ask them to fill the cards. And then again, closing everything with a discussion. I would like to point out that formative assessment figures heavily in this because we saw the projects, we saw the product, the cards, how was it created. Other students were evaluating the work because of the discussion and the teachers were doing self-assessment at the end with a survey and so on. This was a very effective method. They started to feel comfortable with the cards. They were able to apply them. They said that they had good understanding how the cards could align the curriculum and they wanted more cards. WE have a whole repository of these cards. We are still working on a website the teachers can access and use the cards in the classroom.The other project, which was a little bit older. This terminology was not used here, along with the inquiry based learning there is something like challenge based instruction. We worked with our colleagues in Edinburg, the city next to us. The group of engineering faculty was conducting the workshops together with some people from education because they were already using challenge based instruction for a long time already. Challenge based instruction varies a little from IBL, I will explain that later. It started in bioengineering and in bioengineering you have very challenging problems because you combine a lot of areas into one. Everything was about having complex problems then students need to figure out what is needed to solve the problem from each of the areas. Could be some mathematics, could be something from biology, mechanics, whatever it is. Once they knew what was needed, then they would learn to find the sources. Could be professors, could be books, internet, whatever. Then they would help solve the problem and do the presentation of it and then do the assessment of it and so on. First of all, maybe I'll go to the next slide. We had a diverse audience in the workshop, not only faculty from mathematics but there was some from biology, some from engineering, and so on. It is a big challenge to find a challenge based instruction problem in mathematics because for most of the classes we have very pure concepts which are not a mixture of anything else. Calculus was not the only one who was heavily represented I believe there exists a lot of resources and material for challenge based instruction problems in calculus. Faculty was using the modules we prepared a lot. They were not expected in the workshop to do the whole course just one module and then use the module in their class. Just to take revenge, we invited someone from IBL to show the engineers you can do it a different way, and the person we invited is here I think called Michael Starbird. He gave us a very nice workshop. I would say the perception of the faculty from mathematics was this one was much better for mathematics than challenge based instruction. Some faculty even use IBL to teach complex analyses, the whole course.Very last project. We started doing this flip classroom project eight or nine years back. We looked at some existing products like for instance a lot of information on NCAT, which is National Council for Academic Transformations. But none of the models worked for us because of the fact that we didn't have classrooms, the proper infrastructure. You need to really force attendance to class or they wouldn't learn anything. During this period of time we managed to redesign college algebra, contemporary mathematics, elements of statistical methods, and pre-calculus. There was some attempts to do calculus but those were not completed. Another model, students were supposed to read or watch course material before they come to class. Everything was on the blackboard and they were supposed to do it. About reading. We had this discussion. Do we force students to read, etc. Students in our area don't like to read at all and immediately when we started this project we had the questions coming like "Do you have E-professor in the course?" I couldn't figure out what is e-professor but it was audio, someone reading to them so they can listen to it instead of room. So along with the regular file materials that were printed, we developed short videos. We have the whole college algebra and precalculus 100, five to eight minute clips for each objective. Immediately students fall in love with this. They ask "Can I just watch the videos and not come to class?" Then the next activities discussion of the material they were supposed to read or watch. We recorded minilectures when some examples were introduced, a summary of the lecture and so on. And that was half of the time for the week. The other half of the time is laboratory activity when students work problems on the computer but it has to be at a specific time to come to the laboratory because in the laboratory we have instructor together with some teaching assistants and they walk away and help students and students can use any resource to solve the problem but they need to submit the problems at the end to receive some credit. This is a flowchart of this method, work at home, then this class, then come to the lab when you do the first practicing of the problem, then you can go home and do the homework online, which has more attempts, then you have the quiz and that's one attempt. Each course has two serious exams which are proctored and that's how it goes. I think my time is up. Thank you.(applause)BouniaevI would really appreciate your input on this statement. Active learning demands active assessment. Personally I have some challenges with this, however I would really appreciate if you will comment on this. There is a kind of difference between inquiry based instruction and active learning. You would have inquiry based instruction, from my point of view, I would throw you active learning assessment. Think about this, make some conclusions. Over here I have made a couple steps more, I have made scaffolding. First of all, I put forth the hypothesis then I suggested some steps but at the same time I said if you don't like my steps, build your own. Anyways, this was just a comment.(audience) Our table talked a little bit about maybe the difference between what we viewed as active assessment and assessment that asks students to do tasks at diffeernt higher versus lower cognitive levels. I was of the opinion that maybe these are two separate things. That really any assessment you give them they're doing something, even if you are just quizzing them on definitions in some way that is more active than just telling them a definition. Maybe that's a very low level cognitive task. But we weren't really sure what was meant by the word active and we had some discussion about that. (audience) We started at the same place and we decided we needed to come to an agreement on the second question before we could agree on a statement. So we needed to decide what we were going to accept as a definition for active learning and we ended up going with thinking of active assessment more in the sense of formative assessment that's happening quickly throughout a class. In that sense we decided yes we agree with the statement but active learning doesn't only use active assessment. It has a more fly-by-the-seat-of-your-pants feel to where you can go in with an activity that is awesome, someone asks a great question and you completely go off of your plan or they don't get it and you realize it quickly and fix things. As far as the last one goes, we talked about it briefly, but we all agreed that objectives affect our formal assessment strategies the most even though that's not necessarily what's the most important to us because we use formal assessment strategies to have data to back it up in case an administrator comes to us and asks why did so and so get a grade when really I'd rather just have them write proofs on the board all day and I can get a feel for whether they got it or not. I have to have numbers somewhere. It's math, right?(audience) So at our table we said that active assessment can be active learning with a grade attached but maybe it shouldn't have a grade. It should be formative and change the way someone might think. You can give a quiz that you grade and give back but that's not really active assessment. We gave some examples like having group discussions in class based on something that could instructor facilitated like Ted modeled for us very well. One of our colleagues said she had students put up presentations like a poster session in class and students go around and put comments on others problems with post it notes.(audience) Our table was trying to answer the question in the sense of active learning as a collection of techniques to facilitate the creation of shared meaning between the students and the teachers and active assessment as a part as one of these tools. We started to focus our conversation around active assessment as giving substantial feedback to the students but also students giving substantial feedback to the instructors. As many others have mentioned, it's not just point added up X's and O's or arrows pointed in different directions. It's a dissection of the products that the students produce but also a dissection of the products the instructors produce all in the service of creating shared meaning.(audience) As we were struggling with what active assessment was, we thought of an assessment opportunity that provided a back and forth feedback. An example I gave was that students were creating an infographic in a quantitative literacy course. The goal of it was to show an issue related to immigration. Could be immigration in education, immigration of the labor force, some area. They gave presentations about their infographics. The active assessment part of it allowed myself or other students in the class to ask for more clarification where they might not have clearly defined something so there was some give and take and an opportunity for a student to further clarify their understanding in the midst of the statement and that made it an active opportunity.MahavierAlmost a repetition of what Rebecca just said. Our table said you ask very specific questions. A student is presenting something and you ask a very specific question of another student in class. Angie, can you tell me how John got from step four to step five? There are many reasons to ask this question. One, it could be invalid. But more likely you are trying to figure out does Angie realize that that follows from the definition of continuity convergence, the extreme value theorem, does she understand this. If she does not, can the speaker justify it to Angie so that Angie then understands it. So you ask very specific questions that help you assess a minimum of two students and then more than that, can someone else answer this question so the onus isn't always on Angie. (audience) I would say the point of formative assessment is to for the instructor, the student, and the other students to see what a student is thinking. You are trying to get a view into someone's head to see what they are thinking. What someone is thinking is changing all the time. It's not I know this then I know this, it's additive. It's a very complex thing.(audience) We didn't answer the question. We had a robust discussion about what the question was. From my perspective, I was just making this comment to David. As mathematicians we will be very specific and clear about definitions. We will go to this process of making conjectures and refining hypotheses and as soon as we step into talking about scholar teaching and learning we drop all of these skills entirely. So I've been thinking about assessment recently. The first thing you have to ask is what am I assessing. Am I assessing student learning, am I assessing teacher implementation of active learning tasks, am I assessing the efficacy of a given task across multiple teachers? Once you've laid down then you have to ask what do I want to know that it does. These are different questions. Does active learning demand active assessment when you are talking about does it just help students, like are students reaching their outcomes versus does active learning demand active assessment in terms of the efficacy of a given active learning practice. I see this in conference after conference that I go on this topic, we often times start in the middle of an investigation and don't step back and really clarify what the constraints are of what we want to talk about. For me, this would be third question down the line and the first thing I would want to do is put it in a specific context. That's just my take.(audience) Wanted to share an anecdote. I taught number theory for the third time out of Marshall, Odell, and Starbird this spring and for the first time students received weekly written feedback. Students presented at the board so we had active formative assessment that way but in addition they had problems sets to turn in every week and they got written feedback. They received no grades of any kind throughout the entire semester. This is very disconcerting for a lot of students. The work of Dylan William and others shows that as you mention as soon as they receive a grade they stop looking at what you have written. They had opportunities to turn in repeated versions of their work until the semester was essentially over. (inaudible audience) I did assign grades and I will do some version of this course again. I think it went extremely well. Over half of my students were pre-service middle school teachers but I also had computer science majors, physics majors, so I had a variety of majors. They started off the year going to the board in small groups but I weened them from that. They presented at the board individually and ultimately everybody presented projects as their final exam, taking topics that we had not covered in class and presenting them to the class. It went very well. All the students who participated passed the course. We had many breakthroughs, students who did not believe they could do proof at all or that proof was a totally unrealistic expectation for them embraced the idea of writing and presenting proofs. (audience) I'm Jim Factor and I'm from Alverno College. It's an all women's college. We haven't given grades since 1970. This is the paradigm that's been used, no grades. Right now we have three of our math students in their PhD programs and some that have already got their PhD's and we've had many positive aspects talked about. We have an integrated approach across all disciplines. We work together to share the common aid abilities that application across every area. In terms of not giving grades we are constantly having the students self-assess, know what they know or they don't know. That's a very important part of active learning, maybe that's the most important because it's only going to cause real change that the student owns and not impose from outside. We have found that our students when they come from high school and "I don't get a grade, why" but once they are involved in the culture and are in there awhile they focus on learning. One of the key abilities we have is social interaction so they know how to work with people. Once they get into that mode we now have them think about "Oh I got an A, I got a number of points, I don't have to do anything". No. It's important to continue to learn. They need acknowledge what they know and what they don't know and then the confidence that they can learn what they need to know. This is the paradigm for the 21st century when we have exponential growth in national. This is a type of thing that works very well across all disciplines. Feedback is a major part of what we do. We require them to know content. We evaluate do you know and can you survive at this level. For each of the areas, say, for problem solving, in the context of calculus we have six different levels that they have to demonstrate knowing the material from observation all the way to the part where they have to apply a framework to different areas. They will go through all of those in the class and depending on where they are at they are constantly you are at this level here and you need to get to this level to pass this course. The idea is for them to develop these skills and use the same kind of skills in different areas.(audience) I'll be very brief. Bob Sachs from George Mason. We had a discussion of examples. One of my colleagues has basically perfected a lovely thing that is an active assessment that isn't totally separated from active learning. We use it even in non-active learning classes. Before major exams we have this thing called oral reviews. It's voluntary for the students. They show up in a room, they are all up at the board, they work on interesting problems and a facilitator runs around the room, and she's talking about it on Friday so if anyone wants to know about that. Her name is Mary Nelson, she was at Mason and then at CU-Boulder and then back at Mason.16 08-03-16 Suzanna Klaf Zachary KornhauserKlafGood morning everybody, I'm Suzanna Klaf and I'm the Associate Director for Teaching and Initiatives at Columbia University Center for Teaching and Learning and I'm joined by my colleague...KornhauserZach Kornhauser, Associate Director for Assessment and Evaluation also at the Center for Teaching and Learning and Columbia.KlafIt's a real privilege to be here with you today. Full disclosure: neither Zach nor I are mathematicians nor really from STEM fields but we work on a regular basis with STEM faculty so we really are privileged in the kind of work we do to get people to think about active learning strategies in the courses and curricula and also thinking about assessment. What we'd like to discuss today is to reiterate the points we've been hearing over the last day and a half of this symposium, thinking about the why and the what of active learning but also bringing in a slightly different perspective, and that's the perspective from a center for teaching and learning and the kind of approach we use. I'll actually go into more depth of the kind of professional development activities that we engage in. I know we've heard about some other forms of workshops and things like that to promote the effective learning and teaching strategies that we'd like to see. And we'll be emphasizing, and this is where Zach's expertise comes into play, some of the tools for assessing active learning. And then finally, closing out, I know one main question people were asking yesterday was what can we do? How can we be the agents of change? And we really do want to emphasis the scholarship of teaching and learning not just to inform our practices but more so, how can we all be contributors to the scholarship of teaching and learning and we hope by engaging in the activities and worksheets we have for you that you'll walk out of here thinking about what unanswered questions are out there and how can I be an active contributor to making teaching learning visible and advancing the effective practices around active learning and the assessment of active learning.We do have worksheets at the center of the table piled up. I just want to walk you through those worksheets quickly. If you can grab one. If you don't have one we have a whole stack at the front and we do have extras neither of which Zach nor I want to carry those, they are heavy, so help us lighten our loads please and feel free to grab additional worksheets. What you have in terms of worksheets and handouts, you have a resource and reference sheet. This will hopefully be of use to you as we cite some sources from the literature. The next handout is a summary of various assessment tools which Zach will be discussing. We also have a handout of 50 classroom assessment techniques which is something that distribute regularly to our faculty. We also have a blue worksheet which is the assessment loop and we will highlight that in the example that we give and have you think about that. And finally we have a worksheet for contributing to the scholarship of teaching and learning. On the back side there are some references to general SOTL resources but then also so possible journals which can be some great outlets for the research that you're engaged in. As I mentioned a moment ago, we do hope that we can get you actively thinking about your own practice and hopefully leaving here today with some sort of action plan in terms of how can you contribute or continue to contribute to this work. We will jump right in.KornhauserThis has been discussed already a couple of times at this symposium and I don't want to belabor these points but I think it's always helpful to frame this in terms of what we are discussing, and I'm going to frame it in terms of the issues that have been facing STEM education. In the last decade there has been a decrease in the number of STEM degrees conferred as a proportion of the total number of degrees conferred, and that has been happening consistently. 40% of students who intend to enter a STEM field actually complete that STEM field. Accompanying this in the job market, there's been much more of a need for people with STEM degrees. When jobs were added during the economic recovery period, there was about 3 million jobs added, close to a million of those were in STEM fields and current projections have by 2020 there will be 2.6 million more STEM jobs. So if there is such a need for STEM degree holders, why aren't there more STEM degree holders? The way the market should function there should be. A lot of people are pointing to a lot of reasons, but one of them is perhaps the college classroom which is where folks are turning their attention to now. At the college level, research indicates that even high performing STEM students find themselves to be uninspired by the way courses are taught and low performing students find that the instructors are difficult to work with, they're unwelcoming, they don't structure their classrooms in ways they can succeed so the STEM pedagogies are in need of improvement and more engaging teaching methods have been considered an avenue for that improvement. In particular, adoption of active learning techniques has been seen as a way to address this issue. When we talk about what is active learning, and I don't want to spend too much time on the definition. There's two quotes that work well here. One of them is anything that involves students in doing things and thinking about the things that they are doing, and the second one is anything course related that all students in a class session are called to do other than simply watching, listening or taking notes. Rather than focusing on a specific definition I think it's better to focus on the characteristics of active learning classrooms. Not all active learning classrooms look alike and I think that's pretty important to realize, but there are some important characteristics that occur across these classrooms, and these are students doing more than listening, having more emphasis on developing skills rather than transmission of information, targeting these high order advanced thinking skills rather than more lower level basic skills, engaging students in activities and placing emphasis on exploration on students' attitudes and values. In practice, the way active learning happens in classrooms can be very different. The literature discusses many ways of what active learning classrooms look like and that includes small group discussions, peer instruction, using clickers which was discussed yesterday, problem based learning, case studies, solving problem sets in groups, even things like computer simulation and games, and concept mapping. In practice, active learning classrooms can look very different from each other.Why use active learning? This was discussed as well. There's a growing body of evidence that suggests that active learning provides improvements over lecture. That's not too much in dispute but one thing that Carl Weiman notes is that when you look at the analyses as we know are just collections of a number of studies produced on a topic, when you look at individual studies the effects of active learning classrooms, there's a large distribution of those effects. Saying that something is an active learning classroom doesn't mean that you are going to enjoy the benefits that are provided by other types of active learning classrooms. The term fidelity of implementation was mentioned so what can you do to implement an effective active learning classroom because active learning classrooms cannot be considered as one and the same.KlafOkay, what do we try to do in the Center for Teaching and Learning to promote active learning practices? Just a bit of background. Our center for teaching and learning was created about a year ago and merged different entities, primary our Center for New Media Teaching and Learning and our Graduate Student Teaching and Learning Center. At the foundation our mission is to try to create this culture of teaching and learning on our campus. I know that yesterday many people were thinking about how do we change the culture of teaching and learning and Zach and I really do the center as that space where faculty that volunteer or graduate students or really any instructor that is teaching at the university can come to us to experiment on their practices and to think about how their practices can be informed by the scholarship of teaching and learning. So we really try to create an environment in which that experimentation for active learning can take place and also to think very intentionally about what one is doing in the classroom and why and ultimately communicate that to the students. How do we be very transparent? Yesterday people were mentioning how do we frame this for our students and that's a very critical component. We were work very closely with our instructors to think about how they're communicating, what they're doing, why they're doing it. Students can be resistant and think "Active learning, this is busy work. Why are you having me do this, I can learn in different kinds of ways". It is very critical to think very deeply and intentionally about one's innovative practices and really help students change their own perceptions about what teaching and learning should or should not be.We try to do this in the center by offering a range of different programs. We are purely voluntary. People come to us, we don't impose our work on anyone. We do this through various means, whether it's one-off workshop or more intense kinds of programs and offerings like our multiple day institutes and different types of programming like a reflective seminar that we offer or one on one customized opportunities to work with individuals that come to us with various teaching and learning needs. These one on one consults really start the conversation where we pose questions and get people to really reflect on their practices, the what, the why, the how of what they are doing and how they would improve over time. So one critical component of how we get people started reflecting on their practices a lot of times it really comes down to the design. The point I made earlier about intentionality so some of the foundational frames that we use, one of which is actually backwards design. Anyone in the room familiar with understanding by design? Particularly towards the back there a lot of people that are familiar. Williams and McTighe really were writing to mostly a K-12 audience when they published these particular works, understanding by design. Their argument is that one should begin with the end in mind. Rather than planning your courses, your curricula or your course units by thinking about your content or the cool activities that you want your students to do you should really think about what really are your student learning objectives. By the end of the course, by the end of the semester, by the end of the program, what should you students, be able to do, or value differently. That's really what the starting point should be. That should drive the decisions that you make in terms of your assessments so stage 2 on this graphic is determining that acceptable evidence. How will you know it when you see it? How might you articulate the different types of assessment strategies or methods you will use? Finally, thinking about, ok, how am I going to operationalize this, what kinds of active learning strategies am I going to utilize, what kinds of engaged pedagogies am I going to use in my classroom in order to help students achieve those outcomes? The critical component is that all three components are aligned. When we work with faculty we have them reflect on all of this. They may come to us with a great idea like ooh, I want to try this innovation, I want to flip my class" and we have to press the pause button and say let's step back. What are you trying to do, why are you trying to do it, why would this be the most effective way to achieve these student learning outcomes? So really getting them to think critically about their practice. Backward design is something we'd also like you to think about today. Conferences and symposia are great opportunities for us to reflect about our own practices, why do we do what we do. Here in this slide we have some questions to think about. I know yesterday people were mentioning a lot about this notion of situational factors and the context we operate so questions here might be who are our students, what are their particular learning needs, so this recognition that our students are very different from us, perhaps we are experts in our discipline but they are novices but also they come from different socioeconomic backgrounds, different cultures, so we need to factor this all into play. They may have education baggage, as I call it. I know some people were saying yesterday our students are kind of hung up on "I can't do math" or there's certain barriers or this fixed mindset. In the situational factors, we have to think about who we are as instructors, what do we bring to the table and how can help the students learn.Getting into the three components that I mentioned a moment ago, the three stages of backward design. Objectives, what do we want your students to know or be able to value differently by the end of the course? What kinds of evidence of that student learning would be acceptable for you to actually gauge that your students achieved those outcomes? And finally, what kind of learning experiences are you going to create for your students? Just in the interest of time what I'd like you to think about is the context in which you are operating for the context of teaching and learning and then the second component, regarding the objectives for a specific course or curricula or program you are working on. So you might want to jot those down for yourself and what I'd love you to do is to actually find someone at your table which you have not discussed with and share with one another the context and the student learning objectives you have. This is a great way for us to hear what others are thinking and find commonalities across and we're also trying to model the kinds of things we have our instructors who participate in CTL programming engage in. If you would take a few minutes to do that and would love to get some lively conversation going here. Again the first two, situational factors and student learning objectives.There's a lot of lively discussion here, that's great. Hopefully you've had an opportunity to share across the table thinking about who are your students, what are some of the situational factors that are going to impact the kind of design for your particular kind of course or curriculum and also what are the student learning objectives. So that is the first step. Soon we'll actually be thinking about the actual assessment component, and Zach is going to walk us through various strategies. Before we get to that, this activity models the sorts of things we try to do in the center. One example is something that is really a deep dive into active learning, something we started which is the Active Learning Institute. This is actually an iteration of a previous institute that a colleague of ours, Jessica Brodsky, brought from Brown University which is actually a flipped classroom institute. There was this recognition that there is a lot out there about the flipped class room and people are really excited about that but at the foundation the flipped classroom is all about how do we engage our students more in the teaching and learning process. We launched this active learning institute and primarily we had our STEM faculty apply to participate in this. As you can see on this slide our flier for this particular intensive and also some photographs of our instructors working on mapping their courses and really thinking intentionally how all the components align, what does the student experience look like, why include certain strategies versus certain others, how is this informed by the teaching and learning literature and by what we know works. One thing we really try to stress when we talk about active learning, aside from the definitions and the literature Zach shared earlier, we have our instructors think about taking a holistic view of active learning. This comes from D. Fink's work "Creating Significant Learning Experiences". According to Fink, we present our students with information and ideas and we may have them do active things with that, things like recall or lower level blooms where they are doing very basic cognitive tasks. But then as instructors we can create certain learning experiences for them that provide much more rich, higher level kinds of skill development. Whether it's simulation or observing the stress is on the doing piece which was in the definitions that Zach shared with us from Bonwell and Eison. And then finally there's that metacognitive piece. I know today there was mention of the development of metacognition but this is really all instructors' responsibility. We're not just here to deliver content but to help students reflect on the active learning experiences that they engaged in. Having them not just think of the content and subject but also reflecting on how learning works or learning how to learn certain skills. Some of the examples we have been hearing the last couple of days on engaging in active learning with clickers. We may pose a clicker question which is a great opportunity for students to self-assess to know what they know and what they don’t know because they see the instant feedback. But it's an important opportunity for us as instructors to be able to adjust our teaching as a result. It's a missed opportunity if we don't engage in some kind of debrief conversation with our students following the question rather than just moving in. So that reflective component is really key and at the center for teaching and learning we try to emphasize that. What was funny was when our instructors actually mapped out their courses and the learning activities that they engage in very few of them had reflective component. It became very apparent to them that that was missing from their map and they realized that was something that they wanted to build in so their students could have a significant and transformative learning experience.A couple of resources we share with our instructors and that are listed on your resource and reference sheet, so you don't have to write this down, are some accessible texts that are out there or some short papers. For instance, we have the idea centers short papers on active learning strategies in face to face classrooms, we have Student Engagement Techniques by Barkley, and then also Collaborative Learning Techniques. The Student Engagement Techniques and Collaborative Learning textbooks are really great because they also provide strategies for adopting these techniques in the online environment as well for those of you interested in hybrid techniques to teaching and learning these also have that going for them. Something else that we really stress in our institutes but just in general in the center is putting assessment where it belongs not as an afterthought but foregrounding it in the design process. One thing that we do in the active learning institute is walk people through this assessment loop and really help them think about creating an assessment plan for their course and think intentionally about what they are going to assess and what they're going to do with the results they get back. Hopefully this becomes an iterative process where they can then make adjustments to their course or their curriculum design. The blue sheet in your packet actually is an adapted version of this and later we will ask you to think about this in your own context.Another component for the assessment piece, as I mentioned, this idea of creating some sort of assessment plan, we really do stress that the assessment center selected align with the student learning objectives so you are not just giving tests for the tests' sake but that it actually measures something that you said was important from the get-go. Or actually look at particular skills that you want your students to develop. You are not going to ask students to recall facts on an exam if you are looking for higher order problem solving or critical thinking skills, then you'll need to rethink or adjust your assessments. We also have our instructors think about what's going on in terms of assessments inside and outside of the classroom, why limit your assessments to just a face to face test. But then also what kinds of methods are you going to use, so this is where we engage our instructors in thinking about not just the summative or high stakes assessments, you have a midterm, you have a final at the end of the class. The formative pieces, how do you stage your assessment strategies so that your students have opportunities to improve their performance, they have opportunities to check in to know where they are at, and give them opportunities to not just self-monitor but for you to make adjustments based on the needs of your students. I know that these were referenced earlier, but the formative piece, you also have a handout which are the classroom assessment techniques so these are great tools from Angelo and Cross from 1993 so the cats are wonderful ways to integrate active learning but also classroom assessment techniques so these two can go hand in hand. Generally not graded but you can assign some points to motivate your students to participate in them. Again, they make your students more self-aware to where they are at but they do help you as the instructor, they don't take a lot of time, and they can be adapted for large class context or smaller class context, online world or face to face world. And finally as our instructors develop their assessment plans or develop their course units or redesigns of their curricula, we really have them think about how do you know if what you've done in your changes or your innovation are actually working, so we have them develop an evaluation plan. Have them think about what sources of evidence will help inform their decision making. Whether it is asking for student feedback, asking a colleague to sit in on a class and give them feedback, maintaining a reflective journal or log throughout the semester so they can keep track of what worked through their own observations. We do try to emphasize this idea of moving beyond just the end of the semester generic form that everyone has to have their students fill out to thinking more intentionally about what is it that they really want to get feedback on and how will they use that information to make changes. This is our emphasis on taking a scholarly approach to one's teaching and something we will come back to in a little bit, but first we will have a deeper dive into the assessment tools.KornhauserIn the literature, the way folks are assessing active learning there is no one way that it's happening but I think that it's important to point out multiple ways that active learning is currently being assessed. One common indicator that is used is academic success. That can be operationalized differently, sometimes as grades, sometimes as general persistence through college . You'll see a lot of pre and post-test comparisons, and these comparisons are run by looking at the gain in students' skills or content knowledge. So if you target a specific skill for students you are looking to promote, having a baseline measure of that skill and then having measure of that at the end of the course. There's also currently different articles of discussion using performance on test measuring higher order thinking skills, which were discussed order, including critical thinking, written communication, synthesis. And then there's also some social cognitive measures being used which I think is interested because it might frame how you think about active learning differently. Folks are looking at interpersonal relationships between peers and instructors and how that changes throughout an active learning classroom. There's definitely literature on that as well. There's also literature looking at the overall college experience. Does being involved in active learning classes create more satisfaction with the college experience? And finally, because of the issue a lack of interest in STEM has posed for college, some folks are looking at attitudes towards learning and subject areas through participation in an active learning classroom. All these are commonly used indicators at the moment.We are going to talk a bit about how you might measure active learning in your class and the handout lists more measures than I am going to discuss here. There's just a few that I want to spotlight, one of them is the CLASSE. How many of you are familiar with the NESE? It's developed by similar folks. What they are looking at is asking students how frequently they engage in various practices and then asking instructors how important those practices are in facilitating student success. You can contrast the responses from the students and the instructors, you can find out which practices are occurring more or less frequently than you expected. And then there's the Science Motivation Questionnaire, that's developed by Glenn, that is a very resource for a lot of purposeful thought about assessment in active learning classrooms. It's a scale that assesses student motivations to learn science in college and in high school. You can determine students motivation to learn STEM at the individual level or at the class level so you can distribute the students and find out what the motivation level is like at the beginning of the course to learn STEM and then you can perhaps interface with particular students who have lower level of motivation to learn STEM or you can also look at it as a function of the entire class and it's a great measure to use to evaluate change. If one of the things you want to do is dispel some of the fear that students might have with an advanced mathematics course this would be a great tool for doing that.The second questionnaire that's being used is the Study Process Questionnaire. What this does is evaluate college students' learning approaches. Are they learning at deep levels or surface levels? This helps categorizes through a questionnaire types of approaches students are using. If your goal is to create deep learning how do you know if that's happening? Questionnaires like this are very valuable because they will give you an indication of whether or not you are reaching that goal. Then there are observational tools. Now we are getting to things that are more complex. Two I want to highlight. One is the COPUS, which was developed by a few folks including Carl Wieman, and what the COPUS does it allows you to characterize how faculty and students are spending time in the classroom. If you say that you have an active learning classroom and this is what time is being spent on then this is an observational tool that can actually tell you is the time is being spent as you say it is. It will code behavior into what students are doing and what instructors are doing in the class. The advantage of the COPUS is that out of all the observational tools out there this requires the least amount of training of all the measures. Typically an hour and a half, according to the authors, is needed to code the interactions that happen in the classroom. A more intensive measure that's been around a little bit longer is the TDOP. It's similar in that it's a protocol to really get at the nuances of teaching behaviors in a descriptive manner. It really describes what's going on in a classroom at the moment. The observer that's using the TDOP will record behavior every two minutes like in the COPUS. There are more codes in the TDOP, it captures a more expansive array of behaviors in the classroom. The main downside of the TDOP is the intensity of the training needed. It's usually a couple of days of training in order to be a reliable observer. That's why the COPUS was developed. There's also outcome tests you can use, like the Collegiate Learning Assessment which Doris could talk to you more about. That measures critical thinking and written communication. It's a non-content-based test. If you wanted to look at growth in skills agnostic to the content, an assessment like that could help you do that. Another one that has been used a lot is the Classroom Test of Scientific Reasoning. It's been around awhile. It was adapted again in 2000 and maybe after that. What this looks at is its ability to apply aspects of scientific and mathematical reasoning to analyze a situation or to make a prediction or to solve a problem. It's looking at skills you need in scientific and quantitative reasoning. The downside of this one if you look at some of the items they look a little basic so it might not be appropriate beyond introductory college students. Suzanna mentioned the classroom assessment techniques. They are great tools. They are ungraded, they give you a lot of formative feedback that can be inducted quickly. You have 50 in that handout there, and the important thing is that you can find the ones that speak more truthfully to your course context and you can embed those into what seems more natural or most connected to your course. This was mentioned earlier today, but the AACNU Value Rubric. They produce rubrics for the 16 of the most frequently assessed learning outcomes in higher education. A lot of the outcomes you are thinking of there are rubrics for. What the rubrics do they break down the learning outcomes at different performance levels. So looking at more advanced, more emerging or developing performance levels, they categorize those outcomes for you. I encourage you to look more at the tools but this was a rundown of some of them. KlafNow that we've heard about some of these tools and now you've thought more about your student learning outcomes for a course or program you offer, we'd like you to refer to your blue sheet, I see some of you have pulled those out. This is an adaptive version of something we use in our active learning institute in our center. What we hope for here is for our participants in thinking about their course or curriculum design is to really work through these various components of this loop. Thinking about intentional planning, thinking about the implementation of particular strategies that ultimately align with your student learning objectives, thinking about what kind of evidence you are going to gather, how are you going to analyze that, and what are you going to do with that information, so hopefully you will close that loop and really think about ways you can improve or bring about change. We do have these two questions that we'd like you to reflect upon. How do you know if the active learning strategies that you've been using in your particular course or program have been effective and what changes would you make to the learning experience based on your particular instruction. Hopefully you can identify certain assessment tools that make sense for your particular context depending on what you are trying to measure and matching those tools. You do have the resource sheet with the assessment tools so hopefully that can be a useful resource. Thinking about these questions for your own context, if you can just jot down a few notes on your blue sheet. Hopefully this will be a nice take away and something you can keep working on as you think about as you think about closing the loop in your own context. Hopefully you've made some notes for yourself and we will continue on in thinking about taking a very scholarly approach to your teaching, what are the trends, and what kinds of questions might you think of posing. KornhauserYesterday there was discussion about how you can contribute to the field and where can you generate knowledge that would be of value to others. I just want to say a couple of things. In terms of if you are looking through active learning literature in STEM fields there is far less literature in the mathematics field than there are in the other STEM fields. It really is ripe for discovery in that sense. I want to let you know a little bit about the current trends, what are people studying now in active learning. Some of them are flipping the classroom, as Suzanna mentioned, is a big focus and folks are getting a lot more creative with it, looking at it in different ways. I know a lot of you have mentioned teaching very large courses so they are focusing on different strategies for flipping the classroom in courses that might have 100 students. Another focus is getting faculty to adopt active learning classrooms. I know there was a lot of discussion yesterday about how do you have faculty and what's that process like. One article here focuses on the issues that are posed by faculty adopting active learning classrooms. Another focuses on a mentorship model, having senior faculty mentoring junior faculty in order to encourage the adoption of active learning classrooms. There has been research now conducted on active learning to address the achievement gap in a class itself, if there are students more likely to have poor performance than others how do you close the gap using active learning strategies. Then there are other alternative ways of promoting active learning in class so there's different studies like students modeling molecular models with their bodies, promoting active learning through students answering online questions outside of class, and using storytelling to engage students in a genetics course. There's different ways that folks are approaching active learning in more alternative manners.KlafIt was mentioned earlier this wonderful resource that you have through the MAA, doing the scholarship of teaching and learning. What we want to really encourage is that all of you here today try to contribute to the scholarship and think about what kind of questions you would propose. There are these great resources as have been mentioned earlier, there have been minicourses but there's also this print on demand resource through your organization. We'd like you to refer to the Contributing To the Scholarship of Teaching and Learning handout. On this handout it walks you through the various stages to think about creating a SOTL project, starting with what kinds of questions would you propose, what are you interested in learning about or finding out about, what is the literature saying, how are you going to build on what others have said and others have done, figuring out what your research design and methodology will be, what kind of instruments will you use, perhaps some of the ones Zach mentioned, perhaps you will develop new ones, what type of data will you collect, how are you going to analyze that data, and what types of venues will you use to report those findings? Perhaps you will be on a panel at the next MathFest or perhaps you will publish something in one of the scholarly journals. If you flip that worksheet over there are some resources. It is by no means comprehensive but it does provide sample journals or venues where you might propose one of those questions. With the time we have left we would like you to think about how might you contribute to the scholarship, what questions might you pose based on your context? We'd like you to share those at your table and then we'd like to invite you to come up to the mic to volunteer and share what those questions are because, chances are, you might be asking similar questions. One thing we try to stress in the Center for Teaching and Learning is creating a community of practice, and encouraging people to come together to create support structures to encourage writing circles or SOTL circles. Faculty and graduate students coming together that are engaged in the scholarship of teaching and learning to mentor one another and give each other feedback and to perhaps conduct joint studies so looking at similar questions in different institutional contexts and publishing those results, so encouraging you to think about what opportunities you might have to network with equally passionate people in this room here today. To reiterate, what we'd like you to do is to focus on that first row, identify possible research questions that you might contribute back to the teaching and learning literature and discuss those at your table and then come back up to the mic to volunteer to share what questions are you pondering about, what remains unanswered in the literature and how can you be an active change agent, not just on your campus but publicizing those findings beyond your particular context.Hopefully you've been able to share some unanswered questions or question you have that you'd like to explore further and contribute to the literature. So we'd like to take some volunteers, so can we have two volunteers to share back?(audience) I'm Debbie Borkovitz, Wheelock College. I'm not doing exactly what you just said. What is burning for me is what lasts. I don't care so much about how this technique or that technique does at the end of class, I care about what happens ten or twenty years down the road. I'm not quite in this world and I'm always a bit skeptical about this. I think we are asking the questions we can answer but I don't know if we can answer that question because there's too many variables. I was talking with my table, one study that I have thought about, because we have this math for elementary teachers class we have these things called baseline proficiency which are these quizzes they have to pass to pass the course. We've been doing this for ten years so I know students have passed the course. Number bases is a topic that most of them haven't seen before and probably haven't seen since so I thought we could talk to them ten years out and see what they know but my table is telling me no, that's not what we are really looking at. My concern is really that blue sheet that you give, with this whole process, if we're focusing on things that we can measure are we missing on the most important things? The best teachers that I know if they have decades of experience I might trust their intuition more than I trust an inventory. Certainly an inventory I'm going to make tomorrow to get things that are measurable. I haven't really delved into this scholarship of teaching and learning world in the same kind of way. Part of what happens for me, the question that keeps me up at night, don't ask that question because it's too hard, ask these other questions instead. If anyone can help me with this it would be great and I'd love it if you've got something to tell me no, this is how you do this, this is the next step, because I get stuck here. That's not an answer to your question but there you go. My table wanted me to come up (laughter)KlafThat's great. Often time our instructors do come to us and they are struggling. How do I measure the unmeasurable? How do I measure life-long learning, how do I measure some of the transferable skills that are perhaps outside of the purview of my content or expertise area? These are things that we grapple with and I think the value rubrics are a great way to think about how do we think about student growth, student development over time and not to measure or focus on can they regurgitate certain facts or certain formulas, can they do some of the higher order thinking skills or critical thinking skills that aren't as easy for us to assess? Just in the interest of time because we do want to leave some times for questions and Zach would like to share another opportunity that we have in our center plus opportunities we have for grad students because how do we prepare the next generation of teacher scholars and what resources we have out there. Zach?KornhauserJust to address that question as well, I think the other thing you need to think about is are we putting students in the position to succeed? There's a growing body of evidence on who are the students that are most likely to succeed after graduation and what are the characteristics that they possess. Gallup has conducted a lot of research into this and they've identified certain factors that are descriptive of students that are likely to succeed after college by their outcome measures. You can definitely promote the skills that make it more likely to succeed than not, and that's definitely something that should be thought of.The last thing I want to mention is the CIRTL network which Columbia has joined recently. The CIRTL network is, I mentioned this yesterday just as a comment to a presentation. Some folks were talking about as a graduate student they weren't feeling any support at all with teaching. The CIRTL is supposed to address that. It's the Center for Integration of Research, Teaching, and Learning. It's an NSF funded center. It really uses graduate student education as a leverage point to address the issues that are currently existing in STEM education. Those institutions that join the network, CIRTL can operate differently within the institution but they are committed to CIRTL's core principles which are: teaching as research, which they call TAR, learning diversity, and learning community. Grad students who are a part of CIRTL, the term they use is CIRTL-ization. If there's CIRTL-ization of the program, they will be encouraged to contribute to being reflective on their teaching and contribute to the research in that sense as well as the other two principles. Those students that do participate in CIRTL, their time to completion is not any longer than students who do not. In addition to that, they are more likely to find tenure track faculty position in R1 universities. It also improves the education offered at the institution where there is CIRTL. There is a lot of evidence, and they do a really good job of documenting their effectiveness. This is just another thing I wanted to share with you as well.We just have a few more minutes for questions from the session so if anyone has questions now would be a great time.(audience) I thought I would make a quick comment to offer a counterpoint on the potential value of SOTL in mathematics. Perhaps this isn't a strong enough statement from where you folks are coming from, but encouraging teachers to engage in any SOTL-like behavior would hopefully raise the level of awareness of the importance of being an intentional instructor. So for someone like my colleague who was saying maybe she's not sure how SOTL would serve the purposes that she thinks are most important. I'd venture the guess that you are already intentional in your instruction so perhaps you aren't the principle target of the SOTL movement. I think that it still has a lot of value to get people to talk about the notion of identifying an aspect of a course and studying it worth any kind of rigor. (audience) Emma Wright, Plymouth State which is in New Hampshire. We have a center for excellence in teaching and learning and we've had it for several years. I like their mission but I'm not sure how much of a university impact that they have. It seems like something you could use or you could ignore. As we talk about creating an environment for these techniques I'm not sure if that's what they are doing. They target people who want to come there but not reaching everyone. I was wondering if you had any advice in terms of taking your center for excellence in teaching and learning and making it impactful across the whole university.KlafThat's an important point. Not all centers are created equal and have equal clout on their campuses. I have been very fortunate to have been part of other centers in my career and when a center is invited to the table to inform decision making, I know yesterday people were talking about active learning classrooms, and these are extremely costly, and who's deciding what those active learning classrooms look like? Can teaching and learning centers be invited to the table to show the scholarship of teaching and learning that perhaps can impact the kinds of decisions being made. I would encourage you to invite your center to these various spaces so if you serve on different kinds of committees, centers can be great advocates for faculty but we need to be invited to the table. This is how you can leverage the various hats you wear on campus whether you engage the center or your faculty development committee because it may be one faculty member on your campus designated to focus on teaching and learning. Encouraging them to be a part of those conversation regardless of what they may be can be very impactful. Hopefully your center is endorsed by strong leadership or can work with leadership to impact decisions. I know, for instance, at my previous center we worked very closely to help inform rank and tenure guidelines to encourage that SOTL be recognized not just as an alternative scholarship but as a legitimate form of scholarship. That was a really impactful kind of change. Centers can be change agents, but it really does require everyone to make an effort to bring centers to the table and to encourage them to advocate for faculty along the way.ZahnerI think we are out of time. Two things, thank you Zach and Suzanna for your presentation (applause). Before we adjourn. One, I just have a comment regarding something Tara mentioned yesterday. With all the mathematicians in the room, when you tell someone that you are a mathematician and the reaction you get, just consider what it would be like if you had to tell someone you were a psychometrician. I'm just saying, a little perspective. And lastly, Ron has some closing words for the group. Thank you.17 08-03-16 closing remarks (Douglas)DouglasI want to start out by thanking Doris and David for helping me in organizing this and let me thank the other presenters and panelists for also participating. And of course all of you. I think many of us have seen this as a group of people. It is unlikely that everyone in this room knew everyone in this room before this meeting occurred. There are different points of view toward active learning but I think one of the reasons for trying to organize this symposium was the idea that the practitioners, the people that teach, the people who are doing all these things, it's time to broaden the participation of people doing this but to also think of this this as a part of a larger enterprise. I hope this has helped to accomplish that. I hope you will be able to take advantage of the contacts and connections that you've made here. Again, let me thank you all for coming. (applause) ................
................

In order to avoid copyright disputes, this page is only a partial summary.

Google Online Preview   Download