AMERICAN INSTITUTES FOR RESEARCH

National Implementation of TeamSTEPPS Webinar 7

Moderator: Alexander Alonso
February 11, 2009
12:00 pm ET

Operator: Ladies and gentlemen, thank you for standing by. Welcome to the American Institute for Research Designing Evaluation Systems for TeamSTEPPS conference call. During the presentation, all participants will be in a listen-only mode. Afterwards we will conduct a question and answer session.

At that time if you have a question, please press the 1 followed by the 4 on your telephone. If at any time during the conference you need to reach an operator, please press the star followed by the 0. As a reminder, this conference is being recorded Wednesday, February 11, 2009.

I would now like to turn the conference over to Dr. Alex Alonso. Please go ahead, sir.

Alexander Alonso: Thank you, Mohammed. I’d like to welcome everyone to the first Webinar in 2009 as part of TeamSTEPPS National Implementation Program. The topic for today’s Webinar is Designing Evaluation Systems for TeamSTEPPS. And we have a very honored speaker here today, Dr. Sandra Almeida, who will be providing us with her insights on Designing Evaluation Systems for TeamSTEPPS.

Before we begin though, I’d like to turn it over to Cori White as part of the AIR team who will tell us a little bit about Live Meeting.

Cori White: Hi. So those of you who are joining us in Live Meeting should currently see the slide up, that has the title page that should say TeamSTEPPS and the title of the Webinar that we’re doing today. The most important function aside from that for all of you is the Q and A tab at the top of your screen.

The Q and A tab is how we’re going to have you ask questions today. So if you click on the Q and A tab, that’ll give you an opportunity to type in your question to us. We will then take breaks at designated times during the program to answer questions. If it’s a question that we think can be answered privately, then we will do so. If it’s something we think everybody should know, it will be answered publicly under the Q and A tab.

Another very important part is that if you are looking for the handouts for today’s meeting, those can be found at the top right hand side of your screen. There’s an icon that looks like three little pieces of paper. And clicking on that will allow you to download the slides in PDF format, both in full slide form and as three slides per page with space for notes.

Next to the icon for downloading handouts is an option that says Feedback with a colored square. That is to let you give us an idea of how you’re feeling about the call. If you think we’re going too fast or if you’re having trouble hearing, then you can change the color that shows up next to your name in our attendee list. We ask that if you have a question, rather than turning yourself purple, you use the Q and A tab.

We will be asking two quick poll questions at the end of the Webinar through Live Meeting. And that is just sort of to help us know how we’re doing. So you will end up seeing those later. But again, the most important part of Live Meeting for all of you besides watching the slides is the Q and A tab, which is where we will manage questions for today.

Alexander Alonso: Okay. Thank you, Cori. Before we begin, I’m going give a brief overview of what we will be talking about today. First we’re going to describe who we are. And by “we” we mean the National Implementation Program Team. Then we’re going to talk about the National Implementation of TeamSTEPPS Program.

We’re going to get into the topic for today which is why measure, when to measure, how to measure, providing an overview of Kirkpatrick’s Hierarchy of training evaluation criteria, and then providing a case study with some bare data.

Then we’re going to provide opportunities for questions, although we will also provide opportunities for questions as we transition from section to section. Then we’re going to finish out the call today with a couple questions for you, followed by a reminder about contacting us if should you have any further questions and providing you with some contact information.

We represent the American Institutes for Research, which is the prime contractor for AHRQ and the Department of Defense on the National Implementation of TeamSTEPPS Program. We are not-for-profit, non-partisan, D.C.-based research firm.

We have 11 U.S. offices and 12 international offices that have staff ranging from health services researchers, nurses, physicians, social and behavioral scientists working on projects involving health, education and workforce topics like team training in the healthcare workforce or the healthcare arena.

Our overall mission is to better society through our research. The National Implementation Project was designed to create a national infrastructure to support the adoption of TeamSTEPPS, through numerous arms of the Department of Health and Human Services sponsored organizations, including Quality Improvement Organizations, and the Patient Safety Improvement Corps under AHRQ.

The goal is to make training available to early adopters of TeamSTEPPS such as the High Reliability Organizations, ACTION Partners, our partners on the Action Research Network, Academic Medical Centers and other organizations. The goal overall is to spread TeamSTEPPS and to create up to 1200 new master trainers. Most of you here today are master trainers.

The team for the program includes four team resource centers. Those include the University of Minnesota, Creighton University, Carilion Clinic, which is part of the Virginia Tech upcoming medical school, and Duke University.

We have two organizations supporting us with recruitment of QIOs. One is Lumetra. And the other is Delmarva here in the D.C. area. We also have a couple research organizations providing support for data collection and data evaluation. Obviously AIR is in the lead. gOE is known as the Group for Organizational Effectiveness based out of Albany, and Booz Allen Hamilton based in the D.C. area as well.

Our title sponsors are the Department of Health and Human Services, and specifically the Agency for Healthcare Research and Quality and the Department of Defense, and specifically the TRICARE Management Activity and the Health Care Team Coordination Program.

The AIR project team is led by Dr. David Baker, who has a dual appointment with Carilion Clinic as the Director of Quality Improvement and Quality Clinical Outcomes. I am Alex Alonso. And I’m the Deputy Project Director for Research Products.

Deborah Milne is the Deputy Project Director for Outreach User Support and all activities surrounding TeamSTEPPS outreach. Rachel Greenberg is an administrative leader on our project as is Cori White who you may have heard already on this call.

We are all part of an interchangeable team and can all answer questions regarding TeamSTEPPS or get your questions answers regarding TeamSTEPPS. This is our contact information. And I will leave this up for a second. But be aware that you will see this later on in the call. Okay. Let’s see here.

As far as the objectives here today, participants will be able to identify reasons for measuring effectiveness, describe the multi-level TeamSTEPPS evaluation model and discuss evaluation methodologies and tools. However, before I do this I want to tell you about our guest speaker here today.

Dr. Sandra Almeida is a board certified preventive medicine physician, with over 20 years experience in evidence-based population medicine. Her expertise includes the application of scientific methodologies and evidence for innovative healthcare program and technology development, evaluation and implementation.

Dr. Almeida has extensive experience in organizational safety and teamwork promotion. She served for over ten years as an active duty U.S. Naval flight surgeon and Aeromedical Safety Officer, supporting both the Navy and the Marine Corps at numerous commands.

During that time, she was one of the Marine Corps’ first air crew coordination training instructors, and the lead Human Factors and Safety Specialist for the Osprey Tilt-Rotor Aircraft system research and development team. Dr. Almeida has held a number of faculty positions including Director of Research and Thesis Affairs at the University of California, San Diego and San Diego State University’s Preventive Medicine residency.

She has been consulting for over ten years for both government and private sector organizations. Since 2003, Dr. Almeida had worked extensively with the Department of Defense Patient Safety Program, including the Center for Education and Research in Patient Safety and the Healthcare Team Coordination Program. She has assisted in various aspects of CERPS and HCTCP Program, and curricula development including instruction, implementation and evaluation.

Specific recent contributions include authorship of a primer for patient safety, which was released by the Center for Education and Research in Patient Safety in 2006. She was a leading contributing author to TeamSTEPPS which was developed by the DoD and AHRQ prior to 2006.

She also serves as a senior technical advisor on the AHRQ TeamSTEPPS National Implementation Program. We are very honored to have Sandy here today with us. Sandy is one of the people I enjoy working with the most on TeamSTEPPS because we always have creative juices flowing. So I’d like to turn it over to Sandy right now.

Sandra Almeida: Thank you, Alex. Welcome to all of you. Thank you for joining us. What we hope to do over the next hour or so is actually to make your life easier, if you can believe a scientist telling you that. What I would like to do is provide you with some tools and strategies for measuring impact of your TeamSTEPPS initiatives.

This is not always the most popular task for TeamSTEPPS change teams. But I look at it as similar to exercise. We always have a million excuses why we can’t do it, why we don’t want to do it. It’s too hard or I don’t have the time. But if you actually drag yourself out there to the gym or to your class or for a run, it feels really good once you get going.

It feels even better once it’s over. That’s how I feel about measurement. It’s once you get this train moving, it just propels itself along. The outcome of making the effort to measure is enormous for both you and your initiative and your change teams. It becomes easier and easier to do.

So what we’ll do, as Alex has said, is we’ll review the reasons why we should bother to measure, and describe the evaluation methodologies. I’m going to go over first generation of measurement instruments and methodologies, as well as second generation, qualitative versus quantitative methods, and the future.

We have some technology-based tools coming out which we think are really exciting. And we hope they’ll be available to you all in the not too distant future. Okay.

Why should we bother to measure? When to determine if the goals were achieved? And that’s the most important thing.

How do you know if your intervention actually has worked? What were you trying to improve with TeamSTEPPS? And how do you know if you got there? And with data you will establish program credibility, which will lead us to the next point. Generate buy in from your key leaders and your frontline staff, which is critical to ongoing success of your initiative.

The next reason is continuous improvement. You can identify opportunities to improve the TeamSTEPPS training program itself, your implementation processes and your sustainment plan. Finally the data itself will promote sustainment and spread, because within a facility once the data gets out that a particular unit has had significant impact on clinical process and patient outcomes.

Then the other unit starts saying, “Hey, you know, I’d like to have this, too. You know, we think we have these same sorts of problems.” There are actually groups in quality improvement, such as the Institute for Healthcare Improvement and other groups like that who actually go out on a limb.

I have to say that I’m in this camp; that unless you’re going to make a commitment to measurement, some of these larger initiatives are probably not worth your time and effort unless you’re going to try to do this. Because the chance of them sticking and really being successful is much lower if you don’t make an effort to show that it’s worth the time and money to continue the initiative.

All right. When to measure -- you’re all master TeamSTEPPS trainers. You’re very familiar with this particular slide. This is the three phases of a TeamSTEPPS initiative -- Phase 1, 2 and 3. Measurement actually occurs at all three phases. Phase 1 is site assessment.

You measure A -- to determine if your facility or unit is ready for the initiative, and B -- to identify opportunities for improvement with teamwork. Some of those measures can be established as your baseline. Those would be the numbers against which you compare post intervention measures.

Then you get into Phase 2 -- planning, training and implementation and after training is that red arrow -- testing. That’s essentially where you’re measuring: Is my initiative working? Is the intervention that I’m implementing achieving the aims that I establish?

For example, I put a checklist and a pre-op brief in the OR because I wanted to improve the quality of our pre-procedure time outs. I wanted to reduce post-op complications. Well how do you know that happened? You don’t know unless you measure. Then finally your sustainment phase, that is continuous improvement.

You can continue to look at some of the same measures to continue to see that you maintain the gains, or you continue to see improvement in your time outs and reductions in your post-op complications. You may start looking for other things as well, like there may be some other things going on with efficiency in the OR that you now want to add some other measures to, so measure all through these three phases.

All right. How do you do it? There’s the big question. Finding good Dr. Almeida -- now how do I actually do it? All right. We have this comprehensive model for training evaluation in TeamSTEPPS. And at its very core is what is called Kirkpatrick’s 4 Level Evaluation model.

Kirkpatrick has developed this model for organizations other than healthcare many, many years ago. It’s been used in many industries to evaluate the impact of organizational training. The premise behind it is that you can’t see effectiveness at the higher levels if you don’t see effectiveness at the lower levels. Let me explain this to you.

Okay. Level 1 at the very bottom in the blue is reaction. That’s essentially measuring did the participants like the training? What are they planning to do with what they learn? You see I write right next to it “like”- that’s an easy way to remember. Level 1 is liking it.

Level 2 is did they learn anything. Did they learn the skills, knowledge and attitudes that you wanted them to? And by how much did those KSAs change? All right. That’s learning. Level 3 -- now assuming they learned it, did they then apply it on the job? Did the participants change their behavior on the job based on what they learned? Did they use their TeamSTEPPS skills? And how much did they use it?

Okay. So we’ve got like, learn, use. And then finally Level 4 -- this is what everybody cares about -- what is the impact on the organization? Did we see results? Did that change in behavior at the unit or facility level result in what we want to see -- the bottom line, which is improved patient outcomes, clinical processes and/or staff morale?

Now what we mean about “you can’t progress to the next levels until you’ve been successful at the preceding level,” it’s kind of like if any of you have kids. If you’re the younger generation yourself, all those new computer-based games the Nintendos, the Wii, the Wii Fit -- you can’t go to Level 3 unless you’re successful at Level 1 and Level 2.

And the purpose behind this hierarchy, this model, is you can identify for example, say you don’t get the results you wanted at Level 4. You want to ask why. Is it because my training wasn’t any good? Or is it because something went wrong along the way?

For example, Level 1 -- if they don’t like the training, they think it’s a waste of time and they don’t ever plan to use it, probably not going to learn anything -- Level 2. If they don’t learn the skill, how can they ever apply it on the job? And if you were sitting in the classroom here right now, I’d probably ask you a question.

I would say, “Okay. So where do you think we usually fail in TeamSTEPPS?” And all of you would jump up and raise your hand and say, “Well it’s usually we don’t quite ever get to Level 3.” It’s because they learn it. They like it, but we never quite make it -- we don’t really know why.

Okay. So what we did is actually expand on Kirkpatrick’s model. This is based on continued training sciences and lessons learned in organizational change theory. We found out that even before you get to Level 1, there are individual pre-trainee experiences and attitudes that will impact the probability of them even liking the training to begin with.

For example, if you have staff members who have witnessed or been a part of a bad patient outcome because of failed team communication, they’re more likely to come into the classroom ready and motivated to learn.

Then you have the other side where, you know, you have some folks are like, “You know what? If we’re just quiet, this whole team initiative thing will go away like everything else. This is a huge waste of my time. I’m just going to nap through this anyway.” So you know what? He’s probably not going to learn anything either. So we measure this as well.

Now here’s the biggie coming up. Organizational factors -- these are huge. These predict whether everything they learned in your class actually is going to be transferred to the job -- organizational factors.

I’ll show you the list of that. That’s things like leadership support. You’ll all recognize it when you see them. We’ve added this fifth “return on investment” because everybody asks this sort of question.

So essentially what we’ve been doing over the past couple of years with a team of experts from within the DoD, AIR and other consulting groups, is developing tools and methods to measure training at each of these levels. Now these tools will become available to you all publicly through AHRQ’s TeamSTEPPS Web site as they are tested and validated.

Okay. Here are those organizational success factors about which I spoke. And these are the ones that impact whether your learned TeamSTEPPS skills actually get transferred to the work environment. And I probably could have asked you what these were without you even seeing this list. And you would have known if you’ve worked in TeamSTEPPS because you’ve heard this a thousand times.

You have to have a supportive organizational culture and learning climate. What we mean is they have to believe in patient safety and teamwork, shared vision, leadership to the front line, visible leadership support, and apparent subordinate support. Many times we have executive leadership support.

But those clinical front line leaders -- the peers who are like, “You know what? That’s not how we do things here.” And so it doesn’t matter. If you’ve got champions there for teamwork but those other -- the peers and the subordinates are like, “All right. He’s a geek with teamwork. We don’t do it that way in this shop.” So it won’t work unless you have support all along the way.

Reinforcement, rewards, recognition -- that’s just basic human behavior change. You get rewarded for good behavior, held accountable for bad behavior. Minimal delay between training and practice -- it’s important to know that. If you give them TeamSTEPPS training but you don’t actually implement these skills for months down the road, they forget them.

Ongoing training -- this has been a lesson learned in the DoD. We now have a pretty robust coaching program where we’re continually touching bases with our facilities. How are things going? Are you running into barriers? Can we help you out? Do you need a refresher and new staff training?

Commitment to measurement, ongoing improvement -- that’s why I’m here today. You need a sustainment. How are you going to integrate these TeamSTEPPS skills into your normal way of doing business, and resource availability of course.

All right. I’m going to stop here right now and take a couple of questions if we have any before I move on. I know I’ve given you a lot of information. Any questions out there?

Cori White: We’ve had two questions through the Live Meeting client. Again, if you have a question, please go to the top of your Live Meeting screen and use the tab that says Q and A. The first question was asking if -- they were having trouble with audio.

I guess if you’re already on the phone you know that. So I answered that publicly through LiveMeeting in case anyone is looking for the number.

The other question was about the slide presentation. It is available at the top right of your screen. There is an icon that looks like three little pieces of paper. And that will allow you to download the slides. Operator, do we have any questions on the phone?

Operator: There are currently no question on the phone lines.

Cori White: Okay. We have had a few come through on the Live Meeting client, Sandy.

Sandra Almeida: Okay.

Cori White: One is, “Do you have a tool to assess the organizational factors?”

Sandra Almeida: We’ll get to that.

Cori White: Okay.

Sandra Almeida: That’s coming. Okay.

Cori White: The next is, “Have there been studies demonstrating impact of medical team training on outcomes?”

Sandra Almeida: We’re getting to that, too.

Cori White: Okay. The next one is, “How do you measure return on investment? Examples.”

Sandra Almeida: Yes. We’re getting to that, too. Okay. It’s all coming. All right.

Cori White: It sounds like people have a lot of good questions.

Sandra Almeida: Yes, lots of good questions. I’m glad that you’ve raised them. If I don’t answer them adequately by the time we get to the end of the presentation, please ask again and I will expound on them. Okay?

Cori White: Okay.

Sandra Almeida: All right. Very good.

Cori White: This one might also be covered later. It says, “Can you talk a bit more about the pre-level one?”

Sandra Almeida: You’re talking about the pre-training attitudes and experiences? Yes. We will talk about those, too, as we talk about the specific instruments. But let me -- I will expound on that a little bit. There is a new tool. My colleague, Alex, will talk a little bit about the tools that we have since developed since we released the TeamSTEPPS binder.

The tools I focus on -- the measurement tools -- are the ones that are already available to you in the binder or out in the literature. But there is a pre-training questionnaire. Some of you may have seen it or used it already either as a student yourself in TeamSTEPPS or in your follow on training sessions.

We basically asked them questions like how they feel about the importance of teamwork. If they’ve had an experience where they’ve seen a bad event that they felt was because of, you know, poor teamwork or they ask them about the organizational climate -- is it a safety climate? You know, are we able to speak up when we think that there’s a safety issue going on?

But there is a tested and validated tool out there already. It’s called the pre-training questionnaire. Dr. Alonso will talk a little bit about it. I’m going to talk a little bit more about how to measure those pre-training experiences and attitudes.

But if you think about it from a practical standpoint, you all know this. How many of you been into training yourself going, “I can’t believe I have to do this again. This mandatory stuff is the biggest waste of my time.” And you, you know, you read or you snooze or you daydream.

You never get to leave Level 1 there. So that’s why it’s so important. And yet if you have personal experiences or attitudes that make the training meaningful to you, you’re much more likely to progress through those levels of training impact.

Okay. So we’re ready to move on? All right. So how can we evaluate TeamSTEPPS effectiveness now and keep it simple? I wish I could have shown you the slides I developed in my evaluation plan that started two years ago.

It was the most scientifically beautiful plan you ever wanted to see. It would take me about an hour just to explain the model. I’m now down to Like It, Learn It, Use It, Improve. So you’re all very fortunate to be in this later class.

All right. First generation measurement tools -- and I call them first generation because those are the tools that were released with the binder. These are in your binder under Tab A, which is the Change Management. So after the presentation, go pull them out and take a look at them.

All right. I’m sorry that it’s a little bit obscured. But there are actually four different measurement tools in your binder already that measure different levels from Level 1 liking it, Level 2 learning, and Level 3 do they use their TeamSTEPPS tools and strategies.

I’m going to talk about each one of these, the pros and cons and how you can use them, their limitations -- their uses and limitations. We developed these and included them in the binder to provide you with some tools to use locally for your quality and safety improvement projects, not for research.

These tools have not been tested for validity or reliability. Any of you in research know that you need that sort of stuff for measurement tools if you plan on publishing study down the road. There are no standardized user manuals. There’re no standardized scoring methods.

However, let me qualify these statements by telling you that each and every one of these tools was developed with scientific methods in mind. Some of them are actually developed based on other tools that were used in other settings or for other purposes that were validated and tested.

So these can be very valuable tools to you. You can customize them and use them the way you might need them at your facility. So I suggest that the people you train do the same.

So let’s look at them. Okay. Okay. I’m going to go through each and every one of them. This one is the Course Evaluation Form. This is Tab A, Appendix B in your binder. This measures Level 1 -- do they like it and did they learn it?

Oh, and by the way I have to tell you it’s good news and bad news about these levels -- Level 1, Level 2, Level 3, Level 4. It is true that you can’t advance to the next level unless you’re successful at the preceding level. What is not true is that just being effective at a lower level does not mean that you are successful at the higher levels.

For example, just because they like your training and they learned it does not necessarily mean that they’ll transfer it to the job or that you’ll have impact. There are lots of reasons for it, one of them being those organizational success factors.

Unfortunately, most organizations stop at this level -- Level 1. They do this cost evaluation form. This is similar to the ones that you see out there whenever you take a course for CMEs or CEUs. Did you like it? Was it well organized? Did you learn the objectives? This has very low association with whether they actually use it on the job.

All the studies have shown there are many other factors besides liking. You’ve got to like it in order to learn it and move on. But that’s not the only factor that determines whether or not this actually has impact in your organization. So it’s not enough just to do this -- do the course evaluation form.

But anyway this evaluation form in your binder -- it’s going to provide information on what the trainees thought of the instruction and whether they learned the training objectives. So there’s some Level 1 did they like it, and Level 2 did they learn it. And in the yellow box there’s some sample questions, you know, the typical.

To what extent was the speaker knowledgeable, organized and effective? You know, did you like it? Number two -- to what extent did you achieve the objectives? Okay. Did you find it productive -- blah, blah, blah -- all that stuff. You can use these for CME and CEUs. That’s how they were originally developed by us for that purpose actually.

And by the way, probably more what we have found is that if you really want to be a very popular instructor, you bring food to your class because those studies have actually shown a very strong association between the quality of the food that you provide, and your trainees’ responses to how well they liked your training. It’s got nothing to do with your knowledge base but how good the treats were that you brought.

So all right, your next one -- your learning benchmarks. This is Appendix F under Tab A. This is like a little test. This is a little knowledge test. The 23 item multiple choice test that was written by some of our HCTCP team. And it measures your knowledge of teamwork -- Level 2 did you learn it. And the only limitations -- these items do tend to be a bit too easy.

Even people who have never taken TeamSTEPPS can tend to figure out how they should answer these questions. But it still can be useful. You can either do it pre- and post-TeamSTEPPS training or just give it after TeamSTEPPS training and say, “Hey, did they actually get the knowledge that we wanted them to get with this training?”

All right. The next one is the Team Assessment Questionnaire. This is actually was developed using very rigorous scientific methods for a different organization and environment. Then we were actually given permission to use it. This is a 55 item questionnaire that staffs complete. It measures their perception of the quality of teamwork that they experience in their workspaces.

For example, if you look at the questionnaire on the right and it’s broken down by teamwork, principles and steps of competencies for example team function. The first question is the team has a clear vision of what it’s supposed to do. Then you have ‘strongly agree’ all the way down to ‘strongly disagree’.

You can use this in several different ways. You can use it as part of your site assessment to help you identify areas in which to hone in on. For example, that very first question -- the team has a clear vision of what it’s supposed to do -- get a bunch of bad marks.

You didn’t quite show your leadership. Look. There’s confusion amongst the team about what they’re supposed to do. And that could be, “Hey, you know, we might want to do that TeamSTEPPS intervention -- then everybody know the plan.” The “know the plan interventions”, right, with the morning team brief. You know, what are we going to do today? Who’s going to do what?

You can do this as your site assessment and then administer this questionnaire again three months, six months, nine months down the road whenever you’re ready to do it. Now you have a pre- and a post- measure of team behavior and team functioning based on staff perception.

So this can be a very useful tool. You can score this. You can come up with a scoring method. Give them, you know, one point for strongly agree and, you know, all the way up to four or vice versa for strongly disagree.

The limitation to this is it’s not directly aligned with TeamSTEPPS. It wasn’t built around TeamSTEPPS. But it’s built around major principles of team performance. So it can be used. But sometimes it’s a little bit difficult to translate directly.

Okay. The next one -- the final one, is the Team Performance Observation Tool. This is Appendix C, Tab A. This is an instrument that you would use when you have an independent observer who’s directly observing a medical team and rating their performance.

This is nice because it’s very specific to TeamSTEPPS. It looks at TeamSTEPPS’ specific tools and strategies. This was actually developed from another tool that was used and validated in a MedTeams project. Then we made it specific -- not I specifically, other groups before me, made it specific to TeamSTEPPS.

It has a rating system up in the upper right corner. The way I’d recommend you use this, is you see you’ll have things like team structure and leadership. These are competency sets. So you can give them a rating score of one from very poor to five excellent. Then the maximum score for each of these is somewhere around 20 to 25. You give them a rating for each.

Your team structure rating was 20 out of 25. Your leadership rating was 15 out of 25. So you can use this either during a site assessment looking for opportunities for improvement, and you can also use it again post-intervention to see have things gotten better in what we observe in team and team performance.

You just look at the scores pre- and post-. A limitation to this is it does require some training and practice. But what we found, even in our training sessions, is it doesn’t take a lot. We have our students take a look at these tools and score some video vignettes from TeamSTEPPS. We find out that they actually get pretty close, you know, the inter rater reliability sort of subjectively seems pretty good.

Okay. Questions on this portion?

Cori White: We have a few administrative questions that I think I’d like to cover first. One is again, you should be able to download the slides by clicking on the icon at the top right of your screens. It looks like three pieces of paper. However, I’ve been told some people are having trouble with that.

Sometimes that’s a firewall issue. If you are having trouble, you may send an e-mail to teamsteppswebinars@air.org. I’m sitting at my computer now. If I see it come through, I will send you out copies of the slides as soon as I see your e mail. But please try to download them yourself first.

Another thing that is going on administratively is if you have a question when you go to the Q and A tab, please type it in so that we can see it instead of just using the little hand icon. Because if you don’t type it in, we don’t know what your question is. And now we have a content question for you...

Alexander Alonso: Cori, before you do that though, I do want to say that for those of you that are concerned whether or not the slides will be made available after the presentation, if you contact us at teamsteppswebinars@air.org, we will e-mail them to you. Also, Cori, correct me if I’m wrong but do we publish these on the TeamSTEPPS Web site?

Cori White: The presentations themselves are not published. We will put a transcript online on the TeamSTEPPS Web site, the slideshows that go with each of those Webinars are available by request from teamsteppswebinars@air.org.

Alexander Alonso: Thank you.

Cori White: Okay. Now we have some content -- or this one might be slightly administrative as well. It’s we have someone who says that their binder does not contain these tools. Are they on the CD?

Sandra Almeida: They should be, yes. They should be on your CD.

Cori White: Okay. Another one says, “Is there a training guide for using the team performance tool that clarifies and/or provides anchors for scoring?”

Sandra Almeida: No. That’s one of the limitations of these first generation tools is they don’t have user manuals. And we don’t have benchmarks or standardized scoring methodologies. These can be customized locally. Those are the sorts of things that will come with the second generation of tools that will become available through the Web site.

But if you actually take a look at these tools, pull them out and look at them you can see that they wouldn’t be hard, you know, to come up with a scoring because -- a scoring system for them. Also, when you start doing benchmarks, you get into kind of some problems because you have to compare apples to apples.

You know, some of your facilities are really small and you’re doing different types of initiatives. And then you have larger facilities doing a different TeamSTEPPS initiative. So benchmarking hasn’t really occurred yet. Again, these tools have not been tested for validity and reliability.

We hesitate to give a benchmark on a tool that’s really not scientifically rigorous yet. So the answer is no. But we do hope to provide those types of tools and manuals down the road, with the more rigorous measurements that are coming sequentially over time. Okay? Any more or are we good to move on?

Cori White: It looks like we’re good to move on for now.

Sandra Almeida: Okay.

Alexander Alonso: Check and see with the operator. Do we have any questions?

Operator: Ladies and gentlemen, as a reminder to register for a question, please press the 1 followed by the 4 on your telephone.

Alexander Alonso: Do we have any questions at this time, (Mohammed)?

Operator: Sir, we have no questions at this time.

Cori White: Okay.

Alexander Alonso: Okay. Thank you.

Sandra Almeida: All right. Now let’s move on. Okay. What are some other options for Level 3 that are available to you now? Level 3 again is looking at are they using your TeamSTEPPS tools and strategies on the job? Are they implementing them? Or if you want to be really quantitative, you go look into the literature.

There are several teamwork observation tools out there, some of which have been tested for validity and reliability within certain clinical settings. I’ve listed a few of them. There are CATS and there are ANTS and there’s OTAS. Again, what they’ve found is that they use them for specific clinical settings. Some of these are quite good. You can use them if you want to.

The second one -- and this is the one I’m always asked -- is can we do it in a simple way that doesn’t require a lot of training and practice? The answer is yes. These are qualitative methods. What we’ve discovered is that we’ve actually gotten some very compelling information this way and need qualitative just are subjective reports.

What we’ve had is staff report staff. And the facilities, our change teams, when we call them on coaching calls, they just tell us. We ask them, “So how’s the implementation going?” We started getting things like, “You know what? We’re using the morning, brief, the whiteboard rounds, you know, about half the time.”

So we started writing this down -- half the time. All right. “Well have you had any impact that you’ve seen because of this?” They tell us these stories. We call them stories -- success stories, and we started documenting these.

After a while we started asking them to become a little more quantitative and get a little more quantitative with rates. You can say, “Well about how often are you using those whiteboard rounds in the morning in ED?” “Well now, you know, Doc, we’re up to about 50%,” or “We’re up to 75%.” And these are not scientifically rigorous. But it’s a start.

What we’ve found is that once we start feeding this information back to them, they want to become a little more quantitative because they start to see improvement. It’s that thing about once you run that first half mile, you feel better and you want to run a little bit further most of the time. If you were actually in my class right now, I might have you actually stand up and do some jumping jacks.

Okay. Here we go. Now how to measure Level 4 -- now how else can we do Level 4 which is the result? Did they impact clinical processes and patient outcomes? Well the bad news is there’s probably never going to be a standardized tool and measure that will fit all clinical settings and all TeamSTEPPS initiatives, although we’ve looked for this.

That’s a very hard do. The reason for it is Bullet Number 2. It’s because you have to directly align these results with your initiative aims. What were you trying to fix to begin with? What are you hoping this initiative will do for you? And how will you know you got there?

How can you measure it to say, “Hey, you know what? I actually achieved what I was trying to do. I was actually trying to reduce post-op complications. I was actually trying to improve communication in my emergency department.” Well how do you know that you’ve improved communication?

Well you might have an adverse event and a near miss reporting system in the ED. And if it’s granular enough, it will tell you that, “Well this near miss was because of an information flow problem or a communication problem.” Well you always start tracking those over time, looking at how many near misses and adverse events do you have in the ED that were related to poor communication? Those should start going down over time.

It is very best to use measures that already exist. Or if you’re going to implement something new for TeamSTEPPS, it’s got to be very simple and low effort. Don’t let them try to tell you, “Oh we’ll do design our own survey.” Bad idea. It’s just too hard to do. And most clinical clinics are too busy to begin with. You can’t add more work for them to do.

Another option is to start looking in the literature for measures that have been shown to improve with teamwork intervention. There are plenty out there now in surgery especially where they’ve looked at the pre-operative brief and the checklist. And they start looking at some outcomes.

And there are things like they’ve improved some of these ORYX measures, the prophylactic antibiotic. They’ve improved prophylactic venous thromboembolism medications. They’ve improved post-operative complications, you know, aggregate. So there are some measures that have been shown to improve already that are already documented in the literature.

All right. Level 4 again -- and here are some examples of Level 4s that you can use quantitative measures. Existing measures again are Joint Commission ORYX. There’s the AHRQ Hospital Survey on Patient Safety Culture. You’ve probably listened in on some Webinars on how you can use that.

We’ve already mentioned the near miss and adverse event databases, staff satisfaction survey, local clinical measures. I’ll tell you one caveat to all of this is when you look especially at something like Joint Commission ORYX measures, things that are measured tend to improve period because now people are paying attention to them.

ORYX has been around for a while. So if your facility is already up around, you know, 95% with each of these measures, your chance of improving them further with TeamSTEPPS are not very high. You’ve got to take measures where there’s room for improvement. So keep that in mind.

We’ve had that problem with staff satisfaction surveys where we’ve had some facilities go through some great lengths to develop some local staff satisfaction. Well their baseline is their patients are already happy. So you get tiny little improvements with TeamSTEPPS.

But it doesn’t show great impact. So you have to be really careful that your baseline’s not too high. And again, simple, simple, simple -- you’ll hear me say this over and over again -- simple measures.

All right. Qualitative – here are some qualitative simple things to do. Again, staff reports of what we’re calling good catches and glitch capture improved efficiency, improved staff morale. And you’re on those coaching calls. You’re just going in periodically once a month and saying, “Tell us your success stories. Do you think anything good’s happening because of TeamSTEPPS?”

And they’ll say, “You know what? We had a corpsman -- a tech speak up because he was empowered by TeamSTEPPS. He thought something might be going wrong with this particular patient and said, ‘You know, I think something’s not right with this patient.’

And sure enough, there was something going on with the patient. There was medical intervention that occurred. The patient’s outcome was likely maybe even a life saved because of it. That’s a success story. That’s essentially a good catch is a near miss that was caught because of TeamSTEPPS.

Glitch capture -- these are systems-based problems that occur repetitively that because now you’re doing debriefs or you’re doing teamwork, communication, information and so on, you identify, “You know what? We have the same problem every, you know, once a month we see this same thing happening that the labs are not getting back to the ED long enough. This is why our patients are always here.”

So now somebody goes and talks to the lab. We have this collaborative relationship going on. They solve the problem. That’s glitch capture. And these are just simple reports coming back. You can get more quantitative with glitch captures and good catches just by doing rates.

For example, the percent of your operating room cases where they have a good catch. Let me give you an example. This is the World Health Organization Surgical Safety Checklist. Many of you have probably seen other types of OR checklists in place. But it basically -- here’s the sign in. The checklist you go down and make sure the patient has everything done that should be done.

For example, are there any allergies? Is there likely to be a difficult airway aspiration risk? And the middle is the time out. This is the pre-procedure time out that you go on verifying that you’ve got the right patient, the right procedure, the right site. Then the sign out, you know, basically have we done a count for all the equipment? We’re not leaving anything in the patient. You know what went well, what didn’t go well.

Well some of our facilities are now adding this little added box here with each of these -- a good catch. For example, did we identify that somebody on the team knew there was an allergy, but most of the team members didn’t? And we stopped giving any antibiotics because we discovered this patient’s penicillin allergic, and we gave something else. That’s a good catch.

Waiting time events -- these essentially are a type of glitch capture. These are things that occur in the OR that delay the case progress. For example, you don’t have the right equipment. You don’t have the right labs. You don’t have the right imaging studies. You don’t have the surgeon -- whatever. But you write them down.

And then over time, you start saying, “Well, you know, we’ve had this many types of good catches.” And you present it to your leadership. And over time, you start seeing, “We could have really have averted some really bad things from happening because of this. And now we’ve solved some of these waiting time events, too.”

Let me just show you -- these are actually reports that have come out of our military facilities. These are just these qualitative subjective reports that have occurred in various settings including the OR as well as some non-surgical sites. These are things they’ve told us.

When you have case after case after case, report after report after report like this and we present it to our leadership, these become very powerful. We’ve had staff saying they have a clearer direction of the patient plans because they’re using whiteboards, decreased patient harm incident reports -- glitch capture, meaning they’ve identified that some members of the team didn’t have the right knowledge or the training to be doing what they’re doing.

So now the staff goes on to teach them or to explain to them what should happen in this particular case. Equipment gaps and problems -- I’ve identified ongoing repetitive equipment problems that they’ve now corrected. Increased staff and patient satisfaction reduced nursing report time -- you can look at this. These are huge.

Everything from efficiency to safety to staff morale has been reported back. Now we start showing this and we feed this back to the facilities. They get a little more excited about, “Wow. We’re doing this.” Now they actually want to measure more quantitatively.

All right. Here’s the answer to the question that was raised about how do we measure these organizational success factors. Remember these are factors that can act as barriers or facilitators to transferring your training to the job meaning, you know, they’ve learned it. They loved your training. They learned all the skills. Did they actually get it and use it on the job?

In the blue box is the list of those factors again. I’m not going to read them because you know what they are. In the yellow box there are some measurement options. We’ll tell you we haven’t developed that standardized tool yet, although we are in the process of doing that in the Department of Defense. And we’re mostly doing it through our coaching process.

Again, our coaches are our master trainers who call about once a month or on a periodic basis back to the facilities. “So how are you doing?” We’re actually developing standardized tracking forms for this. But right now we’re measuring these again through those qualitative reports at this point, asking, “So what has worked for you? What’s been a barrier? What seems to facilitate making this happen? What seems to stand in the way?”

When you ask those questions, you get some very compelling responses. Now if you want to try to be quantitative, the things that are out there right now -- the HSOPS. That’s AHRQ’s Hospital Survey on Patient Safety Culture. But again, those are sort of things that change over the long term. But you can see some of these organizational factors are definitely measured through the HSOPS.

The Team Assessment Questionnaire that’s in your binder -- this will measure several of these as well. You know, when you ask -- and then the Team Assessment Questionnaire is the one that’s filled out by the staff. And they tell you their perceptions of how well they are working as a team in their work unit.

They ask them a lot of these questions like, are your leadership supportive? Are your peers supportive, you know? They ask lots of them. So you can get a feel if you do this at baseline if some of these organizational success factors are in place. Then you can do it again nine months down the road and re-look at it.

So we do have some quantitative measures. But it’s not perfect yet. We are looking at other ways to develop this and take a look at it. But let me show you what we’ve discovered in the Department of Defense through our qualitative coaching reports.

Again, just asking the facilities what’s a barrier to TeamSTEPPS down there? What’s a success factor? No big surprise what we’re seeing. If you look at your barriers, for us in the DoD we have lots of problems with staff turnover and deployment. It’s a huge problem for us. Just as people come on board, they leave.

This very consistent with the literature -- a lack of visible leadership support, lack of frontline staff support, and bad actors -- no accountability system and limited time. Then all the success factors -- these align directly with what all the literature studies have shown what organizational change theory shows.

So our qualitative reports are producing some very compelling and results that are consistent with everything we would expect to find. As we start again reporting these back to facilities and to our trainers and to our leadership, it’s like, “Wow. You know, now we’ve got a place to start.” These are sorts of things that where we can start looking at measuring more quantitatively.

So in summary -- got it all on one page for you -- what are the measure options available now? On your left hand side, these are the levels. This is your evaluation model. These are your options. Now I will tell you -- you don’t need to do all of this. As a matter of fact, most places don’t.

We don’t do it all and the DoD at every single site. But you can decide what you want to take a look at and pick and choose from here. Now down here I’ll tell you at this very bottom, this is the Teamwork Attitudes Questionnaire. This is a new tool. Dr. Alonso will talk a little bit about this.

It’s about to hit the streets via the AHRQ TeamSTEPPS Web site. This measures baseline staff attitudes about teamwork. How important is it to you? Would you think it’s valuable? Do you think it will make a difference in patient safety?

So you can use it to measure their baseline attitude before they ever go into training. Then you can use it again as, you know, down the road did their attitude change after training? It’s a learning -- an L2 level measure. You can measure it again six-nine months down the road as an outcome, sort of a cultural outcome.

You know, are we changing the attitude here, the culture here, the teamwork culture? So that’s pretty exciting. That one’s been tested for reliability and validity. So that will be very useful to you we hope. And there’s a user manual with it and a scoring method. So you should be seeing this soon. And here you go. I’ve listed them all for you -- all your different options. And we’ve discussed each and every one of them. That’s one page for you.

Okay. Questions? Any more questions for me? I have a few more things to cover -- not much.

Cori White: Okay. Again, I would like to remind you that if you have questions, we would prefer that you go through Live Meeting and use the Q and A tab at the top of your screen to type in your question. If you don’t type in a whole question, then we won’t be able to answer it. If you are listening only on the phone, you’re able to press 1 and then 4. And the operator will handle your questions.

We have a question here that the participant says, “We are using various forms of trigger tools to measure Level 4 which are fairly objective measures of adverse events.” They’re wondering about your opinion regarding this as one measure of improved teamwork.

Sandra Almeida: Oh yes. No. I think there are lots of groups that use these standardized trigger tools. You may be referring to one of them that came out of IHI. Absolutely. Measuring, you know, indicators for evolving adverse events. Absolutely. It’s very legitimate -- again, especially if it’s linked directly to the aim of your initiative.

That’s always the key question is what are you trying to achieve with your TeamSTEPPS initiative? Have you designed the TeamSTEPPS intervention -- meaning the tools that you’re using and the setting you’re putting it in -- where it should impact this.

So I would say yes, as long as it’s directly in line to your aim. Almost all TeamSTEPPS initiative reducing adverse events of course is the primary aim. So that’s a very good point that there are those trigger tools methodologies out there. They’re quite standardized. That’s very good.

Cori White: All right. The next question is, “When you have multiple changes occurring in an organization...”

Sandra Almeida: Yes.

Cori White: “...how can you specifically attribute the outcomes to TeamSTEPPS, specifically the outcomes patient experience and quality?”

Sandra Almeida: We take all the credit for all good outcomes. Next -- no I’m kidding. That’s actually a criticism that’s come out of all of these safety and quality improvement initiatives is how do you really know? The truth is that unless you’re doing a really rigorously designed study, you can’t really know.

But if you read things like the, you know, Institutes of Healthcare Improvements 5 Million Lives campaign, 100,000 Lives campaign, you’ll discover that they address those issues and that you’re right. It’s probably a combination of initiatives that are having the impact. It is difficult to tease out which one is really making the difference. But the longer you do this -- and this is a whole research question.

There are ways of teasing it out. But it’s not something that most of you all are going to do at your level. But if you continue to show improvement over time as -- and you continue to show that you’re implementing, and you’re spreading and implementing TeamSTEPPS more and more and more over this time that you’re continuing to see improvement, you have more confidence that it’s likely related to TeamSTEPPS.

Also, there are ways of doing it. It’s called time theory. If you look at a facility had teamwork and then the champion left, this happens to us in DoD all the time. So the initiative kind of went south for a while. You may find that those measures go south, too. Then when you re-energize the initiative, you have a new champion. Those measures improve again. That’s called time theory. That’s actually pretty compelling.

But you’re right. That’s an issue that always comes up. But we say, “Everybody else takes credit. We’re going to take credit, too.” But there are ways to sort of look at it. It’s probably a combination of things.

Very good question. Next one?

Cori White: The next question is, “When do you recommend to use the Teamwork Attitudes Questionnaire for pre-training and post-training? What timeframe?”

Sandra Almeida: At what timeframe? Depends on what you’re trying to measure. And I’ve actually talked to one of the major developers, Dr. Baker from AIR. If you’re just trying to measure if the training itself changed their attitudes, you can administer the baseline immediately before they start training. They’re all in a classroom -- captive audience. Give them the Attitudes Questionnaire.

Then you administer it immediately after you conclude training. So now you know the question was, “Did my training itself change their attitudes?” Now if you want to look at it more globally, does the actual practice and behavior of TeamSTEPPS change their attitudes about its value? Sometimes you see this behavior actually changes the way they think.

So you actually go back to the staff maybe six or eight months or whatever point where you feel like TeamSTEPPS has been implemented fairly well. There are lots of people using it. They’re kind of excited about using it. You measure it again then.

It may be nine months, six months or whatever because we sometimes find that okay, now we’re using it. And we’re seeing the value. And we’re seeing these success stories in these reports. So you can use it multiple times if you want too.

It just depends -- again you can do it at baseline before training, immediately following training and then nine months later or six months later and then a year later to kind of get a pulse recheck. How we doing with that sort of our team work attitudes, that sort of culture of the importance of teamwork.

So you can use it at multiple times. Actually using it over time is actually a very good thing to do because you can continue to show the impact of your initiative as well as if the initiative starting to fall off. It’s like Okay no we’re starting to see this drop in the attitude about teamwork, we better do something about it. That’s all part of the sustainment and continuous improvement, that’s the purpose of continuing these measures.

So you can use it multiple times, depending on what you’re trying to achieve. Okay? Next?

Cori White: We have a whole pile of questions about the Attitude Questionnaire.

Sandra Almeida: You know what we’re going to do, I’m going to -- let’s hold on to those because Alex is going to address those again a little more. I’m just going to finish my last couple of slides and let Alex do his piece, and if we haven’t answered those questions we’ll raise them again.

Cori White: Okay we have one other question online that’s not about the Attitude Questionnaire. Then we can go to the phone.

Sandra Almeida: Okay

Cori White: This question is -- are there available examples of the huddle SOP’s? We’re looking to write a command on unit level SOP as our facility is too small to maintain a rapid response team.

Sandra Almeida: You mean huddle? The term huddle is used in so many different ways. A huddle technically, like TeamSTEPPS is when the team gets back together very briefly because things have changed. For example in the ED all of the sudden you have this emergency case that’s about to come in, a multiple trauma case, and now the team has to re-huddle. Okay what are we going to do now for this particular case?

If you look under the sections in the TeamSTEPPS binder on what sort of information should be exchanged in a huddle it has those check lists -- the type of information that should be discussed and transferred and shared among all team members.

Now lots of people in the real world in the facilities use huddle when they really mean a morning brief -- getting everybody on the same page. Meaning that the team gets together before the day and they -- and again in that pocket guide and also in your TeamSTEPPS instruction binder there are check lists, there’s information that should be covered and information that should be shared and exchanged between the team members for the brief, the morning brief and also again for the huddle.

So, if that doesn’t answer your question, whoever asked this, send us that question via email to teamsteppswebinars@air.org and they can send it to me and I’ll get back to you. Okay? Next, anything else?

Cori White: That’s all we have on line right now. Operator do we have any questions on the phone?

Operator: There are no questions at this time then.

Sandra Almeida: Okay. I’m going to move on because I want to give my colleague time to cover his information. These are tips for measurement success and you can read these, don’t panic, this doesn’t really need to be difficult. I hope I’ve demonstrated to you some simple ways of getting started.

The first thought is very important. Align your measures with your TeamSTEPPS aims and other requirements integrated. If there’s a Joint Commission National Patient Safety goal you’re trying to achieve, then you can align your TeamSTEPPS with it -- do that. Get your leadership in front line staff involved in the measurement plan, so that you’re using measures that meaningful to everybody.

We may want to look at safety but then the surgeons like, “You know what; I never have the right equipment in my room.” So add that, add the equipment issue to your -- you know glitch capture and waiting time. The surgeons are all excited about the waiting time events to add to their check list.

Select existing measures and simple new measures -- have a plan. Who’s going to measure what, when, where and how and stick to it. Use a pre and post intervention design. You know the simplest way to do it; don’t forget to use your baseline, that’s the point of that. Look for short term wins. You’ve got to have room for improvement. We see this mistake all the time.

Do not pick a measure that’s already good, that baseline; pick one that’s lousy -- really lousy. Go in and look at your timeouts in your OR’s -- if you’re in the surgical community: Are they really doing their timeouts, team base, the way they say they’re going to do them? Feedback your results to your leaders and your front line staff so that they can start to see the successes of their efforts.

Alright the only thing I’m going to tell you about these Second Generation Measurement Tools -- these are the new things that are coming on your right hand side these are the new tools that have been developed, tested and validated. Alex will talk a little more about those.

All I’m going tell you - he’s only going to talk briefly about them, is that keep checking back at the AHRQ TeamSTEPPS Web site because as these become tested and available and we’ve gone through all the process to making them publicly available you’ll get them and you’ll see them and they’ll have user manuals with them. Mostly you can see that we’re in development, in development, in development. We’re getting there, slowly but surely.

Okay, very briefly, just to let you know some very exciting things coming down the road where we’re going to use automated technology to measure for us and we have two tools on the skyline. The Team Effectiveness Accelerator and The Medical Team Performance Assessment Tool, Alex will talk a little bit about this.

But this is the MTPAT here and it’s a tablet, an electronic tablet where essentially observers, independent observers, watch a clinical event occurring. It can be simulation based or a real clinic event and they measure and rate technical clinical performance and team work performance and it’s very cool and I’ve actually seen this action. Alex will talk more about it.

The Team Effectiveness Accelerator, this is really a debriefing tool. The intent of this is to help teams identify areas where they really need to improve so that their debriefing, the teamwork debrief is targeted towards areas where they are likely to have impact. To very quickly show you: this is how it works.

You know the team performs either a training activity or a real clinical event and then a lead person sets up a debriefing session -- alright they go over to, this is going to be Web based, and they set it up and then all of the team members, in the red box next, they answer the debriefing questions and I’ll show you some examples of that. You know a leader was established, we all knew what was supposed to happen and it can be some key clinical things like the patient didn’t have any allergies and they all put their answers in.

Then this software analyzes the team’s answers and it looks for gaps. Meaning was there some team members who thought this went really well when others who said no it really stunk I had no idea what was going on. So the leader who might think that everything went really is able to know if his team members were confused. Then the software will develop a customized debriefing guide that focuses on these areas where there were gaps in knowledge or attitude or whatever, and then that directs your brief.

These are just some of the questions that this -- TEA - the Teams Effectiveness Accelerator, the types of questions that the team members would answer. You know, the room was well prepared. This one is actually written for a resuscitation team, a trauma team. But we are actually in the process of developing kind of a generic version of this for any clinical setting and there will be some very basic evaluation of this as well.

Meaning, that if there were lots of gaps in knowledge and attitude amongst the team members they’re going to get kind of like a red score. Meaning lots of room for improvements. If there is very little gap, everybody felt like they really knew what they were doing, they really knew the plan then they’ll get like a green score in the end. But, this will be very nice and these are the sorts of technologies that we’re developing to make this evaluation simpler and more automated.

And that is the end of my session and I am going to turn this presentation over to my colleague Dr. Alonso.

Alexander Alonso: Okay thank you Sandy.

What I’m going to do is jump into the next slide and take a little bit about the Second Generation Measures and Tools. And in the interest of time I’m going to talk a little bit, briefly, about the ones that we have listed here as well as the Automated Performance Measurement Tool, the MTPAT. But then jump into our case study which should not take me very long.

I’m going to jump over to a different presentation now and I’m going to show you information about the pre-training measure that includes information about the team attitude questionnaires. So this is actually the Team Attitude Questionnaire, before we called it that. It really looks at three basic subscales.

The first was motivation to learn. The next was safety culture and then there was belief in the importance of teamwork. Now the key for these is these are critical prerequisites to successful training. Research has shown that participants who are motivated to learn and have a collective orientation will benefit from team training. Whereas, those that do not, will not benefit from it.

Another critical component is that a positive environment must be set for the team training to transfer. Let’s see here, so some of the questions that have come up include when to administer and we talked about it being at the beginning of training. But it can also be done several weeks before training as part of a site assessment, if you will. It really hinges upon what Sandy said, which was that, “What it is that you intend to get out of the information you collect.”

What questions does it really answer? Well one to answer is whether trainees are motivated to learn. It also answers whether or not they value team work and whether or not the culture supports safety. I would use these results and recommend using these results only to explain reactions to training as well as explaining differences across different implementations of TeamSTEPPS.

Here are some of the sample items that you’ll see by dimensions. So we talked about motivation to learn and you’ll see that one of the items might say, have you ever worked in a medical team where you did not feel comfortable voicing your professional opinion? Many of the items are similar to this. You’ll also see other items for in teamwork that deal more with the fact that teamwork deserves more attention in health care. And these are treated along the Likert Scale.

The other items regarding safety culture might tap into a concept such as disruption or discontinuity in patient care, for example shift changes, patient transfer and how they can be detrimental to patient safety. What you’ll note is that when you score these you are really scoring them on a 0 to 5 scale for your motivation to learn items and you’ll go ahead and you’ll answer these and -- add up all the ones that would be yes answers. For the beliefs in teams you would want to add up all the ratings and they could range from 5 to 25. For your safety culture you would range from 5 to 20, by adding up all the ratings.

This is basically the context within which the Team Attitude Questionnaire or the Team Assessment Questionnaire was developed for training for TeamSTEPPS. Many of the questions that have kind of come through here deal with whether or not we can notify folks when it’s been released. We will gladly do that, we will post it on the Web site and we will notify the listserv of master trainers. So please know that it will be coming out in 2009, but it is not necessarily ready at this point.

One of the questions that has come up is whether or not the measure is validated. What we can tell you is yes, that it is validated; we have done the basic normative data base, created a normative data base and validated it at two different locations. So it is something that is currently still under development, but has been shown to have reliability and validity so far.

Let’s see here, I’m going to go ahead and switch back to our other presentation. Okay when we talk about the Level 1 reactions and we’re talking about post training questionnaires I’m going to go ahead and go back to the other presentation that I had.

We’re talking about a second generation of post training measures that deals specifically with not only effective reactions to the training but also instrumentality and utility of the tools that you learn on the job. This is kind of different, or an extension of what was used in the initial post training measure or the post training course evaluation for TeamSTEPPS.

These are critical outcomes of successful training, specifically positive effective reactions leave staff having good things to say about the training and set future expectations for their behavior or their use of the behaviors. Instrumentality is correlated with training transfer more than learning itself. The reliability for this type of measure is usually around .92.

The typical place to administer the post training measure would be immediately after training, although for DoD pre and post training measures they are usually copied on the back of and the front of a single page and done immediately after training -- both before and after training.

What questions does this answer? Well specifically, do they find the training useful and can they apply it on the job? Do they like the training, which is usually the ground floor of training evaluation.

So I would use this as a proxy for transfer of training, which is really what we’re looking for with TeamSTEPPS. Can you transfer the behaviors that you are learning in the training itself? Now having said that, is it the only way that I would measure transfer of training? No, I would also use behavior observation such as the TeamSTEPPS performance observation pool. It is an indication about what will be said about training though, in the actual workplace which is critical for the success of any TeamSTEPPS base initiative.

Okay, let’s see here. So moving back, we talked about the revised course evaluation form, we’ve talked about the TeamSTEPPS attitude questionnaire a little bit and I apologize that we do not have more information on it. We will make more information available to you as soon as we have it. The key thing with this one is to remember that it assesses attitudes towards TeamSTEPPS tools and strategies but it also takes into account safety culture and motivation to learn and believes about teamwork in general.

Here are some of the sample items from the T-TAQ for you to go ahead and keep in mind, in the yellow box. Now, another tool that is kind of under development is the Teamwork Perception Questionnaire and it is similar to the TeamSTEPPS Assessment Questionnaire but it is aligned with TeamSTEPPS specifically and aligned specifically to the TeamSTEPPS curriculum.

It will measure staff perceptions of organizationally culture, team structure, leadership situation monitoring and communication. Okay, one thing that is not there is that it will measure some perceptions of mutual support.

Another thing that is currently under development is the Teamwork Direct Observation Tool. Sandy I don’t know if you would like to speak to this particular tool, but my knowledge of this one is fairly limited.

Sandra Almeida: Yes this is essentially we’ve had discussions about basically modifying what we already have in the binder which is the Team Performance Observation Tool. That’s the one again where an independent observer watches a team and rates them on how well they’re doing with these particular activities. We’re still in an early strategizing phase with that on the measurement team, you know how to make this TeamSTEPPS specific but still you know as scientifically valid as we can and that’s a discussion item on the next measurement team meeting in March.

Alexander Alonso: Sure. I think it is important to note that these particular measures that we’re talking about really deal with the observation of behavior and or the attitudes towards behavior or perceptions of behavior.

One other tool that Sandy brought up was the MTPAT which is the automated tool for observing team behaviors. You’ll notice that this window here doesn’t do it justice because it is so small on your screen. But specifically what it does is it looks at trigger events that should cause or clinical events that should lead to expected team behaviors which you will see all the way on the right hand side. Then the rater can go ahead and rate what exactly was seen or observed as far as the expected team behaviors.

In the bottom hand portion of the screen you will also see that there are tools for writing in specific notes. This typically comes in the form of a tablet where you can write in your specific notes about specific events, specific team behavior, specific trigger events and also write in general notes about the overall team episode or event log. Let’s see here, and that is also under development at this time. In the interest of time, I am not going to, I’m going to skip over the questions and take them at the very end so that we go ahead and get in the example of what TeamSTEPPS evaluation looks like in practice.

Specifically we have a case study that was provided to us by an emergency department that wishes to remain anonymous. It is a 20 bed emergency department, with 5 trauma bays. It has about 80 staff nurses, 22 physicians and 40 support staff. Its history is one of punitive culture and this was measured using the AHRQ HSOPS survey.

Specifically their goal was to target handoffs and really handoffs to other units because other units kind of picked on this specific ED for their inability to hand off critical information.

They did experience an event involving the transfer of a patient from the ED to the ICU and specifically the event was about a stabilized trauma patient who had third degree burns and had been given narcotics. But the transfer, during the transfer the ICU staff didn’t receive any information about the narcotics that were given to the individual. And as such, a mistake in medication was administered to this patient. Now the patient was fine, but none the less it was an event.

Clearly there was poor hand off all the way around. The ED did not have a standard way of communicating vital medication information to other units or support units. And specifically the group or the hospital administration asked, “Why are nurses not involved in the transfer as a standard practice?”

Specifically the nurses who are part of the patients care. They decided to make that standard protocol and went ahead and used SBAR based handoffs and conducted TeamSTEPPS training focusing on handoffs and SBAR. They instituted a handoff check list.

Now, their overall plan was to collect reactions tools, using the post training measures. They also did post training quizzes on a bi-weekly interval about SBAR communication and hand off check list use. What they had was a coach who would go ahead and hand out a basic scenario and ask them to provide a SBAR communication and it was a quick quiz that was given to the staff. And it actually became a fun activity for the staff.

Level 3 they conducted behavior observation using a hand off check list usage. Specifically, they used two internal staff members to observe on a monthly interval. They also went ahead and used the TeamSTEPPS performance observation tool for this.

The goal for Level 4 was really a reduction in time spent working up patient history in ICU transfers. However, this is really more a process than an outcome measure. That’s one thing to keep in mind. That’s really not an outcome measure per se.

As far as the results at Level 3, they used TeamSTEPPS performance observations tools as I mentioned earlier. When they looked at the frequency of hand off check list completed per transitions in care, they noted that it went from .57, so about half the time when there was a patient transition in care the folks were completing a hand off check list or some form of check list.

Whereas, when they went ahead and instituted the check list, every patient received one. Finally they also had about 83% of all verbal hand off to the ICU perform post training met the basic standard of including SBAR information.

As far as their Level 4 process measures. They reported time spent working up a patients’ medication history decreased considerably. As far as all transfers from ED to ICU about 48% of them required a complete work up medication history, prior to TeamSTEPPS, after training 13% required them a complete work up.

As far as time spent on medication work up, per case they went from 32 minutes per patient from the ED to 7 minutes per patients from the ED following -- so they had certainly quite a bit of success.

At this point I want to open the floor for questions and then I want to leave room in the last two minutes to conduct our polls and our own evaluation of this Webinar.

Cori White: Alright, we have someone who had a suggestion for the person who asked about huddle SOPs. This person suggests that if this question was from a military institution that you can forward the respective PS Rep (Air Force, Army or Navy) and they can further expound for you. So that question is one that you should pass on to someone in your institution.

The next question is: We have not had TeamSTEPPS training yet, how do we find out when and where the training is scheduled? If you go to our Web site which is www.ahrq.gov/teamstepps there is a lot of information about TeamSTEPPS training there and that is the best place to start if you want to find out more about receiving TeamSTEPPS training, again that’s www.ahrq.gov/teamstepps.

We have a comment from someone that says in measuring results we are getting much more traction in looking at outcomes rather than process measures. So for timeouts it’s much more engaging to clinicians who look at number of wrong site surgeries or near misses for wrong site surgeries than simply looking at how well the timeouts were performed.

Sandra Almeida: Though that’s absolutely true, that and from a clinical standpoint the stakeholders the clinicians they absolutely are more interested in the outcomes, the patient outcomes. The problem that makes that difficult sometimes and I don’t discourage you from trying to find those numbers because those are the ones that matter, is that hopefully you don’t have many wrong site surgeries in your institution.

So, that rule of trying to find a measure that can show change over the short term, if you just measuring something hard like that, you know wrong site surgery, it’s difficult to use that unless you have multiple facilities involved over a long period of time.

So what we try to find is marker measures for that. That’s the whole purpose of the timeout. Theoretically if you do the timeout properly you should prevent that. There is a survey in the literature, it’s called the Orbat, the O R B A T there’s an article that was written where they actually measured a marker of that and they asked the staff, the surgical team, you know pre and post the teamwork intervention, do you feel more confident about the right side surgery?

Do you think you’re less likely to make an error like that now that you’ve had teamwork? Again these markers they are not exactly the best, but you’re correct, clinical patient outcomes in searching for those measures and it’s a good thing to do, just make sure that you’ve got room for improvement.

Cori White: Right the next question...

Sandra Almeida: Very good point though.

Cori White: Thank you. The next question is when will MTPAT be available?

Alexander Alonso: I can answer that Sandy. It is still currently under development under a project with the Department of Defense and as such it’s way way out in the process. It is being piloted at a couple of facilities both military and civilian. And what I would tell you is that you should look for it sometime in the future, perhaps beyond 2010 I would say. Sandy I don’t know if you have any indications as far as that...

Sandra Almeida: Yes I think that that’s probably correct. We are trying to develop a simpler version of the MTPAT that would be generic. Because right now they are specific to trauma and labor and delivery. But we are trying to make them simpler for all clinic settings and sort of a broad teamwork behavior, but you know probably 2010 at the very earliest.

Alexander Alonso: Yes. Cori I see one question that asks what perceived limitations of the AHRQ HSOPS survey led to the creation of lower level assessment tools, i.e., tools for Level 4?

I think that there are no limitations per se -- I would not say that there are limitations of the AHRQ HSOPS that led to development of tools for TeamSTEPPS. The only thing that I will say is that several tools along the evaluation criteria are not aligned, or were not aligned with TeamSTEPPS and the AHRQ HSOPS survey is partially aligned with TeamSTEPPS.

But when we tried to delve into more specific TeamSTEPPS or team oriented things the HSOPS does not have a straight forward team focus or is not only team focused and so that’s part of the reason why I think other tools were developed, specifically to align them to the TeamSTEPPS initiative. But they do go hand in hand perfectly and we do strongly encourage using AHRQ HSOPS with any TeamSTEPPS basic initiative.

Sandra Almeida: Right and the other piece of that response is the lower levels, the Level 1, 2 and 3. If you don’t see a change in your HSOPS you should ask the question why not? The question might be that your team training was never transferred to the job, they never actually implemented it.

So unless you measure at those lower levels you don’t know if you were ever in a position to see a Level 4 patient outcome culture change. So it’s not that any of these Level 4 measures are not good, it’s just that if you don’t see a change you want to know why not.

Was it because the training wasn’t effective? Or is it because it was some organization factor where you staff were never actually able to implement these tools in the real setting? That’s the real premise behind these multi level evaluations, is you can’t get to Level 4 results unless you’re successful at Level 1, 2 and 3. Does that make sense?

Alexander Alonso: Absolutely. The last question I see here is would the TeamSTEPPS Attitude Questionnaire work for selecting the change team prior to implementing the team training program?

Sandra Almeida: That’s pretty good.

Alexander Alonso: Yes, I would say that it’s not been used for that focus, but I don’t see why it wouldn’t be used for that. The only thing that I would be careful with is to not make that your only criteria for selecting members of the change of team.

Sandra Almeida: That’s a good point.

Alexander Alonso: Let’s see here. What I do want to do now is to turn this over for one second and I want to thank Sandy very much for participating here today and for facilitating this today. I’d like to conduct a small poll to have the remaining participants vote on whether or not the information provided here today was useful. And it looks like some of you have already begun answering. Go ahead and answer and then I will have Rachel close out the poll, or Cori close out the poll.

Cori White: We’ll give you just another minute to answer; we want to make sure that everybody who watched gets a chance to put in their response. If you’re in a group full of people, you will have to figure out who gets to the mouse first to answer.

Alexander Alonso: Sandy in the meantime someone did ask whether or not there is a list of sites testing the MTPAT. I don’t have that list available. What I recommend is calling the or contacting the folks at the Health Care Team Coordination Program at DoD.

Sandra Almeida: Yes, I know that it’s being tested at the Army Trauma Training Center in Florida. Again, it’s been developed for trauma teams as well as for labor and delivery. It’s also being tested and developed at the Naval Medical Center Portsmouth in Virginia, that’s the L&D piece. My understanding is there’s been some discussions about moving some of the testing to private sector. I think there are some discussions going on with David Baker about doing some of the work at Carilion, but I don’ know the exact clinical setting.

Alexander Alonso: Sure.

Sandra Almeida: That’s my understanding of this.

Cori White: Okay I just closed this poll question, I’m going to -- people usually like to see the answers, so I’ll give you a response. It looks like most of you found this pretty useful.

The next question we have is whether or not you would recommend these Webinars to others. So this is open again. Please let us know what you think.

Again if you have questions that you think would be better answered in an email, you are welcome to email us at teamsteppswebinars@air.org. Also we find that some questions are best answered by looking at our Website which is www.ahrq.gov/teamstepps. Alright, I’m going to close this one as well and then let you all see the results. Looks like many of you are referring us to others and we thank you for that very much.

Alexander Alonso: Okay, I’d like to thank you all for attending. We greatly appreciate you spending your lunch hour with us for some of you or your breakfast hour with us. I want to thank Sandy again for providing us with such useful and wonderful information.

Sandra Almeida: Thank you for the invitation and thank you all for listening and for your commitment to promoting team base care throughout our health care system. It was an exciting and challenging opportunity.

Alexander Alonso: Thank you all and we look forward to seeing you again next month when we have a Webinar scheduled for March, 25 if I’m not mistaken. So once again, thank you and have a great day.

Operator: Ladies and gentlemen, that does conclude the conference call for today. We thank you for your participation and ask that you disconnect your line.