National Implementation of TeamSTEPPS Program Webinar 19:
SOCIO-TECHNICAL PROBABILISTIC RISK ASSESSMENT OF TEAMWORK IN HEALTHCARE

Moderator: Alex Alonso
February 16, 2011
12:00 p.m ET

Operator: Ladies and gentlemen, thank you for standing by and welcome to the Socio-Technical Probabilistic Risk Assessment of Teamwork in Healthcare. During the presentation all participants will be in a listen-only mode. Afterwards we will conduct a question-and-answer session. At that time, if you do have questions you may press the 1 followed by the 4.

If you require operator assistance at any time during the conference you may press star 0. As a reminder, this conference is being recorded Wednesday, February 16, 2011.

It is now my pleasure to introduce Alex Alonso. You may proceed, sir.

Alex Alonso: Thank you, Operator. I want to welcome everyone here to the 19th webinar in the TeamSTEPPS National Implementation webinar series. We are very honored to have you all here today. So good morning and good afternoon depending upon your location.

Today we are going to be discussing a topic known as Scio-Technical Probabilistic Risk Assessment and especially as it relates to assessment of teamwork or the development of teamwork measures. We’re joined by Dr. Anthony Slonim who is the Chief Medical Officer for Shady Grove Adventist Hospital and is part of the Virginia Tech Carilion School of Medicine.

Before we begin the webinar I want to remind everyone that we ask you to be considerate of others while participating in this webinar, that you mute your phone to reduce background noise, that you do not put your phone on hold if you have music or advertisements, and that you remember that any conference call is only as good as its weakest link or connection.

Today’s agenda, as I alluded to, focuses on who we are, what the National Implementation of TeamSTEPPS program is, what is the socio-technical probabilistic risk assessment or STPRA, and how can it be applied for healthcare teamwork. And then we’ll provide you with information for contacting all of us.

As many of you know, the American Institutes for Research is a prime contractor for the National Implementation of TeamSTEPPS and we are a not-for-profit, nonpartisan D.C.-based research firm with 11 - it actually should be 13 US locations now and 12 international offices focusing on health education in the work force. Our staff is comprised primarily of PhD level health services researches, nurses, physicians, and social and behavioral sciences. Our mission is to better society through our research.

The National Implementation Project was funded by AHRQ and the Department of Defense initially to create a national infrastructure to support the adoption of TeamSTEPPS, initially with staff from quality improvement organizations and from the Patient Safety Improvement Core. The overall goal is to make training available to early adopters including those at high reliability organizations, members of the AHRQ Action Network, academic medical centers, and other healthcare professionals.

The goal, obviously, is to spread TeamSTEPPS and the ultimate end goal was to create at least 2,400 new master trainers. As you know our team is made up of initially four team resource centers and they were joined by a fifth -- these include the University of Washington, the University of Minnesota, Creighton University, Duke University, and Carilion. AIR, of course, is the lead group for this entire effort.

As I mentioned, our sponsors were the Department of Health and Human Services and specifically the agency for healthcare research and quality and the Department of Defense and specifically the Tricare Management Activity.

The project team at AIR is led by Deborah Milne and I am a task leader for research and webinars for the Implementation Program. Alexa Doerr is the research assistant for all project activities. We are an interchangeable team. We are also fortunate enough to be joined today by a former team member who is assisting us with the large volume for this webinar and Cori White will now orient us to some of LiveMeeting if you will.

Cori White: There are a few things that you need to know about LiveMeeting as a participant in the webinar and someone who’s following along with us. The most important one that we get the most questions about is where to get a copy of the slides. At the top right-hand side of your screen, there’s an icon that looks like three pieces of paper that if you mouse over it it should say handouts. If you click on that button you’ll find the slides available to download in both full slide per-page format and three slides per page with space for notes.

The other thing that you should be aware of is also at the top right-hand side of your screen where it says feedback. This basically provides the feedback that we can’t get because we’re not all in the same room. So you let us know if you can’t hear or if we’re going too fast. The only thing that we ask is that you do not use the question icon under feedback. Instead, we ask that you use the Q&A tab.

The way that we’ll be managing questions today is we ask that all questions go through the Q&A tab at the top of your screen. You can type in a question there for us and it will go into our question queue. When we get to a designated spot for questions, either Alexa or I will read out the question for the presenters so that they can answer it. If you have a question that can be answered privately we’ll do so.

If it’s something that we think will benefit the group but is more of a question about LiveMeeting we’ll answer it publicly in that Q&A tab. So again, if you’re joining us through LiveMeeting and you’d like to ask a question please do so through the Q&A tab. If you are only with us on the phone then you’ll be able to ask questions through the phone and if Operator you could give us another quick tour of how to do that that would be helpful.

Operator: Absolutely, ladies and gentlemen, at that time if you wish to register for questions you may press the 1 followed by the 4. To unregister, 1 followed by the 3.

Cori White: Thank you very much. So those are the major things you need to know, where to get handouts and how to ask a question. We will be asking two very quick poll questions at the end of the webinar. They just basically ask how we’ve done and, again, those are very quick. You can answer them through LiveMeeting.

If you are only with us on the phone and you’d like to answer the question or provide more formal feedback we welcome all emails that you would like to send. The email address for that is teamsteppswebinars@air.org. I think that’s about it.

Alex Alonso: Okay, thank you, Cori. Here is some contact information for all of us and as Cori mentioned you can also reach us at the generic email which is teamsteppswebinars@air.org.

At this point I’m going to go ahead and turn it over to Dr. Tony Slonim and I do want to go ahead and give you some brief highlights as far as his illustrious career, namely he is a Chief Medical Officer for Shady Grove Adventist Hospital. He’s the President of SAMIKE Consulting. He has a faculty appointment at the Virginia Tech College of Medicine. He is also the professor of Basic Sciences, Medicines, and Pediatrics there. He has over 100 publications and has edited 12 textbooks, three focused on medical errors and patient safety.

In addition to this, hehas really been very active in the world of socio-technical probabilistic risk assessment, namely he has been funded for an AHRQ A08 grant investigating safety of blood product transfusions in pediatric patients. Hehas also been funded for the American Society of Healthcare Risk Managers and AHRQ respectively for the use of STPRA in healthcare for projects ranging from hospital-acquired infections to serious infantile events during hospitalization.

Currently, I also have the pleasure of working with Dr. Slonim as the principle investigator for a task order we’re working with the Agency for Healthcare Research and Quality to identify risk factors with surgical site infections in ambulatory surgery centers.

So at this point, Tony, with great pleasure, I hand the webinar over to you.

Dr. Anthony Slonim: Alex, thank you so much for your kind introduction. I want to spend a moment just to thank our sponsors, the Agency for Healthcare Research and Quality and the Department of Defense for not only helping us to get these tools promulgated to a broad audience of people who can help our patients, and who can help our providers provide better care for our patients, but I get the luxury of being able to present to all of you this afternoon and I would be remiss without acknowledging the team of people, teams of people, who have worked with me on many of the projects and the points that we’re going to be talking about today. So I want to acknowledge their support and the great work that they do everyday.

We’ve broken the presentation down into two major components and the first component we’re going to spend some time thinking about and talking through a tool that I’ve become familiar with over the last few years called probabilistic risk assessment. We’ll give you some of the basics and hopefully in a manner that allows you to understand what the tool is and what it can do for you and your patient safety programs and also give you some of the challenges in terms of using it. We’ll pause after that component of the presentation to answer any questions you might have about the tool.

The second component of the presentation will go into additional detail about how we use the tool and applied it, and work that was funded by the Agency for Healthcare Research and Quality to improve said patient safety in several high-risk event areas where teamwork was thought to be operational. So what we tried to do was identify the risks associated with teamwork as a part of that work. We’ll share some of the findings and some of the going forward opportunities that we think we have.

So with that we’ll jump in. I want to provide a backdrop about some of the ways - I’ve had the luxury of working with some of my engineering colleagues at Virginia Tech School of Engineering. I’m not an engineer and I don’t claim to be, but I have learned quite a bit about the ways in which engineers approach the way systems work. In healthcare, we’ve tried to understand the way we deliver quality through an assessment of risk. The engineering folks tend to look at this from the other side of the coin where they’re looking at reliability and how systems work rather than how they fail.

They are a little bit two sides of the same coin but nonetheless as important because if you can understand what the risks are and how risk can be mitigated you may be able to drive performance at the system level to a new opportunity. I think that’s important for us as we’ve tried to make sure that we don’t have individuals to blame in this world of healthcare, but we are looking at system design and process improvement work to drive the care that we deliver for patients.

Probabilistic risk assessment is a tool. It’s a comprehensive tool and it’s very structured in the way that you apply it. The idea of using this tool is that it can help you identify risk to improve safety. So if you know what the risks are in a complex system you can identify them and work on them one-by-one. The tool has been used in a variety of high-risk industries and is particularly helpful for those things that we consider in healthcare serious and sentinel events.

So for example, the child that gets abducted from the hospital, the wrong site surgery- those things that don’t happen very often but probabilistically will occur over the course of time in healthcare institution once every five years, once every three years. So you want to try and figure it out. This is one of the things that we struggle with everyday is we’ve got a serious or sentinel event that occurs, many of us will do a root cause analysis on the ground and try to figure out what happened and why and then we do a whole lot of performance improvement work to make sure that it doesn’t happen again.

The challenge is these are events that are not happening every day or at least we hope not. You may have six months that go by without another event happening, a year that goes by without another event happening, but you’re not sure that the fact that you’re not seeing events is because probabilistically they’re just so rare that you’re not seeing it or that your fixes to the system actually do something that reduce the likelihood of those events. PRA helps us get to some better understanding about what those risks are and how we can manage them.

At the end of the day, the fundamentals of PRA are that it is a process analysis tool. Let’s think back if you will with me to the mid to late 1970s when the grandfather of quality in healthcare, Donobedian, helped us to think about structure, process, and outcome. When he applied that construct to healthcare it helped us to think about processes or those sequential steps that govern our interaction between patients and providers and between providers and one another.

We’ve now added to that construct the interactions that providers have with equipment and technology because as you know technology in hospitals has become more complex. For example, if the providers don’t know the right ways to troubleshoot the ventilator in the ICU or the medication pump on the medical surgical unit, our ability to deliver safe and effective care to our patients will be compromised.

TeamSTEPPS in particular helps the middle bullet where providers need to work with one another to understand the ways in which they can do that more effectively. It’s using shared mental models. It’s improving the way we communicate with one another. It’s identifying our opportunities to be better team members so that we can provide more comprehensive and coordinated care to our patients.

Well, as a part of the work that we’ll explain in a little while, there are risks associated with the way in which providers interact with each other. If we could identify those risks, we thought that, we might be able to help provide a better team experience for patients.

Well, looking at processes is important, as a matter of fact, there are several tools that we already use in healthcare to try and help us understand process work at the system level and do a better job for our patients. All of us will use root cause analysis as a tool when a bad event happens. It’s simply a way to breakdown the system of care and understand by looking back and putting a bunch of smart people in a room, what happened and why so that we could prevent that from ever happening again to another patient.

There are other examples and a more prospective manner: failure mode and effects analysis. FMEA, is a tool that doesn’t look back, but looks forward at those high-risk scenarios so that we could mitigate care risks before they actually occur. Over the last few years, joint commission has allowed us the opportunity to look forward and make sure that we were doing one of these FMEAs on a high-risk process at our organizations each year to get ahead of the safety curve if you will.

Many of these tools and techniques are complementary and the way that they approach, I’ll say, more traditional PI methodologies, PDSA, focused PDCA, plan-do-check-act, plan-do-study-act methodologies, they all have a component where they’re looking forward trying to understand the process, figuring out what is the potential to make it better, and then we’re running cycles of improvement. The DMAIC model is something that has come to us from the world of Six Sigma as another opportunity to do performance improvement or process improvement techniques.

I want to spend a half a minute on HAACP, the hazards analysis and critical control points which has been a prospective process analysis tool used within the food services industry for a number of years and then promulgated more broadly through the FDA as an opportunity to understand what are the risks, particularly in the meat packing industry where these kinds of things -- you want to prevent infectious spread through the food supply --where these kinds of opportunities to reduce risk present themselves. A whole family of process analysis techniques performs a probabilistic risk assessment, which is one of those techniques in that take a broader look at process work.

You all have, I’m sure, worked with and think about how you map processes in your clinical care and the delivery of patients. Here’s a high-level overview just to provide us with some understanding about what this process looks like. We usually have a starting point where, in this example, we admit a patient to the ICU and we determine if they need a central line and if they don’t, well, then we’re done and we exit the process.

But if they do, we have to make sure that we’re prescribing for them the insertion bundle or the maintenance bundle because those two bundles of care and checklist have opportunities for the potential for getting an infection.

If they don’t get an infection, they simply exit the process and we collect data on them and make sure that our rates are commensurate with what’s expected for us from national benchmarking organizations and others in our community. But if we do end up with an infection, how do we look back at our work and see where the breakdowns were so that we might prevent those breakdowns for other patients that come into our care?

Another performance improvement technique, the cause-and-effect diagram also known as a fishbone provides important opportunities for us to organize our work for those things that may come up as a nosocomial infection or a blood stream infection. So we tend to look at the insertion bundles or the maintenance bundles here whether or not our policies and procedures were followed. Do we have a lack of a team in the way that we’re providing this?

Hand washing and resistance in our environment is so important to how we drive some of this care but importantly, what I’m trying to frame out for you here, is that probabilistic risk assessment is no different from any of the other tools that we use in healthcare to look at how we deliver care for patients with our process analysis techniques.

So a little bit more detail on what PRA is. For you, PRA includes those process analysis techniques, but also includes some decision support so that providers can make better decisions when they’re caring for patients. Importantly, it identifies the risk points as we alluded to before. You want to be able to identify not only the parts of the system where you’re system is functioning well but identifying those breakdown points so that you can get directed towards interventions.

That’s one of the important and unique things about this tool that is not only do you catalog the risks like you do in an FMEA, but probabilistic risk assessment points you to the interventions, importantly the ones that are most likely to impact the outcome that you’re interested in.

The other elements of PRA that are important is that it is hierarchical and I’ll go into some detail about that in a second, and it’s what we call probabilistic. We can disentangle the risks that occur at the patient level. This is a patient who presents immunosuppressed and is at higher risk for an infection during hospitalization. Those risks that are associated with the providers, they did or did not follow the bundles for example, and those that are involved at the system level. We were able or not able to get an antibiotic to a patient within the first hour of their presentation with pneumonia.

So disentangling where the risks are so that you can identify the intervention steps that need to go on in those different, if you will, hierarchies.

In addition, PRA is what we call probabilistic. It allows you to assign probabilities so that you can prioritize what your risk reduction strategies might look like. Then you’ll see us use some jargon as we move forward in a little while around the socio-technical component, what we like to call - and then that moves this from being PRA, probabilistic risk assessment, to socio-technical probabilistic risk assessment, what we call STPRA because it allows us to embed in there some of the elements of our culture into the modeling.

Let me give you an example of what that looks like. All of us know how important it is to wash our hands during an episode of care, pump-in, pump-out, make sure that you’re washing your hands prior to a procedure, etc. We also recognize how important culture is and climate is to our need to wash our hands. If I’m likely to get caught for not washing my hands there’s a greater propensity that I will wash my hands if there’s some potential harm that might affect me as a provider. I might get suspended; I might get caught on camera; I might have a couple other things happen.

We also understand that these cultural norms dictate how safe we are as we present ourselves to our patients. So this is important. If we recognize that it’s important for us to always wash our hands at 100%, but we recognize that on a particular unit the director doesn’t much keep track of how well people are washing their hands and there are no consequences for not washing your hands, the cultural norm may be that, okay, I’m really only going to wash my hands about 50%.

Well, that obviously impacts our ability to deliver at the level of risk. So we need to in some way embed that cultural norm into these models. Socio-technical PRA allows us to embed those cultural elements in here as we’re moving forward.

So I really like this tool because it allows you not only to include, if you will, the numeric values. It allows you to include some of the elements of what you understand about your culture or your climate as you’re moving forward for your patients.

How do you do a PRA? Well, you identify what you’re trying to fix or where your patients are experiencing an adverse event, example mortality from hip surgery. Then you use some of those other tools that we’re familiar with: a process map, expert opinion, past or published data from the literature. You go out on Google and you figure out who’s done this better than you, and you steal liberally from them because that’s what we try to do is learn from others, and then you package this into all of the possible event combinations that can occur that may lead to that adverse outcome.

You’ve tried to predict the likelihood of that adverse event occurring before it actually occurs. So this is, if you will, a little bit more like FMEA in that it’s prospective versus RCA, waiting for the risks to occur and then going back and trying to figure out why it happened.

This is a very complicated diagram, but I do want to walk you through it. We will often take a look at two major components of quality, so called, quality of fact where, , what is our rate of medication errors, what are our fall rates, what are our length of state, any other number of metrics that you might be looking at for your patients.

We also need to include as we’re doing our analyses those socio-technical pieces, what is the quality of perception, what have our patients told us about bad things that happen or complaints, what have they told us about how satisfied they are with the experience, how patient-centered are we as we orient our care for that singular patient in the bed. Again, trying to understand PRA in this hierarchical approach is critical to moving forward.

PRA allows us to understand the risks one patient at a time, but as we move to Panel B it also allows us to operationalize this one unit at a time, whether we’re looking at unit-based quality and how the scores on Eight West might look in comparison to the scores on Twelve West. You get the idea.

We can use this and dissect out and use the tool, PRA, for the benefit of understanding the risk at the individual level or the unit level or what I really like, across the system of care whether you’re talking about that system of care being the hospital, the region, the state, or more importantly as we start to address the issues of how accountable care organizations will move forward in delivering population-based care, how are we addressing the risk at the level of the population we serve.

So probabilistic risk assessment is a very robust tool that lets you not only understand probabilistically what’s going to happen, but allows you to dissect out, at what level of analysis you might want to incorporate your findings so that you can derive interventions to move forward and improve patient care. In this way it capitalizes on some more sophisticated statistical techniques like cluster analysis and modeling and other things that get incorporated into the models.

I’ve found one of the best ways you could use probabilistic risk assessment are for those very low occurrence, high-impact events. That’s why it was promulgated early on in the nuclear regulatory industry or in the aviation industry because you don’t have a nuclear reactor, thankfully, blowing up every other month. You want to be able to use this for the really high-impact events, but those that occur low frequency and gets to the sentinel event issue in healthcare.

It’s also useful for those places where variable processes come into play because it directs you to the fixes so, again, where you’ve got death or major harm from a process of care, it allows us understand this is some work that we’ve done for Ashram about trying to figure out where across an eight-hospital system, the risk of care might be identified and then mitigated through deliberate work that we have going on.

Process analysis helps us to identify those risks and prioritizes those interventions based on the probabilities. Again, if you build it right, it allows you to give decision-making support to the providers who are providing the care. This where particularly for teamwork occurrences we thought it would be particularly helpful.

So it’s helpful for examining low-based rates events and Six Sigma is a methodology where we’re talking about errors - while healthcare is focused on the two decimal point occurrence rates. We’ve got hospital associated infections appearing in the 1% range or where we’re looking at rates that are defined as two decimal points. Six Sigma wants to talk about the occurrences of one in 10,000, one in 100,000. This tool helps to support the work that goes on in Six Sigma.

It also allows you to get away from the root causes and build into the model the contributory causes that you might have - be able to identify as you would with a root cause analysis. Because this is computer generated, it lets you try an intervention in the computer model before you apply it to patients. When you all know systems are very complex and making a change in one part of the system creates opportunities to make more risk in another part of the system, it really does provide us an evidence-based and empirically-based approach to an intervention when we go out and make it.

In addition, it allows us to learn related to the impact and modify our behaviors. If I know that we’re building a model where patients on the socio-technical level are only washing their hands 50% of the time, I need to be able to provide an intervention that allows me to address those provider behaviors and thinking through what those interventions are can really come out of this STPRA modeling.

Importantly as well, we can use the tool to do some higher level sophisticated statistical techniques like Monte Carlos simulation and changes in the likelihood of outcomes that may be important for us as we move forward.

So here’s a conceptual framework if you will, again, drawing on what STPRA can do for us as a global tool is it allows us to address the hierarchies of the institution, the providers, and the patients while at the same time drawing on probabilities from the literature and what we know through our instant reporting systems as well as the more qualitative elements of our culture and our climate to be able to drive improvements for our patients.

So really it is an all-in-one tool that I think helps in some respects, but I want to make sure that we’re recognizing this is not a one-tool-fits-all approach. It’s not the tool for every single scenario. You’ve got to apply the right tool for the right indication. So PDCA may be the perfect performance improvement tool for that you’re addressing everyday. On some occasions you may have a process where you’re trying to get the improvement to be Six Sigma-like and you may want to use the DMAIC methodology.

When bad events occur or have occurred in your organization you certainly want to use a root cause analysis so you can get to an understanding of those events in a relatively short period of time and fix them. PRA does have its place within that toolbox and how you choose to use it in your organization may be beneficial for you as you go about advancing your patient safety programs.

So if we think about the ways in which PRA works, it uses four questions. It asks the question about how the system works. It asks the question about what can wrong and what are the sequences or processes or scenarios that in combination lead up to those bad events occurring. It asks about the consequences of what goes wrong. Then it creates understanding of how it affects the adverse events of interest and it tells you by the probabilities how likely those scenarios are.

Within Six Sigma, you can see that there are any number of inputs that we have go into the way that we provide our care. PRA helps us to think through in this DMAIC methodology part of the analysis piece. So you’ve defined what goes in. You’ve measured it. Now the PRA, the risk assessment tool, helps us to analyze what goes on. Then as we’re moving towards our interventions it identifies our major risk points and prioritizes the areas where we’re likely to have the most bang for the buck.

Importantly, once you’ve built the model it’s iterative. So if we build the model around wrong site surgery and you make a process change you can build the process change into the model and not have to create the model all over again to move forward.

So when we’ve created in the past, through some funded work with ASHRM, the PRA model at the level of the organization with our serious and sentinel events, this allows us an opportunity to update the model based on our new risk profiles. What were we now learning about in the software, our risk reporting software, is that it allowed us to improve the model and drive our patient care to new levels.

PRA can answer four questions. What is the most cost effective strategy? By including cost into this algorithm, you can really help to identify not only what is the most likely to make a big difference, but also what is the most cost effective in making the difference.

You can do what are more statistically called sensitivity analyses where you run the model through a base state and then alter your assumptions over a range of indications. Imagine that you want to double the risk of a wrong site surgery from 1% to 2% and see what, then, becomes your high-risk area to focus on. The model allows you to do that and figure out over a range of occurrences what your most important interventions are.

It allows you to understand if you’re meeting your safety goals before you set out and tell the board what you’re doing so you can actually see if you can actually make the difference. Also, it allows you to, using a more quantitative methodology understand by how much you might reduce the risk.

The way you create a model is you identify the outcome of interest, the so called top level event. That’s PRA jargon for the thing we’re most interested in. Then we look at all of the contributing basic level events or the lowest level terminating event. For example, one of the analyses we did several years ago was looking at ECMO, extra corporeal membrane oxygenation, which is essentially a heart-lung machine.

We wanted to at the basic level event try and figure out, well, what if the machine became unplugged? What would happen to the patient? At the most basic level it allows you to incorporate into your modeling those - what I’ll call low-level events that lead up higher hierarchy to the highest-level event. It also allows you to incorporate your conditional events, which are in other words saying that your contributing causes that are part of your root cause analysis.

The way that you incorporate those events is important and they have very logical relationships underneath which there is a whole mathematical profile; we’re not going to go into at this point for the purposes of this call, but that I would be happy to talk offline about that if you’d like to contact me. I’d be happy to have those conversations.

How do we identify the fact that the provider did not wash their hands and did not follow the insertion bundle for the insertion of a central line? Or how do we identify that the provider did not wash their hands or did not follow the maintenance bundle for a central line? Do you get the idea? It’s a logical way to organize the multiple-level basic events and move them up so that we can understand the risk associated with the top-level event.

This is a graphical representation about how we do this modeling. Up here at the top of the diagram goes your major level events, the outcome of interest, then the wrong site surgery. We might identify the wrong site surgery comes from either the fact that we don’t perform a timeout or we don’t do a site marking or - you get the idea.

What are the ways in which those more fundamental events occur? Then if we’ve got a point where we didn’t do a timeout, well, why didn’t we do a timeout and what are the basic level events? Well, the surgeon wasn’t paying attention and nobody on the team helped them to understand that it was their job to intervene. Do you get the idea?

So we’re looking at very basic level events and the ways in which providers interact with one another organizing them to a series of and or logic gates and then seeing how all of these risks add up to a top-level event with an overarching probability of occurrence.

We’re going to pause there as we come to the end of our first segment of the presentation and see if there aren’t any questions that we might be able to help before we describe how we use this tool in an effort to try and understand how teamwork is operative in some of the serious event work that we did about a year ago.

Alex Alonso: Okay, thank you Tony. At this point, Alexa, do we have any questions online?

Alexa Doerr: We do have one question. The question is would you supply good references for more detailed information on how to conduct an STPRA?

Dr. Anthony Slonim: Absolutely, if you go into the literature, there is actually very little written in healthcare. Some of this our teams have tried to contribute to the literature in a couple of ways.

There’s some work out there in the Journal of Quality and Safety that describes probabilistic risk assessment at the, if you will, at the highest level in a how-to-do-it approach. The manuscript was written by David Marx and myself several years ago, probably five or six years ago.

There are some other examples of focused work and using probabilistic risk assessment in critical care and using probabilistic risk assessment in hospital associated infections. There’s some additional work we’ve got coming out around serious and sentinel events. That’s about the limitations of it in the healthcare literature.

The other places you’ll find the tool being used, again, particular. There’s some literature out there about using the tool for cancer cluster analysis- so very epidemiologically-based tools, as well as, the non-healthcare literature around things like, as I mentioned before, nuclear regulatory commission, airline crashes, aviation industry.

In fact, engineering colleagues of mine have shared with me how they’ve used this tool for things that we all would recognize as recently as the shuttle disaster and how they used this to try and understand from a very engineering-oriented approach where things went wrong and what were the likelihoods that those things went wrong.

So in healthcare, there is very limited exposure, but we hope to contribute to that more broadly. Outside of healthcare, there is a whole range of opportunities to get to know this. If you just Google probabilistic risk assessment or do a search using that term you’ll be able to find a few things.

Are there any other questions, Alexa?

Alexa Doerr: The next question is what do the numbers at the bottom of the last slide indicate? Weights?

Dr. Anthony Slonim: The numbers at the bottom of the last slide were just an opportunity to identify that this basic level event and this basic level event were the same. This basic level event and this basic level event were the same. So for example, I’ll use an example here. The basic level event here might represent hand washing.

Well, hand washing may be the basic level event in the contribution to a series of infections that might lead to excess mortality. Hand washing might also be operative in the way that we handle our patients before an operating room case. So it falls under a different tree if you will but nonetheless is a fundamental event in a way that needs to be organized across this model because it impacts so many different places.

It’s a good question as well because at the end of the day your basic level events are where your interventions are driven. So the fact that hand hygiene may be represented by any - by two of the four trees here indicates that that might become important for us from a probability perspective if I were going to get on top of what this high level event is. So again, PRA not only lets you qualify what the basic level event is, it then organizes them through these logic gates so you can better understand where to put your efforts from a performance improvement perspective.

If I get at the target of hand hygiene I can affect not only this outcome, but this outcome as well and both of those in combination will affect my highest level event.

Alexa Doerr: The next question is are any patient safety organizations using this tool, i.e. those states doing mandatory reporting?

Dr. Anthony Slonim: I don’t know about states. It’s deep embedded in the Joint Commission is an allusion to probabilistic risk assessment as a tool that organizations should be using as they move forward to proactively assess risk. So about two years ago the Joint Commission started to include this in their language about tools that you could use to be able to move forward and in understanding the risk of your patients.

In fact, it’s included in the leadership chapter of the Joint Commission and proactive risk assessment methodologies that your organization may use. I think one of the important pieces of our funding efforts and really what launches this to the next horizon is it’s not often been used in healthcare and I think the reasons why are you can see that this modeling is pretty complex to do. It requires data. It requires conversation. It requires some expertise honestly around the use of the modeling software so that for the average healthcare organization you may or may not be able to do this kind of modeling.

Our team believes that there are opportunities to help address that shortcoming of PRA by creating any number of opportunities to make this tool more readily available so that it can become a plug-and-play software tool that you can include in your quality improvement department and it would be, again, one of the tools in your toolbox just like root cause analysis is or just like FMEA is moving forward, particularly since it seems to have caught the attention of the Joint Commission and other regulatory bodies.

But at the moment, most of its orientation in healthcare has been around the research domain where we’re trying to figure out if it’s a useful tool and how it can be used more broadly.

Alexa Doerr: Great, thank you. There are no more questions on LiveMeeting right now.

Alex Alonso: Actually, Alexa, I have one other question here that was sent to me via email. Does the outcome of interest need to be rare? Could this, for example, be applied to readmissions?

Dr. Anthony Slonim: Good question, really good question. The strength of the tool - short answer, yes, it could be applied to readmissions. Longer answer, the important part of PRA - because it is probabilistic it gets over the limitations of FMEA or root cause analysis in that for those you never really know what the likelihood of change is. Going back to my example before on the wrong side surgery, if it doesn’t happen for two years does it mean I fixed it? Maybe not necessarily because the event likelihood is so random, that’s where PRA has its biggest strengths.

Importantly as well, this goes back to the last question that was asked, is anybody requiring it? Using this approach you really get two bangs for the same buck here. One of the byproducts of probabilistic risk analysis risk assessment is that you get an FMEA produced as a byproduct with this work. So you’re investing in time that allows you to meet the requirements. Imagine that readmissions was the thing you wanted to focus on this year and you wanted to prospectively identify the risks of readmission and wanted to do an FMEA around it.

Well, you could certainly do a PRA around it and then your FMEA would be there already hardwired by the push of a button. So it really does capitalize on the work. The other good news about that is if you’ve done an FMEA on a particular process - we did an FMEA last year on infusion pumps. We were then able to take that FMEA and put it in the PRA software and build upon the great groundwork that had been done without duplicating it and that’s been particularly helpful for us as we go about trying to figure out how to use this tool more broadly.

Alexa Doerr: Great, the next question is which commercial PRA software tools do you recommend?

Dr. Anthony Slonim: We’ve been using for our purposes Relex. Relex is a software company in Pennsylvania. I have no disclosures to make. I don’t get any funding from them nor do I have any proprietary interest in them nor do my family members, but I think it’s important - this is a tool that’s been used and when you go to training at Relex it is a little bit daunting because you’re surrounded by a whole bunch of really, really smart people who are chemical engineers and mechanical engineers and aviation engineers who have flown in from around the country from federal organizations like NASA and the airline industry.

You’re sitting in this room with them talking about things like wrong side surgery and they look at you in disbelief and say, you really have people that operate on the wrong body part?

It is interesting to be sitting in that arena where they’re launching space shuttles and thinking about ways in which they could use this tool to advance so many of their opportunities. It’s a little bit of a humbling experience as a healthcare provider to be mixed in with that, but nonetheless that training is valuable if you’re going to set about this.

Our team is looking at ways in which we could help others by making a more available technology that allows you to do this kind of analysis without having 20 hours of training and tools that are very complex to use.

Alex Alonso: I also do want to stipulate, and this is because we are part of an AHRQ funded project here, that AHRQ is not providing an endorsement for Relex by any stretch nor is AIR. This just happens to be the tool that was used by Dr. Slonim and as he stated, he has no interest in that software at all.

At this time I think we’re going to go ahead and ask the Operator if we have any questions on the telephone line.

Operator: Thank you, ladies and gentlemen, if you would like to register for questions please press the 1 followed by the 4 on the telephone. You will hear a three-toned prompt to acknowledge your request. If your question has been answered and you would like to withdraw please press the 1 and a 3. Once again, to register for questions you may press the 1 followed by the 4.

Alex Alonso: Do we have any questions at this time, Operator?

Operator: No questions at this time, sir.

Alex Alonso: Okay, given that we’re going to go ahead and jump forward and I’m going to turn it back over to Tony.

Dr. Anthony Slonim: Thank you, Alex and thank you folks for asking questions. We’re now going to jump into the second component of our presentation, which really focuses on how to use this tool. Again, you see there the socio-technical probabilistic risk assessment and teamwork.

As a TeamSTEPPS master trainer, so much of the work that we try to do around teamwork, and I have to admit one of the most rewarding pieces of my career was being the principle investigator for the team resource center at Carilion, really good work where we tried to hopefully expand and explore our experiences with teamwork training more broadly throughout the country as others want to learn how to use these techniques.

Well, one of the things that we really got stymied by was what are the risks of teamwork? Well, if teamwork doesn’t work care becomes more fragmented and less coordinated. I don’t know about you folks, but I often see in my complaint and grievance experiences at our hospitals, the fact that we’re not coordinating our work as much as a team and as a result we may be falling down in our ability to deliver effective care. So one of the things that we said might be helpful is could we use PRA - STPRA for identifying measures of teamwork and figuring out where teamwork breaks down, particularly for really bad events in a hospital setting.

That’s what this work was focused on. The three top level events that we used or we were interested in understanding was a foreign object - a foreign body left in the surgical patient, shoulder dystocia during labor and trying to figure out what you might be able to do about it, and cardiogenic shock during trauma. We chose these three events because we thought they had the largest teamwork-based risk factor components to them that contributed to the event.

So for example, in a foreign body left in a surgical patient, many of our organizations have policies and procedures to assure that this doesn’t happen. Nonetheless, it happens. Trying to figure out where the breakdown components are- Have we rushed through closure so that we don’t allow the count to occur on time? When the count is off have we followed the series of steps that we need to to assure that there’s no foreign body left in? What are the elements of provider interactions? Is a surgeon being intimidating because they want to go to the next case?

Despite the fact that they’ve been told that the count is off they want to proceed nonetheless and we don’t follow our process with getting an x-ray or doing the next thing in the chain of command. So what are the breakdowns that occur in teamwork specifically that lead to these events occurring and similarly with shoulder dystocia and with cardiogenic shock?

The basic steps in our research were to conduct the PRA and, again, I get the luxury and the real opportunity to present to you. I happen to be the spokesperson for a whole team of people that helped to get this work done and I really do want to acknowledge the work of that team. Our PRA work helped us to identify the precursors of those events and to focus on the teamwork breakdowns.

We then identified using a tool in the STPRA model called Based Cut Sets. These are areas where it tracks through that hierarchy to identify the biggest combinations or cut sets of events that leads to the breakdown in teamwork and it highlights them for you so that you can prioritize your interventions. So we then developed a framework of those cut-sets, validated those using data that we had available to our risk management system, and then began to design measures that we thought would improve teamwork in those areas.

We used expert panels for two activities. In each of those categories of events we identified the fault trees. That diagram that I provided for you before, we identified one of those for wrong side surgery, one for shoulder dystocia and one for cardiogenic shock. We also input into the model, those fault trees, the estimates of probability at each of the top level events as well as each of the basic level events.

So for example, for the patient who had a foreign body left in during surgery, we went to the literature in the Joint Commission website and any other number of quantitative sources and identified what was the probability of leaving a wrong site - leaving a foreign body in a patient. Using nationally represented data like the Healthcare Cost and Utilization Project, the National Inpatient Sample, and other expanded databases, we were able to identify the probabilities.

Are those probabilities perfect? No, but remember that PRA allows you the opportunity to do a sensitivity analysis. You can anchor the risk of a foreign body left in using one of those datasets and then you can alter that risk over a range of observations to see how it affects your models. That’s a little bit about what we did.

In addition, we used a group of expert panels because you can imagine that there are often things that come out in the interactions between patient - providers and their patients, providers and one another, and providers and the equipment that they use, going back to that process work. There are things that come out in that conversation that are not found in the literature.

How often does the surgeon who is concerned about a foreign body being left in snapping at one of the OR staff and saying, we’re not going to do the x-ray? What does that cultural norm look like? Well, this allowed us through an expert panel to say, well, here’s what a variety of cultures might look like and this might affect your risk estimates as you’re moving forward.

So it allowed us to build in some of the socio-technical components as well. I don’t mean to be picking on surgeons here. It could easily have been the anesthesiologist, but know that wouldn’t happen.

What are the probabilities for each of those precursor events? Again, what we were trying to do was identify the biggest bang for the buck. We want to be able to figure out where we could put our efforts from intervention perspective to be able to make sure that this outcome did not occur. We use that by identifying the key areas for risk in each of those areas so here, cardiogenic shock, what we found was that the getting an ABG and interpreting the ABG was a major risk point.

Okay, how can we build that in, if you will, to a checklist kind of an event and figure out that any patient who presents in shock gets an ABG? Also importantly, is that it is looked at as part one of a problem, and interpreted correctly. Helping the teams understand where they came into play if they didn’t understand what was going on with the blood gas, how could they escalate that to a member of the team who could affect the change understanding that body of work was important?

Then, there is making sure that we got and interpreted appropriately an EKG. So the patient may have had a perioperative MI, but we weren’t thinking it could have been an MI because, why? Well, we weren’t getting the EKG. It didn’t cross our mind. The team wasn’t thinking about the ways in which the patient’s heart might be affected. This was actually another area that quite a bit of robust dialog.

How are we interpreting our cardiac monitor rhythms? Are we putting too many patients on the monitors so that we can’t figure out who really needs it and who doesn’t? Therefore are we being inundated with data that we don’t have to pay attention to and then segregating out or diluting the affect of those patients to who really needed to be monitored?

In addition, there were some important interventions around the administration of nitrates and the maintenance of a clear airway that were important in this issue. The fact that patient’s in cardiogenic shock, there are implications from assuring that they are also managing, remember that the cardio respiratory status is an important interaction there and making sure that we were managing their airway and breathing was as important as managing their heart.

This was important body of work for the teams that were involved and we had a whole range of expert members on our panel that included cardiac surgeons and cardiologists and nurses and members of a variety of teams that helped us to get some of this work done.

From a shoulder dystocia perspective, it went back to the basics. We get so wrapped up with high-tech error these days and our patients are more complex, but there’s nothing that can replace a complete patient history and that admission assessment on the labor deck was so important to being able to understand who might be at risk.

In addition, the ability and understanding of what maneuvers to perform when necessary, how to do them, how to escalate the tear from a team perspective if and when you need help in the fundamental maneuvers not working really did help us understand better how the team worked together. So that these moments when a mom and a baby are at risk do become particularly tense for the episode of teamwork. You need a designated team leader. You need to make sure you have a shared mental model of what’s going on.

You need to understand who’s doing what and when with feedback loops that allow us to get to where we need to be. These are the fundamentals of TeamSTEPPS that we teach and they’re the fundamentals that breakdown in an area where shoulder dystocia is a problem for mom and a baby.

So again, we believe that harkening back to hardwiring those tools that TeamSTEPPS teaches us are so important for the work no matter how it might be presented to you.

In addition, some of the risks for teamwork for the foreign body left in was to prevent objects from being retained in the first place by patients. Do we have too many instruments on the field? How are we doing that check and double-check from a verbal perspective that says, “Scalpel”? “Scalpel back”? “Scissors”? “Scissors back”? How are we making sure that the team is communicating directly? It doesn’t take any more time, but it has to be deliberate communications from one place to another and then making sure that we follow the process.

If the process and our risk mitigation step is to get an x-ray if the count is off, let’s make sure we don’t forget to get the x-ray or we don’t get ourselves into a place where we’ve got confirmatory bias. The count’s got to be wrong because we couldn’t possibly have left an instrument in the patient. We couldn’t possibly have left a sponge in the patient. Well, if the count’s off we might have. Let’s do what we’re supposed to do to assure ourselves that we’re working through the process in a deliberate and focused way.

What do our findings mean? Well, we think that some of the important part of identifying where things go astray provides us a roadmap for how we might be able to fix the things that lead up to the top level event. Importantly here, many of the people who participated were TeamSTEPPS master trainers who were very familiar with the tools and strategies that we used to enhance teamwork performance. So it was an opportunity to break down those processes and try and understand the places where TeamSTEPPS might be fine-tuned to improve better approaches to teamwork and healthcare.

It was the opportunity to step back and see really for the first time how you take generic tools like TeamSTEPPS and apply them to specific events to see if they could mitigate risk. That was important for us because now, as was asked in the question period, we may be able to apply some of this learning towards things like readmissions, and towards things that we face everyday like hospital associated infections.

We’re currently working on a project right now that Alex alluded to in ambulatory surgery centers to try and reduce the risk from hospital associated infections and in that context, certainly, teamwork is important, but TeamSTEPPS hasn’t been rolled out in a variety of places. So thinking together about what might be the next steps for TeamSTEPPS is important as well.

Then what we’ve tried to find out are the ways in which we could use these risks to inform the interventions that we have ahead of us so that we could combat those breakdowns in teamwork. So helping our physician colleagues to understand what they need to do as a group of doctors to help drive performance improvement at the level or in the practice of medicine.

How do we help our pharmacy colleagues understand what they need to be doing from a discipline specific approach to improve the way that they deliver care when they come to partner with other members of the care team to deliver medications safely to patients?

How do we work with our nursing colleagues to assure that they have a critical role and they understand their critical role in raising questions in an important and respectful manner, but nonetheless are advocates for their patients, are raising those questions to other members of the team so that we don’t overlook something that they may have identified? These were important ways for us to understand how to apply these findings in other areas.

We believe that STPRA has opportunities for us in surgical site infections and are so appreciative of the work that went on with an action network of being funded for doing some of this work. Now we’ve looked in terms of trying to understand the ways in which we design facilities for taking these risks and then looking forward to how we might design. We’re currently here designing trying to figure out what are our next steps around the delivery of oncology care. So using these steps and informing the design of that unit is important.

How are we thinking about breakdowns outside of the hospital setting? Certainly there had been modules that have come up around the use of TeamSTEPPS principles for rapid response teams. Are there other more specific settings and examples where TeamSTEPPS can be modulated and operationalized differently to make sure that we’re working forward?

Importantly, and I think this is really important work that we’re learning from our most current project on ambulatory surgery centers is, despite the robust nature of the datasets that are out there, there are always opportunities to redesign these national datas because they don’t give you firm understanding of some of the fundamental elements that you need to incorporate into these risk models.

So informing the great work that these people are doing with designing these databases and improving them over time, how can we help to enter that dialog so that they can give us the data that we need to drive some of the risk modeling that we’re doing?

Again, I would like to take the time to think about and share our appreciation, not only for our sponsors AHRQ and the Department of Defense and all the great work they’ve done to move TeamSTEPPS forward, but also Dr. Jim Battles, Dr. David Baker, the co-investigator for the work that we’ve described here, Dr. Sigrid Gustafson, Ms. Deepa Ganachari from AIR and all of the great folks at Carilion Health who helped us not only on the patient units to help identify where some of these opportunities landed, but also in the simulation lab where we tried to do some of this work, operationalize it, and then test it in a simulated environment to make sure that we were getting to the outcomes that we thought we were as we moved forward with this work.

With that I’ll turn it over to Alex, again, so that he can go forward and help us with some of the other acknowledgement. Thank you so much for your time and for joining today and then we’ll take some additional questions.

Alex Alonso: At this point I want to go ahead and direct everyone to contact Dr. Jim Battles for more information of AHRQ about the particular slides that are available or for more information about the measures. His email address is James.Battles@ahrq.hhs.gov. At this point, Alexa, can you provide us with information about the questions that are present?

Alexa Doerr: The first question is, do you use observation to determine some of these issues and the probabilities?

Dr. Anthony Slonim: Yes, we’ve used a hybrid approach because we believe that that’s what gets you the best estimate. Again, a one-size-fits-all doesn’t work here. So we’ve tried to inform the models at the basic level with rates and proportions from national data when we could so it’s very clear now that compliance with insertion bundles, for example, that literature exists and we can get estimates about those fundamental risks.

There are some places where we can’t get estimates of the risk and we put a bunch of smart people in a room and we kind of try and figure out what the risks are and that’s a little bit about where the qualitative and quantitative pieces come together. Then the third way we do this is by looking at, even if they’re local, the risk management systems or the risk - you know, your incident reporting systems, or alternatively direct observation. So a little bit of secret shopper work helps if we need to inform some basic levels of the models.

Alexa Doerr: Okay, the next question is, have you or AHRQ published any material about PRA?

Dr. Anthony Slonim: Yes, again, there are several publications out there in the healthcare literature, a couple of things that we published early on were overviews or specific applications in the ICU, Household Associated Infections, and we have more stuff that’s in the queue that’s being published that’s not yet available.

It’s interesting, not only are the healthcare people interested in this but our colleagues in engineering - they now are at schools of engineering health management tracks for engineers. So we see this work now being funded by National Science Foundation and being published in operational engineering journals where previously you might not have expected that work to be found.

So for example, the most - some of the work that we accomplished in my QOA award on blood product transfusions was very heavy technically-oriented in its discussion around how you do the modeling in healthcare. That went to an operations engineering journal.

So you may have to look a little bit outside of the healthcare arena to be able to try and find some of these references.

Alexa Doerr: Great, we currently have no questions on LiveMeeting.

Alex Alonso: Okay, operator, do you have any questions?

Operator: We have none registered at this time, but as a reminder to register for questions you may press the 1 followed by the 4.

Alex Alonso: Okay, in the meantime while we wait for questions, Cori, why don’t you go ahead and run the polls?

Cori White: Okay, we have just two quick questions for you today. The first question, which is currently open it says, how useful was the information presented here today to you? This is, again, just a way to find out how well we’re doing on our webinar series. If you’ve been on one of these TeamSTEPPS webinars before you will have seen this question before as well as the next one that we’re asking. If you are in a room with a group of people we usually say please collaborate and make a decision or whoever gets the computer first gets to answer.

Operator: We have no questions on the audio lines at this time.

Cori White: Okay, thank you. It looks like our responses have pretty much stabilized. You guys in general did find this useful. I can show the results and I will go ahead and close the poll.

The next question that we have for you is would you recommend this webinar series to others? Again, this is just sort of to let us know how we’re doing. I’ll give you just ten or 15 seconds to respond to that. All right, and it looks like, again, the numbers are starting to stabilize so I will share the results on that and close the poll.

Alexa, did we have some more questions come in while we were doing the polls?

Alexa Doerr: Yes, we did.

Operator: On the audio lines as well.

Cori White: Okay, why don’t we start with the ones on LiveMeeting and then we’ll go to the ones on the phone.

Alexa Doerr: Okay, the question is, can the presenter discuss more of the research work in oncology he mentioned?

Dr. Anthony Slonim: In oncology the research work has been focused primarily on - the use of the tool has been published mostly around, at least the literature I’m familiar with, has been focused mostly around the identification of cancer clusters as an epidemiologic tool to help folks understand where environmentally a toxic waste dump might be buried, for example, or where a stream may be contaminated with pollutants or high electrical wires - those kinds of risks that advanced cancer risk may be identified. That’s the literature I’m aware of in the oncology world.

That is also in the healthcare literature, not as much in the epidemiologic literature.

Alexa Doerr: Okay, that’s the last question in LiveMeeting right now.

Cori White: I’ll just give you a second to show that I’ve put up the slide with Dr. Battles’ contact information so if you have questions about the measures you’ll find his contact information as well as an email address on the slides that are up at the moment.

All right, well, let’s go to the phone then I guess.

Operator: Our first question from the line of Catherine Jones with the University of Nebraska Medical Center. You may proceed.

Catherine Jones: I have two questions, number one, if there are any organizations that conduct training in PRA for the frontline healthcare risk manager or patient safety officer? Then number two, if there’s any experience in applying PRA to less procedure intensive adverse events such as a fall?

Dr. Anthony Slonim: Great questions, I think while we were in between questions I took the liberty of actually doing a quick little search on socio-technical probabilistic risk assessment and you get a whole range of hits, not the least - which brings you back, in fact, to several AHRQ resources that allow you to understand where you might be able to get additional information on this. Always, Wikipedia pops up in there as well.

There are also in that - in that quick Google search there were a couple of places that were identified as for training so to answer that question, a quick Google search on probabilistic risk assessment got me to the training episodes. A couple of them were actually healthcare specific.

Then the question about the use of this for less procedure oriented work, yes, we have used this in a couple of areas. Again, we use this in some work that was funded by ASHRM, the American Society of Healthcare Risk Management. We actually looked across the organization. The idea was how could we improve our patient safety program? Recognizing that everyone has limited resources, the objective was to identify where we would have the biggest opportunity with the least resources to be able to improve care. So we modeled using our incident reporting system, the risks of our adverse events.

So let me give you an example, we took if you will, the numerator of medication errors over all of our medication doses dispensed and tried to figure out what the probabilities of that were and built it into the model.

We identified what the delays in care were from our incident reporting system and identified what were the multiple episodes of care to create a denominator opportunity so that we could get the probabilities of risk and incorporated that into the model.

So you can apply. It does not necessarily have to be a procedural-based application and, again, many of the applications outside of healthcare have not been procedurally based.

Operator: We have no further questions at this time.

Alex Alonso: Okay, if that’s the case you see here that we have Dr. Battles’ information up there. I want to thank everyone for participating today. Also, we’re going to switch back and provide you with our contact information again.

Tony, do you want to go ahead and provide your email address to folks so that they can contact you if they have questions?

Dr. Anthony Slonim: Absolutely, sure. It’s TSlonim@ahm.com.

Alex Alonso: Great. If you want further information for contact Dr. Slonim you can also reach us at TeamSTEPPSwebinars@air.org. For further information about TeamSTEPPS you can try the resources onscreen right now, TeamSTEPPS.AHRQ.gov is the website and you also have the DODpatientsafetyuses.mil website for information about the TeamSTEPPS program.

For points of contact at AIR regarding TeamSTEPPS you can contact Project DirectorDeborah Milne at either the email or the telephone number there. You can also contact myself or Alexa.

I want to thank you all for being here today and look forward to seeing you in May when we continue our webinar series with a sharing of success about TeamSTEPPS and through the TeamSTEPPS National Implementation Program. Thank you and have a good day.

Operator: Thank you ladies and gentlemen, that does conclude the conference call for today.

We thank you all for your participation and ask that you please disconnect your lines.

Have a great day everyone.

END