ROI Framework for Healthcare Simulation

Although the role of medical simulation in improving patient safety outcomes and healthcare education is well established, the Return on Investment (ROI) for healthcare simulation activities is sometimes harder to quantify. A recent complex article by Bukhari, Andreatta, Goldiez and Rabelon (2017) provides a detailed methodology for determining the monetary return on investment for medical simulation. The authors were interested in creating a new framework which could used to measure not only tangible and quantitative benefits of simulation but the intangible and qualitative benefits as well, which we cover today:

Costs and Benefits to Help Determine ROI:

1. Costs such as development, manikin purchases, maintenance and operations.
2. Quantitative Benefits such as reduction in training costs, reduction in errors  and faster time to competence.
3. Qualitative benefits such as improvement of patient safety, quality of care, employee satisfaction, the reputation of the organization, and others.

If a benefit can be converted to a monetary value, then calculation of ROI is easier.  The authors identified previous methods reported in the literature that have been used to convert qualitative benefits into tangible monetary values. Note, not all qualitative benefits can be converted to monetary value.  In this case, the benefits are listed as intangible benefits.

Methods from Literature Review:

1. Value Measurement Methodology (VMM)
   1. The methodology allows for a decision framework to be personalized and adapted to the specific requirements of a project.
   2. The authors determined that medical simulation has five values
      1. Direct value—benefit of simulation in training users
      2. Social value—benefit to society (e.g., quality of care, fewer complications, etc.)
      3. Operational value—decrease in length of stay
      4. Strategic value—patient safety culture, sustainability
      5. Financial value—increase revenue, reduced costs
2. Frost and Sullivan.
   1. This method pre-identified benefits/values from the list above by use of questionnaires and/or interviews of experts
   2. According to the authors this is the only method available for determining ROI for computer-based simulation.
3. Dr Jack Phillips approach in measuring the ROI in health care.
   1. This approach allows a monetary value to be placed on improvements/benefit from specific training.

Article Framework to determine ROI using a combination of the above methods.

The authors combined the methods above to create a new framework for measuring ROI for simulation.

1. Determine Total Costs:
   1. Project Costs
      1. Initial Analysis and Assessment.
      2. Implementation
      3. Support and Overhead
      4. Project Development
      5. Acquisition
   2. Operational Costs
      1. Materials and Supplies
      2. Maintenance
      3. Labor
2. Use VMM categories to identifying the key parameters and measures (note some categories are found in both quantitative and qualitative data.
   1. Quantitative Data
      1. Financial Value
      2. Social Value
      3. Operational Value
      4. Strategic Value
      5. Direct Value
   2. Qualitative Data
      1. Social Value
      2. Operational Value
      3. Strategic Value
      4. Direct Value
3. Transform qualitative and quantitative measures into monetary value and tangible values (Using Phillips methodology)
   1. Tangible Value – consisting of
      1. Quantitative Value above
      2. Qualitative data above that can be converted to a monetary value
      3. AKA Total isolated effects of the simulator
   2. Intangible Value
      1. Qualitative data that cannot be converted to monetary values but which has high credibility.
      2. Intangible value is reduced when the valuing process is:
         1. Too subjective.
         2. Inaccurate.
4. ROI = Net Benefit of Simulator/Total Cost of Simulator x 100.

This is a complex methodology with many considerations.  The authors provided an illustration of how they used their framework to determine ROI.

Illustration:

• The authors decided to apply their framework to an existing study by Andreatta, P, Saxton, E, Thompson, M, Annich, G. Simulation-based mock codes significantly correlate with improved pediatric patient cardiopulmonary arrest survival rates. Pediatr Crit Care Med. 2011;12(1):33-38)
  • The study showed that survival rates increased by 50%.
• However, the authors wanted to look at physician turnover. They used their framework to:
  • Identify the cost of the study
  • Consider the intangible factor of turnover rate of physicians
    • This has a strategic and financial impact on health care organizations.

Implementation of Framework

• Cost structure of the simulation training,
  • Included all costs such as pay for physicians, nurses, educators, simulation technicians as well as manikin costs, equipment costs, consumables, maintenance etc
  • Scenario development, designing assessment and evaluation tools etc.
  • Total cost $391,600.
• Identification of the key parameters to be included in the ROI assessment using the different VMM categories.
  • Direct – cost of orientation
  • Social – loss of experienced physician familiar with other staff
  • Operational – Training
  • Strategic – patient safety
  • Financial value – increased neonatal/pediatric CPA survival rate
  • Intangible value – “loss of organizational history, knowledge and expertise, disturbance of the morale of the remaining employees, and potential adverse publicity for the organization”.
• Conversion of qualitative and quantitative data to tangible values
  • Isolation of the effects of training to evaluate the ROI of the training compared to its cost.
    • The average cost of replacing a physician (From Waldman)
      • $36,743 hiring cost
      • $89,800 training cost
      • $43,250 average loss of productivity
      • Total $170,000
    • The average cost of replacing a nurse including
      • $1,635 hiring cost
      • $15,825 training cost
      • $10,026 average loss of productivity
      • Total cost to $27,500.
• The author’s’ assumption:
  • One doctor and one nurse would not leave every year for 4 years (no data provided)
  • Annual cost saving $170,000 + $27,500 = $197,500.
  • Four year savings = $790,000
• ROI(%) = $790,000-391,600/$391,600 = 101.7%

This article helps to identify the many facets that can be used to calculate an ROI. Converting non-tangible benefits into monetized values helps create an ROI that will be more likely be persuade those who control finances to support simulation. Additional articles on this topic recently published on HealthySim include “7 Steps to Make the Business Case for Healthcare Simulation” and “How to Pick Key Performance Indicators (KPIs) that Optimize Healthcare Simulation Outcomes“.

Read More About an ROI Framework for Healthcare Simulation!

Today’s article was guest authored by Kim Baily PhD, MSN, RN, CNE, previous Simulation Coordinator for Los Angeles Harbor College and Director of Nursing for El Camino College. Over the past 16 years Kim has developed and implemented several college simulation programs and previously chaired the Southern California Simulation Collaborative.

Have a story to share with the global healthcare simulation community? Submit your simulation news and resources here!