Mixed and Augmented Reality Can Facilitate Seamless Medical Communication

ar-healthcare-simulation

MedicalResearch.com recently interviewed Birmingham City University Associate Professor Dr. Ian Williams PhD about the work of the DMT lab on mixed and augmented reality for healthcare simulated training. Make no mistake, VR and AR are the future of healthcare simulated training:

Dr. Williams: Our work at the DMT Lab (dmtlab.bcu.ac.uk) focuses on developing a novel Mixed Reality (MR) medical presentation platform which allows practitioners to interact with patient data and virtual anatomical models in real time. The system enables the presentation of medical data, models and procedures to patients with the aim of educating them on pending procedures or the effects of lifestyle choices (for example the effects of smoking or excessive alcohol consumption).

The system employs an exocentric mixed reality environment which can be deployed in any room. It integrates a medical practitioner in real time with multimodal patient data and the corresponding result is a real time co-located visualisation of both the practitioner and the data, which they can interact with in real time.  We implement a natural interaction method into the system which improves a user’s level of direct interaction with the virtual models and provides a more realistic control of the data.

The system can also be used in a fun educational setting where patients, students, children or any naive user, can learn about medical anatomical information via a real-time interactive mixed reality “body scanner”. This fun system overlays the MR information onto their own body in real-time and shows them scaled and interactive virtual organs, anatomy and corresponding medical information. We are aiming for this system to be used not only in patient education but also in engaging and informing people on lifestyle choices.


Sponsored Advertisement:


MedicalResearch.com: What types of medical or surgical problems do you envision can be enhanced with the use of free hand gestures to manipulate patient data?

Dr. Williams: Mixed reality has enormous potential within the medical field, with healthcare being profoundly affected by some recent developments. Mixed reality technology can also provide the platform for facilitating a seamless doctor-patient communications in real time. The system we are developing can provide a real time augmented view of the patient’s data which can be overlaid onto the patient, or interacted with via freehand interaction without the use of complex wearable devices.

Many current mixed reality systems rely on bespoke sensors and cumbersome wearable devices (for example haptic gloves) whereas we work in freehand interaction without the need for a medical practitioner or patient to wear any complex wearable device. This interaction method enables a more natural virtual interface and via the use of naturally inspired physical interaction models (for example common real grasping types) we bridge the gap between users and technology. This form of natural interaction can also enable an interaction which can be perceived as more realistic to the observer.


Sponsored Advertisement:


Serious Games Conference Features Future of AR and VR in Healthcare – July 18-20 VA

ar in healthcare simulation

Is your program or institution looking specifically at VR and AR for healthcare applications? Check out this Serious Play Conference July 18-20 at the George Mason University!

Use of VR for Game-Based Learning Growing; Healthcare, Govt, Education Apps at Serious Play Thought leaders leading sessions at Serious Play Conference in July will share their experience using Virtual Reality to create education and training games. Speakers at the gathering, hosted by the Virginia Serious Game Institute (VSGI) at George Mason University’s Science and Technology Campus outside DC include many speakers and topics such as these presentations focusing on healthcare:

  • Mobile VR & AR Games for Healthcare by David Metcalf Institute for Simulation and Training, UCF
  • Clinical Tools VR for Complicated 3D Healthcare Structures Bradley Turner
  • How VR and AR Will Revolutionize Healthcare by Walter Greenleaf Stanford School of Medicine

Serious Play is a gathering where creators and learning professionals can have critical conversations about game design requirements and share their knowledge with peers. The focus of the conference is exploring opportunities, challenges and the potential of game-based learning. Their goal is to provide a forum for visionary educators, chief learning officers and heads of training programs in health care, government/military or other fields that want to learn how to improve the effectiveness of their program, and use the data collected to do even better.

Healthcare Track Sessions Include:

  1. Thomas Talbot USC Institute for Creative Technologies: Time to Leave the Lab, What Will it Take to Make Useful Games Viable for People and Businesses?
  2. David Metcalf Institute for Simulation and Training, UCF: Mobile Games Developed for Military Healthcare Training
  3. Scott Simpkins Applied Physics Laboratory Johns Hopkins: Using Games to Improve Clinical Practice and Healthcare Administration
  4. Alexander Libin Georgetown University: Predictive Analytics for an Embedded Assessment Framework: Developing Data-based Multimedia Technologies
  5. Students From Univ of Washington iSchool with Ran Hinrichs: Serious Play in Government Leadership Training
  6. Kevin Holloway Center for Deployment Psychology Uniformed Services Univ Of the Health Sciences: Virtual Professional Training in Evidence Based Psychotherapies, Gaming for Behavioral Health Providers
  7. Ran Hinrichs 2b3d Studios: Using Games to Study the Psychological Impact of Military Deployment
  8. Walter Greenleaf Virtual Human Interaction Lab Stanford University School of Medicine: How Virtual and Augmented Reality Technology will Revolutionize Healthcare
  9. Brad Tanner: Clinical Tools 3D Virtual Reality Using Oculus to Teach Complicated 3D Structures in Healthcare
  10. Doris Rusch DePaul University: Integrating Academia, Healthcare Professionals and Patients to Create a Learning Game for Chronically Ill Patient Diseases
  11. Dmitriy Babichenko Lorin Grieve, Jonathan Velez University of Pittsburgh: To Scope or Not To Scope: Challenges of Gamifying Clinical Procedures Training
  12. Kimberly Hieftje Yale Center for Health & Learning Games: Re-purposing Serious Games: Making the Development Time Count Twice (or More)
  13. Kenneth Bibbins PrepWorld Philliph Mutisya NC Central University: Trauma Informed Game Based Learning for Kids
  14. Beth Rogozinski Pear Therapeutics: The Challenges of Creating Mobile Games for Regulated Health Situations
  15. Lisa Marriott OHSU/PSU School of Public Health: Working with Local Schools on Nutrition Education
  16. David Wortley GAETSS, UK: Trends in Serious Games for Health and Well-Being

Learn more and Register on the Serious Games Website today!

 

CAE Healthcare Leads Innovation at IMSH with New Games, Eyes, and AR Technologies | IMSH 2017 Video Interview

augmented reality ultrasound trainer

Furthering HealthySim’s coverage of the IMSH 2017 trade show floor, today we are sharing our exclusive interview with CAE Healthcare to learn more about their latest innovative healthcare simulation technologies including simulated eyes and augmented reality ultrasound trainer! Today we are seeing three new innovative products from CAE Healthcare: simulation-based anesthesiology training in a gaming-style online learning environment, the future of healthcare patient simulator eyes, and the first holographic computer that allows you to freely move and interact with your healthcare simulation learning environment. Watch our video interview below to see it all!

Screen-Based Anesthesiology Simulation Product Highlights

The American Society of Anesthesiologists (ASA) and CAE Healthcare have agreed to co-support the design and development of an interactive/gaming-style training product which will help learners enhance their medical skills. The screen-based simulation training product is intended to assist with improving performance in the management of anesthesia emergencies. The ASA screen-based simulation product will allow flexible, asynchronous training in a self-paced, just-in-time environment. This means users may engage with simulated, online interfaces in a totally immersive, 3D setting and progress at their own pace. Imagine the benefits to training when a student, or new learner, can “enter” an operating room in virtual reality, treat a virtual patient with varied symptoms and conditions, and have that patient respond as if it were all real. There is no risk of harm to a living person. The learner can stop and start the self-paced instruction at will, and develop progressive skills mastery according to their own ability.

The possibilities for improved learning and competency acquisition are endless. Other features include:

  • learning can take place in the comfort of the user’s own home, via their own computer
  • as learners enhance their clinical and decision-making skills, they can also see how they score in comparison to their peers
  • the product will include five (5) medical scenarios with realistic medical instruments and equipment for medical diagnostics and monitoring

See the Future of Simulation with SymEyes

The quality of the digital screen eyes was quite remarkable! Standard with CAE Healthcare’s Lucina High-Fidelity Maternal/Fetal Training simulator, you can now display even more realistic patient symptoms and conditions, including jaundice, hemorrhage, keyhole pupil, cataracts and bloodshot or droopy eyes. (Note: The flicker of the eyes was only due to the frame rate of our camera.) Our team was very impressed by the innovative thinking with these new eyes and believes they are the new standard for patient pupil display!

Augmented Reality Ultrasound Training

The expanded learning opportunities in healthcare with augmented reality are becoming increasingly clear. CAE Healthcare has reset the bar for augmented reality technology in healthcare simulation by combining Microsoft’s Hololens with their leading Vimedix Ultrasound trainer, designed to make learning more engaging and intuitive. The manikin-based system allows for the development of the psychomotor skills needed to handle ultrasound probes. The innovative software tools of Vimedix accelerate the development of the cognitive skills needed to interpret ultrasound images, make diagnoses and clinical decisions. With the Hololens addition, you can now gather your learners for a shared, immersive experience that will inform and delight. Our HoloLens-based solutions will inspire deeper understanding from the start, and awaken their imaginations to better treatments and tools to improve patient care.

Freed from its two-dimensional environment inside a monitor, our Vimedix ultrasound simulator leaps to life, displaying anatomy you can enlarge, turn, rotate or command to return into its manikin body so you can view the interrelatedness of all of its structures. Witness in real time how the ultrasound beam cuts through the human anatomy.

See all of these simulation technologies and more at the next CAE Healthcare HPSN event!


Sponsored Advertisement:


VimedixAR from CAE Healthcare Uses Microsoft Hololens for Futuristic Ultrasound Training

cae healthcare hololens ultrasound simulator ar

During our IMSH 2017 “favorite products” recap I shared about my first look at the Augmented Reality version of CAE Healthcare’s Vimedix Ultrasound simulator. The CAE VimedixVR ultrasound simulator leaps to life with the Microsoft HoloLens module. Freed from the limits of a two-dimensional environment inside a monitor, users can display, enlarge, turn, and rotate realistic-looking anatomical parts, or command them to return into the manikin body. Learners engage in augmented reality to view the inter-relatedness of internal structures, and witness (in real time) the ultrasound beam as it cuts through human anatomy. CAE representatives explained to me that that you can “gather your learners for a shared, immersive experience that will inform and delight as our HoloLens-based solutions will inspire deeper understanding from the start, and awaken their imaginations to better treatments and tools to improve patient care.”

This video is one of a great series you can check out on CAE Healthcare’s Vimeo page!

More from CAE about the Hololens

“We are on the cusp of a new frontier in simulation for healthcare,” said Dr. Robert Amyot, president of CAE Healthcare. “Augmented and virtual reality can accelerate learning and provide shared training experiences in a more immersive and engaging clinical learning environment. Our engineering team is just beginning to explore possibilities with the Microsoft HoloLens, and we look forward to offering it as a key training solutions technology to our industry partners.”

The CAE Healthcare team has already begun to develop training prototypes with the medical device industry that incorporate the Microsoft HoloLens and are expected to accelerate professional  education for new technologies.  With CAE Healthcare’s virtual views of human anatomy and the Microsoft HoloLens, physicians will be able to practice placing cardiac devices or implants with speed and precision before they perform procedures on real patients.

“At Microsoft our goal with HoloLens and mixed reality is to help customers visualize and interact with 3D content in ways that offer new possibilities for creation, collaboration and consumption of information,” said Lorraine Bardeen, General Manager, Microsoft HoloLens and Windows Experiences. “It is inspiring to see how CAE is integrating HoloLens into its healthcare simulation portfolio, and we are excited about the opportunities mixed reality presents to revolutionize the future of patient education and training through the use of holographic computing.”

The “aha” moment for me was being able to see my device insertion past the physical walls of the anatomy. Now I can see exactly what was happening outside AND inside the body at the same time! Genius!

Learn more and sign up for future updates about CAE Healthcare Hololens products here!

Virtual and Augmented Reality Market To Reach $162 billion by 2020

virtual-reality-utilization-engagement

Sim Champs before you know it AR and VR technologies will become a mandatory part of our healthcare educational programs. The opportunities to learn and train in high-cost risk-heavy environments in a safe and affordable manner will continue to expand through advanced learning technologies like augmented and virtual reality. HealthySim will continue to report tidbits of progress in this arena as its applications to healthcare become ever clearer. Today, we share the expectations of the industries growth to demonstrate the upward trend in utilization:

BusinessInsider.com reports that the total revenue for virtual reality (VR) and augmented reality (AR) is projected to increase from $5.2 billion in 2016 to over $162 billion in 2020, according to the IDC:

  • More than half of the revenue will come from VR/AR hardware sales. Additionally, service revenues are projected to increase over the period as demand grows for enterprise-class support. Software was also mentioned as a smaller, but notable revenue source, growing more than 200% year-over-year (YoY) in 2016. Nevertheless, services revenue will quickly surpass it, largely due to demand in the enterprise segment.
  • AR systems will ultimately contribute more revenue than VR systems. Games and paid content will be strong sources of revenue for VR systems, particularly in the next two years. However, this revenue will be eclipsed as AR systems are integrated into healthcare, product design and management-related uses.
  • Most revenue through 2020 will come from the US. The US, Western Europe, and Asia Pacific (excluding Japan) are projected to account for three-quarters of revenue for VR and AR. The US is projected to contribute a larger amount as time progresses.

The adoption of AR and VR headsets will be driven primarily by the introduction of less expensive models to the market, first powered by smartphones before mainstream adoption of stand-alone headsets. While early adopters will drive the initial wave of purchasing, sustainable growth will likely come from VR and AR app developers building a robust and engaging ecosystem of content that entices slower adopters. Lastly, as the underlying technology powering these devices increases, so too will the capabilities, creating new use cases in entertainment, workplaces, and education.

Read the full Market Report on BusinessInsider


Sponsored Advertisement:


Case Western Reserve, Cleveland Clinic Collaborate with Microsoft for Mixed-Reality Technology for Education

microsoft medical simulation

Shared from Case Western Reserve University:

Case Western Reserve University Radiology Professor Mark Griswold recently shared how “HoloLens” can transform learning across countless subjects, including those as complex as the human body. Speaking to an in-person and online audience at Microsoft’s annual Build conference, he highlighted disciplines as disparate as art history and engineering—but started with a holographic heart. In traditional anatomy, after all, students like Ghodasara cut into cadavers to understand the body’s intricacies. With HoloLens, Griswold explained, “you see it truly in 3D. You can take parts in and out. You can turn it around. You can see the blood pumping—the entire system.”

In other words, technology not only can match existing educational methods—it can actually improve upon them. Which, in many ways, is why Cleveland Clinic CEO Toby Cosgrove contacted then-Microsoft executive Craig Mundie in 2013, after the hospital and university first agreed to partner on a new education building. “We launched this collaboration to prepare students for a health care future that is still being imagined,” Cleveland Clinic CEO Delos “Toby” Cosgrove said of what has become a 485,000-square-foot Health Education Campus project. “By combining a state-of-the-art structure, pioneering technology, and cutting-edge teaching techniques, we will provide them the innovative education required to lead in this new era.”

Because the technology is relatively easy to use, students will be able to build, operate and analyze all manner of devices and systems. “[It will] encourage experimentation,” Buchner said, “leading to deeper understanding and improved product design.”

In truth, HoloLens ultimately could have applications for dozens of Case Western Reserve’s academic programs. NASA’s Jet Propulsion Laboratory already has worked with Microsoft to develop software that will allow Earth-based scientists to work on Mars with a specially designed rover vehicle. A similar collaboration could enable students here to take part in archeological digs around the world. Or astronomy students could stand in the midst of colliding galaxies, securing a front-row view of the unfolding chaos. Art history professors could present masterpieces in their original settings—a centuries-old castle, or even the Sistine Chapel.

“The whole campus has the potential to use this,” Griswold said. “Our ability to use this for education is almost limitless.”

Read the full Hololens article here!

 

SimX Augmented VR Glasses Win IMSH Serious Games Showcase | Video Interview

augment reality

At the IMSH 2015 HealthySim had a chance to interview Ryan Ribeira, CEO of SimX, and practicing resident physician in emergency medicine at Stanford. Ryan was showcasing SimX at the 5th Annual Serious Games and Virtual Environments Showcase and Arcade which took place at IMSH 2015 in New Orleans, LA and provided over 300 attendees demonstrations of more than 21 entries. The event, started by Dr. Eric B. Bauman, continues to attract exceptional talent from around the world and did a fantastic job of highlighting some serious technology advances in our field such as SimX – which walked away as the WINNER of the Small Business or Corporation category.  (Click here to visit TheClinicalPlayground.com for a full showcase awards recap).

More about SimX:

Virtual Patients: SimX’s software replaces your physical simulation mannequins with a customizable, high-definition, 3D virtual patient, that can be projected onto any empty hospital bed. Whether obese, pregnant, young, old, vomiting, missing limbs, bleeding, or expressing any number of other physical signs and symptoms, SimX’s software allows you to reproduce patient presentations with unprecedented visual fidelity.

Case Builder: Build complex cases in minutes using the powerful visual case building system. Drag & drop events onto the field, determine the environment, and set patient data with just a few clicks. Use SimX’s powerful case monitoring and feedback system to see the case from each trainee’s perspective, and adjust case parameters on the fly.

Global Case Marketplace: SimX allows you to access thousands of cases from top hospitals across the globe! Let your trainees learn from specialists at the cutting edge of their field. Tap into the expertise of your own simulation specialists. Market your cases to the world and turn your expertise into revenue.

Direct Integration: SimX is built so that your trainees can learn using the tools you already have right at your institution. You can use your own beds, monitors, ultrasounds, stethoscopes, even your existing simulation manniquins! It’s easy to ensure that SimX software will recognize your tools, and allow your trainees to use them in the simlation.

Cloud Based Software: SimX’s marketplace, case authoring tools, and moderator interface are all located on the cloud, so you don’t have to worry about downloads or compatibility issues. Write and run cases from your desktop, laptop, or even your mobile device! If it has a browser, it works with SimX.

Reporting:  SimX’s case authoring and moderating tools come with powerful reporting features built right in. Every order, request, and event is recorded on the case timeline, so your team can debrief and see how cases might have gone if their decisions had been different. Create powerful reports that can help you pinpoint where trainees need the most help, and can track the improvement of teams or individuals over time.

Learn more at SimXAR.com!


Supported Organization:


SimX Offers Augmented Reality Medical Simulations Through META 1 Wearable System

simx-meta1

*Jan. 2015 Update: Watch Our IMSH 2015 Serious Video Games Showcase Interview with SimX who won “best small business company”!*

‘The technology of medical simulation continues to rapidly evolve with a new announcement from Immersive wearable headset manufacturer Meta that their new “Meta 1” Augmented Reality goggles have recently started shipping. CNN says the Meta 1 “makes you a real life Tony Stark”. Below are some breakdowns about the hardware and software being developed specifically for medical simulations with a video demonstration recorded at TechCrunch: Disrupt. Augmented reality overlays 3d animations in real time over your physical envrionment. So in the photo above, SimX CMO Dr. Srihari Namperumal sees a virtual patient lying on the bed because of the QR code that is placed on the gurney surface. This provides learners with an opportunity to see virtual patients existing in real environments!

meta1

Key Features of the Meta 1 Device

  • True Scale Holograms: See the physical and holographic worlds merge through our 3D stereoscopic display in real size, depth and parallax.
  • Markerless Surface Tracking: Look around the room and watch as holographs stay anchored to physical tables, floors and walls – thanks to our low-latency, 360º tracking.
  • Natural User Interface: Grab, pinch and touch 3D objects in the real world, and drive a touch-based holographic user interface.

About SimX Augmented Reality Medical Simulation Software

“SimX’s software replaces your physical simulation mannequins with a customizable, high-definition, 3D virtual patient, that can be projected onto any empty hospital bed.  Whether obese, pregnant, young, old, vomiting, missing limbs, bleeding, or expressing any number of other physical signs and symptoms, SimX’s software allows you to reproduce patient presentations with unprecedented visual fidelity.

Build complex cases in minutes using the powerful visual case building system.  Drag & drop events onto the field, determine the environment, and set patient data with just a few clicks.  Use SimX’s powerful case monitoring and feedback system to see the case from each trainee’s perspective, and adjust case paramaters on the fly. “

Watch the demos above and learn more at the SimX and META websites!

Sheffield Hallam University Uses Augmented Reality to Increase Manikin Fidelity

augmented reality medical simulation

Sheffield Hallam University was recently featured on BBC news for their augmented reality system which increases realism of stagnate simulators by overlaying recorded standardized patient videos on top of the manikin. This is a step in the right direction towards increased fidelity for patient simulator engagements. Because this is such an innovative project, I have copied several links to additional information below:

From the Sheffield Hallam website:

Sheffield Hallam University has become the UK’s first higher education institution to use a new piece of cutting edge technology that assesses empathy and compassion in healthcare.

Augmented reality (AR) has been introduced into the University’s nursing and midwifery curriculum which sees videos of patients, played by actors, superimposed onto training manikins. The computer-generated images or video of the patient is overlaid onto the dummy via an iPad tablet and provides a ‘real’ account of the patient experience.

It is designed to give trainee nurses a range of scenarios to test their reactions and their patient communication skills. Jean Flanagan, assistant dean and head of nursing and midwifery, said ‘The introduction of augmented reality has been a hit with our students and staff and it has allowed us to realistically assess how our students are going to perform when they are out on the wards.’ Read the full article on the SHU website.

nursing simulation augmented reality

Click to watch the awesome BBC coverage of the AR system“Medical mannequins can be a useful training aid but interacting with them can feel unnatural. But at Sheffield Hallam University augmented reality is now being used to ‘turn’ the mannequin into a real person. The app contains various scenarios, recorded by actors, to make the situation feel more realistic. Once the introduction has been made using the app, the mannequin can be remotely controlled to continue reacting in keeping with the simulation.”

You can also download the presentation slides from the session entitled “Human Patient Simulation – Now with Added Reality” from HPSN given by Mel Lindley, Senior Lecturer in Physiotherapy within the Allied Health Professions department; Deborah Clark, Senior Lecturer within the Nursing and Midwifery department; Robin Gissing, Technology Enhanced Learning Advisor for the Faculty of Health & Wellbeing.

Finally, download this report entitled “Using Augmented Reality” from UK’s Council of Deans: “Student feedback indicates that learning was furthered by the interactive workbook being created by students for students as it enabled students to better understand the level of knowledge required for their stage of training. Annual course review and module evaluation have both demonstrated that students value this module above all others due to its level of interaction, realism and preparation for clinical practice.”