List of Papers By topics Author List
| Paper Info | Reviews | Meta-review | Author Feedback | Post-Rebuttal Meta-reviews |
Authors
Sasan Matinfar, Mehrdad Salehi, Shervin Dehghani, Nassir Navab
Abstract
We introduce a general design framework for the interactive sonification of multimodal medical imaging data. The proposed approach operates on a physical model that is generated based on the structure of anatomical tissues. The model generates unique acoustic profiles in response to external interactions, enabling the user to learn about how the tissue characteristics differ from rigid to soft, dense to sparse, structured to scattered. The acoustic profiles are attained by leveraging the topological structure of the model with minimal preprocessing, making this approach applicable to a diverse array of applications. Unlike conventional methods that directly transform low-dimensional data into global sound features, this approach utilizes unsupervised mapping of features between an anatomical data model and a sound model, allowing for the processing of high-dimensional data. We verified the feasibility of the proposed method with an abdominal CT volume. The results show that the method can generate perceptually discernible acoustic signals in accordance with the underlying anatomical structure. In addition to improving the directness and richness of interactive sonification models, the proposed framework provides enhanced possibilities for designing multisensory applications for multimodal imaging data.
Link to paper
DOI: https://doi.org/10.1007/978-3-031-43996-4_20
SharedIt: https://rdcu.be/dnwOU
Link to the code repository
N/A
Link to the dataset(s)
N/A
Reviews
Review #1
- Please describe the contribution of the paper
Authors proposed an interactive sonification model for multimodal medical imaging data to aid the navigation and ROI selection tasks. Preliminary results also reveal the feasibility and usefulness of the proposed sonification technique. The model is surprisingly simple yet effective!
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
1 - The document is very well written and structured. Only minor formatting issues and a few clarifications are required to ensure publishing quality.
2 - A novel and simple interactive sonification model for multimodal medical imaging data is proposed.
3 - Preliminary results reveal the effectiveness of the proposed sonification model.
4 - Preliminary results reveal a research path in this topic-
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
1 - mathematical formatting must be revisited
2 - Figure 1 could be improved to provide greater clarity regarding topology concepts
3 - this work is more a technical note as it considered a single abdominal CT volume; more case studies are welcome
4 - some technical details are missing.
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
In general, the numerical steps are easy to follow. However, the image pipeline was not entirely clear (for instance, how were the images registered to each other?) nor has any software tool and/or programming language been mentioned.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
ABSTRACT
(MMIMS), as a general –> (MMIMS) as a general
KEYWORDS
consider removing “Augmented Reality” from the list of keywords as it becomes redundant (“Additive Augmented Reality” is enough).
INTRODUCTION
simultaneously, i.e. when –> simultaneously, i.e., when
1.1 Sonification and State-of-the-Art in Medical Domain –> 1.1 State-of-the-Art of Sonification in the Medical Domain
Section 1.1 starts with a definition of sonification that is hard to follow. Please rephrase the sentence and provide a more clear definition.
Avoid the use of “etc.”; “data exploration, etc.” –> data exploration, among others.
Please clarify why the “issues of integrability and pleasantness of the resulting sound signal” are arguably the most significant challenges in surgical sonification. This last sentence of sub-section 1.1 is very vague.
there is no need for two separate sub-sections, consider fusing sub-section 1.1 and 1.2 together as the division just breaks the reading fluidity.
2 Towards Model-based Sonic Interaction with Multimodal Medical Imaging Data
Consider the following format for introducing the contributions of your work: instead of ‘Contribution’ as a sub-title just place it inline with the paragraph followed by a colon (all bold text) —> Contribution: This paper presents a novel (…)
3 Methods
domain and m, the dimension –> domain, and m the dimension
A sequence of physics-based sound signals –> A sequence of physics-based sound signals, S, expressed as
Figure 1 does not explain the concepts of ‘input mass’ and ‘output mass’, nor how topology is built based on image data. Only later in the text, when the reader reaches sub-section 3.2, does it becomes clearer …
3.1 Physical Model
Newton’s 2nd law and following equations are unappropriatly formatted: please consider using vector notation as the adopted format represent all variables being scalar.
Consider removing “F = m.a = m.d2x/dt2.” from the text because it is redundant (the community is aware of such basic physics knowledge).
recurring typo: mass/-es; please rectify.
3.2 Experiment and Results
as shown in 2. –> as shown in Figure 2.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
7
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
A surprisingly simple and interesting technical contribution is shared in this report, which paves way to an interesting research path. Moreover, it is a very well written document with just a few minor revisions to consider.
- Reviewer confidence
Very confident
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Review #4
- Please describe the contribution of the paper
The authors describe a strategy for transforming spatial features of multimodal medical imaging data (CT) into a physical model capable of generating distinctive sound. The physical model captures many complex features of the data (shapes, textures, etc.) and converts them to sound, with the goal of enhancing the ability of a physician to rapidly interpret complex anatomical data and structures.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
As current augmented reality guidance systems continue to incorporate additional visual virtual stimuli in the form of augmented virtual entities, the idea of sonification of complex 3D medical imaging data as a means for communicating tissue type and structure to augment a physician’s performance is quite interesting and novel. The authors present a detailed description of their proposed mass-interaction physics methodology to enable the formulation of physical models from 3D data and describe how wave propagation is influenced by the model topology and structure of inter-mass connections. As a proof of concept, the authors present spectrogram results (and a supplemental video) demonstrating the sonification of an abdominal CT dataset and show different sound profiles across multiple tissue types.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
A weakness of this work is in the lack of discussion surrounding relevant clinical use cases (I can certainly envision a few). The authors do mention the possibility of an auditory equivalent of 3D visualization for medical imaging, enabling a more immersive and intuitive experience. Another opportunity could be in the sonification of intraoperative medical imaging data (ultrasound or fluoro) to augment the visual information which a physician would receive, alerting them as to the tissue type or structure which their surgical instruments are approaching. One other point of discussion/limitation of this work is with regards to generalization of the model-based sonification approach across different imaging modalities.
- Please rate the clarity and organization of this paper
Excellent
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
The authors do not provide code or relevant data from their work. They do provide a detailed description of their mathematical derivation and approach and supplemental video demonstrating their algorithm performance.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
For an imaging modality where there is less consistency with intensity values across scans (like MRI), how consistent and repeatable would the sonification of tissue structures across different patients be? When sonifying complex 3D medical data, the spatial location of the sonified tissue structure is equally important to communicate – how do you envision conveying this spatial information to a physician? In the accompanying supplemental video, I had a difficult time distinguishing between heart, liver, and muscle tissue based on the audio. Similar to how color perception is very subjective, do you anticipate different physicians will interpret/perceive the sonified information differently?
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
7
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
Novel idea, well written manuscript, detailed description of the methods, and interesting proof of concept results. Excited to see where this work goes.
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Review #2
- Please describe the contribution of the paper
The paper tackles model-based sonification with a novel approach. The authors propose an unsupervised, high-dimensional multi-modal based sonification approach. Proposed method relies on multi-modal imaging data and a topology definition, coupled with a user interaction applied on a certain position in the imaging space, all fed into a physical model generating sound profiles based on this user interaction, through time. Experiments results design aim to validate the generation of discernable sound profiles correlated to the underlying anatomical structures. An abdominal CT volume was taken in consideration, physical model was excited through equal forces applied to the model center. 3D isotropic cubic regions of interest, from the heart passing through the lung and ending in the air. The mel spectrogram used for visualization of the generated sound profiles suggest that distinguishable sound profiles are generated per tissue types.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
Previous works, such as parameter-mapping sonification (PMSon) and model-based sonification (MBS) focus on sonification of imaging data embedded to a low-dimensional, suffering from too much abstraction of the underlying data patterns. Moreover, current state-of-the-art approaches rely on defining case specific mapping functions from imaging data to sound profiles, making it a laborious process. To remediate to these limitations, the authors propose an unsupervised, high-dimensional multi-modal based sonification approach. The proposed approach does not suffer from previous state-of-the-art limitations, regarding the use of low-dimensional imaging data and case specific physical model fine-tuning. The approach is novel and could potentially be useful.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
Studied approach relies on underlying imaging data intensities, however a study of the variability and robustness of different sound profiles per tissue type was not studied across vastly different imaging modalities, such as MRI. Additional experiments could have been done in that regard. Empirical and case-specific topological definition is still necessary for case-specific imaging, making it case-specific and potentially tedious, as mentioned by the authors.
- Please rate the clarity and organization of this paper
Satisfactory
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
Physical model definition is presented, however full derivation of the parameters is to be desired. The interaction module especially is not mathematically defined, more precisely regarding the mapping of the input data intensities to parameters of the mass-interaction system.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
- I suggest the author could disuss the solution of simplify the process of the empirical and case-sepecific topological definition.
- I understand some information is included in the video. I think it is still better to have some detailed method in the paper.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
5
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
Approach is novel compared to SOTA. I think it could be a potential approach. However, the empirical and case-specific topological definition is still required, makes the process tedious. Also, the experiment is not sufficient to evaluate the proposed method. I hope the authors could improve it during rebuttle. If so, I think it is worth to accept the paper.
- Reviewer confidence
Very confident
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Primary Meta-Review
- Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
The authors propose an unsupervised, high-dimensional multi-modal based sonification approach that maps tissue to sound with the aim of enhancing the ability of a physician to rapidly interpret complex anatomical data and structures. The model works by capturing features from the data (shape, texture, etc.) and converts them to sound.
All reviewers agree that this is an interesting work and a novel approach to medical sonification that would be of interest to the MICCAI community. I urge the authors to address the minor comments of the reviewers in their resubmission (example of clinical use cases, generalization of model, details from video added to paper, definition of model, etc.)
Author Feedback
Dear Reviewers,
We would like to express our sincere gratitude for your constructive comments and recognition of the potential contribution of this research to the MICCAI community. Your insightful feedback has provided valuable suggestions to further enhance the quality of the manuscript. We will improve the camera-ready version considering your comments regarding the discussions, method descriptions, mathematical formattings, and figures.
Best regards, The Authors