Trajectory Planning using Surgical Device Orientation in Ultrasound Guided Minimally Invasive Surgical Procedures

Minimally Invasive Surgical Procedures (MISP) are an exciting choice of option for patients where surgery is performed with localized anesthesia and small incisions to the skin. Surgeons still have to decide if this option is applicable to patients based on each prognosis. If applicable, patients would greatly benefit from the procedures can be carried out in a day care setting with faster recovery time and reduced risk from the use of general anesthesia. Ultrasound is one medical imaging modality which can be used to visualize the underlying tissue during MISP. This gives the surgeon a real-time view of the anatomy without ionizing radiation. Various surgical devices and tools are utilized in MISP. This paper presents a trajectory planning for surgeons using rotational and translation orientation of three-dimensional Cartesian coordinates. The artificial Intelligence software precision guidance system then provides surgeons the ability to pre-plan in realtime the execution of the procedure and improve the procedure's efficacy. Increased development of such software and hardware solution could lead to further adoption of MISP and after that increase the wellbeing of patients. Keywords—Minimally Invasive Surgical Procedures; trajectory planning; ultrasound


A. Minimally Invasive Surgical Procedures (MISP)
In recent years, surgeons (both juniors and seniors) are being trained to utilize ultrasound (u/s) to diagnose and immediately perform surgical procedures using Minimally Invasive Surgical Procedures (MISP). A common alternative to MISP would be the invasive surgical procedures which require large surgical incisions to remove the cysts and commonly carried out the in operating theatre under general anesthesia. Although such classical surgical incisions are appropriate in certain situations based on the surgeons' discretion, the use of u/s has been evaluated with promising outcomes in treatments such as breast biopsy [1], prostate biopsy [2], biopsy of cysts in the pancreas [3] and wrist biopsy [4]. Del Cura et al. [5] has reviewed a list of medically established surgical procedures utilizing u/s. Currently, a various medical specialist such as anesthetist, orthopedic surgeons, gynecologist, radiologist, and sonographers are utilizing u/s to guide them in their procedures. The current practice uses a free hand approach where no additional software or hardware tools are used.
As shown in Fig. 1(a), u/s probe generates a beam of Ultrasonic sound waves which bounce off human anatomy at a variety of speed and the resulting echo is picked up a highly sensitive microphone array. The image generated by u/s is shown in Fig. 1.(b).

B. Medical Problem Statement
U/s procedures can be categorised into two types which are aspirations or injections. Aspiration consists of the process of either removing liquid or semi-solid substance such as abscess or tumors from the body. A biopsy is one example of aspiration where a small part of a tumour is removed in order to check its characteristics. The second type of u/s guided procedure involves injections of medication into the human body. Fig. 2(b) illustrates the process of surgical injection. The challenge for the medical doctor would be to align the surgical instrument in one hand while holding u/s probe in his/her other hand. As depicted in Fig. 2(a), the instrument would only appear in u/s screen if held at the same plane as the plane of the narrow u/s beam shown by the horizontal midline. The positions of both hands can be noted in Fig.  2

C. Related Work
Various hardware and software solutions exist to guide the medical practitioner to reach their target in a precise manner, and these have been reviewed thoroughly [6]. Existing solutions pose a challenge which may not be practical from the surgeon's viewpoint. Hardware/ medical device based solutions include such as laser beam projection guide [7] and the mechanical needle guide [8]. Both these hardware-based solution do not incorporate a software trajectory projection hence unable to predict the direction of the needle path in u/s screen. A software solution exists to guide users to reach the target [9]. However, this system requires the patient to undergo a Magnetic Resonance Imaging (MRI) or Computer Tomography(CT) scan. An MRI scan can be costly and does not provide realtime data as the patient must be scanned before the execution of the procedure. A CT scan involves the use of ionizing radiation which may have side effects on the patient [10]. Surgeons then expect a solution which is purely based on u/s imaging and can predict a trajectory path for the instrument insertion before they insert the instrument into the skin.

A. Syringe Detection and Alignment
Syringe detection and alignment involved image capture via camera and subsequent image processing in detecting the syringe. It was challenging to detect the barrel of the syringe because of its variety of shapes and almost translucent plastic material. Such challenges were solved using a combination of methods as described in this section.

B. Circuit Attachment on Syringe
The first stage involved the detection of the presence of the syringe. This was achieved by mounting four (4) infrared (IR) Light Emitting Diodes (LED) on a striped Veroboard. A microswitch and a trimmer potentiometer were also mounted on the board. All circuit connections were terminated with solder to ensure reliable light emission. Three one point five (1.5) Volts model LR45 button batteries were used to power the LEDs. The circuit was then mounted on a 3D printed bracket and tied onto a syringe as shown in Fig. 3. An accelerometer was attached, and the corresponding tilt measurement was derived [11].

C. Camera Attachment on u/s probe
In order to detect the syringe, a digital Complementary metal-oxide-semiconductor (CMOS) camera was utilized as a camera input device. The camera that was utilized had a two-megapixel sensor without an IR filter. An IR pass filter was then affixed on the front of the camera lens using a custom-made 3D printed casing. Fig. 4 depicts the camera attachment positioned on the probe. It was necessary to use the lens without the IR filter in combination with the IR pass filter to allow only the IR light from the IR LED to be detected by the camera. This method was then able to increase the precision of the image processing methods. In order to formulate the trajectory of the needle, it was also necessary to calculate the distance from the syringe to the IR camera. The distance calculations were based on the methods proposed by [12] and [13] using a single camera. The camera transferred video feed data to a laptop computer with eight (8)

D. Camera calibration to solve lens distortion
It is known that camera lenses may have lens distortion because of manufacturing processes which would result in inaccurate measurements when the image captured is used for calculations of depth or distance [14]. In order to prevent this distortion, the camera was calibrated using a checkerboard calibration pattern which was printed onto a board. This ensured that the camera was able to capture images which are accurate regarding undistorted pixels especially on the edges of the image.

E. Image processing to detect the syringe
The image that was generated by the camera feed was processed in realtime using a custom developed software application, STIG_main. The raw image was processed with the threshold filter in order to create a black and white image. Subsequently, the contours on the image were detected. The number of contours present was counted, and if the number was found to be numbering more than one, then this application was able to detect the presence of the syringe. A global boolean variable for syringe presence was marked as positive. If no contours were detected, the global boolean variable was marked as negative to denote that no syringe was present.

F. Image processing to detect alignment of the syringe
Subsequently the circular features were processed further to derive the centroid of each circle. The four centroids were saved as a top circle, bottom circle, left and right circle. Calculations were carried out on these four circles to detect the alignment of the syringe. These calculations were termed as the kite algorithm named after the shape of four corners of a kite pattern for the location of the LEDs. If the patterns as shown in the flowchart in Fig. 5 were detected, then the syringe was interpreted as to be aligned and a boolean variable was flagged to be positive.

G. Determination of Angle and y-axis Intercept of Trajectory Line
An assumption was made that the needle does not bend during the procedure. In reality, only certain cases were noted where the needle curved during insertion while the doctor is performing the procedure. Given the values of tilt of the syringe (a1), camera (a2) and probe (a3), and the distance between the camera and the probe (d2); the angle and y-axis intercept of the trajectory line was derived using trigonometry as shown in Fig. 6. The line labeled as L1 was the final trajectory path that was calculated with the value c as the y-intersection using the Eq. 1. In three dimensional (3D) Cartesian, the z-value of both u/s probe and syringe must equate with the same value.

H. Evaluation of System using Calibration Images
In order to prove the functionality and usage of the system, it was necessary to evaluate it on calibration images. An image with multiple small boxes on the right edge and bottom edge was prepared. Each box had a white border and was filled with a black background. The system was then used to point to each box, and the angle at which the box was highlighted was noted.

I. Evaluation with Trajectory on u/s Images
The system was subsequently evaluated on a mock to setup the mimic human tissue prepared using the method for medical u/s training [7] with marble as a target. The system was also evaluated using u/s acquired from 50 MISP procedures with informed consent from patients and doctors.

A. Results on Syringe Detection and Alignment
The image that was captured by the IR pass camera was as shown in Fig. 7.(a). The image had four (4) circular features positioned in a pattern. Regardless of the pattern, if any one of the circles were visible on the image, then the scene interpretation algorithm interpreted it as the syringe was present. The pattern analysis of the kite algorithm was as shown in Fig. 7.(b).

B. Results on Evaluation of System using Calibration Images
The result of the testing of needle trajectory was as shown in Fig. 8.(a). The system output showed that the box at position 3 was detected with an angle of 14 degrees. The angle was verified with ImageJ, an image processing software developed at the National Institute of Health (USA) [15]. The verification is shown in Fig.8(b) using the angle measurement tool.

C. Results on Trajectory of Needle
The trajectory of the needle overlaid on agar is shown in Fig. 9. outlined as a cyan colored line. In Fig. 9, the target can be seen in the bottom right. The mock target consisted of the marble. The thin dark blue lines in Fig. 10 denote the tilt angle of the accelerometer positioned on the syringe.

D. Discussion on IR based Image Processing
Various methods to detect the syringe and alignment were attempted before the usage of the IR pass lens. It was observed that the room lighting in the doctor's procedure room was frequently changing because the doctor preferred to dim the room lighting during u/s procedure in order to focus his/her attention on the display shown on the monitor. Ambient lighting has been known to affect image processing and object detection [16]. The detection of the syringe was carried out using IR imaging method. This allowed a more precise and reliable method to calculate the position of the syringe accurately. The use of IR imaging also allowed certain image processing algorithms to be skipped and not utilized such as the color space separation algorithm and color detection algorithm. By leaving out the color processing algorithms, indirectly the application's overall speed and stability of the image processing was able to be increased. The use of the kite algorithm to calculate the deviation of the syringe from the midline of the probe was noted to be very efficient and highly precise.

E. Needle Trajectory
Needle trajectory was an important feature of this whole software where it allowed the user to preplan the direction they want to penetrate the skin. This then allowed them to move the syringe in various angles and positions and then to insert the needle in one go. Based on the three experimentations that were carried out, the results showed that the target was able to be detected accurately. The guidance system was an able to detect alignment of both planes of u/s beam and instrument in 3D space. It has been noted that novice medical practitioners would tend to have multiple attempts of needle insertions before the correct insertion of the needle to reach the correct target and this practice may result in injuries to the other surrounding anatomy such as injury to the nerves [17]. It was noted that one of the errors was an incorrect choice of entry site and angle.

F. Surgical Instrument Detection
During the procedure, instrument detection guides the medical practitioner to see the whole instrument and its tip. The tip of an instrument such as the needle is usually sharp and may cause damage if it hits any unintended anatomy [18]. The display of the needle in a clear contrast to the rest of the anatomy also gives the user a sense of localization which is a primary important aspect of the procedure given the fact that u/s beam is guiding the user all throughout the procedure. This is in contrast to visual eye inspection, such as experienced in open surgery. The localization of the instrument using u/s helps to complete the treatment as planned and subsequently to be able to withdraw the instrument out of the tissue after the target is reached.

G. Robotic Needle Trajectory
There have been attempts to develop a needle trajectory system which have utilized using robotics systems [19] [20]. Robotic methods even allowed trajectory planning using a curved path [21]. Another novel attempt utilized a projector to project a target image onto the skin of the patient to give the surgeon a live view of his/ her trajectory [22]. The utilization of the robotic arms in clinical problems may seem as grandeur progress, but robotic solutions are still experimental and have not been tested in human subjects due to the risk of injury by malfunctioning movement of the robotic arm. The use of robots in clinical settings requires extensive human-robot collaboration studies before ethical approval for clinical tests may be acquired.

IV. CONCLUSIONS
In order to further substantiate the solution as presented in the article, effort should be carried out to validate the system by using the established clinical trial methods. In the medical field, clinical trials are a routine method to validate the use of new drugs or new medical devices. There exist guidelines to for medical software development [23], and these should be adhered to with a stringent policy in order to derive a solution which is stable and with risks mitigated [24]. The medical doctors especially the surgical community would also be more receptive to the system is presented with results from a systematic clinical trial. This would then lead to the increased adaptability of u/s guided MISP indirectly benefiting the patients who undergo such treatments.
ACKNOWLEDGMENT U/s images published in this research were obtained from XXX Specialist Centre, XXX with the informed consent given by participating patients for research and publication use and with medical ethical committee approval RSC/EC/2017/3796. The results of the doctors testing the prototype u/s guidance device have been included with informed consent and with XXX University's ethical committee approval XXXREC 2017/047 for human subject testing. The authors thank XXX Pte Ltd for provisions of research equipment.