Atharva Peshkar1*,
Danna Gurari1,
Sergi Pujades2,
Michael Black3 ,
David Thomas4

1 University of Colorado Boulder, USA, 
2 University of Grenoble, France,   
3 Max Planck Institute for Intelligent Systems, Germany, 
4 University of Colorado School of Medicine, USA. .

To enhance the accuracy of surface-guided RT (SGRT) for abdominal SBRT by designing an artificial intelligence (AI) enhanced computer-vision (CV) patient setup technique that predicts skeletal anatomy from surface imaging.

We have designed a modified SGRT technique, 'avatar guided-RT' (AgRT), that employs patient-specific "avatars” based on the recently published Sparse Trained Articulated Human Body Regressor (STAR) model. STAR is a realistic 3D model of human surface anatomy learned from >10,000 3D body scans that considers gender and BMI for pose-dependent surface variation and can be fitted to CT-based surface contours or surface meshes acquired from 2D video/depth images. We utilize a pre-existing neural network trained on 2,400 soft-tissue/skeleton training pairs obtained from dual-energy X-ray absorptiometry (DXA) scans to predict the skeletal anatomy from the body surface of a patient in treatment position.

AgRT was tested using a calibrated multiple camera system. Real-time 3D pose extraction from multiple 2D images was tested in a virtual treatment room to optimize camera numbers and positions. Testing was then conducted on a healthy volunteer to track various treatment poses. The patient's 3D pose was mapped to an avatar with matching gender and BMI. The skeletal alignment technique was assessed on XCAT phantom data and retrospective patient CTs. Skeletal anatomy was predicted from surface imaging with sub-cm accuracy.

Real-time acquisition of 3D human pose and shape is feasible using video inputs and CT data. Inferring the skeletal anatomy can enable alignment to a patient’s X-ray imaging and improve the correspondence between surface imaging and internal anatomy. Realistic body modelling in SGRT can potentially address issues caused by insufficient surface anatomic variation that can lead to poor correlation and large random errors in current SGRT techniques. Considering patient gender, BMI, pose, and body type can enhance SGRT's accuracy and reliability.

Surface Guided Radiation Therapy (SGRT) techniques have been introduced to improve patient alignment without increasing imaging dose. However, current SGRT techniques are limited in their accuracy to track internal tumor/organ motion when the target is not close to the surface. Additionally, higher BMI has been linked to decreased accuracy of SGRT, raising a health-equity issue for patient populations with already reduced outcomes. Current SGRT matching algorithms underestimate motion of internal anatomy for flat chested patients and overestimate motion of internal anatomy for large BMI patients. Realistic body modelling will overcome current issues with lack of anatomic variation which can lead to poor correlation and large random errors in SGRT.

Figure 1 shows an example of skeletal misalignment between days of treatment for a pelvic patient based on a typical SGRT matching algorithm, due to the limited field of view available to the SGRT algorithm. Realistic full- body modelling has the potential to overcome current issues due to lack of surface anatomic variation which can lead to poor correlation and large random errors. 

Figure 2 (a) camera system was synchronized and calibrated using a 50cmx50cm checkerboard to account for intrinsic and extrinsic camera distortions. (b) real-time 3D pose was extracted from multiple 2D images of a healthy volunteer in various treatment positions. (c) each pose was calculated and mapped to a 3D avatar with matching gender and BMI.

This project consists of two major innovations. Our first innovation is the use of a novel patient-specific skin mesh model to enable marker-less 3D human anatomy tracking. Our patient-specific skin mesh model is based on a recently published Sparse Trained Articulated Human Body Regressor (STAR), which is a realistic 3D model of the human body learned from >10,000 3D body scans. It allows estimation, synthesis, and analysis of 3D human pose and shape with sub-centimeter accuracy (Fig. 2). Our second innovation is an improved internal to external anatomical model that fuses patient specific body models to their internal skeletal anatomy, improving the correspondence between surface imaging and internal anatomy. We have adapted the technique outlined in Keller et al, 2022, using a neural network trained on 1000s of dual-energy X-ray absorptiometry (DXA) scans to predict the skeleton from the body surface when patients are in the treatment position (Fig. 3b-c). Accounting for patient pose and body type will be particularly valuable for overcoming two health-equity issues of current SGRT techniques, 

Figure 3 (a) a virtual patient positioned on the treatment couch in a virtual treatment room environment. (b) one of five virtual camera views monitoring patient pose. (c) the overlay of the 3D avatar on the 2D image in (b). (d) real time tracking of avatar, couch, and gantry motion during treatment.

Figure 4 (a) Flow chart showing the method for inferring the skeleton from surface. (b) example of a surface acquired from CT. (c) model input data. (d) optimization of skeletal position. (e) inferred skeleton from (a). (f-g) method showing patient in treatment position in the virtual treatment room. (h) breathing motion can be extracted from skeletal anatomy; overlap of deep inhalation vs exhalation from the XCAT phantom shown.

3D human pose and shape can be acquired in real time with sub-centimeter accuracy from video inputs (Fig 1-2). Skeletal anatomy can be predicted from surface imaging to enable alignment to a patient’s day-of-treatment cone-beam CT (CBCT) and improving the correspondence between surface imaging and internal anatomy (Fig 3). This technique can be also used to extract breathing motion for gating purposes as shown in Fig 3(h), with clear differences in the rib positions between deep inhalation and exhalation (XCAT phantom data shown).