A novel 3D atomistic-continuum cancer invasion model: in silico simulations of an in vitro organotypic invasion assay

We develop a three-dimensional genuinely hybrid atomistic-continuum model that describes the invasive growth dynamics of individual cancer cells in tissue. The framework explicitly accounts for phenotypic variation by distinguishing between cancer cells of an epithelial-like and a mesenchymal-like phenotype. It also describes mutations between these cell phenotypes in the form of epithelial-mesenchymal transition (EMT) and its reverse process mesenchymal-epithelial transition (MET). The proposed model consists of a hybrid system of partial and stochastic differential equations that describe the evolution of epithelial-like and mesenchymal-like cancer cells, respectively, under the consideration of matrix-degrading enzyme concentrations and the extracellular matrix density. With the help of inverse parameter estimation and a sensitivity analysis, this three-dimensional model is then calibrated to an in vitro organotypic invasion assay experiment of oral squamous cell carcinoma cells.