I can't think of any approaches to improve the performance other than the ones I've already tried. I have a sneaking suspicion that I could do a better job with the architecture (the way the 2D inputs are split/combined in the model) and perhaps improve the loss calculation (how the two losses are combined).