Looks like the hot swap fan could use a “gasket” between the case and the fan.
Ie, you want the fan to draw air out of the hot swap case, not from the ambient.
Some strategicly placed double sided foam tape may make all the difference.
Airflow is a funny thing. For example to have my fan wall working well, I sealed all the unused chassis screw holes forward of the fan wall. It’s about making sure the fans draw air through the path you want. Not the easy path.