my issue: I mod a small (4 runners) game that is on 4 platforms (PC, Switch, Xboxone, PS4). I did a rough timing of the loads on 3 of the platforms (12s-22s-34s), Should such a small game be concerned with removing those load times? if not, when would we be concerned with removing them.

for a more general question, how does a game decide when to remove load times?

Ask the community. If the community is concerned about load times then ranking the leaderboards based on load-removed time should be looked into.

Though it definitely should be considered that the moderators are probably going to have to manually re-time every run that comes in to get the load removed time (at least for the console versions, for the PC version you should look into creating a load-remover plugin for LiveSplit to make everyone's life easier).