Yes it all depends on how much float is used.
There are experts wich made benchmarks on it and off course then with software specially written to accentuate differences. Mostly those programs come from cpu manufacturers . (in dutch they say ge moet daar een korreltje zout bij nemen )
However what apart from such tests is true.
If You have an fpu and use sofloat some weird issues may happen
If You do not have a fpu and do use hardfloat some weird issues may happen.
In general the performance in both cases does not make that big difference.
Like You said
With fpu use hardfloat to squeeze max performance out off the binary s and also the sometimes weird things do not happen.
Without fpu use sofloat.
There was even somewhere a article about it from Linuz Torvald. Yes he can be very radicale. I can't find it anymore.
But he wrote to stop with those emu's and hacks in core to cover the non fpu processors and just use the right compile settings hard or softfloat end off line.