MFU Calculation has potential precision issues #364

The test_compute_mfu test had a hardcoded expected_mfu of 0.4275 given a world_size of 8 ranks and 532 samples/s throughput. Varying the world size therefore lead to failing the test, as we were not compensating for different world sizes.

For instance, if the same throughput would have been achieved on a single GPU, we would exceed the GPUs peak performance resulting in a MFU of 0.4275*8=3.42. Note, wile this would be infeasible in practice, the calculation would be still correct.

I adapted the code by replacing the expected_mfu with expected_single_gpu_mfu, to support arbitrary world sizes: d309a83

While the changes seem to be correct up to a small percentage, I had to increase the tolerance a bit. We should check where the discrepancy is originating from.