Adding weight may slow the acceleration rates and cornering potential of all the cars. But it will do nothing to equalize the performance within the class as you are suggesting.
Some of my Citation in other classes carry as much as 50 lbs. ballast. That much ballast gives one the potential to optimize the weight distribution. Getting the weight distribution optimized and the CG lower can give quite an advantage over the competition. That is fact not speculation.
Leave the weight where it is.