At low levels the Dex isn't going to be really fast anyway. It takes a while before anyone gets fast.
...if you enter battle with Cloud at 31 Agi and Barret at 24 Agi,
then the Normal Speed would be:
Normal Speed = RU((31 + 24) / 2) + 50
= RU(55 / 2) + 50
= RU(27.5) + 50
= 28 + 50
= 78
...
Your party members get the following increase:
Turn Timer Increase = [(Total Dex + 50) * V-Timer Increase / Normal Speed]
While monsters get this per tick instead:
Turn Timer Increase = [Dex * V-Timer Increase / Normal Speed]
It actually takes more higher-dexterous party members for Barret to get noticeably slower and could be disadvantageous if there are too many lower dex allies. V-Timer is only dependent on the battle speed and haste/slow/stop.
For Barret to be half as fast as Cloud, if it were just the two of them, Cloud's Dex would have to be 52 and Barret's would have to be 1. Cloud won't reach this until 56 at the earliest without sources.
In conclusion, don't expect Barret to be extremely slower than anyone else, but he will be slower.