The cost of Moray fitness card is leaving many less well off families trailing in its wake, a councillor has claimed.