As far as I know, there is no requirement that gets triggered based solely on # of units above first floor.
You didn't say whether this is private housing or public housing.
If you are doing publicly funded housing, then ADA would require minimum 5% of your total # of units (including one of each type of unit offered (e.g. 1 bedroom 1 bath, 2bed/1bath) to be mobility accessible, and on an accessible route. In that case, as long as you could get 5% of your total # of units (including one of each type of unit) to be on the ground floor and be mobility accessible, then you would not need any units on the upper floors to be mobility accessible.
However, if you decide to install an elevator for other reasons (such as because it qualifies as a high rise per IBC 403.6.1, then the presence of the elevator triggers all floors having FHA covered multifamily adaptable units. And if it was public housing ADA would require dispersion of the ADA accessible units, so you couldn't just put all 5% on the ground floor like you otherwise could in a nonelevator building.