Section 508 is an area many infrequent code users do not fully comprehend and use to their advantage. The section is very flexible in that Section 508.1 states within the last sentence that the building must comply with "Section 508.2 508.3 or 508.4, or a combination of these sections."
That bold part is something that many code users do not take advantage of, albeit because it requires a little more effort and analysis. Therefore, you can have separated occupancies and accessory occupancies. As an accessory occupancy, the 10% floor area limitation is still measured against the entire floor area for that story and not the separated area within which it is located. However, the Group B occupancy would be included as part of the main occupancy area. So, if the Group A occupancy is the main occupancy, the floor area for the Group A occupancy would include the floor area for the Group B accessory occupancy and would be a part of the Group A occupancy ratio when calculating separated occupancies for the rest of the building.
This situation can also be considered a combination of separated and nonseparated occupancies (still acceptable by Section 508.1). Essentially, the Group A and Group B occupancies are grouped together as "nonseparated" occupancies. That grouping is based on the most restrictive occupancy--Group A. Thus, the grouping is considered a Group A occupancy for calculating ratios per the requirements for separated occupancies for the rest of the building.
There are many ways to play this game, but it does take a bit of creativity and some time to evaluate each possible scenario to determine the most advantageous outcome.
