Not sure I have all the information or not but our zoning would not allow a secondary or accessory building to stand alone on a seperate lot. Zoning also typically requires a BL (Build Line) established when a subdivision is platted. Building faces are what determines the front of the building. Concrete stoops and stairs and maybe a soffit is the only thing that is typically allowed over the BL. Your post indicated 1-inch maybe you meant 1-ft?
As building official I have no way of determining if a building is 1-inch over the BL, I get a plot plan and that would be difficult to verlfy, however I suppose a surveyor can, but at who's cost?
A BZA board (Board of Zoning Adjustment) may see your way in regards to oak tree displacement, but keep in mind, no one is forcing you to build a barn so it's not considered a hardship.