Building fa?ades can feature different patterns depending on the architectural style, function- ality, and size of the buildings; therefore, reconstructing these fa?ades can be complicated. In particular, when semantic fa?ades are reconstructed from point cloud data, uneven point density and noise make it difficult to accurately determine the fa?ade structure. When inves- tigating fa?ade layouts, Gestalt principles can be applied to cluster visually similar floors and fa?ade elements, allowing for a more intuitive interpretation of fa?ade structures. We propose a novel model for describing fa?ade structures, namely the layout graph model, which involves a compound graph with two structure levels. In the proposed model, similar fa?ade elements such as windows are first grouped into clusters. A down-layout graph is then formed using this cluster as a node and by combining intra- and inter-cluster spacings as the edges. Second, a top-layout graph is formed by clustering similar floors. By extracting relevant parameters from this model, we transform semantic fa?ade reconstruction to an optimization strategy using simulated annealing coupled with Gibbs sampling. Multiple fa?ade point cloud data with different features were selected from three datasets to verify the effectiveness of this method. The experimental results show that the proposed method achieves an average accuracy of 86.35％. Owing to its flexibility, the proposed layout graph model can deal with different types of fa?ades and qualities of point cloud data, enabling a more robust and accurate reconstruc- tion of fa?ade models.