The optimization of virtual power plant power regulation is of great significance for enhancing the flexibility and economy of the power grid. Traditional optimization control methods rely on strong assumptions and have insufficient adaptability. This paper proposes a virtual power plant power regulation optimization strategy based on the aggregation of multi-agent in buildings. A hierarchical multi-agent graph reinforcement learning framework is constructed, dividing the virtual power plant into local regulation layer and global state evaluation layer; based on this framework, a hierarchical power regulation optimization model for virtual power plant and intelligent buildings is established; combined with graph attention network and multi-agent deterministic strategy gradient algorithm, an improved multi-agent graph reinforcement learning algorithm is proposed. The case study analysis shows that this algorithm outperforms traditional reinforcement learning methods in terms of convergence speed and stability, can quickly respond to power regulation requirements and maintain a low power deviation, and can effectively solve the problem of excessive power allocation among buildings, providing an effective solution for virtual power plant power regulation.