The vibration signal of planetary gearbox of wind turbine is a kind of non-linear and non-stationary complex signal.The traditional fault diagnosis method can deal with this kind of signal well in a limited range. The convolution depth belief network is established for planetary gearbox fault diagnosis. In order to prevent the wrong selection of hyper parameters from causing insufficient recognition accuracy, particle swarm optimization algorithm is introduced to optimize the hyper parameters of the network, and the chaos initialization of particles improves the global search ability of particles. Firstly, the original signal is decomposed by VMD to extract the eigenmode function which is relatively concentrated in the impact information as the input data of the network. Then, the training set is used to train, the chaos particle swarm optimization algorithmis used to determine the hyper parameters of the network according to the minimum fitness function, and the layer-by-layer greedy algorithm is used to continuously update the network parameters. Finally, the extracted fault features are classified by a classifier. This method is verified that it can diagnose the fault of planetary gearbox under different conditions.