Many emperors of the southern dynasties (420-589) were devout Buddhists, in line with the so-called "gentry Buddhism" that was said to have emerged in South China. According to legend, during the Liu Song Emperor Wendi's reign (424-453), the Buddhist Monk Beidu sojourned in Tuen Mun, suggesting that Buddhism was established in Hong Kong at that time.

Buddhism flourished during the Jin and Southern dynasties and many monks came from India and central Asia to preach in China. It is said that Monk Beidu, who arrived in north China during the Liu Song regime, stayed at Tuen Mun to await his ship home. The monastery and the mountain behind it were later named Beidu Monastery and Mount Beidu. However, the monastery has since been demolished (The Hong Kong Story).

It appears that, in comparison to the religious worship of Tin Hau, which originated among lay people, Buddhism was less of a commoner's religion to begin with. As it appealed most to the educated aristocracy initially, it only gradually became practiced among all social classes.