Self-supervised monocular depth estimation has garnered significant attention in recent years due to its practical value in applications, as it eliminates the need for ground truth depth maps during training. However, its performance usually drops when estimating weakly textured regions and boundary regions, primarily due to the limited depth representation capability of traditional Convolutional Neural Networks (CNNs) that do not support topology. To address these issues, we propose a Graph Semantic Model (GSM) to improve self-supervised monocular depth estimation by utilizing graph learning and semantic information. Our focus is on improving feature representation through graph semantic information. Therefore, we incorporate semantic segmentation and depth estimation into one framework and enhance the interaction of different modal information through the Inter-Directed Graph Reasoning (IDGR) module. In addition, we design the Semantic-Guided Edge Graph Reasoning (SGEGR) module, aiming to boost the network's ability to perceive local depth. Extensive experiments on the KITTI dataset show that our method outperforms the state-of-the-art methods, particularly in accurately estimating depth within weakly textured regions and boundary regions.