Track fasteners play a pivotal role in infrastructure inspection for high-speed rail. Yet, images taken by drones often capture shadows cast by electrical towers flanking the high-speed rail tracks. These shadows can hinder the visibility of the track fasteners, thereby impacting detection efficiency and accuracy considerably. The present paper introduces an end-to-end shadow removal algorithm, rooted in generative adversarial network training. The comprehensive network framework is segmented into three sub-networks: pseudo-mask generation, shadow removal, and result refinement. We have integrated a Fourier convolutional residual module to bolster the feature extraction capability of the generator network. This integration ensures the network retains a global receptive field, even in its more superficial layers. By employing an overall weighted loss function, we enhance the quality of the images produced without shadows. Further, a perceptual loss function has been incorporated to retain the structural information of objects, setting the stage for subsequent defect detection. Our results highlight that Pse-ShadowNet adeptly eradicates fastener shadows while maintaining vital visual features, including object position, structure, texture, edges, and other key visual elements. Consequently, the reconstructed images are detailed and showcase superior image quality.