Digital image correlation (DIC) is a speckle image-based optical measurement technique for measuring deformation of object. In recent years, deep learning has been widely used in 2D-DIC, while research in the field of 3D-DIC is in its infancy, although 3D-DIC can measure 3D shape and deformation compared to 2D-DIC. 3D-DIC achieves 3D deformation measurement through temporal matching and stereo matching, and it is difficult to perform two matching tasks through an end-to-end network due to the different deformation types of them. To solve this problem, we propose an end-to-end speckle matching network for 3D deformation measurement, called 3D-DICNet. Considering that the difference in deformation types between the two matching tasks is mainly manifested in the deformation scales, we extract the features of different receptive fields, propose a new attention connect volume and multi-scale cost aggregation to achieve the deformation measurements at different scales. Experimental results show that the network can perform 3D deformation measurement with high accuracy and efficiency.