Despite its proven record as a breast cancer screening tool, mammography remains labor-intensive and has recognized limitations, including low sensitivity in women with dense breast tissue. In the last ten years, Neural Network advances have been applied to mammography to help radiologists increase their efficiency and accuracy. This survey aims to present, in an organized and structured manner, the current knowledge base of convolutional neural networks (CNNs) in mammography. The survey first discusses traditional Computer Assisted Detection (CAD) and more recently developed CNN-based models for computer vision in mammography. It then presents and discusses the literature on available mammography training datasets. The survey then presents and discusses current literature on CNNs for four distinct mammography tasks: (1) breast density classification, (2) breast asymmetry detection and classification, (3) calcification detection and classification, and (4) mass detection and classification, including presenting and comparing the reported quantitative results for each task and the pros and cons of the different CNN-based approaches. Then, it offers real-world applications of CNN CAD algorithms by discussing current Food and Drug Administration (FDA) approved models. Finally, this survey highlights the potential opportunities for future work in this field. The material presented and discussed in this survey could serve as a road map for developing CNN-based solutions to improve mammographic detection of breast cancer further.