A deep multimodal system for provenance filtering with universal forgery detection and localization

[EN] Traditional multimedia forensics techniques inspect images to identify, localize forged regions and estimate forgery methods that have been applied. Provenance filtering is the research area that has been evolved recently to retrieve all the images that are involved in constructing a morphed image in order to analyze an image, completely forensically. This task can be performed in two stages: one is to detect and localize forgery in the query image, and the second integral part is to search potentially similar images from a large pool of images. We propose a multimodal system which covers both steps, forgery detection through deep neural networks(CNN) followed by part based image retrieval. Classification and localization of manipulated region are performed using a deep neural network. InceptionV3 is employed to extract key features of the entire image as well as for the manipulated region. Potential donors and nearly duplicates are retrieved by using the Nearest Neighbour Algorithm. We take the CASIA-v2, CoMoFoD and NIST 2018 datasets to evaluate the proposed system. Experimental results show that deep features outperform low-level features previously used to perform provenance filtering with achieved Recall@50 of 92.8%.