Tag-based search on Flickr by combining folksiness information and image visual features

In recent years, there are explosive photo sharing websites such as Flickr that users can upload and annotate images with arbitrary keywords called tags. Consequently users have been overwhelmed by the huge numbers of images that effective retrieval technology for this volume of images is needed. In tag-based image retrieval most existing methods use tags or visual characteristics and do not consider users information. In this paper we propose a system for tag-based image retrieval on Flickr. Base on extracted information from ternary relationships between users, images and tags in folksonomy and simultaneously visual features that extracted from images, the similarity between images and tags derive and use for tag-based image retrieval. A folksonomy can be viewed as a three-dimensional space of users, tags, and images; consequently this three-dimensional space can be projected onto three two-dimensional matrices user-tag, user-image and tag-image. We then extract Tag-Image similarity from tag-item matrix. To calculate similarity between images based on visual features we use cosine similarity between local features. To validate the effectiveness of our proposed approach on real-world web image datasets, we conduct extensive experiments on the image dataset NUS-WIDE. Since the NUS-WIDE dataset do not contain user information and only contain visual features and tag information, we use Flickr API to access users tag assignments. Experimental evaluations demonstrate that using simultaneously folksonomy and image visual features can improve results of tag-based image retrieval than using separated folksonomy information or image visual features.