Introduction - HMC Computer Science



centertopA Software Framework For Tattoo MatchingAnonymous IJCB 2014 submission00A Software Framework For Tattoo MatchingAnonymous IJCB 2014 submissionAbstractTattoo matching is a compelling problem at the intersection of biometrics and vision. Tattoo-matching systems offer potential aid in identifying groups or individuals who associate with or can be identified by their tattoos. Because tattoos of the same class (e.g. tattoos of hearts) can appear very different, many computer vision algorithms provide helpful partial solutions to the problem, but finding the optimal combination of those algorithmic tools is not at all trivial. This paper describes a software framework designed to compare a large set of computer vision algorithms as efficiently and thoroughly as possible. It provides one-to-one, one-to-many, and many-to-many comparisons of performance, while being as simple as possible to run. Although the original motivation served tattoo-matching, Although designed with tattoo-matching in mind, the framework and its architecture also provide a foundation for efficiently exploring other vision-based biometric challenges.Introduction32899351534160Figure 1: (Top) Examples of similar-class tattoos, e.g., painted woman and lion. (Bottom) Different keypoint- matching pipelines run on the pairs: at left, BRISK and hamming distance; at right, SURF and RANSAC00Figure 1: (Top) Examples of similar-class tattoos, e.g., painted woman and lion. (Bottom) Different keypoint- matching pipelines run on the pairs: at left, BRISK and hamming distance; at right, SURF and RANSACMore and more common, tattoos are of increasing interest and importance to the biometrics community. Depending on the specific application at hand, systems for matching tattoos can leverage a wide variety of pixel-processing approaches. Figure 1 (top) hints at the variety of tattoos that might be considered "matching": the right pair are the different images of the same tattoo and the left pair are distinct renderings of the same figure. Resources such as OpenCV [13] and others offer collections of algorithms for exploring this challenging matching problem, but they provide less structure for composing and comparing those algorithms for specific tasks. The bottom panel of Figure 1 shows two very different combinations of feature detection, extraction, matching, and scoring for the two tattoo pairs above. Because challenges such as tattoo-matching potentially benefit from many pixel-processing approaches, this work presents the design, deployment, and results of from a software framework that combines computer vision algorithms to gain insights into the potential and challenges involved with using tattoos for biometric purposes. The system allows users to test out and analyze the performance of different keypoint detectors, descriptor extractors, descriptor matchers, match scorers, and other algorithms, described in section 3. Many of these algorithms have been implemented in OpenCV, but several others do not; the framework's interface enables easy incorporation of almost any off-the-shelf resources. Since tattoo matching is a relatively unexplored problem, our system's structure is designed to thoroughly test all variations and combinations of existing computer vision algorithms and can compare multiple metrics for the system's performance. Such a framework may serve researchers of other image-based biometric tasks, as well. -222258001000BackgroundTattoos could be very important for the field of biometrics. While there exist techniques that are able to match two dimensional image patterns, tattoos have a number of specific issues that make them difficult to match using traditional computer vision algorithms. Issues include lighting, skin imperfections, the changing surface they exist on, artistic ability, etc. We would like images to have a better score based on their similarity. For example, two snake tattoos should be a very close match, a snake tattoo should be a match with a different snake tattoo, and a snake tattoo should not be a match with a dragon tattoo.Tattoos and BiometricsTattoos have a high potential for use in identification since they are larger and more accessible than other physiological characteristics such as fingerprints or irises. They could be used to distinguish a gang member or identify a person after a natural disaster, along with many other security and safety applications.Problem space for tattoo-matchingBecause tattoos are on bodies, they can appear stretched or warped in different images of the same tattoo. Perspective changes can also change a tattoo’s shape in a two dimensional image. In order for a computer vision algorithm to perform well, it has to account for these visual differences by using algorithms that are invariant to many geometric transformations, including shearing, translation, rotations, and scaling; and fortunately, many feature comparison algorithms are invariant to these transforms. Tattoos are also affected by lighting and context changes that are common problems in image processing. Images taken with surveillance cameras may not be high quality and are unlikely to be taken under good lighting conditions. Finally, tattoos are often partially obscured by clothing or may be cut off when they are not the main focus of the image. Beyond image processing challenges, tattoos carry their own set of problems. Varied skin tones can make it difficult to detect where skin ends and the tattoo begins, especially when skin is close to the color of a tattoo that is only an outline as opposed to a fully filled in design. Also, a tattoo artist’s skill or different interpretations of the same image can greatly impact the kind of tattoo produced. Thus, due to differences in the design, interpretation, and production of a tattoo, tattoo images that are of the same class (e.g. a tattoo of a heart) can appear very different.We wanted our system to be accurate no matter what database of images it is used on. It would not be possible to create a single program that takes two images and gives them a score because images taken under one set of conditions would have different results than another set of images taken under different conditions. Since the actual use and types of images of tattoos the system will be used on is unknown, we created a system that allows users to swap out algorithms and parameters to find the best combination for their particular set of images.AlgorithmsMany algorithms can serve tattoo-matching. Most of them start by seeking corresponding keypoints between images. Keypoint-matching generally has three stages: feature detection, descriptor extraction, and descriptor matching. These can overlap - some algorithms can both detect and extract keypoints - but are distinct enough to form the backbone of our system design, Figure 2: our current system uses the OpenCV implementations of the algorithms noted. The final stage, match-scoring, quantifies the overall match results. The framework allows easy combination and comparison of various detectors, extractors, matchers, and scorers in order to determine the combination with the best precision, accuracy, recall, and/or performance.79375919480Figure 2. (Top) The four stages common to many image-matching applications. (Bottom) The command-line interface to the framework allows a user to mix and match those algorithms arbitrarily. 00Figure 2. (Top) The four stages common to many image-matching applications. (Bottom) The command-line interface to the framework allows a user to mix and match those algorithms arbitrarily. 31755016500Keypoint DetectorsThe most common type of algorithms implemented, detectors locate all areas of interest in an image. These areas of interest can include keypoints, curves, or regions. Keypoints designate regions where the image gradient is changing in a locally distinct way, e.g., at a corner. Often this means that such a point would be re-identifiable regardless of modest rotation, warping or other simple transforms. The detector algorithms return keypoint locations for use in the following extraction stage. The current system uses SimpleBlob, BRISK[1], Dense, FAST[2][3], GFTT[4], HARRIS, MSER[5], ORB[6], SIFT[7], STAR[8], and SURF[9] detectors; others would be straightforward to incorporate, as well.Descriptor ExtractorsAfter finding the locations of an image’s keypoints, the system passes those locations along to the designated extractors. An extractor algorithm creates a vector characterizing the image region around a keypoint. For example, if the descriptor has found a point where the gradient changes in a useful way, the extractor might include a histogram of the orientations along which the gradients change. Implemented extractors include BRIEF[10], BRISK, FREAK[11], ORB, SIFT, and SURF. In addition to returning a descriptor vector, some extractors also provide a scale and orientation reference: this information can increase the set of conditions for which matching, the next pipeline stage, will succeed.Descriptor MatchersThe extracted vectors are matched by the algorithms in this stage of the pipeline. Matches are made by pairing keypoints with the most similar visual characteristics. For instance, typical matchers will compare the gradient-orientation histograms, and some methods will also account for characteristics such as the difference in pixel location. They can be augmented by the geometric consistency of a set of correspondences, e.g., through RANSAC. Matchers currently implemented are variations on brute force (Hamming, L1), radius matching, KNN, K-Nearest Neighbors, and a fast KNN approximation known as FLANN[12].Scorers3366135880110Figure 3. Stages of the optional image segmentation.00Figure 3. Stages of the optional image segmentation.In the final stage of the system, the list of keypoint matches is passed along to a scorer algorithm that quantifies the matching of the two images. Our current tattoo-matching task uses two scoring algorithms, known as Simple and Distance scoring.The Simple scorer consists of the ratio of the number of matches found to the number of possible matches that could have been made, i.e., the number of keypoints in the sparser of the two images. The Distance scorer yields the average vector distance between the matched descriptors, i.e., the visual similarity found. Flexibility in handling other algorithmsWhile most algorithms fit into this set of stages, some do not fall into precisely one of these categories. Our framework can also incorporate approaches that work outside this basic pipeline. Rather than finding keypoints and passing information along from stage to stage, many algorithms provide a global descriptors of an image, already prepared for matching against the corresponding descriptors from other images. In this vein the current system includes contour-matching by ShapeContext [16], GIST [14] and local binary patterns, LBP [15]. In each of these cases the upstream processing is idiosyncratic, but all of them take in two images and output a match score between them. From a high-level perspective, then, these algorithms add three more scorers to the two noted above.Ensemble LearningThe framework's primary strength is in its organization of these component algorithms and its facility for creating combinations of them. Ensemble learning naturally builds from this foundation, and our system includes an implementation of AdaBoost [17], an adaptive boosting algorithm. To use AdaBoost, the user simply appends a flag to the algorithms they want to run (see Section 5). The AdaBoost algorithm will then use each pair of images as an instance and each specified computer vision algorithm as a weak learner. The result is a composite, learned algorithm whose prediction is a weighted sum of all of the weak learners’ predictions. Tattoo SegmentationThus far, we have assumed that the images are primarily of tattoos and nothing else, but that is generally not the case when images are pulled from surveillance cameras or other realistic scenarios. Most images that contain tattoos are not solely of tattoos. For example, there could be an image of a man who has a tattoo on his arm. Certainly, there might be a Frisbee, a dog, a slide, birds, or any other various things in the image, but we only care about the tattoo on the man’s arm. 31756921500Thus, we have incorporated tattoo-segmenting software into our system that is run before any of the other steps. The segmentation software generates a mask, i.e., a grayscale image in which intensity represents the probability of that pixel being a part of a tattoo. When specified, the framework uses a threshold to binarize the image and extracts the the convex hull of the tattoo region. Finally, a bounding box of the convex hull is created and the corresponding area of that bounding box in the original tattoo image becomes the image that the computer vision algorithms will operate on. (Figure 3.) As with ensemble learning, this is an option that can be included – or not – at run time in order to determine the performance of the matching and/or segmentation portions of the system.FrameworkThe framework seeks to balance the conflicting demands of being both user-friendly and extensible. Although OpenCV has a wide array of vision algorithms, it is not as user-friendly in providing support for testing combinations of those algorithms. Our system, built alongside OpenCV, simplifies the task of running "contests" among component algorithms in order to determine which pieces work best for a particular vision-based biometric task. Thus, though initially tattoo matching, the framework is not limited to that domain – it will support a wide variety of situations in which a user needs to quickly compare the performance of various combinations of vision algorithms. "Contest" supportThe framework efficiently runs many different image comparison "contests," where the contest winner is the most accurate combination of algorithms for a particular set of images. Such contests to be one-versus-one, one-versus-many, and many-versus-many. In terms of tattoo-matching, this translates to one image versus one image, one image versus a folder of images, and a folder of images versus a folder of images. We foresee several possible use cases for each of these contests. For example, one versus many may be used to determine the closest matches for one image against a database. The one versus one, on the other hand, may be used for finding the match score between two particular images. A winner for one contest may not be the same for a different contest with different images.Algorithm SpecificationThe system does not take on the constraints (and dependencies) of a graphical interface; Figure 2 shows an example invocation of the Python-based framework. For flexibility, command line flags are used to specify the algorithms to be compared. For example, a user might enter -d SURF SIFT as an option. The -d specifies the stage of matching of concern – keypoint detection in this case, and SURF SIFT indicates that the matching should use both SURF and SIFT algorithms for detecting keypoints. This will not combine the algorithms (yet), but will calculate each algorithm's match scores separately. Similar flags for extraction and scoring steps. Algorithm specification is designed to be extensible so that it is easy to add algorithms to the system.Parameter SpecificationWhen finer tuning of the algorithms is desired, there is a system in place to tweak the values of the parameters for each algorithm that can be specified as previously described via the command line flags. This system consists of text files containing key value pairs for each of the algorithms. The user of the system can edit these files to tweak the parameters.SerializationIn a one-versus-many contest run (or many-versus-many) the na?ve way of matching the images is to run them against each other N many times. However, this system speeds things up by outputting the intermediate keypoint and descriptor results for the “one” image to a JSON file. Then, instead of re-computing the keypoints—a slow operation—the keypoints are instead read from file. This is done separately for each different algorithm combination. We have found that serialization dramatically speeds the process of matching and scoring similarity between images. Other componentsIn addition to the four general steps in the matching process, the system also features other components. Ensemble learning, e.g., -ensemble, and masking are two such components that can be added in addition to the typical image matching steps. The framework makes it easy to add additional components to handle specific use cases.OutputAfter running any algorithm or pair of algorithms, the output from the run is stored in a csv file whose name contains the information about the run. In addition, the csv file is stored in a directory structure organized by folders or images given as input to the testing framework. Finally, the parameters used for the run are stored in a text file whose name is based on the timestamp. Thus, all information about the input to a run of the framework is encoded in the names of the output, so that users can easily identify and compare the output from various runs. The output, in turn, is stored in columns of a comma-separated values (csv) file, includes: (1) the name of first image being compared, (2) the name of second image being compared, (3) index, (4) score, and (5) truth. The index is an increasing unique identifier for each trial, the score is the final output of the testing framework. The ground-truth column holds 1 if the images are of the same class (we use filenames to indicate class, e.g., butterfly1.png and butterfly2.jpg) and 0 otherwise.AnalysisThe analysis of the data is primarily dependent upon the score, which serves as our prediction and the truth. The metric tools provided in the framework include Receiver Operator Characteristic curves and Cumulative Match Characteristic curves. We use R to generate these visualizations from the framework's output, noted above.Receiver Operator CharacteristicReceiver Operator Characteristic (ROC) curves display the true positive rate (sensitivity) on the y-axis and the false positive rate (fall-out) on the x-axis. In order to convert our scores into binary predictions, we apply a threshold so that scores above the threshold are converted to 1 and scores below are converted to 0. Then, to vary the false positive rate, the threshold is changed (decreasing the threshold increases the false positive rate by increasing the number of scores that are considered positive). This metric shows the user how well their algorithms perform at recognizing when a pair of images is a match, as shown in Figure 4.Cumulative Match CharacteristicCumulative Match Characteristic (CMC) curves display the percent likelihood that an image of the same class is in the top X images, where X is shown on the horizontal axis. Thus, the program looks at every image (let’s say imgA) and the score between it and all other images are examined in order of the images that have the highest score when matched against imgA. The program keeps track of how many images it has to look at before it finds an image of the same class, and this number is then kept track of for every imgA and used to derive the probabilities Figure 4 shows a large run comparing CMC curves of many algorithm combinations, along with the AdaBoost ensemble of all of them, further to the upper left, a clear improvement in performance over any single component.8953510033000313753553249008953564770Figure 4: (Left) Example of a multi-algorithm ROC curve. Each curve on the plot represents a different combination of the detector/extractor/matcher/scorer algorithms; an AdaBoost-learned ensemble algorithm is also included, though for the task of maximizing the ratio of true-to-false positives, it is not the winner. (Right) In terms of CMC performance, the ensemble algorithm is a clear improvement over the components. At right the curves summarize "how far one must look through the search results" in order to find a category match. Both approaches have broad use in biometric applications. The contribution of this paper's framework is increased human efficiency in exploring many different combinations/ensembles of off-the-shelf routines. Both use a dataset of 200 tattoos distributed across 40 different categories. The file-based, caching framework eases the transition to larger datasets, as available.00Figure 4: (Left) Example of a multi-algorithm ROC curve. Each curve on the plot represents a different combination of the detector/extractor/matcher/scorer algorithms; an AdaBoost-learned ensemble algorithm is also included, though for the task of maximizing the ratio of true-to-false positives, it is not the winner. (Right) In terms of CMC performance, the ensemble algorithm is a clear improvement over the components. At right the curves summarize "how far one must look through the search results" in order to find a category match. Both approaches have broad use in biometric applications. The contribution of this paper's framework is increased human efficiency in exploring many different combinations/ensembles of off-the-shelf routines. Both use a dataset of 200 tattoos distributed across 40 different categories. The file-based, caching framework eases the transition to larger datasets, as available.ConclusionThis work originally sought to find a "single" algorithm that would best suit the important biometric application of matching images of tattoos. Yet after studying the challenges involved in tattoo matching, it became evident that progress depended on a framework that would allow investigators to quickly try out any existing popular computer vision algorithm and, singly or in combination, generate relative performance statistics. In this way, our framework enables users to explore any matching-based computer vision problem, not just the problem of matching tattoo images. If a user has a database of images representative of a phenomenon they hope to study by matching, then this framework can efficiently and exhaustively explore a wide variety of existing computer vision algorithms, their variations, and their combinations. Whether trying to match images of kittens or images of tattoos, this framework empowers users to find the best algorithm for their specific problem.AcknowledgmentsThe authors thank Nick Orlans and Mikel Rodriguez of the MITRE corporation for their support, suggestions, and guidance in this work. This work is made possible by the MITRE corporation and the Harvey Mudd College computer science clinic program.ReferencesS. Leutenegger, M. Chli, and Y. Siegwart. BRISK: Binary Robust Invariant Scalable Keypoints. Porc. IEEE Int. Conf. Computer Vision (ICCV), 2548–2555, 2011.E. Rosten and T. Drummond. Fusing Points and Lines for High Performance Tracking. IEEE Int. Conf. Computer Vision (ICCV), 1508-1511, 2005.E. Rosten and T. Drummond. Machine Learning for High-Speed Corner Detection. ECCV, 430 – 443, 2006.J. Shi and C. Tomasi. Good Features to Trak. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 593-600, 1994.J. Matas, O. Chum, M. Urban, and T. Pajdla. Robust Wide Baseline Stereo from Maximally Stable Extremal Regions. Proc. Of British Machine Vision Conference, 384 – 396, 2002.E. Rublee, V. Rabaud, K. Konolige, and G. Bradski. ORB: an efficient alternative to SIFT or SURF. Proceedings of the ICCV, 13, 2011.D. Lowe. Object recognition from local scale-invariant features. Proceedings of the ICCV, 2,1150–1157, 1999M. Agrawal, K. Konolige, and M. R. Blas. CenSurE: Center Surround Extremas for Realtime Feature Detection and Matching. Computer Vision – ECCV 2008, 4,103-115, 2008.H. Bay, A Ess, T. Tuytelaars, L. Van Gool. SURF: Speeded Up Robust Features. Computer Vision and Image Understanding (CVIU), 110(3), 346-359, 2008.M. Calonder, V. Lepetit, C. Strecha, and P. Fua. BRIEF: Binary Robust Independent Elementary Features. In Proceedings of the European Conference on Computer Vision (ECCV), 2010.A. Alahi , R. Ortiz and P. Vandergheynst. Freak: Fast retina keypoint. Proc. IEEE CVPR, 510 -517, 2012.M. Muja and D. G. Lowe. Fast Approximate Nearest Neighbors with Automatic Algorithm Configuration. International Conference on Computer Vision Theory and Applications (VISAPP'09), 2009.OpenCV Dev Team. (Dec 31, 2013). Common Interfaces of Descriptor Matchers. OpenCV. Available: . [2014, April 7].C. Siagian, L. Itti. Rapid Biologically-Inspired Scene Classification Using Features Shared with Visual Attention. IEEE Transactions on Pattern Analysis and Machine Intelligence, 29(2), 300-312, 2007.T. Ojala, M. Pietik?inen, and D. Harwood. Performance evaluation of texture measures with classification based on Kullback discrimination of distributions. Proceedings of the 12th IAPR International Conference on Pattern Recognition (ICPR 1994), 1, 582 - 585, 1994S. Belongie, J. Malik, and J. Puzicha. Shape Context: A new descriptor for shape matching and object recognition, Proceedings, NIPS pp. 831-837, 2000.J. Friedman, T. Hastie, and R. Tibshirani. Additive Logistic Regression: a Statistical View of Boosting, Annals of Statistics v. 28 1998. ................
................

In order to avoid copyright disputes, this page is only a partial summary.

Google Online Preview   Download