Improving Software Package Search Quality Dan Fingal and Jamie Nicolson

The Problem Search engines for software packages typically perform poorly Tend to search project name and blurb Don’t take quality metrics into account Poor user experience

Improvements in Indexing More text relating to the package Every package is associated with a website that contains much more detailed information about it We spidered those sites and stripped away the html for indexing

Improvements in Quality Metrics Software download sites have additional data in their repositories Popularity: Frequency of download Rating: User supplied feedback Vitality: How active development is Version: How stable the package is

System Architecture Pulled data from Freshmeat.net and Gentoo.org into mySQL database Spidered and extracted text for associated homepages Text indexed with Lucene, using Porter Stemming and a Stop Words list Similarity metrics as found in CS276A

Ranking Function Lucene returns an ordered list of documents (packages) Copy this list and order by the various quality metrics Scaled Footrule Aggregation combines lists into one results list Scaled Footrule and Lucene field parameters can be varied

Scaled Footrule Aggregation Scaled footrule distance between two rankings of an item is the difference in its normalized position E.g., one list puts an item 30% down, another puts it 70% down. Scaled footrule is |.3 −.7| =.4 Scaled footrule distance between two lists is the sum over the items in common Scaled footrule of a candidate aggregate list and the input rankings is the sum of the distances between the candidate and each input ranking Minimum cost maximum over bipartite graph between elements and rank position optimizes

Measuring Success Created a gold corpus of 50 queries to relevant packages One “best” answer per query Measured recall within the first 5 results (0 or 1 for each query) Compared results with search on packages.gentoo.org, freshmeat.net, and google.com

Sample Gold Queries “C compiler”: GCC “financial accounting”: GNUCash “wireless sniffer”: Kismet “videoconferencing software”: GnomeMeeting “wiki system”: Tiki CMS/Groupware

Training Ranking Parameters Boils down to choosing optimal weights for each individual ranking metric Finding an analytic solution seemed too hairy, so we used local hill-climbing with random restarts

Training Results Uniform parameters gave a recall of 29/50 (58%) Best hill-climbing parameters gave recall of 37/50 (74%) We're testing on the training set, but with only five parameters the risk of overfitting seems low Popularity was the highest-weighted feature

Comparing to Google Google is a general search engine, not specific to software search How to help Google find software packages?  Append “Linux Software” to queries How to judge the results?  Package name appears in page?  Package name appears in search results?

Room for Improvement Gold corpus  Are these queries really representative?  Are our answers really the best?  Did we choose samples we knew would work well with our method? Training Process  Could probably come up with better training algorithms

Any questions?