Toronto SharePoint User Group Bill Brockbank Navantis Inc. billb@navantis.com

Architecture and Topology Logical Architecture - Components Crawl Content Processing Analytics Processing Index Administration Query Processing Search Process Topologies - Examples

Logical Architecture

Crawling the Content The crawl role is responsible for crawling content sources. It delivers crawled items – both the actual content as well as their associated metadata – to the content processing component Invokes connectors or protocol handlers to content sources to retrieve data Does not do any document parsing (Content Processing Component does that) Information about content sources, schedules, etc. are synchronized to the registry on crawl role servers from the search admin database The Crawl Database is used by the crawl component to store information about crawled items and to track crawl history Holds information such as the last crawl time, the last crawl ID and the type of update during the last craw.

Content Processing Content Processing Component Processes crawled items and feeds these items to the index component Document Parsing happens through Format Handlers iFilter is supported through a Generic iFilter format handler iFilters are still the extensibility platform for SharePoint 15 Heart of the indexing process: transforms crawled items into artifacts that can be included in the search index by carrying out operations such as document parsing and property mapping Performs linguistic processing at index time (e.g. language detection and entity extraction) Writes information about links and URLs to the Link Database directly Generates phonetic name variations for people search

Analytics Processing Component Search Analytics analyzes crawled items and how users interact with search results. Usage analytics analyzes usage events e.g. like views from the event store When an user does an action (e.g. view a page) the event is collected in usage files on the WFE’s and regularly pushed to event store where they are stored until processed Results are then returned to the Content Processing Component to be included in the search index You can add write code to add custom events Analytics Processing Component supports scaling out: Add more APC roles to have analysis complete faster Adding more Link databases to increase capacity for links and search clicks, and potentially speed up the DB extraction Adding more reporting databases to store more reports as well as improving SQL throughtput in retrieveing reports

List of Sub Analyses Search Analytics Usage Analytics Link and Anchor text analysis Click Distance Search Clicks Deep Links Social Tags Social Distance Search Reports Usage Analytics Recommendations Usage Counts Acivity Ranking

Index Component Used in both feeding and query processes: Feeding: receives processed items from the content processing component and writes those items to index files Query: receives queries from the query processing component and provides results sets in return It also physically moves around indexed content when the index architecture is changed by the Search Administration Component

Query Processing Component Performs linguistic processing at query time: Word breaking, stemming, query spellchecking, thesaurus Analyzes and processes search queries and results. When the component receives a query from the search front-end, it analyzes and processes the query to attempt to optimize precision, recall and relevancy. The processed query is then submitted to the index component(s). As part of this it also decides which query rules are applicable, which index to send the query to, and whether to do any pre- or post- processing of the query The index component returns a result set based on the processed query back to the query processing component, which in turn processes that result set before sending it back to the search front-end.

Search Administration Search Admin Component Runs number of system processes required for search Is responsible for search provisioning and topology changes Coordinates search components – Content Processing, Query Processing, Analytics, and Indexing. Search Admin DB Stores search configuration data: Topology Crawl rules Query rules Managed property mappings Content sources Crawl schedules Stores Analytics settings Does not store ACLs anymore

Search Processes MSSearch.exe Host Controller NodeRunner.exe A Windows Service that supervises NodeRunner process(es) on a given box It restarts failed NodeRunner processes: if a NodeRunner fails, the HostController will detect this and restart it NodeRunner.exe Is the process that hosts the search components There might be several instances of this process on a single box MSSearch.exe Is the Windows Service who hosts the Crawl Component

Search Host Process NodeRunner.exe Crawl Component NodeRunner.exe Content Processing Component Multiple NodeRunner instances can run on the same server Each NodeRunner instance hosts one search component E.g. If you have two index replicas on one server you will have two NodeRunner instances – one for each replica On a default single server install there will be 5 instances of the NodeRunner.exe process NodeRunner.exe Query Processing Component NodeRunner.exe Index Component NodeRunner.exe Analytic Processing Component

Enterprise Search – Query Configuration

Query Configuration Ranking Query Spell Correction Query Rules

Ranking Ranking model describes which criteria are included in sorting, how much they contribute to the rank score and how they relate to one another Custom ranking models are managed through PowerShell and the public OM using XML files as it was in SharePoint 2010 Ranking model for a specific query can be selected at query time by setting the RankingModelId of the query; otherwise default is used NOTE: This property is currently not available in the OOB search results web part

Working with Ranking Models A search service administrator can do the following basic operations on the rank models: List ranking models Specify a default ranking model Change an existing custom ranking model Delete an existing custom ranking model Create a new ranking model (either based on an existing one or from scratch) Import and export a ranking model to XML Specify how much weight the various properties should have in the model These operations are all administered through PowerShell There may be a CodePlex GUI-based solution after RTM

Query Spelling Correction Customizations to Query Spelling Correction is now managed in the term store – both inclusions and exclusions We also still have a dynamic dictionary that is created based on content in the index, or you can switch to the static OOB dictionary Here’s an example: Provides customizable “did you mean” functionality:

Query Rules Query Rules are a new feature in SharePoint 15 that help act upon the “intent” of a query Query Rules are composed of three top level elements: Query Conditions (i.e. matching rules) Query Actions (i.e. what do you do when you find a match) Publishing Options (i.e. when should this rule be active) Rules are created at the site collection level, but you will probably also be able to create them at the SSA level through PowerShell (TBD by RTM) Several Query Rules are provided out of the box and can be found in the Site Settings for each site collection

Query Rules Implementation Query Rules allow you to have search requests from a user trigger multiple queries and multiple result sets. This differs from SharePoint 2010 where there were only simple queries – one query has one set of results. All matching Query Rules can generate results; the query orchestration engine retrieves all the different sets of search results and organizes them for rendering to the user The results themselves are then rendered using a new feature called Query Results display templates, which is discussed later

Query Rules - Conditions Conditions allow you to define the circumstances under which your query rule will apply You can choose to match on these use cases: Query contains a specific word or words Query contains a word in a specific dictionary Query contains an action word that matches a specific phrase or term set Query is common in a different source (like Videos Result Source) Results include a common result type (like file type) Advanced rule – match across a set of terms, dictionary, regular expression, etc. You can also match if a query is against a specific Result Source or category (like a Topic Page category) Each Query Rule can have multiple conditions

Query Rule Actions There are three types of actions: Assign a “promoted result” to the query Promoted result is similar to Best Bets with SharePoint 2010 or Visual Best Bets with FAST Search 2010, but much more flexible Create and display a Result Block Result blocks specify an additional query to run and how to display results The feature includes a full query designer so you can build and test queries before finalizing them You can include the results above those returned by core results, or interleaved by ranking You can also choose custom display templates instead of the default for the result Change the query that returns the core results Change the original query by changing the query terms, adding additional terms, apply an XRANK formula to the query, etc. You can add multiple best bets or results blocks for each query rule

Query Rules Publishing Options Publishing options for a Query Rule determine when it will be used It works just like rules for publishing a page in a publishing site template You decide whether the rule is active You decide when to start and when to finish using the rule, or don’t set dates at all and it’s always used You can configure a review date, which is basically an email reminder for you to look at the rule again Publishing a rule is particularly useful in commerce scenarios. For example, queries that should return items you want to percolate to the top and sell during a particular period of time

Enterprise Search – New Search Results Display Options

Search UI Configuration Result Types Display Templates Search Navigation Search Refinement Query Suggestions Thumbnail Previews Site Level Search Admin Summary

Result Types SharePoint 15 includes a new framework for presenting search results to end users called result types Result types have several elements that enable each type to be rendered differently:

Result Type Rules You can create a variety of different rules on which to match: Equality(= or !=), comparison (< or >), or logical (AND or OR or NOT) You can have multiple rules as a condition You can use managed properties in a rule In B1R there will be a new UI for creating these rules, so what you see in the B1 bits will change quite a bit Example Name Rule Matches… Word FileExtension=docx OR FileExtension =doc All Word documents Spec ContentType=Spec AND (FileExtension=docx OR FileExtension =doc) Specs which are also Word documents

Result Type Management Result types are managed as follows: Result Types can apply to an entire search service application or to a specific site or web OOB Result Types are immutable but can be overridden with a copy There is an inheritance hierarchy to result types in a site: the current web, plus the site collection root, plus the Search Service Application. So if you add a new result type to some random web, the sub webs under it won’t inherit that type unless you are working in the site collection root web

Display Templates Display templates contain the following characteristics: They define the visual layout of a Result Type A template is really just HTML You can edit it with ANY HTML designer you want – DreamWeaver, SharePoint Designer, Visual Studio, Notepad – doesn’t matter You add placeholders to your HTML file where managed properties should be emitted It gives you a real WYSIWIG experience when designing templates A set of display templates is included OOB; you can find them in the Master Pages/Display Templates/Search folder Site collection admins can upload new display templates.

Result Type with Display Template Example Here’s an example from one of the Hands On Labs that demonstrates building a custom result type and display template for .TXT file search results:

Steve Peschka Sr. Principal Consultant Microsoft Corporation Result Types demo Steve Peschka Sr. Principal Consultant Microsoft Corporation

Search Refinement There are two different modes for the refiner web part: search results and faceted navigation With search results the refinement data works essentially the same as SharePoint 2010 With faceted navigation it uses a term from the term store to filter what kind of data should be displayed (explained on next slide) Refinement is different with SharePoint 15 in that you can define display templates to use for rendering different kinds of refinements In SharePoint 2010 you had to write your own custom refiner

Faceted Navigation with Search Refiners With Faceted Navigation, it is used in conjunction with term sets that are used for navigation With each term you select which managed properties should be used as refiners with that term Within the managed property you need to configure it as “Refinable” Example: You have term store terms Camera and Laptop You have managed properties Megapixel Count, Color and Manufacturer For Camera term, you add refiners for Megapixel Count and Manufacturer For Laptop term you add refiners for Color and Manufacturer Another Example: Department is the term store term, Customers and Projects could be refiners

Query Suggestions Query suggestions take a big leap forward in usability. It improves on the experience in SharePoint 2010 as follows: Your personal SharePoint activity factors into the query suggestions, i.e. you have a personal query log It includes weighting based on sites that you have previously visited It uses the most frequent queries across all users that “match” the search terms The behavior of the query suggestions turns into more of a “browse and find” kind of experience You can also add inclusion and exclusion lists for suggestions via the SSA admin pages

Query Suggestion Types There are two types of query suggestions – what you see when you are entering a query, and what you see when you get results When entering a query you will see two types of suggestions: A list of items you have clicked on before from your personal query log A list of items that others are typing for their queries When you get query results back, you will get another set of suggestions They are a list of links that you have clicked through at least twice before and match your search criteria

Query Suggestion Examples Here are some examples of how query suggestions might look: Before the Query Suggestion Above are items based on what other users have queried and found Bottom 3 items are ones you clicked before After the Query Suggestion Top 3 items are ones you clicked before

Thumbnail Previews The new Office Web Apps is the engine for thumbnail previews in SharePoint 15 The BIG WIN HERE – you can now browse through the entire document in the preview See all pages, see animations, zoom in, scroll through the entire document The point of this is to allow users to find the exact item they’re looking for right in search results – no more clicking a result, hitting the back button, and on and on until they find the one they’re looking for It also addresses the two major shortcomings of thumbnails in SharePoint 2010: It could only be used with FAST Search It did NOT work with claims authentication  Since there’s only one search engine in SharePoint 15 you get document previews out of the box In a different twist, previews only work with claims authentication – it will not work with classic Windows authentication

Summary: New Site Search Admin Tasks Just to summarize, here are things site admins can do to manage search themselves: Task Site Collection Admin Site Admin Create and/or override Query Rules, including Promoted Results Create and/or override Result types and display templates Create and/or override Result Sources, either for remote locations or as a custom search “vertical” Create managed properties Create refiners Start a local crawl – can even be done down to the list level

Online Content links: IT Pro training for SharePoint 2013 http://technet.microsoft.com/en-us/sharepoint/fp123606 SharePoint 2013 enterprise search Module: http://technet.microsoft.com/en-us/sharepoint/fp123606