Database Technologies for E-Commerce Rakesh Agrawal IBM Almaden Research Center

Outline  Overview of the Pangea B2B marketplace  Technology dives:  Interactive parametric search  Catalog integration  Storage & retrieval of e-commerce data  Research opportunities

Pangea Architecture

Product Selection

Brand and Loyalty Creation

Features (1)  Eureka:  Interactive parametric search  Searches based on example products  Similarity search for product substitution  Dynamic Content generation/presentation (on the fly from database):  Catalog Hierarchy pages with product counts  Side-by-Side Product Comparison  Product Listings/Details  Shopper groups through preference layer

Features (2)  Server side Searching:  Functional description search using classification  Key word & Part Number Search  Restriction of the search to sub-trees of the category hierarchy (pushed down to the database)  Real time Price and Availability Engine:  Ability to crawl, interrogate, extract price & availability data from various suppliers/distributors in real time and present them in side-by-side comparison format  Easily add new distributors/suppliers  Completely client side implementation to prevent blocking

Features (3)  Design Worksheet (Generalized shopping cart):  List of items: an item could be a specific part, alternative set of parts, specifications, other design worksheets (nesting)  Completely integrated with all relevant components (search, content, price, etc.)  Aggregate constraints (e.g. total price)  Multiple projects saved on server  Share projects with other (authorized) users  Mining for suggestions

Features (4)  Design data warehouse creation and maintenance  Crawling technology for extracting part information from websites of various manufacturer/distributor/suppliers  Data extraction from Manufacturer Spec Sheets (pdf files)  Classification technology to build, merge, manage and populate category hierarchies.

Features (5)  Soft content creation and maintenance  Crawling to acquire articles/news/postings from various web news sources, usenet newsgroups, etc.  Agents to cluster, classify, extract concepts, identify associations.  Personalized news/data presentation (based on user defined profile, channels, etc.)  Complete interleaving and integration with part content.  Automatic generation of summary pages for manufacturers (e.g. related news articles, URLs, product categories).

Prototype  Data for nearly 2 million parts, in 2000 categories  Focused crawling of more than 175 manufacturers for datasheets/ application notes/manuals (160,000)  1/4 Terabyte database