Comments on “Expert System Life Cycle …”: (Or what happened to expert systems research?) Daniel E. O’Leary University of Southern California ©2013

Outline 1. AI/ES – Science – Accounting/Auditing 2. Selected Previous Research on which Gray et al. is based – Rogers / Gartner – O’Leary / Moore 3. Gray et al. – Data – Research Questions – Alternative Interpretation 4. Alternative “Paradigm” / Approach? 5. What happened to expert systems?

1-Expert Systems Sound almost magical! Capture expertise … put it in a system – Replace the expert with the system … – Worked really well to help solve scientific problems Artificial intelligence has a similar magic flavor

What is an expert system? When I talk about an expert system, I will use it to represent systems that typically were built by AIS and auditing researchers Circa – Rule-based “if … then …” – Developed for “consumption” by people, e.g., auditors – People were “eyes and ears” of the system – System would ask people for inputs about the state of the world and provide recommendations based on those inputs – There were limitations associated with virtually all of them.

Use was not the same as math, chemistry and geology In accounting and business, the interesting issues required human judgment. – The above rule requires assessment of a number of concepts – Unfortunately, different people make different assessments given the same situation Often proposed as models of, e.g., auditor behavior – As a result, ES found an initial home at the USC Audit Judgment Conference

Sample Rule – Requires an “environmental assessment” If (1) Message control software is complete and sufficient, and (2) Recovery measures are adequate, and (3) Adequate documentation is generated to form a complete audit trail Then there is strong suggestive evidence (.8) that controls over data loss are adequate. Assessments often are inconsistent

2. Gray - Previous Research … O’Leary O’Leary (2008, 2009) used Roger’s life cycle models, Gartner’s technology “hype” cycle and other life cycle models to investigate issues associated with research in information systems – Different data (artifacts) are available at different points in a technology’s life cycle … (data availability) – Different methodologies are used at different points in the life cycle because the available data differs over time Design science early, case studies, empirical later, etc. – Different researchers are likely to research different aspects based on their strengths or interests – Similar to this paper, I used expert system research as an example and used research to illustrate the occurrence in the literature and how examples fit into the life cycle models

Gartner Hype Cycle Artifacts differ across the life cycles 5 % 20-30% 0% Design Science Empirical Cases Biased Data

(O’Leary 2008)

Gray - Previous Research … Moore Moore (2002), a marketing guru, studied the adoption of new products. – Book is interesting, led to development of a marketing consulting firm. – Moore argues that different adoption groups exist at different points in product life cycles – Moore argued that there were “chasms” of firms in the life cycles Adoption groups are not smoothly distributed across the life cycle – One view is that marketing discovered Kuhn’s work on the “structure of scientific revolutions” Technology and technology research as a paradigm shift

Moore’s Contribution Chasm

3-a-Gray et al. paper Searches previous AIS expert system research Studies characteristics of the resulting database of AIS ES papers Considers mapping that research over time into different life cycles – Rogers, Gartner, O’Leary, Moore I particularly liked the data about Ph. D. dissertations. – To me this is an interesting data set – Ph. D. students are a signal of research and innovation

Top #9 Expert Systems Dissertation (Institutions)Dissertation #% Texas A&M University36.82% Georgia State University24.55% Kent State University24.55% Louisiana Tech University24.55% Michigan State University24.55% Oklahoma State University24.55% The University of Nebraska - Lincoln24.55% The University of Texas at Arlington24.55% University of South Carolina24.55% Start 1983 End 1998 Ph. D. Students What does data tell us about expert system research?

AAA Meeting Data AuthorPublication # Baldwin-Morgan, Amelia Annette4 McDuffie, Robert Stephen4 Murphy, David Smith4 Eining, Martha McDonald4 Steinbart, Paul John4 Changchit, Chuleeporn3 Mauldin, Elaine Gay2 Lenard, Mary Jane2 Back, Barbro Christina2 Wensley, Anthony Kevin2 Meservy, Rayman David – 2006 vs ( ) 1998 PH. D. Dissertations Done, AAA Starts? Implication are …? YearPresentation #

Contributors Top Expert Systems ArthorsPublications% 1 Carol E. Brown 73.00% 2 Daniel E. O’Leary 73.00% 3 Robert H. Michaelsen 62.58% 4 Alan Sangster 62.58% 5 Mary Ellen Phillips 52.15% 6 Mohammad J. Abdolmohammadi 41.72% 7 Andrew D. Bailey, Jr % 8 Amelia Annette Baldwin-Morgan 41.72% 9 Martha M. Eining 41.72% 10 James V. Hansen 41.72% 11 Clyde W. Holsapple 41.72% 12 R. Steve McDuffie 41.72% 13 David S. Murphy 41.72% 14 Paul J. Steinbart 41.72%

3-b-Gray et al. Research Questions RQ1: Did expert systems research in accounting and AIS domain go through a similar industry life cycle over time? RQ2: Did the type of research evolve over time as would be predicted by the industry life cycle and the Gartner hype curve? RQ3: Did the type of researcher evolve over time as would be predicted by the adoption life cycle? RQ4: Did the evolution of the type researcher encounter a chasm that slowed or stopped expert system research as would be predicted by Moore?

RQ1: Did expert systems research in accounting and AIS domain go through a similar industry life cycle over time? How are you measuring ES research? – Does it refer to the research artifacts? – Does this mean the approach laid out by O’Leary: design science, case studies, empirical …? – How much research is needed to count as “research”? Does one occurrence count? I am not sure what “… go through a similar industry life cycle …” refers to. – What from industry is being measured? – What is being compared here? – Does this mean research is like industry? – How would this be measured?

RQ2: Did the type of research evolve over time as would be predicted by the industry life cycle and the Gartner hype curve? I am not sure what this means. What is “type of research?” – Research artifacts … taking predictable formats as was illustrated in O’Leary (2008,2009)? What does evolution mean here? – Research uses previous research, is that what “evolution” means? What predictions? – Research is different than industry – Gartner, Moore, Rogers talk about adoption by companies but what would they predict about expert system research? How do we measure/describe research evolution?

RQ3: Did the type of researcher evolve over time as would be predicted by the adoption life cycle? What does it mean for individual “type of researcher” to evolve? – Is this the individual researcher? – Does it mean that groups of researchers change or the composition of the group or their knowledge or … How do we measure “type of researcher”? – We can see the type of research, but researcher? How does adoption life cycle allow us to predict “type of researcher”? – What does it mean to be a “laggard?” – See next slide …

“Laggards would be those researchers who have no interest in the technology per se, but they feel somewhat compelled to do some research in this area or to at least acknowledge this research in their own research portfolio.” ? Applying all of the life cycle schemes do not make a lot of sense to me. Would a new Ph. D. student be a “laggard”? Who is in the “late majority?” This scheme refers to the reaction time, but researchers constantly stream in, e.g., Ph. D. students

RQ4: Did the evolution of the type of researcher encounter a chasm that slowed or stopped expert system research as would be predicted by Moore? What does it mean to talk about the “evolution of the type of researcher?” – How do we measure that evolution? What is a “chasm” here? When is the start of a chasm or the stop of a chasm? – How can we measure that a “chasm” stopped or slowed expert systems research? Doesn’t Moore talk about marketing? – How would we use Moore to predict here? Perhaps researchers – realized that many of the issues had been addressed – realized that there were fatal flaws in expert systems?

What is a chasm here? What does it mean in research terms? Are these faculty? Do faculty act like firms? How does a researcher become part of the late majority? Why become a laggard? Is this reaction time, cluelessness? The issue is the research question, right? How does that fit?

3-c-Alternative Interpretation of Moore? Rather than researchers, perhaps an alternative interpretation is that these are journals. – Those journals are more to less receptive of new or technology research – A chasm would mean no research journals are interested – If no journals go for the new technology, methodology, etc. then researchers need to make “new” journals, ESR, ISAFM, JIS, IJAIS – Was the "gap” after those journals? More Less GAP ESR JAE?

4. Perhaps we should look somewhere else for guidance …Kuhn’s Paradigms? Perhaps we should ask what the expert system “paradigm” is or was? – Was there an AIS ES paradigm? – Was there a competing paradigm? – What was the shared paradigm of expert systems? When we went to AIS ES we assumed that we could use the same approach taken with scientific knowledge BUT in discipline laden with non scientific judgments. – We took the expert system paradigm and did not question it – There are “Layers” of paradigms in applied research such as AIS expert systems, that need to be accommodated

Life cycles are models of moving a theory to become a paradigm “Pre-paradigm schools” – Early in the life cycle, “emerging technologies” – E.g., rule-based systems. Did we prepare people for the new paradigm? – “Men whose research is based on shared paradigms are committed to the same rules and standards for scientific practice.” “To be accepted as a paradigm, a theory must seem better than its competitors” – Is expert systems a better model of auditor behavior? – What were the competitor paradigms?

Tracking the Expert System Paradigm Did older schools begin to disappear? If it was a new paradigm then there is “paradigm building” – “ … the successive transition from one paradigm to another via revolution is the usual developmental pattern of mature science.” The paradigm affects the practice of the group that practices the field. – Some “cling” to the old views … do we see anyone clinging to the old views of, e.g., expert systems? – “Those unwilling or unable to accommodate their work to it must proceed in isolation or attach themselves to some other group. ”

When is it a paradigm …? “In the sciences … the formation of specialized journals, the foundation of specialists’ societies, and the claim for a special place in the curriculum have usually been associated with a group’s first reception of a single paradigm.” – Sounds descriptive of accounting AIS / ES “Paradigms gain their status because they are more successful than their competitors in solving a few problems that the group of practitioners has come to recognize as acute.” – Did AIS / ES help us solve any new problems? Anomalies disrupt paradigms … causing chasms of sorts and breaking up paradigms – There was empirical research that served to attack the paradigm Kuhn talks of “revolutions” as changes of world views

Types of Research via Kuhn Theoretic work comes first, followed by different types of empirical research “First is that class of facts that the paradigm has shown to be particularly revealing of the nature of things.” “A second usual but smaller class of factual determinations is directed to those facts that, though often without much intrinsic interest, can be compared directly with predictions from the paradigm theory” Third, “ … empirical work undertaken to articulate the paradigm theory, resolving some of its residual ambiguities and permitting the solution of problems to which it had previously only drawn attention.”

5. What happened to Expert Systems? (An Evolutionary View) Expert systems were replaced by “knowledge-based systems” – Researchers realized that the focus was broader than experts. – Expert systems got a “bad name” – they just did not work the way we wanted them to. Other forms of AI became interesting, e.g., CBR, qualitative reasoning, etc. The web came along and changed everything … Then it was recognized that it was still useful to have people involved so the world got interested in “knowledge management” Researchers began investigating “ontologies” that were at the basis of both human and machine systems

What really happened to ES? (My personal research view) “One big 4 representative called … (expert systems) a ‘reality mugging,’ meaning what worked in the research lab did not work in the field.” – My own research found that expert systems did not really work, even in the lab. In experiments I found that such systems led to inconsistent decisions Researchers made errors implementing systems and characteristics, e.g., uncertainty factors – I think there must have been a confirmation bias in some of the research – People were more concerned with building the systems than seeing what could go wrong, if the systems actually worked and when they broke.

What did the “chasm expert” Moore say about “artificial intelligence” “Today, however, AI has been relegated to the trash heap. Despite the fact that it was—and is—a very hot technology, and that it garnered strong support from the early adopters, who saw its potential for using computers to aid human decision making, it has simply never caught on as a product for the mainstream market. Why? When it came time for the early majority to absorb it into the mainstream, there were too many obstacles to its adoption …. So AI languished at the entrance to the mainstream, for lack of a sustained marketing effort to lower the barriers to adoption, and after a while it got a reputation as a failed attempt. And as soon as that happened, the term itself became taboo.”

Should we mourn expert systems? I do not think so. There are many problems with expert systems that do not seem very easy to fix. More interesting issues are around now – Other technologies have emerged I would bet that some expert system technology is embedded in a range of other technologies around us

Can I recommend a paper? I did a Google Search on “Why haven’t expert systems lived up to their promise?” – “Auditor Environmental Assessments” IJAIS, 2003

Questions?