1. Software Evaluation in AI

2. The Selection Problem Recency of products Evolving nature of products Variety of products Lack of standards

3. Common Approach to Choice Select a few seemingly important or complicated functions within the products, and compare the products based on them.

4. Shortcomings It is evaluator-specific and unreliable. No product could be judged by only a few attributes. It excludes the relevant issues in a particular application. It lacks an evaluation measure, reflecting the strength of a product or its ratings compared with its competitors.

5. A Structured Approach Contain all of the important features of AI products Make it possible to compare different types of products Accommodate the special needs and requirements of AI project Summarize the results of evaluation into a quantitative or qualitative measure

6. Attribute Hierarchy of AI Tools Financial Aspects Producer Aspects Special Aspects Hardware Aspects Functional Aspects

7. Financial Aspects One-time costs: purchasing cost Periodic costs: licensing, training, and maintenance

8. Producer Aspects Reputation Length of time in business Product line: compatibility issues Technical support

9. Social Aspects User group Number of users Compatible products: technical leadership of a product

10. Hardware Aspects Platform: OS, Networking, Parallel Processing I/O Devices Resource Requirements: RAM, Disk Storage … Efficiency: Response Time

11. Functional Aspects Knowledge Representation Inference Engine Knowledge Management Outside Hooks

12. Knowledge Representation Logic-Based Representation: Rule, Fuzzy Logic Object-Based Representation: Frame, Semantic Net Uncertainty Representation: Bayesian, Fuzzy Logic, Certainty Factors, User-Defined Meta-Knowledge Representation Mathematical Representation: Math Operations, Math Functions, Variables

13. Inference Engine Chaining: Forward, Backward, Mixed Induction: Dec. Tree Object-Oriented: Single vs. Multiple Inheritance Blackboard Conflict Resolution: Recency, Antecedent Ordered, Consequent Ordered, Top-Down

14. Knowledge Mgt. Tools User-Interface Creation: Graphics, Windows, NLP, Voice Input/Output, Help, Animation User Interface: How, Why, Graphics, Uncertainty I/O, Help Debugging Tools: Tracing, Error Messages Knowledge Maintenance: Editor, Menu, Mouse,

15. Knowledge Mgt. Tools Continued System Security: User Access, KB Control Integrated Tools: Databases, Spreadsheets, Forms, Files, Prog. Languages Developer - User Assistance: On-Line Documents, Off-Line Documents, Tutorials, Error-Message References

16. Outside Hooks Components Access Data: Database, Files, Spreadsheets Text Access: Reports, Forms, Word Processing Knowledge Base Access: Multiple Access, Concurrent Access to Different KBs Language Access: Access to Different Languages

17. Outside Hooks Components Continued Portability: Exporting to Other Platforms, Generating Standard Files, like ASCII

18. Legal Issues Data Processing Services Inc. v. L. H. Smith Oil Corp. : The Indiana Court of Appeals upheld a lower court’s verdict that Data Processing Services was liable for professional malpractice.

19. Legal Issues Quad County Distributing Co. v. Burroughs Corp.: This case is significant because the court held that a computer program is covered by the UCC provisions concerning the sale of products. This means that the injured party does not have to prove negligence on the part of manufacturer to recover damages.

20. Legal Issues Users Domain Experts Knowledge Engineers Seller Organizations

21. Users Who is responsible if a decision maker uses a defective expert system to make a decision that leads to damage? There are cases that the user could be held responsible even though he/she has been unaware of the fault. For example, when a software error caused a machine to dispense a lethal dose of radiation to a patient, the doctor was sued alongside the manufacturer of the machine and the institution where the machine was being used.

22. Users On the other hand, there could be consequences for NOT using an available an available system. Take the case of a nurse who does not have access to a doctor, and chooses not to use an expert system that could save a patient’s life. Is the nurse liable for negligence? Need for procedures?

23. Domain Experts Experts who do not have adequate expertise ti stand the test of a court challenge should altogether avoid getting involved in the development of the system.

24. Knowledge Engineer The knowledge engineer could damage the integrity of the knowledge base by his personal biases, negligence, and lack of understanding of the knowledge domain. The documentation process and paper trail of the knowledge engineering process would be of critical importance in auditing the quality of knowledge engineering.

25. Seller If Turbo Tax program is defective and gives wrong tax advice that leads to financial losses, is the company liable?

26. Seller Almost all software companies have limited warranties. As long as a software system is considered a product, such warranties would protect the seller. However, when the system is considered to be providing a service to customers, then such disclaimers could not prevent litigation.

27. Some Questions... Can management force employees to contribute their expertise? What is the value of an expert opinion in court when the expertise is encoded in a computer? Who is liable for wrong information provided by an ES? Who owns the knowledge in KB?