Semi-Automated Design Guidance Enhancer (SADGE) A Framework for Architectural Guidance Development Mohsen Anvaari Norwegian University of Science and Technology.

Slides:



Advertisements
Similar presentations
Language Technologies Reality and Promise in AKT Yorick Wilks and Fabio Ciravegna Department of Computer Science, University of Sheffield.
Advertisements

1 OOA-HR Workshop, 11 October 2006 Semantic Metadata Extraction using GATE Diana Maynard Natural Language Processing Group University of Sheffield, UK.
Modelling with expert systems. Expert systems Modelling with expert systems Coaching modelling with expert systems Advantages and limitations of modelling.
The 20th International Conference on Software Engineering and Knowledge Engineering (SEKE2008) Department of Electrical and Computer Engineering
A Stepwise Modeling Approach for Individual Media Semantics Annett Mitschick, Klaus Meißner TU Dresden, Department of Computer Science, Multimedia Technology.
Two Key ERP Principles Standardisation – Industry level – Firm level Centralisation – Cost saving – eg computing – Control over resources – Organisational.
Learning Semantic Information Extraction Rules from News The Dutch-Belgian Database Day 2013 (DBDBD 2013) Frederik Hogenboom Erasmus.
HOW DO PROFESSIONAL DEVELOPERS COMPREHEND TO SOFTWARE Report submitted by Tobias Roehm, Rebecca Tiarks, Rainer Koschke, Walid Maalej.
Realism in Assessment of Effort Estimation Uncertainty: It Matters How You Ask By Magne Jorgensen IEEE Transactions on Software Engineering Vol. 30, No.
GENERATING AUTOMATIC SEMANTIC ANNOTATIONS FOR RESEARCH DATASETS AYUSH SINGHAL AND JAIDEEP SRIVASTAVA CS DEPT., UNIVERSITY OF MINNESOTA, MN, USA.
Research topics Semantic Web - Spring 2007 Computer Engineering Department Sharif University of Technology.
Chapter 6: Design of Expert Systems
IR & Metadata. Metadata Didn’t we already talk about this? We discussed what metadata is and its types –Data about data –Descriptive metadata is external.
Annotating Documents for the Semantic Web Using Data-Extraction Ontologies Dissertation Proposal Yihong Ding.
Analysis of scientific research Mario Sangiorgio Giordano Tamburrelli.
Info1409 De Montfort University1 Requirements Modelling Systems Analysis & Design Academic Year 2008/9 Info 1409 Lecture 7.
Presented by Zeehasham Rasheed
Trade Study Training Need and Goals Need Consistent methodologies and practices performing trade studies Pros/cons, advantages/disadvantages, customer/management.
Towards Semantic Web: An Attribute- Driven Algorithm to Identifying an Ontology Associated with a Given Web Page Dan Su Department of Computer Science.
Xiaomeng Su & Jon Atle Gulla Dept. of Computer and Information Science Norwegian University of Science and Technology Trondheim Norway June 2004 Semantic.
The Software Product Life Cycle. Views of the Software Product Life Cycle  Management  Software engineering  Engineering design  Architectural design.
Introduction to Systems Analysis and Design
Software Architecture premaster course 1.  Israa Mosatafa Islam  Neveen Adel Mohamed  Omnia Ibrahim Ahmed  Dr Hany Ammar 2.
Erasmus University Rotterdam Introduction Nowadays, emerging news on economic events such as acquisitions has a substantial impact on the financial markets.
Alert Correlation for Extracting Attack Strategies Authors: B. Zhu and A. A. Ghorbani Source: IJNS review paper Reporter: Chun-Ta Li ( 李俊達 )
1 Validation & Verification Chapter VALIDATION & VERIFICATION Very Difficult Very Important Conceptually distinct, but performed simultaneously.
Citation Recommendation 1 Web Technology Laboratory Ferdowsi University of Mashhad.
EXCS Sept Knowledge Engineering Meets Software Engineering Hele-Mai Haav Institute of Cybernetics at TUT Software department.
Automatic Lexical Annotation Applied to the SCARLET Ontology Matcher Laura Po and Sonia Bergamaschi DII, University of Modena and Reggio Emilia, Italy.
Key Challenges for Modeling Language Creation by Demonstration Hyun Cho, Jeff Gray Department of Computer Science University of Alabama Jules White Bradley.
Scott Duvall, Brett South, Stéphane Meystre A Hands-on Introduction to Natural Language Processing in Healthcare Annotation as a Central Task for Development.
PAUL ALEXANDRU CHIRITA STEFANIA COSTACHE SIEGFRIED HANDSCHUH WOLFGANG NEJDL 1* L3S RESEARCH CENTER 2* NATIONAL UNIVERSITY OF IRELAND PROCEEDINGS OF THE.
Creating a Shared Vision Model. What is a Shared Vision Model? A “Shared Vision” model is a collective view of a water resources system developed by managers.
Chapter 10 Information Systems Analysis and Design
UOS 1 Ontology Based Personalized Search Zhang Tao The University of Seoul.
1 Technologies for (semi-) automatic metadata creation Diana Maynard.
OBJECT ORIENTED SYSTEM ANALYSIS AND DESIGN. COURSE OUTLINE The world of the Information Systems Analyst Approaches to System Development The Analyst as.
Combining terminology resources and statistical methods for entity recognition: an evaluation Angus Roberts, Robert Gaizauskas, Mark Hepple, Yikun Guo.
Extracting Metadata for Spatially- Aware Information Retrieval on the Internet Clough, Paul University of Sheffield, UK Presented By Mayank Singh.
Information Systems Engineering. Lecture Outline Information Systems Architecture Information System Architecture components Information Engineering Phases.
1 Learning Sub-structures of Document Semantic Graphs for Document Summarization 1 Jure Leskovec, 1 Marko Grobelnik, 2 Natasa Milic-Frayling 1 Jozef Stefan.
Debug Concern Navigator Masaru Shiozuka(Kyushu Institute of Technology, Japan) Naoyasu Ubayashi(Kyushu University, Japan) Yasutaka Kamei(Kyushu University,
인지구조기반 마이닝 소프트컴퓨팅 연구실 박사 2 학기 박 한 샘 2006 지식기반시스템 응용.
Sergey Gromov Yulia Krasilnikova Vladimir Polyakov (NRTU MISIS, Moscow) KNOWLEDGE BASE CREATION FOR NATIONAL NANOTECHNOLOGY NETWORKS «CONSTRUCTIONAL NANOMATERIALS»
BioRAT: Extracting Biological Information from Full-length Papers David P.A. Corney, Bernard F. Buxton, William B. Langdon and David T. Jones Bioinformatics.
Software Architecture Evaluation Methodologies Presented By: Anthony Register.
27/3/2008 1/16 A FRAMEWORK FOR REQUIREMENTS ENGINEERING PROCESS DEVELOPMENT (FRERE) Dr. Li Jiang School of Computer Science The.
 Copyright 2009 Digital Enterprise Research Institute. All rights reserved. Digital Enterprise Research Institute Expert (and Novice) Finding.
Of 33 lecture 1: introduction. of 33 the semantic web vision today’s web (1) web content – for human consumption (no structural information) people search.
Process Modeling
Volgograd State Technical University Applied Computational Linguistic Society Undergraduate and post-graduate scientific researches under the direction.
Emerging Trend Detection Shenzhi Li. Introduction What is an Emerging Trend? –An Emerging Trend is a topic area for which one can trace the growth of.
SQL Based Knowledge Representation And Knowledge Editor UMAIR ABDULLAH AFTAB AHMED MOHAMMAD JAMIL SAWAR (Presented by Lei Jiang)
Creating Subjective and Objective Sentence Classifier from Unannotated Texts Janyce Wiebe and Ellen Riloff Department of Computer Science University of.
Semantic web Bootstrapping & Annotation Hassan Sayyadi Semantic web research laboratory Computer department Sharif university of.
Comparing Document Segmentation for Passage Retrieval in Question Answering Jorg Tiedemann University of Groningen presented by: Moy’awiah Al-Shannaq
1 Class exercise II: Use Case Implementation Deborah McGuinness and Peter Fox CSCI Week 8, October 20, 2008.
By: M.Gillespie, H.Holmani, D. Kotowski, and D.A.Stacey Presented By: Daniel Kotowski
Virtual Examples for Text Classification with Support Vector Machines Manabu Sassano Proceedings of the 2003 Conference on Emprical Methods in Natural.
ANALYSIS PHASE OF BUSINESS SYSTEM DEVELOPMENT METHODOLOGY.
AUTONOMOUS REQUIREMENTS SPECIFICATION PROCESSING USING NATURAL LANGUAGE PROCESSING - Vivek Punjabi.
Identifying “Best Bet” Web Search Results by Mining Past User Behavior Author: Eugene Agichtein, Zijian Zheng (Microsoft Research) Source: KDD2006 Reporter:
Software Design and Architecture Muhammad Nasir Software Architecture Documentation
Design Evaluation Overview Introduction Model for Interface Design Evaluation Types of Evaluation –Conceptual Design –Usability –Learning Outcome.
Systems Analysis Lecture 5 Requirements Investigation and Analysis 1 BTEC HNC Systems Support Castle College 2007/8.
WP4 Models and Contents Quality Assessment
Towards a framework for architectural design decision support
Architecture Components
CSc4730/6730 Scientific Visualization
Comparative Evaluation of SOM-Ward Clustering and Decision Tree for Conducting Customer-Portfolio Analysis By 1Oloyede Ayodele, 2Ogunlana Deborah, 1Adeyemi.
Presentation transcript:

Semi-Automated Design Guidance Enhancer (SADGE) A Framework for Architectural Guidance Development Mohsen Anvaari Norwegian University of Science and Technology Trondheim, Norway Olaf Zimmermann University of Applied Sciences of Eastern Switzerland Rapperswil, Switzerland

Agenda Introduction An Example Scenario SADGE Framework Development SADGE Framework Operation and Maintenance Framework Evaluation – Setup Framework Evaluation – Results Discussion Related Work Conclusions Future Work Questions Semi-Automated Design Guidance Enhancer (SADGE)

Introduction Architectural decision making is a non-trivial task for architects Researchers have developed several concepts, methods and tools to assist practitioners One of the promising approaches is a decision identification technique It enhances architectural guidance (decisions required) from decisions made previously Practitioners still are reluctant to apply this technique SADGE automatically extracts architectural issues from architectural related documents by applying natural language processing first Then, it manually post processes and fine-tunes the extracted knowledge entities Semi-Automated Design Guidance Enhancer (SADGE)

An Example Scenario The book is a 500 pages text, and architect “A” has spent 50 hours to manually annotate 200 sentences that contain architectural issues SADGE assists the architects to automatically annotate the sentences that contain architectural issues

SADGE Framework Development Automated Annotator: We used ANNIE and JAPE from an open source tool called GATE (general architecture for text engineering) Semi-Automated Design Guidance Enhancer (SADGE)

Annotating Rules and Catalog of Terms High Priority Terms agree on, choose Low Priority Terms approach, articulate, class, component, construct, concern, define, design determine, different, employ, establish, evaluate, exchange, facilitate framework, investigate, limitation, make, philosophy, principle, profile provide, protocol, recommend, refactor, require, schema, select, service several, strategy, support, topology, transaction management, type, various Semi-Automated Design Guidance Enhancer (SADGE)

SADGE Framework Operation and Maintenance

Framework Evaluation Purpose: Evaluate SADGE by comparing the three quality attributes (Precision, Recall, Effort) of manual, automatic and semi-automatic approaches of decision identification – Recall: A∩B/B – Precision: A∩B/A

Framework Evaluation Participants: 19 students of a bachelor’s program in information technology – randomly selected, familiar with the software architecture, but not familiar with the concept of architectural knowledge We divided them into two groups of ten and nine students Stages: In the first stage, students were supposed to annotate the sentences they think are architectural issues In the second stage, they were asked to reject the sentences they disagreed with automated annotator to fine-tune the results Semi-Automated Design Guidance Enhancer (SADGE)

Framework Evaluation - Material The document we selected for the experiment was the one that had the smallest deviation from the mean of precision and recall of annotating all of the tested documents and therefore can be considered as a representative of the tested documents The right figure shows how the output of the experiment can look like Semi-Automated Design Guidance Enhancer (SADGE)

Framework Evaluation - Results Semi-Automated Design Guidance Enhancer (SADGE)

Discussion The effort reduction results are in correspondence to our expectation The recall and precision of student annotations are very low (38 and 25 respectively) In the industrial projects we expect that both precision and recall should be higher The results for automated annotator are relatively high (86 and 57) These show that if the people in charge of enhancing architectural guidance are not expert enough, automated approach will perform more accurate and more sensitive by spending much less time The other part that is not the same as our expectation is the recall rate of semi-automatic approach. The reason may lie in the expertise level of the participants Semi-Automated Design Guidance Enhancer (SADGE)

Related Work Using NLP for knowledge extraction is not novel in software engineering – Existing tools and methods for generating object oriented models from natural language texts by applying NLP (Perez-Gonzalez, 2002)(Harmain and Gaizauskas, 2003)(Bajwa et al., 2009)(Soeken et al., 2012) The majority of work has been done to extract the object oriented data from the documents Extraction of architectural knowledge (specifically architectural decisions) is not mainly in focus (Figueiredo et al., 2012) approach and TREx (López et al., 2012) are two works applying NLP in AK The catalog of terms and annotating rules are not presented in the papers nor publicly accessible It is not possible to replicate the approaches and as a result the comparison is not feasible Semi-Automated Design Guidance Enhancer (SADGE)

Conclusions Applying natural language processing (NLP) techniques, SADGE automatically annotates (highlight) the architectural issues to reduce the knowledge engineering effort Next, a knowledge engineer manually post processes the outcome of the automated annotation process to increase the accuracy of the results Our preliminary evaluation shows that when the practitioners are not experienced, the automatic approach has the highest sensitivity and the lowest processing time while the semi-automatic approach has the highest accuracy SADGE can assist practitioners to identify required architectural decisions more confidently Semi-Automated Design Guidance Enhancer (SADGE)

Future Work Improving the sensitivity and accuracy of automated annotator by applying machine learning algorithms Increasing the training data by receiving more architectural related documents from real projects in the industry Evaluating the framework by conducting case studies that involve expert architects and also include project documents Semi-Automated Design Guidance Enhancer (SADGE)

References Perez-Gonzalez, H. G.: Automatically Generating Object Models from Natural Language Analysis, 17th annual ACM SIGPLAN conference on Object-oriented programming, systems, languages, and applications, ACM New York, USA, pp: 86 – 87 (2002) Bajwa, Imran Sarwar, Ali Samad, and Shahzad Mumtaz. "Object oriented software modeling using NLP based knowledge extraction." European Journal of Scientific Research 35.01: (2009) Harmain, H. M. and Gaizauskas, R.: CM-Builder: A Natural Language-based CASE Tool, Journal of Automated Software Engineering, 10, 2003, pp (2003) Soeken, M., Wille, R., and Drechsler, R.: Assisted behavior driven development using natural language processing. In Objects, Models, Components, Patterns (pp ). Springer Berlin Heidelberg (2012) López, C., Codocedo, V., Astudillo, H., & Cysneiros, L. M. (2012). Bridging the Gap between Software Architecture Rationale Formalisms and Actual Architecture Documents: An Ontology-Driven Approach. Science of Computer Programming, 77(1), (2012) Figueiredo, A. M., dos Reis, J. C., and Rodrigues, M. A.: Improving Access to Software Architecture Knowledge: An Ontology-based Search Approach. International Journal Multimedia and Image Processing (IJMIP), 2(1/2) (2012)

Thanks! Questions? Semi-Automated Design Guidance Enhancer (SADGE)