TOS / TIS Code Architecture Copyright © 2008 Talend. All rights reserved

IntroductionPage 1 Difference between TOS and TIS – Different builds include different plugins – Branding plugins. – Some plugins are used only for TIS. (DB repository, SVN respository…) – Some plugins are even use only for different builds of TIS. (AMC …) – EVERY plugin of TOS are used in all TOS or TIS build. TOS / TIS?

IntroductionPage 2 SVN Organization Different SVN Repository –TOS (Open Source Code) Anyone can access to the code. But need a SVN Account from Talend to commit. –TIS Shared (First TIS part Shared to China) Code not Open Source. This SVN Repository was created to allow people from China to commit some plugins for TIS. This include Joblets, DataPreview, Documentation… All SVN Accounts created in China got this access. –TIS Private This SVN was created first to have all TIS plugins. This access is only for people in France (not other countries). This include DB, SVN Repository / WebApp. People from Talend Beijing should have an access later.

IntroductionPage 3 What about the teams? Plugins for Component Team and UI Team – Components Team: org.talend.designer.components.localprovider org.talend.designer.codegen org.talend.core (folder about mapping) org.talend.designer.components.tisprovider (TIS components) – UI Team: All others… and include also the DB Mapping also depends the tasks. So about 60 plugins maybe in TOS + TIS Shared, and in TIS Private. This doesn’t mean one team can’t commit on the plugins of other team. Only show which plugin you will use most of the time.

IntroductionPage 4 Plugins organization Different plugins to remember for – Code Generation (Components team) – Repository – Job Editor – Routines – Business Modeler – Tools (SQL Builder, Ecosystem…) – External Components (tRowGenerator, tMap…) – Documentation – (…)

IntroductionPage 5 Build System Build Made with both TOS jobs and Ant. –Several types of build exists, these builds will simply list the plugins needed (depends the build) –Build depends of features dependencies. Build Version of TOS or TIS: When Create a new plugin, don’t forget that this plugin should be included in a feature ! If the new plugin is not included in a feature, maybe what you did will work in Eclipse. But this will never be included in the Build version ! When you add dependencies between plugins, never set the version of the plugin. Take care, by default with Eclipse 3.4 it will add the version…

IntroductionPage 6 Main Classes Main classes are linked to: – EMF Model – Job Creation / Edition – Parameters – Code Generation – Link between Job Creation and Code Generation. – Repository – Update Manager – Migration System

IntroductionPage 7 TOS / TIS PLUGINS Organization of the different plugins

IntroductionPage 8 TOS/TIS Plugins Code Generation (1 TOS) –org.talend.designer.codegen (TOS) Repository (2 TOS, 3 TIS Private) –org.talend.repository (TOS) –org.talend.repository.localprovider (TOS) –org.talend.repository.remoteprovider (TIS Private) –org.talend.repository.databaseprovider (TIS Private) –org.talend.repository.svnprovider (TIS Private) Job Editor (1 TOS, 2 TIS Shared) –org.talend.designer.core (TOS) –org.talend.designer.joblet (TIS Shared) –org.talend.designer.jobtemplate(TIS Shared)

IntroductionPage 9 TOS/TIS Plugins Routines (1 TOS, 1 TIS Shared) –org.talend.librairiesmanager (TOS) –org.talend.designer.routines.tisprovider (TIS Shared) Business Modeler (1 TOS) –org.talend.business.diagram (TOS) Documentation (1 TIS Shared) –org.talend.repository.documentation (TIS Shared) –Standard documentation done in org.talend.core. Tools: SQL Builder, EcoSystem, Scheduler... (4 TOS) –org.talend.sqlbuilder (TOS) –org.talend.designer.components.ecosystem (TOS) –org.talend.scheduler (TOS) –org.talend.expressionbuilder (TOS)

IntroductionPage 10 TOS/TIS Plugins External Components (9 TOS) –Simple components: org.talend.designer.trowgenerator (TOS) org.talend.designer.scd (TOS) org.talend.designer.fileoutputxml (TOS) org.talend.designer.alfrescooutput (TOS) –tMap org.talend.designer.mapper (TOS) org.talend.designer.abstractmap (TOS) –ELT Map org.talend.designer.dbmap (TOS) org.talend.designer.dbmap.mysql (TOS) org.talend.designer.dbmap.oracle (TOS)

IntroductionPage 11 TOS/TIS Plugins Metadata wizard plugins –org.talend.repository (TOS) This one contains all TOS wizards (DB, Delimited, Positional…) –org.talend.repository.ebcdic –org.talend.repository.sapwizard –org.talend.wsdl.advancement Contains some other options to existing wizard in TOS.

IntroductionPage 12 TOS/TIS Plugins Main TOS plugins used for development: –org.talend.commons –org.talend.commons.ui –org.talend.core –org.talend.designer.codegen –org.talend.designer.components.ecosystem –org.talend.designer.components.localprovider –org.talend.designer.core –org.talend.designer.rowgenerator –org.talend.designer.runprocess –org.talend.expressionbuilder –org.talend.librairiesmanager –org.talend.model –org.talend.repository –org.talend.repository.localprovider –org.talend.sqlbuilder

IntroductionPage 13 TOS/TIS Plugins Main TIS Shared plugins used for development: –org.talend.designer.cdc –org.talend.designer.components.preview –org.talend.designer.joblet –org.talend.designer.jobtemplate –org.talend.designer.routines.tisprovider –org.talend.repository.documentation –org.talend.repository.ebcdic –org.talend.repository.importation –org.talend.repository.sapwizard

IntroductionPage 14 MAIN PLUGINS / CLASSES Details about the main plugins and classes in TOS / TIS

IntroductionPage 15 EMF Model: org.talend.model Different Model Generated by EMF: – Business Used by the Business Modeler. – Component Used to load the components XML. Modifications must be done first on XSD file, then XSD transformed to EMF model. – Expression Used in the Expression Builder – joblet Used for the joblets, contains only the informations specific to the joblets.

IntroductionPage 16 EMF Model: org.talend.model –metadata Base for the Metadata wizards. Each informations from the Wizard will be saved from this model. – properties Used by any object that can be saved in the repository. Job, Db Connection… (Metadata Wizards model link to this one) – talendfile Job information are saved in this model. Same as the job used in memory, most of the informations are saved in the paramters. Depends the modification of the model, it might need first a validation either from nrousseau or mhelleboid before commit (depends the modification)

IntroductionPage 17 Job Creation: org.talend.designer.core Process (Job in fact!) –Each job is saved into EMF model, but is loaded and modified during edition in memory in other classes –loadXml / SaveXml. Load or Save the complete Job from or to EMF. Node –Graphical object of the components Connection –Object that store informations about the links All these classes extends the class Element. It’s in fact the GEF Model classes. GEF parts related are: ProcessPart, NodePart, ConnectionPart. Some other classes are related to GEF also, for the labels, or some icons, container displayed.

IntroductionPage 18 Parameters (Components setup) ElementParameter (UI Design of parameters) –This class will store parameters of: Process (Job) Node (Components) Connection (links) –These parameters will be used for: Display of information in the UI Code Generation (Mainly…) ElementParameterParser (Use of parameters in the code generation) –This class will be used to read easily the parameters from any object contain parameter (Process, Node, Connection…)

IntroductionPage 19 Global view of the Process ProcessElementParameter Connection NodeElementParameter 1 11 n 1 1 n n n n This is just a global graph to show that one process will contains several nodes, and several connections. And that each contains parameters. (Reality is a little more complex) You can consider that this model exists in: –EMF (Job information saved to repository) –Model part of GEF. So directly UI part. –Model used for code generation.

IntroductionPage 20 Code Generation ProcessorUtilities –Allow to generate easily any code generation for one job. –Code generation MUST call this class, never use Processor classes directly (unless specific use, like shadow process). JavaProcessor / PerlProcessor –These classes help for the code generation for the different languages. This will add all specific code for any language generated. CodeGeneratorEmittersPoolFactory (Comp. team) –Will generate code for each component. It will generate.java files for each jet file. CodeGeneratorService / CodeGenerator –Will generate the code once the components are setup, from a DataProcess (Not from the UI Process)

IntroductionPage 21 Link between Designer and CodeGeneration Take Care ! –Graphical Job is different than the job we generate ! –Generated process can be linked differently, and can have additional components. Difference between : –Process.getGraphicalNodes Get all “Graphical” nodes…. This means get all components directly visible on the This method return Node objects only. –Process.getGeneratedNodes Get all components really used for the code generation. This method can return DataNode and ExternalNode DataProcess, DataNode, DataConnection –This will be a copy of the graphical process, but with the modifications added to generate the process.

IntroductionPage 22 Links between process and code generation Process EMFProcess ProcessorUtilities Can Use graphical Process Or EMF Process Load DataProcess Contains Processor Contains Use to generate the code

IntroductionPage 23 Repository ProxyRepositoryFactory –Main class used to get any item from the repository. Any other repository shouldn’t be called directly, unless specific use. –To get this class from anywhere, for example: CorePlugin.getRepositoryService().getProxyRepositoryFactory() LocalRepositoryFactory –This is one of the classes for repository implementation. Other classes, no matter the repository type, got the same kind of implementation. RepositoryView –Implementation of the view for the repository RepositoryContentProvider –Model part of the repository view, this will get information from the repository used.

IntroductionPage 24 Classes linked to the language Java/Perl JavaProcessor / PerlProcessor –Code generation. Code viewer (tJava / tPerl): –TalendJavaSourceViewer –TalendPerlSourceViewer LanguageManager –Will get informations like current language used TypesManager, PerlTypesManager, JavaTypesManager –Functions related to schema types and types generated in the code.

IntroductionPage 25 Repository Wizards All TOS wizards are in the plugin org.talend.respository org.talend.repository.ui.wizards.metadata.connection All wizards will store informations directly to the EMF Model. Some wizards have links with a ShadowProcess, this kind of process will create a job same as in the designer and will use the components directly to display datas in the table. To get informations from the wizard to any property displayed in the components: –Class: RepositoryToComponentProperty This will allow to get any information from the repository wizard (Emf Model), to use it directly in the properties.

IntroductionPage 26 Component properties or Component Settings View ComponentSettingsView –This one can have different sections (called also category or just “tab”) –Each section can contain different parameters –The section depends on the setup of the components XML Classes MultipleThreadDynamicComposite and DynamicComposite –Will check all the parameters from the current section, and will display the related controls if needed. –Refresh all the composite if need to redraw only. –Refresh all the controls to refresh data if needed. Parameter UPDATE_COMPONENTS (need set to True to force refresh).

IntroductionPage 27 Component properties or Component Settings View xxxController, xxxGenerator –Needed for each control of the properties. –For example there is controller for TEXT, CLOSED_LIST (combo), SCHEMA_TYPE, … Extension point: org.talend.designer.core.generators –One extension point defined for each control. –Extension point will call the Generator class for the control, this one will give the related Controller EParameterName –This enum will define the global parameters with the display name for each. (specific components parameter will never be defined here)

IntroductionPage 28 Migration System Plugin org.talend.migrationTool and org.talend.core contains basic classes for migrations. Plugin org.talend.repository contains most of the migrations needed for TOS. Extension point to use to create migration task: –org.talend.core.migrationTask –For any example, check any migration task in: org.talend.repository.model.migration

IntroductionPage 29 Migration System Take Care ! Migration task will set compatibility or not between each version. If one migration is created at one moment, this means that new job can’t be imported directly in a previous version of TOS. Worse, if one migration is created in one version, then deleted later, a Job created in a previous version can’t be imported in a future version, this should never happen ! Rename a migration task id in the extension point is considered the same as delete a migration.

IntroductionPage 30 Update Manager ProcessUpdateManager –Nothing related with the update from Eclipse. –This will help any job or joblet to be “up to date” with the repository. Means that it will update any property field if anything has been modified in the repository.

IntroductionPage 31 Bugs ? For some parts of the code, need to take care, because it’s quite easy to add a bug… When create/modify a metadata wizard: –Remember that this wizard is or will be linked to one component. So if you do only the wizard and that’s all, it’s not really usefull for the user. The link to the component to set the properties must be added. –Each wizard should also have some drag & drop. –Take care to the property type field, and the link with the repository. Same with the UpdateManager. When modify the “Script”(code) Export: –Don’t forget that there is Java AND Perl mode to test.

IntroductionPage 32 Bugs ? When modify the components: –There should be a migration task at least if: The component has been renamed. One parameter has been renamed or have too much changes. … any time you think the components from a previous version won’t work in development version. Don’t forget that migration system is here to make possible to use a job even from TOS 1.0 to TOS x.y. When modify the link system of the job design, there is some rules for each link… so even a small modification on this, will need maybe lots of tests. …