Collaborative development & Source Code Repository October, 2015 SDMX Tools Task Force

Objectives of this presentation Eurostat: What is CITnet SDMX tools set at CITnet (Confluence, Jira, Stash/Git, Nexus) How to work with CITnet GIT repositories (example of SDMX-RI Tutorial) How to deal with dependencies (example of SDMX-RI Tutorial) Participants: Comment on the proposal Collaborative development and governance October, 2015 SDMX Tools Task Force

What is CITnet? October, 2015 SDMX Tools Task Force

CITNet tools for collaborative development CITNet provides: Source Version Control Software Issue tracker Code explorer and Collaborative Code Review Software code repository management Continuous integration Agile Iteration Management and Planning Collaborative documentation October, 2015 SDMX Tools Task Force

SDMX tools at CITnet Uses Stash from Atlassian Project URL for each tool E.g. SDMX-RI: https://webgate.ec.europa.eu/CITnet/confluence/display/SDMXRI https://webgate.ec.europa.eu/CITnet/stash/projects/SDMXRI https://webgate.ec.europa.eu/CITnet/jira/browse/SDMXRI https://webgate.ec.europa.eu/CITnet/nexus/content/repositories/estat/org/s dmxsource/ https://circabc.europa.eu/w/browse/1c958330-ae5b-42e0-b7dd- 3d77a0141194 GIT URLs format: https://<CITnet user>@webgate.ec.europa.eu/CITnet/stash/scm/sdmxri/<repository> Notes: 1. Stash is highly integrated with CITnet Jira 2. One needs a CITnet account to use Stash / Jira 3. A single account for both Stash and Jira October, 2015 SDMX Tools Task Force

How we develop Project phases, planning and execution Each requirement should be traceable and test case covered October, 2015 SDMX Tools Task Force

Branching strategy Main branches in most projects Release branches X.Y.Z JIRA related branches: e.g. SDMXRI-XY SdmxSource (JAVA) – special case October, 2015 SDMX Tools Task Force

Practical and ready to use Agile and KTC approach Practical and ready to use Confluence Jira Git, Nexus Know The Customer approach October, 2015 SDMX Tools Task Force

Development & supporting tools October, 2015 SDMX Tools Task Force

Collaboration Projects & Status B. SDMX related actions (2014-2015) Collaboration Projects & Status Project Status SDMX-RI & .STAT (OECD) Under arrangement SDMX-RI & ISTAT Established SdmxSource & SDMX-RI & DSWS Transformation service STRUVAL Ongoing, planned Q1 2016 SdmxSource & SDMX Converter & EDIT SDMX-RI & PC-Axis Sweden Under arrangement (on hold) October, 2015 SDMX Tools Task Force

Collaborative development in CITnet User registers in CITNet (request registration) Requests to have access to a project (Stash/GIT) ESTAT includes the user to a group having Read access Registered users can clone, browse and fork any repository within the project create and contribute to pull requests targeting any of these repositories Request merge October, 2015 SDMX Tools Task Force

Collaborative development Link Shared development welcome Ask for access and inform on scope and planning Deliver and request merge Eurostat: Assess the possibilities to merge and publish as part of the tools In case of tools discuss with the system suplier how to publish and ensure evolution System supplier Deliver project supporting documentation Ensure bug fixing and support The expert group: Create guidelines and governance structure October, 2015 SDMX Tools Task Force

Governance proposal – simple and practical Define Actors Roles Responsibilities Scope Time Procedure Ensure Evolution & Quality Transparency Information sharing October, 2015 SDMX Tools Task Force

Items for discussion (1/4) Collaborative development Proposal: Based on already established projects, launch a pilot and work on best practices Discussion: Recommendations: Best practices Proposal: Expert group to create guidelines, project related templates and governance structure October, 2015 SDMX Tools Task Force

Items for discussion (2/4) SDMX development Accessibility and usability Proposal: Use CITnet for information sharing and development; Store all related documentation there and give access rights Discussion: Recommendations: Software sharing Proposal: Updated JoinUP; use Nexus for binaries; use Stash/Git for source code; October, 2015 SDMX Tools Task Force

Items for discussion (3/4) SDMX tools governance ESTAT SDMX tools set Proposal: Owned, maintained and supported by ESTAT; Use-cases relevant to the community implemented by ESTAT Discussion: Recommendations: Enhanced ESTAT SDMX tools Proposal: Bug fixes and support by the system supplier; Eurostat assesses the implementation; Merge if feasible; Maintain based on community needs October, 2015 SDMX Tools Task Force

Items for discussion (4/4) SDMX tools governance Other SDMX tools and frameworks (not based on Eurostat tools) Proposal: Keep informed; Assess integration based on user need and relevant use-cases for the community; Merge if feasible Discussion: Recommendations: October, 2015 SDMX Tools Task Force

Thank you! Questions? October, 2015 SDMX Tools Task Force

Working with SDMXRI repositories Step 1. Get the URL to clone. Visit a repository. E.g. MappingAssistant Click on “Clone” button. Copy the HTTP URL. October, 2015 SDMX Tools Task Force

Working with SDMXRI repositories Step 2. Clone Using a GIT tool (e.g. Git Extensions, SourceTree) Or command line: Important note: When cloning we are using the URL from Step 1 git clone –b <existing branch name> https://username@webgate.ec.europa.eu/CITnet/stash/scm/sdmxri/repository.git October, 2015 SDMX Tools Task Force

Working with SDMX-RI repositories Making changes Fork Clone the fork repository. Make code changes Add and commit changes Push to CITnet (and other remotes) Create a Pull request Discuss, possibly go back to step 2 The changes are merged October, 2015 SDMX Tools Task Force

Working with SDMX-RI repositories Fork From Stash Click at Fork link. To fork you go via Stash to the repository you want to fork and click “Fork” link. 2. A form will appear, where you can select a. the project (usually your own personal project space) b. the name of the forked repository and if we want enable fork syncing, which normally we want. 3. After clicking “Fork repository” the new forked repository will be created and will be shown in the browser. After this step you can clone the forked repository. 4. Then we can clone as we did before. If we already have a local repository with changes then in most cases we can just change the remote. October, 2015 SDMX Tools Task Force

Working with SDMX-RI repositories How to create a branch ? Using CITnet Stash or JIRA And then Clone that branch GIT tools Command line After a Clone. We can also create a branch. There are a few ways to create a branch. E.g. from Stash or JIRA, or some GIT tool e.g. GitExtensions or Visual Studio or the command line. You need to check out to the branch after creating it. There is also a single command do to that, using –b parameter in git checkout to create and change to the new branch. git branch <branch name> # create a branch git checkout <branch name> # change to a branch October, 2015 SDMX Tools Task Force

Working with SDMX-RI repositories Make codes changes October, 2015 SDMX Tools Task Force

Working with SDMX-RI repositories Add modified/new files Commit git add path/to/modified/file.cs git commit –m”SDMXRI-xyz Implementation of the make coffee interface.” Staging: First we stage files to be committed. Then we commit them.. Note: for those with experience in centralized source control software this only commits them locally.. October, 2015 SDMX Tools Task Force

Working with SDMX-RI repositories Push changes to CITnet If the branch does not yet exist on CITnet Else just push git push –u origin <branch name> Push changes: To apply the changes from the local repos to the CITnet repository e.g. the forked repository we need to push them. If the branch does not exist on CITnet we need to tell GIT to create it on CITnet when pushing. Else we just push the changes. git push October, 2015 SDMX Tools Task Force

Working with SDMX-RI repositories Create a pull request Using Stash Discuss To create a pull request and discuss about it we need to use Stash. October, 2015 SDMX Tools Task Force

SDMX-RI Dependencies 3rd party dependencies available via NuGET For NuGet 2.7+ deps downloaded automatically For NuGet prior 2.7 - Enable NuGET Restore SDMX-RI uses NuGet for almost all dependencies. 2. Normally if you are using the latest NuGET then dependencies should be downloaded automatically else NuGet Restore will be needed. 3. Since the current target is Visual Studio 2010, “NuGet Restore” may be needed even for Visual Studio 2013. October, 2015 SDMX Tools Task Force

SDMX-RI Dependencies Inter SDMX-RI Dependencies Only when building from GIT NuGet is used for resolving SDMX-RI libraries All SDMX-RI libraries produce NuGet packages No central repository (yet) Needs a local NuGet repository Correct order and correct branch matching needed Dependency version == branch name E.g. DataRetriever v5.3.0 is in branch 5.3.0 in the dr.net repository Internal dependencies: 1. Inter-SRI dependencies we mean the dependencies that one SDMX-RI project has to another, e.g. Data Retriever depends on SdmxSource and NSI WS depends on both SdmxSource and Data Retriever. 2. While the SDMX-RI packaged source code uses NuGET only for external dependencies, when using GIT the inter-SDMX-RI dependencies are also resolved with NuGet. 3. This difference exists because the packaged source code aims for simplicity, simply link to a DLL, while when developing SDMX-RI NuGet is very useful, but it has complexities. 4. The complexities are: there is no public central repository. NuGet does not have a default local repository like for example maven. So in order to work with GIT code one has to create a local repository, which could a folder on the local/shared disk or full web based repository. October, 2015 SDMX Tools Task Force

Inter SDMX-RI Dependencies Repository NuGet package name SDMXRI Dependencies sdmxsource.net SdmxSource.NuGet None estat_sdmxsource_extension.net Estat.SdmxSource.Extension.NuGet commonui.net EstatCommonUI.NuGet pcaxis.net PcAxis.NuGet common.net MappingStoreRetrieval.NuGet dr.net DataRetriever.NuGet October, 2015 SDMX Tools Task Force

Inter SDMX-RI Dependencies Repository NuGet package name SDMXRI Dependencies sr.net NuGet.StructureRetriever MappingStoreRetrieval.NuGet SdmxSource.NuGet Estat.SdmxSource.Extension.NuGet PcAxis.NuGet MappingAssistant EstatCommonUI.NuGet DataRetriever.NuGet October, 2015 SDMX Tools Task Force

Inter SDMX-RI Dependencies Repository SDMXRI Dependencies NSIClient.NET SdmxSource.NuGet Estat.SdmxSource.Extension.NuGet NSIWS.NET MappingStoreRetrieval.NuGet DataRetriever.NuGet NuGet.StructureRetriever PcAxis.NuGet October, 2015 SDMX Tools Task Force

Inter SDMX-RI Dependencies Repository SDMXRI Dependencies TestAuthConfig.NET MappingStoreRetrieval.NuGet SdmxSource.NuGet Estat.SdmxSource.Extension.NuGet EstatCommonUI.NuGet TestClient.NET DataRetriever.NuGet NuGet.StructureRetriever PcAxis.NuGet October, 2015 SDMX Tools Task Force

Dev Environment (.NET) Visual Studio 2013 Professional/Community or better Re-Sharper might help with SdmxSource... But not free GIT Basic support build in Visual Studio 2013. We use MSysGit + Git Extension. API Documentation Doxygen Sandcastle Help File Builder Ideal Dev Environment: An ideal development environment for developing SDMX RI and SdmxSource would consist of : 1. Visual Studio, preferably Visual Studio 2013 professional/community or better. Currently we target Visual Studio 2010 but we develop in both in 2010 and 2013. 2. Because of the size of SdmxSource some tool like ReSharper does help navigating e.g. finding an interfaces implementing classes or refactoring. But it is not free and requires annual subscription. 3. To access GIT repositories. a. Visual Studio 2013 provides some basic support b. we use msysgit (Command line) and GitExtensions which has a nice GUI and Windows Explorer and Visual Studio integration. 4. If someone is interested in producing API documentation. There are two solutions: a. One use Doxygen which can produce very nice documentation with diagram and such but currently it cannot handle SdmxSource… b.Sandcastle Help File Builder is another option which is used for SdmxSource. October, 2015 SDMX Tools Task Force

Dev Environment (Databases) SQL Server >= 2005, Oracle >=10 and MySQL >= 5.5 For SDMX-RI related development. Tools SQL Server Management Studio 2014 Oracle SQL Developer 4.1 MySQL Workbench 6.3. For SDMXRI development, currently the minimum supported versions are : 1. SQL Server 2005, 2. Oracle 10 3. MySQL 5.5. But since those are very old versions one can of course use a more recent version. Each DB vendor provides an environment to assist with database development and maintenance: 1. Microsoft provides SQL Server Management Studio, 2. Oracle provides the SQL Developer for Oracle 3. MySQL there is Workbench. Of course there are many other tools that could be used with more or less functionality. October, 2015 SDMX Tools Task Force

Support strategy CITnet Jira tool employed as single (ESTAT) repository of Bugs Support requests Suggestions for enhancements Other CITNet elements Source code repository Wikis Support levels 1st level: central problem registration and reporting 2nd level: problem analysis and resolution 3rd level: bug-fixing and new feature analysis (per product) October, 2015 SDMX Tools Task Force

SDMX Tools in CITnet / Project structure Current/ Change Branch GIT Clone URL Create Pull Request Clicking to a repository e.g. MappingAssistant we see the source code of Mapping Assistant. 2. Above that we see the current branch and clicking at it allows us to select another the branch. By default the master branch is selected. 3. On the top right we can get the “clone URL” by clicking at the “Clone” button. 4. You can create a pull request by clicking at the” Create pull request.” 5. You can create a fork by clicking at the fork link. Fork Files (e.g. Source Code.) October, 2015 SDMX Tools Task Force

Branching in Stash/Git Selecting a branch Creating a branch Stash/Git The branch menu allows to search and select branch. It is also possible to search and select a tag. Jira October, 2015 SDMX Tools Task Force