PX-Web development in Kostra PX-reference group meeting 14-15 November 2017 Rune Johansen Joined the family i Copenhagen 2014 meeting Thanks to SCB Upgrade to PX-Web was included in the Kostra project Highlight the areas where we used most of our time.

Servers Citrix Netscaler Varnish Varnish Varnish PX-Web ext. Varnish PX-Api Varnish PX-Web int. Varnish StatbankService Oracle Exadata Cisco Ace was replaced by Citrix Netscaler in 2017 * SSL termination * Loadbalancing * Sticky sessions (using cookies) Varnish in front of external applications, 4 servers also with sticky sessions (using cookies) 2 of every server ODA (Oracle Database Appliance) was replaced by Oracle Exadata in 2017 PX-reference group meeting 2017

Environments UTV (development) TEST QA PROD PX-reference group meeting 2017

continuous integration (CI) hudson-ci Build Test Package Release Deploy SSB heavy java teams Upgrade to newest Jenkins Look into Pipelines (jenkinsfile) PX-reference group meeting 2017

CNMM 2.4 Common Nordic Meta Model 2.4 Copenhagen, May 2015 SubTableVariable set and Grouping Increment size for column ValueCode EliminationCode column for ValueSet table PCAxis.Sql.* Decide to implement before FINAL Implemented with VIEWS The Sortcode field makes it possible to control the sortorder of the valuesets and groupings that are displayed in the dropdown for a variable on the selection page of PX-Web. The Default field will define if a valueset or grouping shall be selected by default for a variable of the selection page of PX-Web. PCAxis.Sql.* Norways main contribution in .NET Begin implementing before 2.4 is final to save time, didn’t happen we started 2107, but model still not final. Implement with VIEWS, because we already have our customized 2.1 model A later database upgrade will be necessary PX-reference group meeting 2017

Saved queries PX-internet/StatbankWeb PX-Web PxsQuery()  SavedQuery() pxs, pxs-xml PX-Web json api queries json saved querys PxsQuery()  SavedQuery() History intro. We use saved queries a lot and need compability with: Tablebuilder, dataset api Since we use serveral servers we need to store SQ in database. First thougt about addig pxs-xml as an export format in px-web and slightly modify our excisting applications. Ended up ful 2-way conversion and not changing anything but out internal api (StatbankService) PX-reference group meeting 2017

StatbankService StatbankService Tablebuilder Dataset API CMS PXS PXS PX-reference group meeting 2017

PX-reference group meeting 2017

PX-reference group meeting 2017

Saved Queries (continued) Converted all queries for PX-Web Validating old and new StatbankService PX-reference group meeting 2017

Interfaces PX-reference group meeting 2017

Indexing ISearchIndex interface Daily incremental Weekly full Performance problems with full indexing All servers maintain their own indexes (in every environment) We are profiling the SQL queries as we speak PX-reference group meeting 2017

Metrics IActionLogger interface ELK (ElasticSearch, Logstash, Kibana) PxDefaultLogger: stats.log PxSSBLogger: log4net.Appender.AdoNetAppender ELK (ElasticSearch, Logstash, Kibana) Google Analytics PX-reference group meeting 2017

Linking metadata IMetadIdProvider interface The aim for the IMetaIdProvider interface is to provide a way to transforms the MetaId property of the Table, Variable and Value in Paxiom to URL:s to one or more metadata systems. PX-reference group meeting 2017

Sign in for SSB employees Iauthorization interface PX-reference group meeting 2017

Short URLs Propose new interface: IRouteExtender Table: /(en/)statbank/table/<tableid> Table list: /(en/)statbank/list/<statistics id> Saved query: /(en/)statbank/sq/<query id> We juse all the provided interfaces an propose a new one: IRouteExtender We still need to document and create 1 or 2 reference implementations PX-reference group meeting 2017

Performance testing Neoload Confirmed problems with old StatbankWeb PX-Web outperformes Work in progress Compare old and new Lucene indexing of menu Work in progress Disposing connections PX-reference group meeting 2017

Validation and Transformation Language VTL is a standard language for defining validation and transformation rules for any kind of statistical data Compatible with existing standards such as GSIM, SDMX, DDI Developed by SDMX Technical Working Group https://sdmx.org/?page_id=5096 History: old version of Kostra had its own language for defining key figures, perl eval  PX-reference group meeting 2017

Validation and Transformation Language SSB implementation at GitHub https://github.com/statisticsnorway java-vtl java-vtl-connectors Connectors describe how to fetch and structure data from a source SsbKlassApiConnector SsbApiConnector PxApiConnector https://statisticsnorway.github.io/java-vtl/reference/#vtl-reference PX-reference group meeting 2017

PX-reference group meeting 2017