Denys Zhdanov @deniszh FOSDEM 2018 What's new in Graphite 1.1 Denys Zhdanov @deniszh FOSDEM 2018.

Slides:



Advertisements
Similar presentations
DynaTrace Platform.
Advertisements

Coursework.  5 groups of 4-5 students  2 project options  Full project specifications on 3 rd March  Final deadline 10 th May 2011  Code storage.
Reimagining the business of apps ©2013 NativeX Holdings, LLC.
Single view of customer Support deposit and loan accounts Fully integrated General Ledger module that can be customised according to customer specification.
27. to 28. March 2007 | Geneva, Switzerland. Fabrice Romelard ilem SA Level 200.
LHCbPR V2 Sasha Mazurov, Amine Ben Hammou, Ben Couturier 5th LHCb Computing Workshop
Data Warehousing: Defined and Its Applications Pete Johnson April 2002.
Migrating to EPiServer CMS 5 Johan Björnfot -
SQL Reporting II Another tool in our IT toolbox. A free with Microsoft SQL that empowers a few levels of users. By Bryan Yates - Programmer.
Getting to Push Button Deploys Moovweb January 19, 2012.
SharePoint 2010 Business Intelligence Module 6: Analysis Services.
SharePoint 2010 Business Intelligence Module 10: Reporting Services.
HBase A column-centered database 1. Overview An Apache project Influenced by Google’s BigTable Built on Hadoop ▫A distributed file system ▫Supports Map-Reduce.
Yokogawa Electric Corporation Copyright © Yokogawa Electric Corporation Release 2.10 Functionality Overview September 2004.
1 ® Copyright 2009 Adobe Systems Incorporated. All rights reserved. Adobe confidential. 1 Building Portlets with ColdFusion Pete Freitag Foundeo, Inc.
Who uses it? MichaelMoore.com What's it all about? Rapid Development Clean, Pragmatic Design.
Copyright © Yokogawa Electric Corporation Release 2.10 Functionality Overview September 2004.
Designing High Performance BIRT Reports Mica J. Block Director Actuate Corporate Engineers Actuate Corporation.
1 MSTE Visual SourceSafe For more information, see:
1 The EDIT System, Overview European Commission – Eurostat.
Distributed Time Series Database
"Data sources index" a web application to list projects in Hadoop Luca Menichetti.
1 A lightweight Monitoring and Accounting system for LHCb DC'04 production V. Garonne R. Graciani Díaz J. J. Saborido Silva M. Sánchez García R. Vizcaya.
ESG-CET Meeting, Boulder, CO, April 2008 Gateway Implementation 4/30/2008.
Monitoring with InfluxDB & Grafana
Aggregator  Performs aggregate calculations  Components of the Aggregator Transformation Aggregate expression Group by port Sorted Input option Aggregate.
ASP.NET MVC An Introduction. What is MVC The Model-View-Controller (MVC) is an architectural pattern separates an application into three main components:
CHAPTER 7 LESSON C Creating Database Reports. Lesson C Objectives  Display image data in a report  Manually create queries and data links  Create summary.
LabKey Server 10.3 What’s New Feature List High-throughput NAb New Customize View UI Next-gen sequencing and genotyping – Galaxy integration for.
Explore engage elevate Data Migration Without Tears Mike Feingold Empoint Ltd Tuesday 10th November 2015.
ConTZole Tomáš Kubeš, 2010 atlas-tz-monitoring.cern.ch An Interactive ATLAS Tier-0 Monitoring.
Metrics at Mantas Klasavičius.
ASP.NET 2.0 Mohammed Abdelhadi Developer.NET Evangelist Microsoft Corporation.
Metrics data published Via different methods Monitoring Server
Data Visualization with Tableau
The Post Windows Operating System
Network measurements with InfluxDB
Ask the Experts – Building Login-Based Sites in AEM
Business Directory REST API
Service Fabrik Manage Enterprise Grade Services
What’s new in SQL Server 2017 for BI?
Monitoring with Clustered Graphite & Grafana
Data Virtualization Demoette… Flat-File Data Sources
Embeddable Discussions Ivelin Elenchev
Docker Birthday #3.
Marc-Elian Bégin ETICS Project, CERN
State of Testing Jason Kenny – ATS committer.
Experience in CMS with Analytic Tools for Data Flow and HLT Monitoring
Spark Presentation.
Using Grafana to show Postgres Statistics
Delivering Business Insight with SQL Server 2005
Azure CLI Deep Dive Neil Peterson Content Developer Microsoft.
Configuration Store in ONAP using Distributed KV Store (As part of making ONAP carrier grade) Consul.
Arrested by the CAP Handling Data in Distributed Systems
Relate to Clients on a business level
Monitoring for large infrastructure
Logging In ASP.Net Core 2.0.
Lecture 1: Multi-tier Architecture Overview
Nate Nelson I*LEVEL, Inc.
Software Metrics “How do we measure the software?”
Cloud Web Filtering Platform
What’s new in ASP.NET Core and Entity Framework 2.2 (Preview 3)
ClueGene: An Online Search Engine for Querying Gene Regulation
Technical Integration Guide
NIEM Tool Strategy Next Steps for Movement
Status and plans for bookkeeping system and production tools
Mark Quirk Head of Technology Developer & Platform Group
Empowering teams with scalable Shiny applications
Login Main Functions Via SAS Information Delivery Portal
API Working Group September 26, 2019 Includes notes from meeting.
Presentation transcript:

Denys Zhdanov @deniszh FOSDEM 2018 What's new in Graphite 1.1 Denys Zhdanov @deniszh FOSDEM 2018

Who am I Denys Zhdanov System engineer @ eCG / Marktplaats.nl Twitter / Github: @deniszh Sysadmin Ninja Graphite co-maintainer Data geek Pythonista Gopher

What? Graphite? Stores Time-Series data Renders it http://graphiteapp.org/

Why do we need Graphite in 2018? Great ecosystem: 80+ 3rd party tools on Tools page! - Collectors (for system and components): collectd / diamond / telegraf / statsd - Dashboards: (mostly Grafana now but also 20+ others) - Monitoring/Alerting: Moira / Seyren / Cabot / Grafana etc. - Compatible components: - relay/proxy: carbon-c-relay / carbon-relay-ng / grafsy / gruffalo - storage: BigGraphite / metrictank / go-carbon / carbon-clickhouse / graphouse - rendering: carbon-api / graphite-api

The good, the bad and ... Pros: - simple metric format <name> <value> <timestamp>\n - push model - rich ecosystem Cons: - no tags! - speed (use pypy or alternative backends) - scalability (use alternative backends) - hard to install (docker installation)

Tags, tags, tags AKA Dimensions metric.name;tag1=value1;tag2=value2 - compatible with Carbon format and Graphite tooling. Writing path: Normalization -> _tagged/xxxx/yyyy/metric;tag=value.wsp -> update TagDB TagDB and Tag API documentation - https://graphite.readthedocs.io/en/latest/tags.html Redis or Django-supported databases (SQLite / MySQL / Postgres)

Tags Querying: seriesByTag('tag=value') seriesByTag('name=~cpu\..*', 'tag1!=value1') - PCRE regex is supported! groupByTags - same as groupByNode seriesByTag('name=disk.used', 'server=~web.*') | groupByTags('sumSeries', 'datacenter') aliasByTags - same as aliasByNode seriesByTag('name=disk.used','datacenter=dc1') | aliasByTags('server', 'name')

Tags: external integrations Grafana: Tag support PR#9230 (Grafana 4.7.0) Auto-completion API for tags and values (Grafana 4.7.0) Tag value regex filtering & multi valued template variables PR#9911 (5.x) Prometheus: Remote read / write - PR#3533 / PR#3635 (based on example adapter) - Criteo’s Graphite-Remote-Adapter (including tags support now) - Prometheus read/write api in Graphite (WIP) - PR#2195 / PR#735

Python 3, finally Python 3 finally supported in all 3 major components - whisper, carbon and graphite-web. Should work with Python 2.7, 3.4, 3.5, 3.6 and PyPy. Graphite-web is running on Django 1.11 LTS (supported up to 2020) .

Custom (user defined) functions Pluggable functions, finally! Put your function in /opt/graphite/webapp/graphite/functions/custom or use FUNCTION_PLUGINS = [ 'some.function_plugin', ] Functions API curl -s "http://graphite/functions?pretty=1" (Support in Grafana planned - PR#10505, will be in Grafana 5.0b1)

Custom function example: ASAP ASAP: Prioritizing Attention via Time Series Smoothing http://futuredata.stanford.edu/asap Please see details in http://www.iamsysadmin.ninja/2018/01/asap-smoothing-in-graphite.html

New clustering code Pros: Local (whisper) and remote (cluster) calls are parallelized now. seriesByTag() calls will propagate to cluster members. http://cluster/render?target=xxx&target=xxx&target=seriesByTag(...) Easier way to write 3rd party finders. Cons: Beware of changes: some of 3rd party tools not ready yet, e.g. - go-carbon (supported in master, but no tags yet) - graphite-clickhouse (supported in master, but no tags yet) - graphouse (doesn’t support 1.1.x yet)

Pipe chaining for functions alias(movingAverage(scaleToSeconds(sumSeries(stats_global.production.counter s.api.requests.*.count),60),30),'api.avg') is really alias( movingAverage( scaleToSeconds( sumSeries( stats_global.production.counters.api.requests.*.count ) ,60) ,30) ,'api.avg')

Pipe chaining for functions alias(movingAverage(scaleToSeconds(sumSeries(stats_global.production.counters.api.requests.*.count),60),30),'api.avg') now can be written as sumSeries(stats_global.production.counters.api.requests.*.count)|scaleToSeco nds(60)|movingAverage(30)|alias('api.avg') or even stats_global.production.counters.api.requests.*.count|sumSeries()|scaleToSec onds(60)|movingAverage(30)|alias('api.avg')

Aggregation functions and xFilesFactor Consistent aggregations’ list: average, median, sum, min, max, diff, stddev, count, range, multiply and last. New aggregation functions aggregate, aggregateWithWildcards, movingWindow, filterSeries, highest, lowest and sortBy. Old functions are aliases for new ones e.g. sumSeries(some.metric.*) -> aggregate(some.metric.*, 'sum') xFilesFactor value to specify how many points in the window must be non-null for the output to be considered valid. https://grafana.com/blog/2016/03/03/25-graphite-grafana-and-statsd-gotchas/#runtime.consolidation

What’s next: beyond 1.1.x Remove Django, but keep metric tree explorer and dashboard view as separate (optional) components Fix for “new metric propagation delay” bug (workarounds are exists, e.g. Graphite-Clickhouse as cache) Separate TagDB server (probably even in Go?)

Thank You! Questions? issues? Contributions? https://github.com/graphite-project IRC: #graphite on FreeNode https://answers.launchpad.net/graphite / graphite@librelist.com