1. Introduction to AWS Redshift
Maryna Popova BI Engineer GO EURO GmbH

2. Maryna Popova BI Engineer at GoEuro www.goeuro.com
LinkedIn:

3. SQLSat Kyiv Team Denis Reznik Eugene Polonichko Oksana Tkach
Yevhen Nedashkivskyi
Mykola Pobyivovk
Denis Reznik
Eugene Polonichko
Oksana Tkach
Oksana Borysenko

4. Sponsor Sessions Starts at 13:00
Don’t miss them, they might be providing some interesting and valuable information!
Congress Hall
DevArt
Conference Hall
Simplement
Room AC
DB Best
Predslava1
Intapp
NULL means no session in that room at that time ☺

5. Sponsors

6. Session will begin very soon :)
Please complete the evaluation form from your pocket after the session. Your feedback will help us to improve future conferences and speakers will appreciate your feedback!
Enjoy the conference!

7. Agenda What is AWS Redshift Columnar vs Row-based storage MPP
Data compression
Distkey and Sorkey
Vacuum in Redshift
Scaling
Features and Bugs
Q&A

8. What is Amazon Redshift
Amazon Redshift is a fully managed, petabyte- scale data warehouse service in the cloud(с) - documentation
My:
Amazon Redshift is scalable DWH in cloud.
It is columnar datastorage It is mpp
Управляемая
Колоночная
Массово-параллельная архитектура

9. What is MPP
In computing, massively parallel refers to the use of a large number of processors (or separate computers) to perform a set of coordinated computations in parallel (simultaneously).

10. Sequential processing
Input
Output
Processing time

11. Parallel Processing with equal workload distribution
Input
Output
Processing time

12. Parallel Processing with unequal workload distribution
Output
Input
Processing time

13. Redshift as MPP
Massively Parallel Processing (MPP): Amazon Redshift automatically distributes data and query load across all nodes. Amazon Redshift makes it easy to add nodes to your data warehouse and enables you to maintain fast query performance as your data warehouse grows.
Массово-параллельная_архитектура

15. Columnar vs Row data storage

16. Row - oriented

17. Row - oriented
In row-wise database storage, data blocks store values sequentially for each consecutive column making up the entire row. If block size is smaller than the size of a record, storage for an entire record may take more than one block. If block size is larger than the size of a record, storage for an entire record may take less than one block, resulting in an inefficient use of disk space.

18. Row - oriented data blocks store values sequentially
If block size is smaller than the size of a record, storage for an entire record may take more than one block. If block size is larger than the size of a record, storage for an entire record may take less than one block, resulting in an inefficient use of disk space

19. Row - oriented
Designed to return a record in as few operations as possible
optimal for OLTP databases
Disadvantage: inefficient use of disk space
In online transaction processing (OLTP) applications, most transactions involve frequently reading and writing all of the values for entire records, typically one record or a small number of records at a time. As a result, row-wise storage is optimal for OLTP databases.

20. Columnar
Using columnar storage, each data block stores values of a single column for multiple rows. As records enter the system, Amazon Redshift transparently converts the data to columnar storage for each of the columns.

21. Columnar data block stores values of a single column for multiple rows
reading the same number of column field values for the same number of records requires much less of the I/O operations compared to row-wise storage
block holds the same type of data ⇒ can use a compression scheme

22. Data compression in Redshift
specifies the type of compression that is applied to a column of data values as rows are added to a table
Applied during table design stage

23. Example create table dwh.fact_bookings_and_cancellations(
reporting_operations_id bigint,
booking_id varchar(255) DISTKEY,
lowest_unit_value_in_euros bigint,
operation_currency varchar(255) encode bytedict,
operation_date_time timestamp SORTKEY,
...

24. Default encodings
Columns that are defined as sort keys are assigned RAW compression.
Columns that are defined as BOOLEAN, REAL, or DOUBLE PRECISION data types are assigned RAW compression.
All other columns are assigned LZO compression.

25. Encodings Raw Encoding Byte-Dictionary Encoding Delta Encoding
data is stored in raw, uncompressed form.
Byte-Dictionary Encoding
a separate dictionary of unique values is created for each block of column values on disk
effective when a column contains a limited number (<256) of unique values
Delta Encoding
compresses data by recording the difference between values that follow each other in the column

26. LZO Encoding Mostly Encoding Runlength Encoding
provides a very high compression ratio with good performance
works especially well for CHAR and VARCHAR columns that store very long character strings
Mostly Encoding
useful when the data type for a column is larger than most of the stored values require
Runlength Encoding
replaces a value that is repeated consecutively with a token that consists of the value and a count of the number of consecutive occurrences
DONT apply on SORTKEY

27. Text255 and Text32k Encodings
useful for compressing VARCHAR columns in which the same words recur often
A separate dictionary of unique words is created for each block of column values on disk
Zstandard Encoding
provides a high compression ratio with very good performance across diverse data sets

29. Here comes the question
Columnar storage, MPP - WHAT is the way to influence the performance?

30. The answer is: Sortkey and Distkey

31. Sorkey and Distkey Applied during table design stage - initial DDL
Can be imagined as indices
Improve performance dramatically

33. Sorkey and Distkey
Both specified at the table design stage

34. Sortkey
Amazon Redshift stores your data on disk in sorted order according to the sort key.
The Amazon Redshift query optimizer uses sort order when it determines optimal query plans.

35. Best Sortkey
If recent data is queried most frequently, specify the timestamp column as the leading column for the sort key.
Queries will be more efficient because they can skip entire blocks that fall outside the time range.
If recent data is queried most frequently, specify the timestamp column as the leading column for the sort key.
Queries will be more efficient because they can skip entire blocks that fall outside the time range.
If you do frequent range filtering or equality filtering on one column, specify that column as the sort key.
Amazon Redshift can skip reading entire blocks of data for that column because it keeps track of the minimum and maximum column values stored on each block and can skip blocks that don't apply to the predicate range.
If you frequently join a table, specify the join column as both the sort key and the distribution key.
This enables the query optimizer to choose a sort merge join instead of a slower hash join. Because the data is already sorted on the join key, the query optimizer can bypass the sort phase of the sort merge join.

36. If you do frequent range filtering or equality filtering on one column, specify that column as the sort key.
Amazon Redshift can skip reading entire blocks of data for that column because it keeps track of the minimum and maximum column values stored on each block and can skip blocks that don't apply to the predicate range.

37. If you frequently join a table, specify the join column as both the sort key and the distribution key.
This enables the query optimizer to choose a sort merge join instead of a slower hash join. Because the data is already sorted on the join key, the query optimizer can bypass the sort phase of the sort merge join.

38. Main Rule for Sortkey For Developers:
Define the column which is/(will be) used to filter and make it a SORTKEY

39. Main Rule for Sortkey For Data Users:
Define which column is a SORTKEY and use it in your queries to filter the data

40. The MOST important Rule for Sortkey
For Developers:
Let your Data USERS know the SORTKEY for the tables

41. Sortkey benefits:
Queries will be more efficient because they can skip entire blocks that fall outside the time range.
Because the data is already sorted on the join key, the query optimizer can bypass the sort phase of the sort merge join.
Amazon Redshift can skip reading entire blocks of data for that column because it keeps track of the minimum and maximum column values stored on each block and can skip blocks that don't apply to the predicate range.

43. Demo - no vertical filter
XN Seq Scan on events (cost=
width=512)

44. Demo - with timestamp filter
cost=
width=512)

45. Demo - with sortkey filter
cost=
width=512

46. Demo Summary
No Filter:
XN Seq Scan on events (cost= rows= width=512)
Any Filter :
XN Seq Scan on events (cost= rows= width=512)
Sortkey Filter:
XN Seq Scan on events (cost= rows= width=512)

47. Killing the sortkey avoid using functions on sortkey
if you need to use a function, specify a wider range to help the optimizer

48. Demo - Killing the sortkey
XN Seq Scan on events (cost= rows= width=512)

49. Sortkey types
Compound
Interleaved

50. Compound Sortkey
A compound sort key is more efficient when query predicates use a prefix, which is a subset of the sort key columns in order
Is the default
Compound sort keys might speed up joins, GROUP BY and ORDER BY operations, and window functions that use PARTITION BY and ORDER BY. For example, a merge join, which is often faster than a hash join, is feasible when the data is distributed and presorted on the joining columns. Compound sort keys also help improve compression.

51. Example ... ,local_session_ts timestamp encode lzo
,vendor_id varchar(80) encode text255
,is_onsite boolean encode runlength )
SORTKEY (session_type, session_first_ts);
alter table dwh.fact_traffic_united owner to etl;

52. Interleaved sort key
gives equal weight to each column in the sort key, so query predicates can use any subset of the columns that make up the sort key, in any order
An INTERLEAVED sort key can use a maximum of eight columns Interleaved SORTKEY (session_type,session_first_ts);

53. Vacuum and analyze VACUUM ANALYZE
Reclaims space and resorts rows in either a specified table or all tables in the current database.
ANALYZE
Updates table statistics for use by the query planner.

54. Columnar and Sortkey
When columns are sorted appropriately, the query processor is able to rapidly filter out a large subset of data blocks.

55. MPP and DISTKEY
Amazon Redshift distributes the rows of a table to the compute nodes so that the data can be processed in parallel. By selecting an appropriate distribution key for each table, you can optimize the distribution of data to balance the workload and minimize movement of data from node to node.

56. MPP and DISTKEY Optimizer decides how the data needs to be located
Some rows or entire table is moved
Substantial data movements slow overall system performance
Using DISTKEY minimizes data redistribution

57. Data distribution goals
To distribute the workload uniformly among the nodes in the cluster.
To minimize data movement during query execution.

58. Distribution Styles
KEY
EVEN
ALL

59. Examples ... provider_api varchar(500) encode lzo,
table_loaded_at timestamp default getdate()
) DISTSTYLE EVEN;

60. Getting data into AWS Redshift
There is only one way - AWS S3
And COPY command
COPY table-name [ column-list ] FROM data_source authorization [ [ FORMAT ] [ AS ] data_format ] [ parameter [ argument ] [, ... ] ]

61. Bugs/features to keep in mind
ALTER TABLE statement
For SORTKEY/DISTKEY changes
⇒ recreating a table only
⇒ keep in mind for CI/CD

62. Bugs/features to keep in mind
Redshift types
Redshift scalability
Automatic backups
Quick table restoring

63. Questions?