1. Forecasting Wavelet Transformed Time Series with Attentive Neural Networks
Yi Zhao1, Yanyan Shen*1, Yanmin Zhu1, Junjie Yao2
1Shanghai Jiao Tong University
2East China Normal University
ICDM 2018

2. Outline
Motivation
Preliminaries
Model
Experiments
Conclusion

3. Motivation
Forecasting complex time series demands time-domain & frequency-domain information.
e.g., stock prices, web traffic, etc.
Various methods to extract local time-frequency features which are important to predict the future values.
Fourier Transform
Short-time Fourier Transform
Wavelet Transform
Use the varying global trend to identify the most salient parts of local time-frequency information to better predict the future values.

4. Preliminaries Problem Statement Wavelets
Given a time series 𝑿= 𝑥 𝑡 ∈𝑅 𝑡=1, 2, …, 𝑇}, predict 𝑥 𝑇+𝑛 𝑛∈ ℕ + , the future value in time 𝑇+𝑛 via a function 𝑓 :
𝑥 𝑇+𝑛 =𝑓(𝑿)
Wavelets
Given a basic wavelet function h(∙), we can get the wavelets：
ℎ 𝑎,𝜏 = 𝑎 ℎ 𝑡 −𝜏 𝑎
Continuous Wavelet Transform (CWT)
The continuous wavelet transform refers to the “similarity” between the signal x(𝑡) and the basis function ℎ 𝑎,𝜏 ∙ ：
𝐶𝑊𝑇 𝑥 (𝜏, 𝑎)= 𝑎 x(𝑡) ℎ ( 𝑡 −𝜏 𝑎 ) 𝑑𝑡

5. 3. CNN feature extraction
Model Overview
1. Input time series
LSTM
𝑥 𝑇+𝑛
f_att(, W)
2. Scalogram
3. CNN feature extraction
4. Attention Module
5. Fusion & Prediction
Preprocessing
Given input time series 𝑿, we denote by 𝐶𝑊𝑇 𝑥 (𝜏, 𝑎) the wavelet transform coefficients matrix. The scalogram 𝑿 s is defined as follows:
𝑿 s = || 𝐶𝑊𝑇 𝑥 (𝜏, 𝑎)|| 2
Source: Wavelet Tutorial by Robi Polikar,

6. Model D 𝑥 1 … … AttentionNet 𝑥 t … … … 𝑥 T … … …… C … … … D …
LSTM
𝑥 1
ℎ 1 … …
ℎ 𝑡−1
AttentionNet
LSTM
𝑥 t …
ℎ 𝑡 … …
ℎ 𝑇−1
𝜶 𝟏
ℎ 𝑇
LSTM
𝑥 T … …
𝜶 𝟐 …… C
𝑥 𝑇+𝑛 … …
𝜶 𝑪−𝟏 … D
𝜶 𝑪 …
VGG output features
Attention Module
Fusion & Prediction

7. Model CNN: extract local time-frequency features
Feed scalogram 𝑿 s to a stack of convolution layers: 𝑿 𝑠 (𝑙) = 𝜙( 𝑾 𝑠 𝑙 ∗ 𝑿 s + 𝑏 𝑠 (𝑙) )
LSTM: learn global long-term trend and get hidden state 𝒉 𝑇 in the last step
Attention module: discriminate the importance of local features dynamically
Given time-frequency features 𝑿 𝑠 𝐿 = 𝒙 𝑖 𝑖∈[1, 𝐶]} and 𝒉 𝑇
Attention score: 𝑒 𝑖 = 𝑓 𝑎𝑡𝑡 𝒙 𝑖 , 𝒉 𝑇 = 𝒘 𝑇 𝜙 𝑾 𝑎 𝒙 𝑖 ; 𝒉 𝑇 + 𝒃 𝑎 +𝒃; 𝛼 𝑖 = exp⁡( 𝑒 𝑖 ) 𝑘=1 𝐶 exp⁡( 𝑒 𝑘 )
Weighted sum of local time-frequency features: 𝒛= 𝑖=1 𝐶 𝛼 𝑖 𝒙 𝑖
Fusion & Prediction: combine local and global features for prediction
𝑥 𝑇+𝑛 = 𝒘 𝑝 𝑇 𝑓 𝑝 𝑧; 𝒉 𝑇 + 𝑏 𝑝
Objective Function
Squared Loss: 𝑳 𝑙𝑜𝑠𝑠 = 𝑖=1 𝑁 ( 𝑥 𝑡+𝑛 𝑖 − 𝑥 𝑡+𝑛 𝑖 ) 2 + 𝜆 ||𝑾|| 2

8. Datasets Stock opening prices Power consumption
Collected from Yahoo! Finance.
Daily opening prices of 50 stocks among 10 sectors from 2007 to
Each stock has 2518 daily opening prices. Daily opening prices from 2007 to 2014 are used as training data, and those in and 2016 are used for validation and testing, respectively.
Power consumption
Electric power consumption in one household over 4 years.
Sampled at one-minute rate.
475,023 data points in year 2010.

9. Main Results Metric: Baselines
Mean Squared Error: 𝑀𝑆𝐸= 1 𝑁 𝑖=1 𝑁 ( 𝑥 𝑡+𝑛 𝑖 − 𝑥 𝑡+𝑛 𝑖 ) 2
Baselines
Naïve: take the last value in the series as the prediction value
Ensemble of LSTM & CNN: feed the concatenation of features from VGGnet and the last hidden state from LSTM into the fusion and prediction directly.

10. Case Study Illustration of attention mechanism
Given an input of 20 stock prices, we show the scalogram, and the attention weights.
The model attends to the local features that are similar to the global trend and helps in predicting the future value.

11. Conclusion
Wavelet transform is able to explicitly disclose the latent components at different frequencies from a complex time series.
We develop a novel attention-based neural network that leverages CNN to extract local time-frequency features and applies LSTM to capture the long-term global trend simultaneously.
The experimental results on two real life datasets verify the usefulness of time-frequency information from wavelet transformed time series and the our method in terms of prediction accuracy.

12. THANK you! Q&A