1. Lecture 4: Econometric Foundations
Stephen P. Ryan
Olin Business School
Washington University in St. Louis

2. Big Question: How to Perform Inference?
Naturally write many problems as solving: 𝑀 𝛼,𝜂 =0
In many econometrics + ML settings, we have injected a predictor with unknown properties
We typically are predicting a nuisance parameter (𝜂) using a ML method
Moment forests: using classification trees to predict the assignment of parameters to observations
IV-Lasso: using LASSO to predict which instruments should be included in IV regression
Natural language processing: using deep learning to parse text into data
How do we perform inference on the 𝜶, parameters of interest?

3. Chernozhukov, Hansen, Spindler (2015)

4. Basic ideas Low-dimensional parameter of interest
High-dimensional (infinite?) nuisance parameter estimated using selection or regularization methods
Provide set of high-level conditions for regular inference
Key condition: immunized or orthogonal estimating equations
Intuition: set up moments so that estimating equations are locally insensitive to small mistakes in nuisance parameters
Application: affine-quadratic models, IV with many regressors and instruments

5. Setting: High-dimensional models
High-dimensional models, where number of parameters is large relative to sample size, increasingly common/used
Big data -> many covariates
Basis functions -> dictionary of terms for even low-dimensional X
Regularization -> reduction in dimension to focus on “key” components required
Need to account for this regularization when performing inference
General approach -> account for model search (we will talk about this more in moment forests later)

6. Orthogonality / immunization condition
The key theoretical contribution is to show the properties of models with orthogonality / immunization allow for regular inference on 𝛼 in:
𝑀 𝛼,𝜂 =0
The key condition is:
𝜕𝜂𝑀 𝛼, 𝜂 =0
Basically, derivative of system of equations is zero in local area with respect to nuisance parameter
This condition can be generally established in many settings
Neyman classic orthogonalized score in likelihood settings (Neyman 1959, 1979)
Will show extension to GMM settings

7. High quality estimators of 𝜂
Process will require high quality estimator of nuisance parameters
Approximate sparsity -> 𝜂 can be approximated by sparse vector, such as Lasso
Reminder, Lasso solves:
𝜂 = arg min 𝑙(data, 𝜂) +𝜆 𝑗=1 𝑝 𝜓 𝑗 𝜂 𝑗
Where 𝑙 is some loss function, 𝜆 is a penalty parameter, and 𝜓 are penalty loadings
Leading example is linear model:
𝜂 = arg min 𝑖=1 𝑛 𝑦 𝑖 − 𝑥 𝑖 ′ 𝛽 𝜆 𝑗=1 𝑝 𝜂 𝑗
However: Conditions here do not require approximate sparsity -> require rate of convergence, usually 𝑛 1/4 .

8. Setup We want to solve system of equations: 𝑀 𝛼, 𝜂 0 =0
𝑀 𝛼, 𝜂 0 =0
Where 𝑀= 𝑀 𝑙 𝑙=1 𝑘 is a measureable map from 𝐴×𝐻 to ℛ 𝑘 and 𝐴 ×𝐻 are convex subsets of ℛ 𝑑 × ℛ 𝑝
Note that 𝑑 assumed fixed, while 𝑝 may grow with 𝑛
Given appropriate estimator 𝜂 , estimator is:
Often 𝑀 is a moment:

9. Adaptivity Condition
We would like to test some hypothesis about the true parameter vector
Inverting test gives us confidence region
Adaptivity is key condition for validity of this inversion:
Key requirement for this to be true is orthogonality / immunization:

10. Conditions for valid inference
Suppose we have: for some positive-definite
Note that these are at the true values, and are basically telling us the underlying DGP and estimating equations are well-behaved
Suppose there exists a high-quality estimator for the variance:

11. Then we get something cool
The following score statistic is asymptotically normal: and the quadratic form:
The first equation is what we would like to use in practice!

12. Proposition 1

13. Valid Inference via Adaptive Estimation
Assumptions:
Derivatives of moment functions exist:
Parameters are on interior of space
Problem is locally identified
Central limit theorem holds
Variance is not too nuts
Stochastic equicontinuity and continuity requirements (bounding variation in underlying functions)
Uniform convergence requirements on estimators, underlying smoothness (usually rate of convergence on 𝜂)
Orthogonality condition (new requirement here)

15. Achieving Orthogonality
Idea: project score that identifies parameter of interest onto orthocomplement of the tangent space for the nuisance parameter (obviously)
Actually, kind of intuitive partialing out of the nuisance parameter
Suppose we have a likelihood function and:
Consider the following moment function: with:

16. Orthogonality with Maximum Likelihood
We have both: and

17. Under true specification
If we assume model is correct, get a nice result: Leading to:

18. Lemma 1 [Neyman’s Orthogonalization]

19. Details

20. Orthogonal GMM Version:

21. Estimator and Variance

22. IV with Many Controls and Instruments
Consider a typical IV model, but with many controls (X) and instruments (Z): Where:

23. IV with Many Controls and Instruments
Consider a typical IV model, but with many controls (X) and instruments (Z): Where: Question: how to estimate with many Z and X, given we care about 𝛼

24. Note that we have a bunch of nuisance parameters here
Letting X and Z be correlated (so that 𝑧 𝑖 =Π x i + 𝜉 𝑖 ), we can rewrite equations as: With:

25. Sparsity
Since dim 𝜂 0 >𝑛, have to do something to reduce dimensionality
Assume approximate sparsity (i.e. low dimension approximation = close enough):
Decomposing:

26. Condition A2 + Estimator
Assume that non-sparse component grows at sufficiently slow rate
(by the way, what does that mean in practice?)
Then, orthogonalized equations are:
Can verify that orthogonality condition holds.

27. Lasso estimator for 𝜂 0

28. Simulation Results

31. Berry Logit Example ala BLP
Simple aggregate logit model of demand for cars:
Concern: price correlated with unobserved quality
BLP instruments: functions of characteristics of other products
BLP reduced dimensionality to:

32. This method applied here
In principle, all functions of other product characteristics will be valid instruments
Which ones to use?
CHS propose all first-order interactions of baseline, quadratics, cubics and a time trend
Apply the IV estimator described previously

34. Conclusion
This paper provides general conditions under which one can perform regular inference after regularizing a high-dimensional set of nuisance parameters
This applies to a very wide set of models
Increasing number of X due to big data
Increasing number of X due to dictionary of basis functions
Two worked examples
Likelihood
GMM
Application to IV
This should be very useful to you all in your own applied work, fairly easy to modify the standard approaches to use