1. CISER Setup Files Creator and Its DDI 2.5 Codebook
Florio Arguillas
CISER
Abstract
INSTALLATION INSTRUCTIONS
SETUP FILE CREATOR FOR FIXED WIDTH DATA
SETUP FILE CREATOR FOR COLUMN BINARY DATA
Most, if not all, Data Archives still possess in their collection ASCII and/or Column Binary data files that have codebooks, but no accompanying setup (or program) files to read them. This lack of setup files deters researchers from using these datasets and is costly cumulatively if multiple researchers independently write their own code to read these datasets. It also threatens the ability to use these datasets into the future, especially those that are still in column binary formats.
The Data Archives of the Cornell Institute for Social and Economic Research (CISER) and the Roper Center for Public Opinion Research have many of these ASCII and column binary datasets that have codebooks, but no setup files. Recognising the threat that this poses to the usability and preservation of these datasets, and given the volume, we developed a software written in Python that would automatically generate SAS, SPSS, STATA and R setup files and datasets, and DDI 2.5 Codebook. The only requirement is to enter variable metadata (such as variable name, record number, starting and ending column location, variable labels, and value labels) in a specific format in Excel, which will be used as input by the Setup File Creator.
This poster showcases the Setup File Creator for fixed-width and column binary data and how we made the software data archive and archivist friendly by simplifying the input and maximizing the volume and usefulness of its output.
Install software on desired folder location.
-Setup File Creator for Fixed Width Data (SFC_FW)
-Setup File Creator for Column Binary Data (SFC_CB)
Required software packages:
- Python 2.7
- Excel
- At least one of these: SAS, SPSS, STATA
Edit the SFC_FW.vbs file
-Change the path of the SFC_FW1_START.bat file.
Direct it to the folder where you installed the
SFC_FW or SFC_CB
Set WshShell = CreateObject("WScript.Shell")
WshShell.Run chr(34) & “<path>\SFC_FW1_START.BAT" & Chr(34), 0
Set WshShell = Nothing
Download and install latest version of Python 2.7
Required pre-loaded Python 2.7 packages:
sys, os.path, datetime, hashlib, tkinter
Required external Python package: xlrd
xlrd is a library for reading data and formatting
information from Excel files, whether they are .xls
or .xlsx files
LIMITATIONS
SETUP FILE CREATOR FOR FIXED WIDTH DATA
Currently, can process only datasets with one record per case
SETUP FILE CREATOR FOR COLUMN BINARY DATA
Currently, can process only datasets with one record per case and columns with only one punch/variable
Currently, can’t process datasets with multiple punches/variables in one column and those with variable values wrapping into other column(s), but can be manually programmed by the researcher in the setup file created by the SFC_CB.
Edit SFC_FW1.bat file and specify correct location of SPSS (stats.exe), SAS (sas.exe) , STATA (Stata-MP-64.exe) exe files, and SFC_FW folders.
set path=%path%;c:\python27
set path=%path%;C:\Program Files\IBM\SPSS\Statistics\24
set path=%path%;C:\Program Files\SASHome\SASFoundation\9.4
set path=%path%;C:\Program Files (x86)\Stata14
set path=%path%;U:\SFC_FW point to folder path
python SFC_FW_MAIN.py
SFC_FW1.bat
Or manually register the above paths in the system environment variables
Welcome Screen