Frank Milthorpe 27 May 2011 Suite of Emme Macros to Enable Flexible Specification of Multiple Data Sources

Outline 1.Goals 2.Text registers within Emme 3.Implementation Approach 4.Conclusions 5.Listing of macros

Goals Flexible definition of input data Easy to distinguish differences between model runs Implemented within Emme macros

Overview of Desired Specification Run 1 Standard Inputs except Employment_A Run 2 Standard Inputs except Population_B Employment_C

Text Registers t1.. t9 are text registers available t9 is global t1.. t8 local registers –can ’ t return arguments Can ’ t “ load-up ” registers with filenames from a lower macro

Implementation Approach Create a LIFO (last in first out) text file Write to text file with label and filename Read the text file in appropriate macro to retrieve and return filename via t9 Also allows an audit trail

Example Data File Created start_default === 27-Apr :23 zonal_data M:\STM_Core\Inputs\ZonalMtx\ emp_data M:\STM_Core\Inputs\Employment\2009_Oct\ pop_adjust_file m:\stm_gma\lu_adjust\pop_adjust_default.mtx emp_adjust_file m:\stm_gma\lu_adjust\emp_adjust_default.mtx timeskim_busonly_macro tskimfac carskim_data M:\STM_Core\Inputs\Calibration_data\Dec_09\ end_default === pop_adjust_file M:\Myproject\PopFac_2011.mtx carskim_data M:\Myproject\carskim_matrices\

Reading Data ~/... Start of macro read_emp.mac %ms02%... ~# ~t2=%t1%read_emp.rep ~<ifexist_delete %t2% reports=%t2% ~# batchin=? ~# ~# --- Find directory for the data ~<get_data_source_item emp_data ~<stm_log emp_data %t9% ~# ~t3=%t9% ~# ~# --- Non-Manufacturing employment: md3 --- ~+;3.11;%t3%nonmanemp%ms02%.mtx;2; ~#

Conclusions Macro suite designed and implemented to meet requirements Can be a filename or a directory Can be adapted for other purposes (eg specify assignment parameters) Text file provides an audit log of specifications

Core Macros delete_data_source_stackDelete the data_source_stack file create_data_source_stackCreate data_source_stack file write_data_sourceWrite an item to a data_source_stack file get_data_source_itemFind a return value for data item strip_leading_blanks Trim the leading spaces from a text string

delete_data_source_stack.mac ~:start ~/... Start of macro delete_data_source_stack.mac... ~# ~<ifexist_delete stmaudit\data_source_stack.dat ~# ~/... End of macro delete_data_source_stack.mac...

create_data_source_stack.mac ~:start ~/... Start of macro create_data_source_stack.mac... ~# ~<ifexist_delete stmaudit\data_source_stack.dat ~# ~# Write something to the data_stack to ensure other macros ~# have a file to open. ~# ~<make_directory stmaudit ~# ~<get_date_time ~<write_data_source data_stack_created %t9% ~# ~/... End of macro create_data_source_stack.mac...

write_data_source.mac Example macro call write_data_source %1% %2% %1% - source_name %2% - directory (or other item) eg ~<write_data_source pop_data m:\popdata\2008\ ~:start ~/... Start of macro write_data_source.mac %t0%... ~# ~t1=stmaudit\data_source_stack.dat ~t2=%t0% ~# ~# Strip leading blanks ~:continue_stripping ~t3=%t2.1% ~+;~?!t3= ;~$>stripped ~+;~t2=%t2.-1%;~$continue_stripping ~# ~:stripped ~# Write to output file ~+;~>>%t1%;~"%t2%;~>

get_data_source_item.mac (1) ~$>start Emme/2 macro "get_data_source_item.mac" Macro to obtain answer (normally directory) for a data item written to the data_source file. This is the last entry written. N.B. If macro used the first entry this may not be the entry that is needed. For example if the model was rerun with different defaults and the stack was not deleted, it would return the first (and wrong original entry). Example macro call ~<get_data_source %1% %1% - source_name ~<get_data_source pop_data Returns %t9% Item (directory or other data label)

get_data_source_item.mac (2) ~:start ~/... Start of macro get_data_source_item.mac %t0%... ~# ~t1=stmaudit\data_source_stack.dat ~# ~# --- Find length of macro argument %1% store in %x% ~t2=%1% ~# ~x=-1 ~:continue_counting ~x+1 ~t3=%t2.1% ~+;~?t3= ;~$>counted ~+;~t2=%t2.-1%;~$continue_counting ~# ~:counted ~# ~# length of argument %1% is %x% ~#

get_data_source_item.mac (3) ~# --- Find line in file with source_name ~# ~# t8 - line of data read in ~# t7 - line of data with a match ~# t3 - substring of t8 for trial match to source_name %1% ~# r1 - count of entries found (set to zero initially) ~# ~r1=0 ~# ~:process_line ~+;~?e;~$>end_of_file ~+;~t3=%%t8.%x%% ~+;~?t3=%1%;~t7=%t8% ~+;~?t3=%1%;~r1+1 ~$process_line ~# ~:end_of_file ~#

get_data_source_item.mac (4) ~# Was the source_name found? ~+;~?r1>0;~$>found ~# ~:not_found ~/ did not find %1% in get_data_source_item ~t9=Error %1% Not found in get_data_source_item ~$>end ~# ~:found ~# --- Extract the item label (probably directory) ~# remove first %x% chars ~+;~t2=%%t7.-%x%% ~# ~# remove leading blanks ~# result is returned in %t9% ~<strip_leading_blanks %t2% ~# returned value is %t9% ~:end ~# Close file ~> ~/... End of macro get_data_source_item.mac %1% %t9%...

Strip_leading_blanks.mac ~:start ~# Does this macro have an argument passed to it? ~x=%0% ~+;~?x>0;~$>valid_argument ~# No argument - nothing to strip ~t9=%t0% ~# ~$>end ~:valid_argument ~# remove leading blanks if they exist ~t2=%t0% ~:continue_stripping ~t3=%t2.1% ~+;~?!t3= ;~$>stripped ~+;~t2=%t2.-1%;~$continue_stripping ~:stripped ~t9=%t2% ~:end ~# returned value t9 is:%t9% ~#... End of macro stip_leading_blank.mac %t9%...

Implementation Macros set_default_data_sourcesWrite the default data sources to data_source_stack file read_empMacro to read employment matrices (similar macros exist to read other data files)

set_default_data_sources.mac ~/... Start of macro set_default_data_sources.mac... ~# ~# Make sure directory where data_source_stack.dat resides exists ~<make_directory stmaudit ~# ~# Delete data source stack if it exists (NOT IMPLEMENTED) ~# Stack operates as a permanent record, use last occurance of keyword ~# ~<get_date_time ~<write_data_source start_default === %t9% ~# ~# --- Set the default data sources ~<write_data_source zonal_data M:\STM_Core\Inputs\ZonalMtx\ ~<write_data_source emp_data M:\Inputs\Employment\2009_Oct\ ~<write_data_source lu_internal_data m:\stm\march_2005_gma\ and etc as per user needs

read_emp.mac ~:start ~/... Start of macro read_emp.mac %ms02%... ~# ~t2=%t1%read_emp.rep ~<ifexist_delete %t2% reports=%t2% ~# batchin=? ~# ~# --- Find directory for the data ~<get_data_source_item emp_data ~<stm_log emp_data %t9% ~# ~t3=%t9% ~# ~# --- Non-Manufacturing employment: md3 --- ~+;3.11;%t3%nonmanemp%ms02%.mtx;2; ~#

General Utility Macros ifexist_deleteDelete a file if it exists get_date_timeObtain and return the date and time make_directoryCreate a DOS directory stm_logWrite to logbook and a DOS file waitWait a specified number of seconds

ifexist_delete.mac ~:Start ~# ~!if exist %1% del %1% ~<wait 2 ~# ~:End ~#/... End of macro IfExist_Delete.mac... ~#

get_date_time.mac (1) ~:start ~# --- Get date string ~# Obtain current day - Global parameter p=2008 ~p=2008 ~x=%p% ~# Obtain current month - Global parameter p=2007 ~p=2007 ~y=%p% ~# Obtain current year - Global parameter p=2006 ~# Depending of status of switch 25 ~# This value may only be a 2-digit year. ~p=2006 ~z=%p% ~+;~?z<2000;~z+2000 ~# --- Obtain month character label ~+;~?y=1;~t2=Jan ~+;~?y=2;~t2=Feb..... ~# Consolidated Date String dd-mmm-yyyy ~# If d is less than 10 need a 0 to pad the field ~t1=%x% ~+;~?x<10;~t1=0%x% ~t4=%t1%-%t2%-%z%

get_date_time.mac (2) ~# --- Get date string ~# Obtain current hour - Global parameter p=2009 ~p=2009 ~x=%p% ~# Obtain current minute - Global parameter p=2010 ~p=2010 ~y=%p% ~# Obtain current second - Global parameter p=2011 ~p=2011 ~z=%p% ~# Consolidated Time String hh:mm ~# If hour or min is less than 10 need a 0 to pad the field ~t1=%x% ~+;~?x<10;~t1=0%x% ~t2=%y% ~+;~?y<10;~t2=0%y% ~t5=%t1%:%t2% ~t9=%t4% %t5% ~# ~# t9 (Date Time) is %t9%~#

make_directory.mac ~:start ~/... Start of macro Make_directory.mac %1%... ~# ~!IF not exist %1% MKDIR %1% ~# ~/... End of macro Make_directory.mac %1%...

stm_log.mac Example macro call stm_log %t0% %t0% - text to write to log file / logbook ~:start ~/... Start of macro stm_log.mac %t0%... ~# ~t1=stm_log.rep ~# ~# Write to output file ~+;~>>%t1%;~"%t0%;~> ~# ~# Write to logbook (with a max of 60 characters) c=%t0.60% ~# ~/... End of macro stm_log.mac %t0%...

wait.mac ~:Start ~# current cpu time since creation of database in 1/10 secs ~p=2005 ~x=%p% ~# current sec (1/10) since database creation %x% ~# ~# Add %1% * 10 to current 1/10 second to get new target ~r1=%1% ~r1*10 ~r1+%x% ~z=%r1% ~# ~# New target is %z% ~:Repeat ~# current cpu time since creation of database in 1/10 secs ~p=2005 ~x=%p% ~?x<%z% ~$Repeat ~#