1. 1 Automated Email Editing Electronic Reporting (ER) Steve Kocsis, NOAA Fisheries Juneau, Alaska

2. 2 Electronic Reporting in Alaska Automate the reporting of the North Pacific groundfish and prohibited species catch Receiving and sending daily and weekly reports from/to shoreside plants and vessels Decision-making for closing fisheries Computing discard and by-catch rates

3. 3 Perl as an E.R. Tool Practical Extraction and Report Language, ‘88, Wall Free, GNU GPL, FSF, CPAN DBI and Oracle DBD modules for dynamic SQL MIME module for Email parsing and sending Regular expression string manipulation

4. 4 E.R. Sequence Client Form Data Email NMFS Server 1) Get files 2) Unique ID Validate Data Bad files Validation errors Load history Load Host Generate RRT NMFS SERVER Email Archive

5. 5 E.R. in AK Region 1998-Now * * 2005 extrapolated

6. 6 E.R. In AK Region 1998 - Now 98% coverage of groundfish shoreside processors Increasing coverage of processing vessels

7. 7 Client data form

8. 8 Client raw data 00,1234,411 03,1234,08/31/2001,N,N,2,N 04,1234,08/31/2001,BSAI,270,P,30,350752.,N 05,1234,08/31/2001,67890,08/29/2001,F1234,JNU,N,,,Y,,N 06,1234,08/31/2001,67890,08/29/2001,G00029549,,PTR,47,189,N 07,1234,08/31/2001,67890,08/29/2001,G00029549,5,655430,110,41,,517,226.,,,N 07,1234,08/31/2001,67890,08/29/2001,G00029549,4,655430,121,41,,517,171.,,,N 07,1234,08/31/2001,67890,08/29/2001,G00029549,3,655430,122,41,,517,5.,,,N 07,1234,08/31/2001,67890,08/29/2001,G00029549,6,655430,141,41,,517,9.,,,N 07,1234,08/31/2001,67890,08/29/2001,G00029549,2,655430,270,41,,517,3994.,,,N 99 Prefix identifies table: 00- sl_rpt 03 - sl_daily_activity 06- sl_delivery_fish_ticket 07 - sl_delivery_product

9. 9 Table fields...

10. 10 Production tables - shoreside transmitted_data_file sl_rpt sl_daily_activity sl_daily_product sl_buying_station sl_delivery sl_delivery_fish_ticket sl_delivery_product 00 03 04 01 05 06 07

11. 11 Report types Six types currently supported Identified by file extension (*.era, *.vwp, *.slg, *.cat, *.dev, *.vlr) Modular software gives flexibility for expansion to new types (design feature) Extension couples types to specific functions

12. 12 Client Server Application Timing Sequence 1 User sends data by Email Read Email Parse ER files Query Database Use query to create Sql*ldr script for loading ER Data into working tables InternetLAN Perl DBI

13. 13 Timing Sequence 2 Oracle Application LANServer Perl DBI Run sql*ldr script to populate working tables Send sql*ldr command Read and parse error messages Return Error messages Load error tables Archive log and ER files Run sql script and Unix archive commands

14. 14 Timing Sequence 3 Oracle Application LAN Server Run ER validation Package Send Sql Move data in working tables to production tables Client Return Error Messages Archive with FTP Internet Send Sql Generate error report Save report Send email with error report to client

15. 15 Perl DBI module ERLSCRIPTERLSCRIPT DBISWITCHDBISWITCH ORACLE DBD PERLAPIPERLAPI PERLSCRIPTPERLSCRIPT Oracle Database *Dynamic SQL *.. File handles, connect/disconnect

16. 16 Perl MIME (Multipurpose Internet Mail Extensions) UNIX mail Perl MIME module Attachments MIME Decoding 1 1 Perl DBI Unique filename by ER type

17. 17 NMFS mailbox Assign unique ID Time scheduled Time stamp Identify ER files Parse & decode attachments Working directory Client Email Queue ER files for processing 00019546.slg 00019547.slg 00019548.slg

18. 18 Perl regular expressions (regex) Pattern matched to a string – Textual analysis and manipulation /^(.+)\.(.+)$/ $file = $1 $ext = $2 $subject =~ /^Subject: (.+)$/ 8128.slg $file = ‘8128’ $ext = slg Subject: NOAATECH2004 $subject = NOAATECH2004

19. 19 Perl regex /^From\s.+\s\w{3}\s\w{3}\s{1,2}\d{1,2}\s\d{2}:\d{2}:\d{2}\sAK[SD]\s\d{4}$/ From Steve.Kocsis@noaa.gov Fri Oct 21 15:15:38 AKD 2003 s/$extpattern$/\.bad/ s/load/mod/;s/ /$fn/; s/$extpattern$// slgload.ctl -> slgmod.ctl -> 0008128.slg 111.era->111.bad 222.slg->222.bad 111.era->111 222.vlr->222

20. 20 Time scheduled Hfdsihfjkhjkhsfh Fdsjhlkjlfkjsfkjf Dsfhkjfhhsfd Fsdhjsjfdfjsfd Fdjkjgkjgjfgkd Dhfdhffjfhjksf … ER Data files run sql*ldr script Load working tables archive Load errors Load history Type?? Working directory 00021348.slg *.slg

21. 21 Validation Working tables Validation package Production tables Load history Generate RRT Email client errors

22. 22 Program Flow 1 NMFS mailbox Get ER attachments Assign unique file name Fetch ER profile sql*ldr ctl file by ER type Time scheduled Import into working tables Data file Ifjkjjjfdsklf Jffkjfkjfksl Fdhfhfh Fshfh fdsjfhfs Client form email

23. 23 Program flow 2 sql*ldr bad recordslog records archive load history validation errors Production tables Working tables

24. 24 Program flow 3 Production tables Load history Generate return receipt Find client email archive Process with client s/w Email

25. 25 Summary – Electronic Reporting Improved In-season fisheries management - goals of increased timeliness and reliability have been met Decreased time and cost of reporting for NMFS and industry - widely accepted by industry - NMFS AK region success - Mandatory for many shoreside processors Cooperation with other agencies In operation since 1998 using Perl/UNIX