1 Automated Editing Electronic Reporting (ER) Steve Kocsis, NOAA Fisheries Juneau, Alaska

2 Electronic Reporting in Alaska Automate the reporting of the North Pacific groundfish and prohibited species catch Receiving and sending daily and weekly reports from/to shoreside plants and vessels Decision-making for closing fisheries Computing discard and by-catch rates

3 Perl as an E.R. Tool Practical Extraction and Report Language, ‘88, Wall Free, GNU GPL, FSF, CPAN DBI and Oracle DBD modules for dynamic SQL MIME module for parsing and sending Regular expression string manipulation

4 E.R. Sequence Client Form Data NMFS Server 1) Get files 2) Unique ID Validate Data Bad files Validation errors Load history Load Host Generate RRT NMFS SERVER Archive

5 E.R. in AK Region 1998-Now * * 2005 extrapolated

6 E.R. In AK Region Now 98% coverage of groundfish shoreside processors Increasing coverage of processing vessels

7 Client data form

8 Client raw data 00,1234,411 03,1234,08/31/2001,N,N,2,N 04,1234,08/31/2001,BSAI,270,P,30, ,N 05,1234,08/31/2001,67890,08/29/2001,F1234,JNU,N,,,Y,,N 06,1234,08/31/2001,67890,08/29/2001,G ,,PTR,47,189,N 07,1234,08/31/2001,67890,08/29/2001,G ,5,655430,110,41,,517,226.,,,N 07,1234,08/31/2001,67890,08/29/2001,G ,4,655430,121,41,,517,171.,,,N 07,1234,08/31/2001,67890,08/29/2001,G ,3,655430,122,41,,517,5.,,,N 07,1234,08/31/2001,67890,08/29/2001,G ,6,655430,141,41,,517,9.,,,N 07,1234,08/31/2001,67890,08/29/2001,G ,2,655430,270,41,,517,3994.,,,N 99 Prefix identifies table: 00- sl_rpt 03 - sl_daily_activity 06- sl_delivery_fish_ticket 07 - sl_delivery_product

9 Table fields...

10 Production tables - shoreside transmitted_data_file sl_rpt sl_daily_activity sl_daily_product sl_buying_station sl_delivery sl_delivery_fish_ticket sl_delivery_product

11 Report types Six types currently supported Identified by file extension (*.era, *.vwp, *.slg, *.cat, *.dev, *.vlr) Modular software gives flexibility for expansion to new types (design feature) Extension couples types to specific functions

12 Client Server Application Timing Sequence 1 User sends data by Read Parse ER files Query Database Use query to create Sql*ldr script for loading ER Data into working tables InternetLAN Perl DBI

13 Timing Sequence 2 Oracle Application LANServer Perl DBI Run sql*ldr script to populate working tables Send sql*ldr command Read and parse error messages Return Error messages Load error tables Archive log and ER files Run sql script and Unix archive commands

14 Timing Sequence 3 Oracle Application LAN Server Run ER validation Package Send Sql Move data in working tables to production tables Client Return Error Messages Archive with FTP Internet Send Sql Generate error report Save report Send with error report to client

15 Perl DBI module ERLSCRIPTERLSCRIPT DBISWITCHDBISWITCH ORACLE DBD PERLAPIPERLAPI PERLSCRIPTPERLSCRIPT Oracle Database *Dynamic SQL *.. File handles, connect/disconnect

16 Perl MIME (Multipurpose Internet Mail Extensions) UNIX mail Perl MIME module Attachments MIME Decoding 1 1 Perl DBI Unique filename by ER type

17 NMFS mailbox Assign unique ID Time scheduled Time stamp Identify ER files Parse & decode attachments Working directory Client Queue ER files for processing slg slg slg

18 Perl regular expressions (regex) Pattern matched to a string – Textual analysis and manipulation /^(.+)\.(.+)$/ $file = $1 $ext = $2 $subject =~ /^Subject: (.+)$/ 8128.slg $file = ‘8128’ $ext = slg Subject: NOAATECH2004 $subject = NOAATECH2004

19 Perl regex /^From\s.+\s\w{3}\s\w{3}\s{1,2}\d{1,2}\s\d{2}:\d{2}:\d{2}\sAK[SD]\s\d{4}$/ From Fri Oct 21 15:15:38 AKD 2003 s/$extpattern$/\.bad/ s/load/mod/;s/ /$fn/; s/$extpattern$// slgload.ctl -> slgmod.ctl -> slg 111.era->111.bad 222.slg->222.bad 111.era-> vlr->222

20 Time scheduled Hfdsihfjkhjkhsfh Fdsjhlkjlfkjsfkjf Dsfhkjfhhsfd Fsdhjsjfdfjsfd Fdjkjgkjgjfgkd Dhfdhffjfhjksf … ER Data files run sql*ldr script Load working tables archive Load errors Load history Type?? Working directory slg *.slg

21 Validation Working tables Validation package Production tables Load history Generate RRT client errors

22 Program Flow 1 NMFS mailbox Get ER attachments Assign unique file name Fetch ER profile sql*ldr ctl file by ER type Time scheduled Import into working tables Data file Ifjkjjjfdsklf Jffkjfkjfksl Fdhfhfh Fshfh fdsjfhfs Client form

23 Program flow 2 sql*ldr bad recordslog records archive load history validation errors Production tables Working tables

24 Program flow 3 Production tables Load history Generate return receipt Find client archive Process with client s/w

25 Summary – Electronic Reporting Improved In-season fisheries management - goals of increased timeliness and reliability have been met Decreased time and cost of reporting for NMFS and industry - widely accepted by industry - NMFS AK region success - Mandatory for many shoreside processors Cooperation with other agencies In operation since 1998 using Perl/UNIX