Mock-ups for Discussing the CMS Administrator Interface

User Name: Full Name: Email: Password: Confirm Password: PIN:

Admin Home Page Import Book Configure Extraction Tools (not clickable) FROntIER (as is, all setup work done outside CMS, but testing done inside CMS) GreenDDA GreenFIE ListReader OntoES (OntoES + snippets with regex tool for creating rules for PCF ontologies) OntoSoar Initiate Pipeline Run Manage Pipeline Run (check progress, kill & restart tasks, manage jobs and batches, submit completed work) Edit User Account (only for super-admins)

Database Schema user (id, username, email, password, name, type) job (id, name, book, max_page) batch (id, pages, status, job_id, user_id) form (id, form_name, version) eventlog (id, user_id, batch_id, session_id, log-info, time_written) status_code (status, next_status, description) user_type (type, description) P: patron A: adjudicator-patron M: administrator S: super-administrator N: not assigned or not submitted S: submitted (initial patron submission) D: done (sent downstream to Gedcomx) Q: quality recheck by patron R: resubmitted (patron resubmission) A: adjudicator re-check C: checked by adjudicator T: testing (under control of CMS admin)

Workflow Pipeline PCF = Person, Couple, Family directories PRF = Precision, Recall, F-score 1.pages 2.tools <toolName> PCF: extracted 2.1.tool-ontology-extracted 2.2.ontology-extracted 2.3.extracted-text-cleaned 3.json-from-osmx 3.1.ontology-merged 3.2.value-cleaned 3.3.date-value-parsed 3.4.constraint-checked 3.5.violation-corrected PCF: osmx & json 4.json-working PCF: json 5.json-final CMS: setup & test 6.osmx-checked 6.1.ontology-merged 6.2.value-cleaned 6.3.date-value-parsed 6.4.constraint-checked 6.5.authority-checked PCF: osmx & json ground-truth PRF-report 7.osmx-enhanced 7.1.target-ontology-generated 7.2.value-standardized 7.3.information-inferred 8.gedcomx 8.1.gedcomx-generated 8.2.reports-generated Q, A, T D, C Q, A, T N, T COMET bypass S, R, C, T

Import Book Browse file of available books Includes browse pages Future organization: books may be categorized, ordered, … Select book from file for import Add meta.xml text shortTitle biblio url (on dithers) auto-assigned Filename (in file of available books) auto-assigned Auto-assign id (six digits + shortTitle) Initialize book authority files (NameAuthority.txt, PlaceAuthority.txt) Initiate PDF-indexer Split pages Generate 1.pages files for each page .html .pdf .png .txt .xml Import Book

Test tool on selected page(s) User-selected list or range of test pages Generate filled in forms (in quality-check mode) Allow filled-in form(s) to be ground-truthed Produce PRF reports for ground-truthed form(s) Test tool in ensemble with other tools Results of a test run For each page/form, the data and form json Regular COMET display of first page and filled-in form Ground-truth reports Differences in COMET display of auxiliary information No “Annotation Actions” or click for instructions link Batch sequence according to list or range of test pages Instead of “Submit Batch”, “Generate Ground Truth” A list of links to any and all ground-truth reports For all tools Test <list, e.g. 031,032,099> or <range, e.g. 030-031> Include Tools: [ ] FROntIER, [ ] GreenDDA, [ ] GreenFIE, [ ] ListReader, [ ] OntoES++, [ ] OntoSoar The tool being worked on is included by default.

For all tools Generate Ground Truth range.030-031.PRFreport-soft.txt Person/range.030-031.PRFreport-soft.txt Person/030.PRFreport-soft.txt Person/031.PRFreport-soft.txt Couple/range.030-031.PRFreport-soft.txt Couple/030.PRFreport-soft.txt Couple/031.PRFreport-soft.txt Family/range.030-031.PRFreport-soft.txt Family/030.PRFreport-soft.txt Family/031.PRFreport-soft.txt

Use FROntIER interface, as is, to create an extraction ontology Use CMS testing interface to test the created extraction ontology Run test (as can be done for all tools) Set ground truth (as possible for all tools) Future: add the CMS regex-checker tool FROntIER

GreenDDA Select training set Train ML tool to training set Select book Apply trained ML tool to book GreenDDA

GreenFIE GreenFIE Regular GreenFIE interface in experimental mode ? Ability to run other tools to initialize the GreenFIE interface ? Ability to selectively save/retract generated rules Changes to COMET interface Add GreenFIE “Generate Rule” button Replace all below the form by “Save” button to save all rules and: ? Initialize from tools request ? List of generated rules with check boxes Make page number a type in box GreenFIE GreenFIE Regex Regex Regex Regex

ListReader ListReader Next Stop Select book and run ListReader (future: also set text abstraction parameters) One record from all three forms on the lhs Highlights on the rhs as generated by ListReader Control buttons Next (to tell ListReader to save the current labeling and ask for the next) Stop (to stop the labeling cycle) ListReader ListReader Next Stop

OntoES Generate extraction rules FROntIER-like rule-creation facilities Re-organized for better UX Built-in regex checker OntoES

OntoSoar

Initiate Pipeline Run Book selection box Page Range – Tools FROntIER GreenDDA GreenFIE ListReader OntoES OntoSoar Run Type with COMET without COMET default Run

Manage Pipeline Run Display progress for book Show progress for each page Report failures Allow admin to kill and restart tasks When running through COMET: Display progress of each job Percent complete Status of each batch: Unassigned/InProgress/Done Status of InProgress batches User assigned Page/form status Eventlog for batch (upon request) Retract batch from user Assign batch to user Reassign batch to user User status Batches completed Batches assigned Eventlog report for user (upon request) Submit completed work to FS Manage Pipeline Run

Edit User Account Select user: id username email name search Find user Set any/all: Password Username Email Full name Change privileges Disable account Edit User Account Select user: id username email name search

Edit User Account User: id username email name Change password Find user Set any/all: Password Username Email Full name Change privileges Disable account Edit User Account User: id username email name Change password Change username Change email Change full name Change privileges o patron o adjudicator-patron o administrator o super-administrator o Disable account done