1. complextractors! (advanced extractors)

2. types of extractors (pyclowder2)
trigger on a single file
can specify type
trigger on dataset change
add/remove file
cannot specify type (yet)
trigger on metadata change
add/remove metadata
trigger on custom combos
extractors-rulechecker
is really dataset > file added
have a PSQL db at your disposal

3. trigger on single file check_message() resource
process_message() resource
includes locally accessible path and pointer to dataset

4. trigger on dataset change
check_message() resource
process_message()

5. trigger on dataset change caveats
receives message every time a file is added to a dataset
if extractor requires input files A,B,C,D,E then you will see at least 5 messages as those are added
have to make sure you don’t truly process until all 5 are present!
the list of files is fetched after message, not as part of it - it will have files that were added later!
we can account for this...

6. trigger on dataset change check if this message is the latest one

7. trigger on metadata change
can trigger on both files and datasets
includes the metadata that was added or removed
useful for extractors that react to updated metadata parameters

8. trigger on custom combos extractors-rulechecker
rules.py - actual rule functions to execute, akin to check_message()
rule_extractor_map.json - mapping of rule functions to which extractors they trigger
accumulate information across datasets into a DB
when criteria are met, trigger a target extractor directly
synthesize two distinct sensor datasets by timestamp
extractors at the collection level - e.g. stitching a day of images
sufficient bounding box coverage of target area is achieved

9. trigger on custom combos rule_extractor_map.json

10. trigger on custom combos rule_utils.py
getFilesByType(resource, extension)
checkRequiredFiles(resource, rulemap, extractor)
submitProgressToDB(rule, extractor, target_output, id_list, jsonobj)
target_output is just a unique key name
id_list is a list of UUIDs you care about
jsonobj is whatever
retrieveProgressFromDB(target_output)

11. trigger on custom combos rules.py

12. trigger on custom combos rules.py

13. trigger on custom combos no rulechecker
RabbitMQ
image combiner queue
image combiner
90 messages
90 messages
90 files uploaded
15 image pairs (30)
15 music sample sets (30)
15 audio recordings (30)
music synthesizer queue
music synthesizer
90 messages
90 messages
audiobook generator queue
audiobook generator
90 messages
90 messages

14. trigger on custom combos YES rulechecker
RabbitMQ
image combiner
30 messages
90 files uploaded
15 image pairs (30)
15 music sample sets (30)
15 audio recordings (30)
rulechecker queue
music synthesizer
90 messages
30 messages
combiner rule
audiobook generator
30 messages
synthesizer rule
audiobook rule

15. future stuff unify file and dataset extractor concepts
return all necessary dataset info with file extraction
add UI element for datasets for extraction events
support collection-level extractors more explicitly
Clowder as workflow engine...? HOW FAR IS TOO FAR