Handling inputs and outputs from recipes

WIP

RecipeOut

A recipe can outputs a lot of things. There are three dimensions to it:

• Is it for out or for log?
• How do we process the object?
• Once it was processed, how do we write it?

Five formats:

1. utf8
2. json
3. bytes
4. table
5. file

You have check which can be called at the beginning of the function (like Airbyte, to check the connection to the IO is alive, and potentially reduce latency)

First, process (basically convert the object to the five formats)

Second, write.

• Can write to Runfile's output_info

Make sure that there is a safe space for runs metadata...

A custom RecipeOut class can overwrite any of these classes.

RecipeOut needs to take into account a certain config...

Ideas:

• configs file (see Recipefile)
• avoid for logging output (e.g. avoid: "*.html"). a way to control the output.

For config: Maybe create a GCS connector just for funsies?

But but but... for now, Recipefile etc WILL be written on disk. Of course.