🔀 Conditional Flows

Flows allow you to build automated data pipelines with conditional logic, branching, retries, and robust error handling. You can define flows that react to the outcome of previous steps, dynamically control their next action, or even skip tasks entirely.

Looking for Legacy Flows? To migrate an existing Legacy Flow to a new Conditional Flow, see the Migration Guide.

Access Flows

Navigate to Conditional Flows > Create Flow. You’ll land directly in the Builder where you can start creating your first flow. Use the plus icon (+) to add different types of actions such as components, conditions, variables, notifications, and more — all of which are explained in detail later in this documentation.

Build the Flow

Phases and Tasks

• Phases group multiple tasks (components, variables, notifications).
• Tasks within a phase run in parallel.
• After all tasks in a phase complete, based on conditions it is determined which phase will be executed next.
• You can define multiple condition rules - only the first matched condition is executed.
• How to end a flow: You can stop a flow at any point using the End Flow option in the ELSE path of a conditional condition. This is especially useful when none of your IF conditions are met and you want to avoid continuing to another phase.

Caution

If too many tasks are scheduled in a single phase, you may exceed the available Storage job slots, causing delays in your flow’s execution. Limiting the number of concurrent component jobs to 10 is recommended. The Keboola Support team can help you adjust parallel limits.

Execute Tasks in Parallel

You can group multiple tasks within one phase. These tasks then run independently in parallel, speeding up the execution. Phases execute sequentially, while tasks within a single phase run in parallel. If you have multiple data source connectors, you can include them all in a single phase, allowing them to run simultaneously.

The same applies to data destination connectors. Also, transformations independent of the connectors can be grouped within the same phase. Note that this does not reduce costs, as each job consumes credits independently.

You can also set up parallelization within a component (configuration), directly in the component’s UI for row-based components like database source connectors using the same credentials to run multiple tables concurrently.

Storage jobs have a parallel limit. They are typically capped at 10 parallel jobs but the Keboola Support team can help you adjust this.

Control Task Execution

Select a task in the Builder to open its settings.

• If you need to temporarily skip something, disable the task. The task will then be excluded from the flow.

• Failure handling is expressed through conditions instead of a “Continue on Failure” toggle — you can branch on task or phase status (e.g., if status == 'error' then ...) to send notifications, run fallback logic, or end the flow. To let selected tasks fail while the rest of the phase must succeed, use the Continue on Failure condition subject. See also Retry for automatic retries of failed tasks.

• To modify the parameters sent to the underlying API call, you can set Task Parameters. Select the task and click Set advanced parameters. When finished, click Set.

Example of the advanced parameter: changing a variable in transformation:

{
  "componentId": "keboola.snowflake-transformation",
  "configId": "0123abc",
  "mode": "run",
  "variableValuesData": {
    "values": [
      {
        "name": "variables_name",
        "value": 12345
      }
    ]
  }
}

Conditions

1. Conditional Logic (IF statements)

Control the flow of execution based on conditions like:

• Task or phase status - such as success, failure, or warning. This is commonly used for handling errors, creating fallback branches, or sending alerts.
• Variable values (e.g. thresholds, results from previous tasks).
• Date and time logic (e.g. run only on Mondays, last day of the month).
• Number of Output Tables - control the flow based on how many output tables a task produces.
• Duration of Task - condition to trigger actions depending on how long a task runs. This is useful for detecting anomalies (e.g., unusually short or long runtimes).

Each of these can be evaluated for a single task, a whole phase, all or any task in a phase, or any task in the whole flow - see What the Condition Compares.

Evaluation proceeds from top to bottom, and once a condition is true, the remaining conditions are ignored - even if others would also evaluate to be true.

Caution

Only the first matching condition is executed.

Example: If you define 5 conditions and both the 3rd and 5th conditions are valid, only the 3rd will be used.

2. Logical AND / OR between conditions

You can use logical operators (AND) and (OR) to combine multiple statements within a single condition statement.

• Use (AND) when all statements must be true for the expression to pass.
• Use (OR) when any one statement being true is enough.

3. What the Condition Compares (Subject)

Every statement starts with a subject - the thing the flow looks at. The picker offers two tabs: Phases / Tasks and Variables (see Using Variables in Conditions).

In the Phases / Tasks tab you can choose:

Subject	What it evaluates
Any Task in the Flow	A flow-level row above the phase list. The statement passes when at least one already finished task anywhere in the flow matches.
phase > Whole Phase Status	The resulting status of the whole phase.
phase > All Tasks in Phase	Passes only when every task in that phase matches.
phase > Any Task in Phase	Passes when at least one task in that phase matches.
phase > Continue on Failure	Passes when every task in that phase succeeded, except the tasks you explicitly allow to fail. See Continue on Failure.
phase > task	A single field from that task’s job result (browse the result tree, as with Dynamic Value variables).

For the aggregated subjects (Any Task in the Flow, All Tasks in Phase, Any Task in Phase), the field is picked from a short list that applies to any task: Job Status, Job Duration, Error Message, Count of output tables, Sum of imported rows, and Min of imported rows.

When you pick a specific phase or task, you can only reference phases that run before the condition (plus the condition’s own phase). Any Task in the Flow needs no phase at all, so it also covers tasks in parallel branches that already finished - not just the ones directly upstream.

Caution

A condition can only ever look at tasks that have already finished at the moment it is evaluated. Any Task in the Flow therefore silently ignores tasks in phases that have not run yet - it never waits for them.

The typical use case for Any Task in the Flow is a single flow-level error branch - “if any task in the flow ended with an error, send a notification” - instead of repeating an Any Task in Phase statement for every phase.

JSON equivalent (useful when authoring a flow as a template or via the API) - the aggregated subjects wrap the inner statement, which uses * as the task:

{
  "type": "operator",
  "operator": "ANY_TASKS_IN_ANY_PHASE",
  "operands": [
    {
      "type": "operator",
      "operator": "EQUALS",
      "operands": [
        { "type": "task", "task": "*", "value": "job.status" },
        { "type": "const", "value": "error" }
      ]
    }
  ]
}

The per-phase variants use the same shape with the ANY_TASKS_IN_PHASE or ALL_TASKS_IN_PHASE operator plus a "phase": "<phase-id>" property.

4. Continue on Failure

Continue on Failure is a subject of its own, picked per phase next to All Tasks in Phase and Any Task in Phase. It expresses “these tasks may fail, all the others must succeed” - the equivalent of the Continue on Failure toggle known from Legacy Flows.

After selecting it, pick the tasks that are allowed to fail in the All tasks of the phase must succeed, except selector. The statement then passes when every other task in the phase finished successfully; a task blocks the branch only if it did not succeed and is not on the list. Leave the list empty to require that the whole phase succeeds.

The condition is summarized as All tasks in Extract Data succeeded / except My GitHub.

Tip

Use this subject instead of combining an All Tasks in Phase statement with per-task failed statements using OR. Such a combination passes as soon as one of the tolerated tasks fails - even when a task that had to succeed failed as well.

JSON equivalent - the operator takes no operands, because the statuses are fixed. It carries the ids of the tolerated tasks instead:

{
  "type": "operator",
  "operator": "CONTINUE_ON_FAILURE",
  "phase": "8888",
  "tasks": ["456", "789"]
}

Task ids that no longer belong to the phase (for example a deleted or disabled task) are ignored, so the condition keeps working when the phase’s tasks change.

Variables

Variables in Flows let you store and reuse values - like dates, task results, or custom inputs - throughout your flow. You can use them to make decisions, control flow logic, or pass dynamic values between tasks. The sections below walk through how to set up and use variables from the UI; each step also shows the JSON shape generated behind the scenes for template authors and API users.

Adding a Variable

1. In a phase, click the + icon and choose Set Variable. The Set Variable panel opens on the right.
2. Enter a Variable Name — this is the identifier other tasks will use to reference the value.
3. Choose a Variable Type — either Static Value or Dynamic Value.

Static Value

A Static Value is a fixed text or number you enter directly. Useful for thresholds, IDs, or labels that don’t change between runs.

JSON equivalent (useful when authoring a flow as a template or via the API):

{
  "type": "variable",
  "name": "max_duration",
  "value": 3600
}

Dynamic Value

A Dynamic Value is computed at run time from a task result, an earlier phase, or a built-in function (see Date & Time function below). When you pick this type, the value picker lets you browse the outputs of tasks that ran earlier in the flow. You can pick any field from the job’s result tree — for example result.output.tables, result.artifacts, result.images, result.configVersion, result.errorMessage, and many more.

If a task produces multiple output tables, the picker also offers aggregations across all of them: Sum, Minimum, Maximum, and Average of a numeric field. For example, Sum of importedRowsCount returns the total number of rows imported by an HTTP data source across every output table.

Caution

Aggregations are not functions. The Sum / Minimum / Maximum / Average options are not built-in functions — the function enum only accepts COUNT and DATE. The picker implements each aggregation as a JMESPath expression placed in the task value field rather than a function block. When authoring a flow via the API or as a template, write the aggregation directly in value:

• Sum → sum(job.result.output.tables[].importedRowsCount)
• Minimum → min(job.result.output.tables[].importedRowsCount)
• Maximum → max(job.result.output.tables[].importedRowsCount)
• Average → avg(job.result.output.tables[].importedRowsCount)

JSON equivalent — behind the scenes the aggregation is stored as a source definition. For example, Sum of importedRowsCount can be expressed as a JMESPath aggregation in the task value. Note that the picker tree displays paths rooted at result.* (for example result.output.tables), while the generated value expression is rooted at job.result.*:

{
  "type": "variable",
  "name": "total_imported_rows",
  "source": {
    "type": "task",
    "task": "extract-data",
    "value": "sum(job.result.output.tables[].importedRowsCount)"
  }
}

COUNT and DATE are the only functions exposed via the function block (see Date & Time function below); they take their inputs as operands. COUNT counts the items a JMESPath expression returns — for example, the number of output tables a task produced:

{
  "type": "variable",
  "name": "table_count",
  "source": {
    "type": "function",
    "function": "COUNT",
    "operands": [
      {
        "type": "task",
        "task": "extract-data",
        "value": "job.result.output.tables"
      }
    ]
  }
}

The value field accepts JMESPath expressions, so you can filter and extract specific items from the task result instead of just walking the tree. For example, picking the name of a particular output table by its ID:

{
  "type": "task",
  "task": "97288",
  "value": "job.result.output.tables[?id=='out.c-test.example'][].name | [0]"
}

Date & Time function

Returns the date/time formatted according to the specified format string, available formats: https://www.php.net/manual/en/datetime.format.php.

This example returns the full textual representation of the current month, such as “July” or “August”.

{
 "type": "function",
 "function": "DATE",
 "operands": [
   {
     "type": "const",
     "value": "F"
   }
 ]
}

Example of creating a variable with the current timestamp:

{
 "id": "set-timestamp",
 "name": "Set Timestamp Variable",
 "phase": "init",
 "task": {
   "type": "variable",
   "name": "current_timestamp",
   "source": {
     "type": "function",
     "function": "DATE",
     "operands": [
       {
         "type": "const",
         "value": "U"
       }
     ]
   }
 }
}

Using Variables in Conditions

Once a variable has been set in an earlier phase, any later Condition can compare its value against a constant, against another variable, or against a task result.

1. In the IF row, click the value picker and choose a variable, a task result, or a phase result from an earlier phase.
2. Choose an operator (Greater than, Equals, Contains, …).
3. Provide a comparison value — a constant, or another value picked from the tree.
4. Set the THEN and ELSE actions: Continue To an existing phase, or end the flow.

Only the first matching IF condition is executed; subsequent IFs in the same Conditions block are skipped.

See also the Conditions section above for the full list of operators and condition types.

How Variables Reach Component Jobs

When a phase runs a component (a job task), the variables you set earlier in the flow are merged into the component’s own variables. A flow variable replaces a component variable only if both have the same name — flow variables whose names the component does not declare are silently ignored. This means: to let a flow drive a value inside a component, declare a variable with the matching name in the component’s configuration; the flow will fill it in when the job runs.

For finer control on a specific job task, an advanced variableOverrides field on the task can restrict which flow variables are merged in.

Retry

You can retry failed tasks automatically and optionally choose to retry based on specific failure messages.

By default, the system retries up to 3 times with a 10-second delay between attempts. Both the number of attempts and the delay can be customized to fit your workflow.

To access the retry settings, click on task to open the configuration.

Example: If you set 3 retries with 10 seconds delay between attempts, it means the task will run 4x total.

Problem: Pipeline fails due to API timeout.
Solution: Configure retry logic for that task.

Delay between tasks

Introduce a delay (in seconds) before executing a task - useful for waiting for external systems or data availability.

To access the delay settings, click on task to open the configuration.

Notifications

Send alerts directly from the flow and conditions using email(s) or webhook(s). Use the plus icon (+) and select Notification.

You can also create the notification inside of condition as a New Phase, name it and once the condition is all set, use the Use the plus icon (+) and select Notification.

Schedule and Automate

Open the Schedules tab in your flow and click Create Schedule. A flow can have several schedules; they work independently, and the flow runs when any one of them triggers. Each schedule has its own toggle, so you can turn one off without deleting it.

A schedule is either Date & Time or Triggered. For Date & Time, pick one of the predefined intervals (every 15 minutes, every hour, once a day, once a week, end of month) or set your own, then click Set Up Schedule. The dialog works in UTC and previews the next runs before you commit.

Scheduling: Commonly, flows are set to run at specific times. To avoid busy periods in a shared environment, consider scheduling slightly off-peak for smoother execution.

Triggers: Set flows to automatically start when certain Storage tables are updated (ideal for managing dependencies across projects). Your projects will stay synchronized and run efficiently.

Note on Triggers: If table updates happen during the cool-down period, the trigger is suppressed, but the tables are marked as ready. Therefore, if all configured tables are updated during the cool-down period, the flow is not scheduled at that time — but once the cool-down expires and any table is updated (causing the trigger to be evaluated), the system recognizes that all tables are already up to date and runs the flow immediately.

Check Run History

Once your flow is running, you can track its progress and debug issues using the All Runs tab.

In this tab, you will find a complete list of all past executions of your flow. Click into any run to see:

• Which phases were executed
• Which tasks ran successfully, failed, or were skipped
• Which conditional branch was taken after each phase (this is especially helpful when debugging conditional logic)
• The values of variables (if used in the flow)
• Number of retries and its status.

You can use this overview to validate whether your conditions behaved as expected - e.g. if the flow correctly skipped a phase on the weekend or retried a failing task.