![]() ExamplesĢa) SQLite check done in configuration.py ( code link).Ģb) When running in standalone mode SQLite compatibility is checked ( code link).Ģc) Sensors in poke mode can block execution of DAGs when running with single process Executors, currently hardcoded to DebugExecutor (although should also include SequentialExecutor) ( code link). Note: This coupling often shows up with SQLite compatibility checks since it does not support multiple connections. To add a new single threaded Executor would require changes to core Airflow code. Some Executors, currently a subset of the local Executors, run in a single threaded fashion and have certain limitations and requirements, many of which are hardcoded. Return redirect_or_json(origin, msg, "error", 400) 2) Single Threaded Executors (SQLite support) Msg = f" does not support ad hoc TI runs" If not getattr(executor, "supports_ad_hoc_ti_run", False): Here you can see it currently being set on the KubernetesExecutor: There is a precedent already set with the supports_ad_hoc_ti_run attribute. ProposalĪ static method or attribute on the Executor base class which can be overridden by child classes which can then be checked by core Airflow code. ![]() Examplesġa) Whether or not to start the serve_logs sub-process, currently hardcoded to LocalExecutor and SequentialExecutor ( code link).ġb) When running in standalone mode, asserts a local Executor is being used, currently hardcoded to LocalExecutor and SequentialExecutor ( code link).ġc) Local Executors need a copy of Airflow configuration created in /tmp , currently hardcoded to LocalExecutor and SequentialExecutor ( code link). To add a new Executor, that would run locally, one would be required to change this core Airflow code. Each include a list of examples along with a proposal to remediate the coupling: 1) Local Executor Compatibility ChecksĮxecutors that run locally have certain limitations and requirements, many of which are currently hardcoded in core Airflow code. Below is a set of coupling types/areas that were identified after a deep dive into the Airflow code base.
0 Comments
Leave a Reply. |