1
- Q：1a745590-1590-4c5b-b6a4-7d4eb2284bd9 Failed to deploy Ingestion Pipeline due to airflow API returned Internal Server Error and response {"error": "Internal error while deploying due to [[Errno 2] No such file or directory: '/opt/airflow/dag_generated_configs/1a745590-1590-4c5b-b6a4-7d4eb2284bd9.json'] "}
- A：webserver pod上没有/opt/airflow/dag_generated_configs目录
2
- Q：openmetadata上配置好doris的元数据ingestion service，test connection成功，但是发布任务报错
- A：需要：pip install pydoris
3
- Q：airflow上任务的日志：*** Could not read served logs: Request URL is missing an 'http://' or 'https://' protocol.
work日志：
[2024-12-20 07:57:42,669: INFO/MainProcess] Task airflow.providers.celery.executors.celery_executor_utils.execute_command[9aec9a2f-7864-4c88-bfe2-4a8e25593053] received
[2024-12-20 07:57:42,675: INFO/ForkPoolWorker-15] [9aec9a2f-7864-4c88-bfe2-4a8e25593053] Executing command in Celery: ['airflow', 'tasks', 'run', '5515133c-4a36-4cf9-b920-58ee6a24dcc4', 'ingestion_task', 'manual__2024-12-20T07:44:03.945304+00:00', '--local', '--subdir', 'DAGS_FOLDER/5515133c-4a36-4cf9-b920-58ee6a24dcc4.py']
[2024-12-20 07:57:52,666: ERROR/ForkPoolWorker-15] Task airflow.providers.celery.executors.celery_executor_utils.execute_command[9aec9a2f-7864-4c88-bfe2-4a8e25593053] raised unexpected: AirflowException('Celery command failed on host: airflow-worker-0.airflow-worker.airflow.svc.cluster.local with celery_task_id 9aec9a2f-7864-4c88-bfe2-4a8e25593053 (PID: 351, Return Code: 256)')
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.10/site-packages/celery/app/trace.py", line 453, in trace_task
R = retval = fun(*args, **kwargs)
File "/home/airflow/.local/lib/python3.10/site-packages/celery/app/trace.py", line 736, in __protected_call__
return self.run(*args, **kwargs)
File "/home/airflow/.local/lib/python3.10/site-packages/airflow/providers/celery/executors/celery_executor_utils.py", line 136, in execute_command
_execute_in_fork(command_to_exec, celery_task_id)
File "/home/airflow/.local/lib/python3.10/site-packages/airflow/providers/celery/executors/celery_executor_utils.py", line 151, in _execute_in_fork
raise AirflowException(msg)
airflow.exceptions.AirflowException: Celery command failed on host: airflow-worker-0.airflow-worker.airflow.svc.cluster.local with celery_task_id 9aec9a2f-7864-4c88-bfe2-4a8e25593053 (PID: 351, Return Code: 256)
webserver日志：
10.60.36.76 - megarobo [20/Dec/2024:07:56:08 +0000] "GET /api/v1/openmetadata/status?dag_id=5515133c-4a36-4cf9-b920-58ee6a24dcc4&only_queued=true HTTP/1.1" 200 2 "-" "Java-http-client/17.0.13"
[2024-12-20T07:56:09.450+0000] {scheduler_job_runner.py:1001} INFO - Exiting scheduler loop as requested number of runs (1 - got to 1) has been reached
[2024-12-20T07:56:09.450+0000] {manager.py:312} INFO - Sending termination message to manager.
[2024-12-20T07:56:09.453+0000] {scheduler_job_runner.py:853} INFO - Deactivating DAGs that haven't been touched since 2024-12-20T07:56:08.262325+00:00
[2024-12-20T07:56:09.474+0000] {process_utils.py:132} INFO - Sending Signals.SIGTERM to group 103. PIDs of all processes in the group: []
[2024-12-20T07:56:09.474+0000] {process_utils.py:87} INFO - Sending the signal Signals.SIGTERM to group 103
[2024-12-20T07:56:09.475+0000] {process_utils.py:101} INFO - Sending the signal Signals.SIGTERM to process 103 as process group is missing.
[2024-12-20T07:56:09.475+0000] {scheduler_job_runner.py:872} INFO - Exited execute loop
[2024-12-20T07:56:09.484+0000] {utils.py:120} DEBUG - Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.10/site-packages/openmetadata_managed_apis/api/utils.py", line 118, in run
scheduler_job.kill()
File "/home/airflow/.local/lib/python3.10/site-packages/airflow/utils/session.py", line 79, in wrapper
return func(*args, session=session, **kwargs)
File "/home/airflow/.local/lib/python3.10/site-packages/airflow/jobs/job.py", line 161, in kill
raise AirflowException("Job shut down externally.")
airflow.exceptions.AirflowException: Job shut down externally.
[2024-12-20T07:56:09.485+0000] {utils.py:121} INFO - Rescan Complete: Killed Job: Job shut down externally.
192.168.4.64 - - [20/Dec/2024:07:56:09 +0000] "GET /health HTTP/1.1" 200 318 "-" "kube-probe/1.30"
- A：找到日志中的['airflow', 'tasks', 'run', '5515133c-4a36-4cf9-b920-58ee6a24dcc4', 'ingestion_task', 'manual__2024-12-20T07:44:03.945304+00:00', '--local', '--subdir', 'DAGS_FOLDER/5515133c-4a36-4cf9-b920-58ee6a24dcc4.py']，去worker pod上执行下，发现worker pod上/opt/airflow/dags和/opt/airflow/dag_generated_configs目录下没有相应的python文件和json，于是把webserver上的文件复制过来，再执行成功了。原因：yaml文件中dag目录没有指定挂盘，要保证每个pod上的dag目录保持同步。
4
- Q：
Airflow中的任务日志：
airflow-worker-0.airflow-worker.airflow.svc.cluster.local
*** Found logs served from host
[http://airflow-worker-0.airflow-worker.airflow.svc.cluster.local:8793/log/dag_id=5515133c-4a36-4cf9-b920-58ee6a24dcc4/run_id=manual__2024-12-20T12:11:25.712723+00:00/task_id=ingestion_task/attempt=1.log](http://airflow-worker-0.airflow-worker.airflow.svc.cluster.local:8793/log/dag_id=5515133c-4a36-4cf9-b920-58ee6a24dcc4/run_id=manual__2024-12-20T12:11:25.712723+00:00/task_id=ingestion_task/attempt=1.log)
[2024-12-20, 12:11:35 UTC] {local_task_job_runner.py:120} ▶ Pre task execution logs
[2024-12-20, 12:11:36 UTC] {server_mixin.py:74} INFO - OpenMetadata client running with Server version [1.5.10] and Client version [1.5.10.0]
[2024-12-20, 12:11:36 UTC] {ingestion_pipeline_mixin.py:52} DEBUG - Created Pipeline Status for pipeline megar-airflow3.5515133c-4a36-4cf9-b920-58ee6a24dcc4: runId='ad6d76ca-9acc-47f3-aabe-d10f06565c11' pipelineState=<PipelineState.running: 'running'> startDate=Timestamp(root=1734696696000) timestamp=Timestamp(root=1734696696000) endDate=None status=None
[2024-12-20, 12:11:36 UTC] {test_connections.py:203} INFO - Running CheckAccess...
[2024-12-20, 12:11:37 UTC] {test_connections.py:228} INFO - Test connection results:
[2024-12-20, 12:11:37 UTC] {test_connections.py:229} INFO - lastUpdatedAt=None status=<StatusType.Running: 'Running'> steps=[TestConnectionStepResult(name='CheckAccess', mandatory=True, passed=True, message=None, errorLog=None)]
[2024-12-20, 12:11:37 UTC] {metadata.py:57} DEBUG - Source type:airflow,<class 'metadata.ingestion.source.pipeline.airflow.metadata.AirflowSource'> configured
[2024-12-20, 12:11:37 UTC] {metadata.py:59} DEBUG - Source type:airflow,<class 'metadata.ingestion.source.pipeline.airflow.metadata.AirflowSource'> prepared
[2024-12-20, 12:11:37 UTC] {local_task_job_runner.py:240} INFO - Task exited with return code Negsignal.SIGKILL
[2024-12-20, 12:11:37 UTC] {taskinstance.py:3498} INFO - 0 downstream tasks scheduled from follow-on schedule check
[2024-12-20, 12:11:37 UTC] {local_task_job_runner.py:222} ▲▲▲ Log group end
openmetadata上配置airflow的元数据ingestion，任务执行失败：Task exited with return code Negsignal.SIGKILL
- A：Yaml文件中没有配置资源，默认资源有问题，设置资源2核4G，搞定
