WARN lineage.LineageWriter: Lineage directory /var/log/spark/lineage doesn’t exist or is not writable. Lineage for this application will be disabled.
Gettnig this?
19/09/17 22:17:41 WARN lineage.LineageWriter: Lineage directory /var/log/spark/lineage doesn't exist or is not writable. Lineage for this application will be disabled.
Resolve it by creating the folder:
[root@cm-r01en01 spark]# ls -altri
total 2244
335565630 drwxr-xr-x. 2 spark spark 6 Aug 17 21:25 stacks
67109037 drwxr-xr-x. 27 root root 4096 Sep 16 03:39 ..
268448735 -rw-r–r–. 1 spark spark 2284695 Sep 17 22:18 spark-history-server-cm-r01en01.mws.mds.xyz.log
134702284 drwxr-xr-x. 2 spark spark 6 Sep 18 07:12 lineage
268447866 drwxr-xr-x. 4 spark spark 87 Sep 18 07:12 .
[root@cm-r01en01 spark]# ssh cm-r01en02
Cheers,
TK
