Gettnig this?

19/09/17 22:17:41 WARN lineage.LineageWriter: Lineage directory /var/log/spark/lineage doesn't exist or is not writable. Lineage for this application will be disabled.

Resolve it by creating the folder:

[root@cm-r01en01 spark]# ls -altri
total 2244
335565630 drwxr-xr-x.  2 spark spark       6 Aug 17 21:25 stacks
 67109037 drwxr-xr-x. 27 root  root     4096 Sep 16 03:39 ..
268448735 -rw-r–r–.  1 spark spark 2284695 Sep 17 22:18 spark-history-server-cm-r01en01.mws.mds.xyz.log
134702284 drwxr-xr-x.  2 spark spark       6 Sep 18 07:12 lineage
268447866 drwxr-xr-x.  4 spark spark      87 Sep 18 07:12 .
[root@cm-r01en01 spark]# ssh cm-r01en02
 

Cheers,
TK