Fail to distinguish alarm emails due to similar subject

Observation

Databricks alarm emails are really hard to distinguish from successful ones. Both successful and failed notification have similar subjects, making them hard to tell at first glance.

This happened at the early times of our pipeline migration from AWS EMR to Databricks. Both of my colleague and I missed alarm for two days because the email subject looks too similar.

Learned

Make alarm email subject has different length as successful ones. Email subject length is an important pattern. We human can tell differences quickly if patterns of an email like length is different. For example:

## Daily Pipeline  completed
## Daily Pipeline  completed
## Daily Pipeline  completed
## Daily Pipeline  completed
## Daily Pipeline  completed
## Failed! 
## Daily Pipeline  completed
## Daily Pipeline  completed
## Daily Pipeline  completed