Good discussion topic @Aaron … I have also heard Graphite in this context, esp customer who already have tighter pairing with Grafana.
To your observation on Prometheus being poll-based, Yes, that infact is the natural mode for scraping metrics. However, its possible to push metrics from ephemeral and short lived tasks to Prometheus using push gateway. Putting aside the complexity of management and any performance overheads, this provides a practical possibility for push based solution with Prometheus.