One way to think about this is that training fits data to a curve. Activation functions specify those curves, and the question is which curve gives the best fit? A linear function would fit the data ...

This is an interesting class of functions, but not nearly interesting enough for calculus purposes. While I do spend lots of time pushing the idea of linear approximations on my students ...