It was definitely not a casual course. I went in hoping for a graduate level course and that's what it was. I had to spend a lot of time on it. I didn't find the math too advanced but understanding the lectures and doing the exercises took a while.

Karpathy's CS231n was indeed another great course. We're really spoiled for choice these days.