I would recommend against running stateful apps in kubernetes. It's not really ready for it. Big problems include routing (it works fine for http requests, but not for DBs, message brokers, etc) and just the pain of setting up stateful sets.
We run stateful apps on Kubernetes. There are obvious rough areas (the lack of persistent volume resizing, for example, which is scheduled for 1.11), but overall, it's great.
What a lot of naysayers leave out, or choose to ignore, is that the challenges running stateful apps on Kubernetes mirror those of running stateful apps anywhere. If you run Postgres on a VM, for example, you're completely reliant on that VM staying up -- this is no different from Kubernetes. Some will also point out the dangers of co-locating lots of software (such as Postgres) on the same machine as many other containers, as they will compete for CPU and I/O; but this is also no different than on Kubernetes, which provides plenty of tools (affinities/anti-affinities, node selectors) to isolate containers to machines. And so on. Containers bring some new challenges, but Kubernetes meets them quite well.
What specific issues do you have? I'm not sure I understand the point about routing. I also don't understand what the "pain" of stateful sets refers to.
1. While "we already rely on VM staying up", with k8s we reply on both VM staying up and kubernetes infra on top of that VM staying UP
2. Maintaining a complex stateful system on k8s _requires_ having and maintaining an operator for that system.
3. You reduce your options when it comes to tweaking systems, e.g. local SSDs on GCP are available in SCSI and NVMe flavors, while GKE supports only SCSI; harder to perform fine-tuning and other tasks on the underlying VMs that would have been trivial with Chef or similar.
4. Enterprise systems like Splunk explicitly mention that their support does not cover Splunk clusters running on kubernetes.
5. As mentioned, you can't even resize a disk without going through dance of operations that would take days or weeks when you're working something like Kafka at scale.
6. Some stateful services like Zookeeper require stable identities and this is far from perfect on kubernetes.
7. More complex traffic routing that involves additional fees because to achieve (6) you sometimes need to expose things publicly.
That's just from the top of my head.
Disclaimer: We run 10+ stateful services on Kubernetes.
I've used Kubernetes a massive amount during the last two years for running stateful apps. In contrast, I do recommend it. Yes, it is challenging, since stateful app are. However, the challenges are all well worth solving in the context of Kubernetes (great benefits from health checks, automated reproducible deployments, etc.). The situation is pretty good these days in my experience; at least, a lot better than 2 years ago!
Author of post here and CRL employee: just for some additional detail, we reached out to Kelsey about the problems he's seen running databases in Kubernetes.
He said "You still need to worry about database backups and restores. You need to consider downtime during cluster upgrades."
These things are totally true. K8s doesn't automate backups (edit: by default; though, it can) and if you need to take K8s down for upgrades, then everything is down. For its part, though, CockroachDB supports rolling upgrades with no downtime on Kubernetes.
As for routing, that is tough problem if you want to run K8s across multiple regions, though we have some folks who've done it.
Databases are just applications with different resource needs. Please stop pushing forward the notion that they can't be run in containers or container orchestration systems. Databases are just programs. If the substrate for running your containers doensn't reliably support flock or fsync or something your database needs, then maybe pick a better substrate that does -- container runtimes these days and kubernetes don't stand in your way these days.
Well with k8s 1.10+ it's also possible to use statefulsets and local volume, so with affinity it's possible to just use k8s as an orchestration system where you "install" your database and keep it up to date with k8s. of course if a node goes down you need to failover, etc. but patroni/zalando postgres works really well with statefulsets and local volume. (as long as a single node is still running, which should always be the case...)
(https://kubernetes.io/blog/2018/04/13/local-persistent-volum...)
I want to note that this has actually been possible since like k8s 1.7, you can just start a DB with node affinity and use hostPath volumes.
That's what I was doing until rook came around. If you're running in something like AWS (or even if you're not), you can also do something like attach an EBS volume (to a local host) and do that. Or, you can set up a plain ISCSI drive (or get one from your provider, even basic providers these days might offer storage that way) and use that.
We run many, many stateful apps on kubernetes. Not without challenges certainly, but I am not sure any of them are really kubernetes specific.
They just don't act like other services, and require more care. That's about it. I think that's what Kelsey is referring to, you can't just treat them the same as other pods.
If you don't believe me, take it from someone who should know what they're talking about: https://twitter.com/kelseyhightower/status/96341350830081229...