I mean, this is pithy and all, but it obviously doesn't rebut my point. You could apply the same logic to any disparate collection of engineering problems.
No, you're right, it's a highbrow dismissal. Let me be a little more detailed.
Once you've built a system that's both a protocol and an OS, you realize that putting an abstraction barrier between them is like building a cow by building two halves of the cow, then sewing them together. It's incredibly hard and the resulting cow doesn't work very well.
Take a problem like identity. Is this an OS problem, or a protocol problem? Earlier I mentioned validation of message / content types. Is this an OS problem, or a protocol problem? Is the spinal column part of the cow-frontend, or the cow-backend?
These are just the most obvious examples. For instance, the standard way of building protocols assumes that message processing isn't transactional, and the OS is a dual-level store that loses its mind all the time. If your OS is a single-level store, you can essentially use persistent sessions. Which means you can get exactly-once messaging. Which is a very desirable feature that's impossible to achieve if you assume that the endpoints can lose their minds.
So the appropriate comparison is: build the front-end of a cow, and stitch it to the back end of an alligator, creating the mighty alligow; or, build a whole cow.
Granted, Urbit isn't perfect and I'd hardly call it done, but the whole stack (counting apps!) is only 25K lines of code. So the "whole cow" doesn't seem appalling. The alligow -- I wouldn't even try.
The notion that the OS and the network are essentially the same thing, a series of procedure calls somehow knitted together either with a stack in memory or as a series of serialized frames on a network, strikes me as a very 1980s way of framing computer science. I feel like the last 20 years of network software development have been a repudiation of that idea.
So yeah, I find it frustrating that Urbit insists that an overlay network is "half the cow", and that to have a "whole cow", we must also adopt an excruciatingly idiosyncratic bespoke programming language.
Mostly, for me, this is coming from a place of deep respect for the concept and potential of overlay networks, and of frustration with designs that seem to sabotage that potential in order to make philosophical/political statements.
(That's my only attachment to Urbit as a thing worth discussing; by way of bona fides:
I actually like that we're having a substantive discussion in a snarky way.
But 1980s? Not to make it personal, but I finished Brown in 1992 and dropped out of Berkeley in 1994 (where I arguably invented ASLR [0], though the basic idea was Larry Peterson's) My field was OS in general and networking in specific, I took Mark Weiser's ubiquitous computing seminar from Mark Weiser, and if there are pieces of paper entitling me to talk about anything, it's '80s networking. "And you, sir, are no '80s networking."
'80s networking as I remember it: at Brown, it was all about TCP/IP versus OSI and crap like that. At Berkeley, the emphasis was more on ATM. And crap like that. Congestion control algorithms, like garbage collection, can be invented indefinitely. Frankly, the only thing more boring and irrelevant than '80s networking: '90s networking. At least in the '80s people still wrote new transport layers and stuff, in the vain delusion that they might get adopted.
And at least in the '90s they still invented their own IETF application-layer protocols that got deployed, or that you could imagine getting deployed. Like that great winner, XMPP. (I spent a particularly depressing afternoon as a fly on the wall in a pre-XMPP working group at some IETF in '97 or '98.) And ACAP? Wherefore art thou, ACAP? And all the calendaring stuff? Various bits of things that were designed to be open networks got fitted into various proprietary protocols of the 2000s.
(Nothing is ever new in CS, and if you're a grad student looking for new ideas about networking, this pattern suggests that the best place to look is '60s and '70s papers. Have fun digging! Or does no one do that these days, either?)
So anyway, we were talking about cows. I get it. You're not interested in cows, only in cow heads. Obviously I share your feelings about cow heads. Love 'em.
So, your problem is: the Internet is full of firewalls and crap. It's a restricted-routing network. This sucks. I remember the Internet when it had no firewalls. You do too. We remember when the Internet was a social network. What is the Internet now? Like we say on the intro page, it's a fing modem, which you use to log in to fing AOL.
So let's create an overlay network with unrestricted routing. Some kind of P2P scheme. (I hope you know Adam Ierymenko's ZeroTier -- he's api on HN. I believe he's basically solved this problem as you define it.)
Great! You have a near-perfect VPN. What else is an overlay network? Have you recreated the Internet as a social network? You know, the Internet that had a distributed Reddit called Usenet -- which, as a digital society, was as far above Reddit as Reddit is above YouTube comments?
Dude, you haven't even come close. All you have is another VPN. Why were those restricted networks put in place? Nobody had a "firewall" when I was at Brown.
To put it another way: what is your overlay network doing? Since you're delegating layer 7 to the OS, you're providing the same basic service as '80s networking, '90s networking, and of course now networking. You're sending datagrams or streams or something between Unix processes on different Unix machines.
For starters, how do you identify these endpoints? A ZeroTier address is an identity in a sense, but it provides no useful information. There's no way to tie it to any identity you're actually interested in.
Who needs a new network for Unix processes to receive data from an effectively infinite set of anonymous untrusted identities?
We already have one of these networks. We call it the Internet. It was a great social network when everyone with an IP address was an institutionally trusted entity. Once that became untrue, we put firewalls on it and it turned into a modem. Anyone can see that this will happen to your overlay network. ZeroTier can be used as a public network or a VPN -- it's a great VPN. With due respect to api, I would focus on that side of the business :-)
But of course there's a bazillion VPNs. You want something different -- you want a cow head. You don't care about the cow body. That's fine. We don't all have to be generalists.
Your theory is: why hasn't someone built a cow head and stitched it onto the alligator body yet? My theory is: (1) if you stitch any head to an alligator and it sticks, it's probably an alligator head; (2) our alligator already has a head, and nothing will be gained by cutting it off and stitching it back on again.
Now, you can see how this process of inferring the rest of the cow goes. You need an identity system, or something, for your network. A PKI. The cow neck. Who holds these identities? Processes on Unix servers? O rly? You're going to infer that person X signed document Y, because person X is connected -- in some way -- to a Unix process with access key to K? Sure, I guess we do that for HTTPS servers, but... for individual human beings?
Ok, you're going to build access to K as a separate component of each application? One keystore per process? Or since it's 2015, per container or whatever? Or the whole computer has access to K? O rly? This is good, this is really good.
So you decide to do what the browser did: create a new opaque layer, above Unix proper and isolated from it, to manage your applications. Imagine if JS "apps" could make system calls through the browser. There'd be no such thing as a Web app. HN would be like a Java applet or something.
An opaque layer (imagine a node that couldn't make system calls, for instance) gives you two wins: it lets you standardize the semantics of a network node precisely, and it lets you run untrusted code with perfect encapsulation. And it offers the even more intriguing possibility of running other peoples' code automagically, which is incredibly useful in a distributed system -- for example, to disseminate protocol validator updates. Thus, "the browser for the server side."
And you go on cow-engineering thus. Until you have the full cow. Then you can drink milk every day and laugh at your strange detour into cow-alligator Frankenstacks.
It must be evident at this late juncture, Chancellor Yarvin, a full half decade since your Durdenesque doppelganger deemed the denizens of earth prepared for his prolix proclamation of Martian supremacy, that the more Carlylean your Urbit apologetics, the deeper your detractors' disposition for derisory dismissal of your loving labors as the grandiloquent, malformed products of, inter alia, luddite nostalgia for the apocryphal elegance of antediluvian computing, a nocturnal-emission inducing fetish for a fascist technocratic telos, viz automation of global autocracy, or a perniciously persuasive strain of neo-Swiftian satire, which advocates Kafkaesque madness with Kaufmanesque earnestness.
That confuses the hell out of me, because I think your plan is brilliant. What gives, broseph?
My only formal CS-education was at summer camp learning Logo when I was nine. In my early 20s, I taught myself LISP from PG's book purely so I could start wasting a few weekends a year coding an urGAI. The first practical coding I've ever done was this year, when I finally bit the bullet and learned Python and Java.
Is your vision convincing to me because I'm clueless about the harsh realities of programming useful things? Or am I able to discern its splendor because I avoided exposure to the false conventional wisdom that blinds other coders to your insights?
If you put a gun to my head, I'd have to pick door number one. Actually, if you put a gun to my head I'd say Urbit was the best thing since Filmer and then apologize for that over-the-top pastiche. But you see my dilemma.
[Edit: Serious question. You came up with Watt/Hoon 5+ years ago, and you claim it's not much harder to use than Lisp. If that's true, where is all the useful Hoon-coded software?
If I gave you five years to write things in Lisp, you could produce an incredible library of offerings... No?]
Hoon didn't really work properly until 2012 or so. Actually I've paid very little attention to it since 2013.
Most of the time since mid 2012 (me for about a year, me and a few other for two for two more) has gone into writing Arvo, a purely functional operating system. Arvo is not enough like anything else to be seen as anything else but research.
This is a fairly typical timeline for CS research. What's unusual is just the depth of the stack. I would also point out that if you look at the normal cost of developing any sort of operating system, even to the alpha level, the metrics are pretty good.
Where the project looks really unproductive is in objective measures of content production. For instance, Nock took me roughly from 2002 to 2008, which is something like a bit and a half of output per day.
Library gap aside, Lisp has an FFI. If you want to use libpng from Lisp, you just make some C calls. If you want to use libpng from Hoon, you first have to re-implement the functionality of interest in Hoon just to have a sound basis for using the C library (as a drop-in optimization, or "jet").
This obviously makes for slower going, especially near the beginning.
And now it hits me, for the first time, shamefully: the sense that Urbit is Pynchonesque... it's often said of Pynchon that nostalgia for the 60s is the animating theme of most of his later work, and here it is made explicit: the chain Usenet -> Reddit -> YouTube. (Clearly, I think this is a good thing.)
Do I miss usenet? I have to say I only ever liked it from within emacs, with glowing characters on a black background. Seeing it on a web page, dark on light, is just not the same. Les neiges d'antan
Once upon a time, it was fun to see what you could learn about an IP address that you found in a weblog. Who bothers anymore?
Perhaps it is snowing, in a way that we are too sinful to perceive!
There are plenty of numbers. We could all have one, forever; or until we die, and then we can, like Charlemagne, will them to our children, who will factor them at leisure
Moreover, the idea that Urbit thinks of the network as "a series of procedure calls somehow knitted together" is a very, very gross distortion. (I guess if you think of "'80s networking" as RPC-heavy, you might have sort of a point about that '80s thing. For me, RPC is already in the OS layer.)
Granted, the high-level interface you want to present to the programmer is something like RPC. The two application-level communication paradigms in Urbit are a transactional "poke" and publish-subscribe.
A key difference is that a successful poke (a) contains no return data and (b) is piggybacked on the packet ack. Actually the whole transaction is piggybacked on the packet ack ("single acknowledgment" or "E2E acknowledgment.") Oh yeah, that's another feature that crosses OS/protocol lines.
This lets us produce a particularly non-leaky network abstraction against not procedure/function calls, but Arvo's stacked event calls (if normal events are a lot like GOTOs, Arvo events are more like GOSUBs). So it's not quite RPC. But it's fair to be reminded of RPC.
But at the actual packet layer, a network is a bus for sharing large (but not too large) unsigned integers, which may or may not arrive anywhere. (Network programming is quite a bit easier, by the way, given a bus-width-independent language that can just model packets and blobs as big atoms, then operate on them functionally.)
Effectively, a packet you hear is something that someone said, not something that someone told you to do. Hearing it is learning something. Idempotence at the packet level is crucial, because if someone tells you something twice, it's the same as telling you once. The protocol exists to answer the question: what happens to me if I learn this number?
It's only a short step from here to defining the entire state of an endpoint as a permanently fixed function of the list of numbers you've learned. The computer's state is a function of its packet log. What could be more natural? Why would anyone define a computer in any other way?
But of course, defining a computer as a pure function of its packet history requires you to define its VM and OS as part of defining the protocol.
And a general rule of protocol design is that your chance of achieving a compatible protocol is inversely proportional to the square of the length of the specification. It's also inversely proportional to the extensibility of the protocol.
So you're going to squeeze all of Unix into your RFC? Or even all of JS, bless its soul? And again, you wind up looking for something like Urbit. You may just be interested in the cow head, but you can't get to it without building the whole cow.
