Hanging out with Timothy Baldridge, talking about Datomic, Datalog, GraphQL, APIs, etc (2019)

In this transcript recorded Jan 2019, Timothy Baldridge and Dustin Getz talk about Clojure and databases. Timothy Baldridge is a software engineer and databases specialist at Cisco, and before that he was a consultant at Cognitect creating web applications for clients using Clojure/Script Datomic. Dustin Getz is the founder of Hyperfiddle, a fast way to make custom web apps using Datomic.

Table of Contents

What Datomic is great at, and what it isn't

TIMOTHY: A lot of the work I've been doing recently, both at Cognitect as well as now here at Cisco is more around logic engines and query engines and that sort of thing. We have a lot of data stored in different places and a lot of data stores have a lot of different properties. You do a lot of work at Datomic, you're well familiar with how Datomic has some things it's really good at and kind of some holes in other areas. Maybe stuff like... What's the word I'm looking for here? Materialized views and if you have a query you wanna run... You wanna get triggers set whenever some data matches a query in the database, that sort of thing.
TIMOTHY: Those are areas that Datomic maybe isn't so good at. I do a lot of work with Elasticsearch now and Elasticsearch has no support for joins, whatsoever. It's not relational at all, [chuckle] but it's really fast and it can... You can store a ton of data in it. And so a lot of the work I'm working on is kind of just providing tools for people to be able to query that amount of data efficiently, so they can hand us a query and we'll analyze the query, optimize it, and find the most efficient way to talk to Elasticsearch. And that uses a lot of the logic engine stuff I've done some talks on, and things in the past. Kind of our own version of Datalog, but even Datalog has its... Datalog is a set-based thing.
TIMOTHY: So in Datomic if you say, "Give me these results, but I only want the first three," all Datalogs are like this, by the way, it'll find all the results, throw away all of them but the first three and hand you the first three. And so that's not so good when you're working with what could be billions of results. You can't find them all, you don't want that, you want a more lazy approach to find your results. And so a lot of the stuff I've worked on really in the past three years or so, is a lot of that, of just saying, "I know I want the first three results, can we get those efficiently?" And then if I can come back later and say, "I want the next set and the next set." And doing that across data stores that maybe don't have any concept of joins, which is fairly common: Cassandra, Elasticsearch, Riak; although I don't know if that's still a thing or not. Those are all data stores that... Databases that have no concept of joins or relations or, that sort of thing. So, that kind of overlaps with a lot of the interest I have in programming languages, and that sort of thing, and that these are kind of optimizing logic engines that have a programming language interface, the user is gonna say, "Find this where this is true" and you gotta kind of interpret that. And so there's a lot of overlap between those two.

What can't you model with Datalog?

DUSTIN: That's really interesting. With Hyperfiddle, one of the things I think about a lot is... Datalog can express a lot by itself, especially simpler CRUD apps. But what type of things can't it express? And what types of things that we might not be thinking about could you express with say some other additional query language, that we can't express today?
TIMOTHY: Yeah. So as long as your data set's small enough, Datalog is gonna be way faster than a Prolog approach or miniKanren or core.logic or something based on those things. It's kind of a lazy... And all at once set-based approach is faster if you want all the results. But the one thing that I would love to see people investigate more, and there's been some very small attempts in this, is that there's a whole 'nother set of engines, logic engines, if you wanna call them that, called RETE networks or rules engines... Where you build the query once and it sits there and the query doesn't change but as you put new data into the engine, it'll trigger new outputs so... So this is the classic like... If you ever say, "I want to know when this query would return an additional result."
DUSTIN: Have you looked into what Niko presented at the Conj?
TIMOTHY: I don't know that I have. That was the one, the talk you did earlier that you sent me, right? I have not seen that yet, but yeah, it's a lot of the same sort of thing of... It kind of runs in the engine. It's interesting because Datalog is this idea of, our query's gonna change and relative to that the data's not gonna change that much. Incremental Dataflows is the opposite. The data's always changing and your queries aren't really gonna change.

Do situated data applications need anything beyond Datomic Datalog + Incremental Datalog?

DUSTIN: So here's the big question then from my perspective at least, is if you've got Datalog on the one hand — pull based — and then you've got the incremental streaming-based RETE network or incremental dataflows or whatever, what are the holes in that? If you can assume both of those are good? What else do we need?
TIMOTHY: I think that's... Yeah, that's pretty much it, aside from maybe a streaming result, a lazy... Something that allows you to lazily produce results from the system and go back later. Prolog had that back in the day, that you could say... You could tell it, "Get me a result for this query," and it runs until it finds one result. And then you can say, "Get me the next one, now get me the next one."

Sorting problem

TIMOTHY: Unfortunately with that, one of the downsides is any sort of sorting is really hard to do...
DUSTIN: It's impossible, right?
TIMOTHY: Yeah, it really is. Unless you do a whole lot of tricks, which I've done in some systems where it's like the backend. If the backend knows about sorting you can kind of let that flow through the whole system. So if Elasticsearch or whatever, can give me the results in a sorted manner, pre-sorted, then I can kind of flow that through the system and my results coming out the other end are sorted. But yeah, it's sorting is... If someone said sorting is no longer a thing in this world that we know of, I would be a very happy person. [laughter]

Pagination – is it even needed?

DUSTIN: One thing that seems interesting, if you look at bigger systems like say Facebook, like Facebook comments on... The President's page has millions of them, they just have the linked list approach of like, "Hey we're just gonna traverse one node at a time and the order is inherent than that." Does that cover all the use cases that people want pagination?
TIMOTHY: That's interesting. I actually haven't thought about that. I think that one of the problems is just people expect arbitrary sorting. That is a sorting based on time that each person... If you said, "I want all the comments made about the President sorted by name of the poster", that's a little bit harder to do. You'd have to then have two lists.
DUSTIN: But that's not an interesting query because there's 10 million of them.
TIMOTHY: Yes and no. In that context, no, Maybe it's not interesting but in a different context, depending what the data set is. A lot of the stuff we work with would be things like... I may want to see all the viruses that we've seen recently sorted by a threat value that we've assigned the virus or then sorted by the domain address that they talked to. I don't know, I'm pulling stuff out of my head here. [laughter] But yeah, there's cases like that. And the true part of the story that I'm sure you're familiar with is that if you're doing UI work, everybody just expects, you click the header at the top and it sorts, [laughter] and you click it again and it inverse sorts. That's just the way the world works. [laughter] Yeah.
DUSTIN: We're gonna find out. [laughter] What you said about sorting by threat value is interesting... Oh, because if you have the incremental data flow, it's like aggregates. You can materialize it and do the sorting in that and view incrementally. If you keep inserting to the right place, you can actually sort it.

Incremental rules engines fix queries at compile time

TIMOTHY: Yes. You can. Now of course the problem there with a rules engine... I have to look at the incremental data flow, but I imagine it's kind of the same sort of approach. The problem with that is that if you want to sort by a new thing or you have a new piece of information, you have to change the network and backflow any...
DUSTIN: You have to know compile time...
TIMOTHY: Yeah. Yeah. Exactly. And there are systems where you can add a new rule at runtime, but a lot of those also involve then re-ingesting history into the network to repopulate anything they... 'Cause a lot of these systems to save memory, will just throw away any information you're not interested in, from compiled query. That's actually a problem that I've seen in some systems is... Okay, great. If we built the system, whenever we add a new rule, we would have to shut down the server and restart it or throw away the data flow or just the fact of ingesting 10 months worth of data would just take a long time. Yeah. Nothing's ever for free. But I've often wanted that sort of thing of being able to have a engine that kind of, can do both. The same query language could express both pull and push-based, because there's almost always a time in a system where we'll say... Someone will say, "Hey, this is great. We can query your system. Can I give you a query and you give me a data feed?"
TIMOTHY: And you're just pushed to this end point whenever you get a new match, and that's hard to do [laughter] without writing them.
DUSTIN: It's interesting that Datalog makes it possible because to some extent, this is all a moving target just like, "What is the world gonna be like in five years and what are the capabilities people are gonna ask for in five years?" And given the operational transparency of the way the Cognitect technology works, I can see a path forward where there's incremental data flows running with Datomic parity, where you can run the same query both ways.

Aggregates

TIMOTHY: Yeah, yeah. There's nothing stopping that from happening with Datomic's Datalog language. You could use the same syntax for both. I worked on that for a while and I thought there was a couple things... Oh, it's aggregates. Aggregates and the like are real interesting when you get into a rules engine, but there's no real way of expressing those in Datalog, and there's really no reason to because you do those in Clojure code, currently. There is a little bit of that, but you could definitely extend Datomic's Datalog to do more.
DUSTIN: Or just re-implement it in different ways, with different operations and characteristics.

Optimizing

TIMOTHY: Backends. Yeah. And that was even the question when I was working on that, was, "Do you then target the Datalog or do you provide an abstraction above it to say, 'Hey you're gonna write in this top level language that understands aggregates and all this, and we'll compile the Datalog or we'll compile to a Clara rules or whatever.'" And I just never got down to writing that. Mostly because these things just take a long time to perfect and build, and I'm still tweaking what I've written for Cisco and I'll probably be tweaking it for years. [laughter] There's always something to improve or change. What's fascinating to me about it though too, is that there's a lot of business logic that you can bake into an engine when you write it on your own. So the classic thing here with Datomic is Datomic does not provide an optimizer for their query language. Because Datomic doesn't store statistics about attributes and there's just not a whole lot that an optimizer could do. And as Rich has said, "The optimizers work great until they don't and then, you're sunk." You know?
DUSTIN: Right.
TIMOTHY: But in a business situation where you know the data you have, you can leverage that a lot more. And I've done that in some projects where, you know, "I know, yeah, this could blow up or this could be highly inefficient or we could have gigabytes of data in this attribute, but we won't and so, we can pre-calculate these values or this thing doesn't change often enough that we care." Statistics would be a good example of that. So I wrote a system that was for a bank and a part of the calculation was optimized based upon the... What was it? The average size of a portfolio of trades or something like that. And that's a value that will change over time. But the assumption was, if the system's been running for years, the average size of a portfolio isn't gonna change that often between server restarts, so you can catch that at the start of the server, when the server starts up, and you're gonna push new code in two weeks anyway and blow it all away.
DUSTIN: So at what layer do you bake in these optimizations?
TIMOTHY: We were putting these in... We would have a layer that the user would use. Kinda like, we were using Instaparse to parse whatever code they gave us that would come out into an AST. And then, we'd run a nanopass compiler over that, so a bunch of passes over the AST to refine it. And part of that, what I was just mentioning, was like a partial evaluator of saying, you know, "We're looking up a portfolio by an ID." So, we could go hit Datomic or we could cache all the IDs in memory of all of our portfolios, 'cause there's only X amount of them, and then we can replace that ID lookup, so that just removes one join from your Datalog query, right? And if you look that up, maybe the next thing that you are going to look up in the chain, you could also partial evaluate. So, there were some times when you would run the partial evaluator and come to find out through the entire query, you're only ever dealing with one result. And that's actually what a partial evaluator would do. It would run a clause and say, "Do we think this clause is only gonna return one result or 10,000?" it would try to look up one result and if it found one result, and that was the only one... It would optimize it away and then we're going up through, through, through. So, there were times you'd run the optimizer and it's just like, yeah, the result is a constant, it's just three or whatever, you know? [chuckle]

Facebook's GraphQL infrastructure

DUSTIN: Facebook has this layer where they... This layer in their GraphQL implementation where you define the queries, but since there are known at compile time before you deploy, you can use that as an interface to inject hand-coded implementations of the query or hand optimized [0:19:57], but without exposing the UI developer to caring that there's an optimized path.
TIMOTHY: Oh yeah, interesting. In fact that's something... I've been doing a lot of work with GraphQL and that's something I want to try to do on a, somewhat of an automated basis. To say, "Okay, you gave me a GraphQL query, I have an AST, I've optimized it, I know what I'm going to do to the servers for that query." I would love to be able to make that a stored procedure, which GraphQL kind of, mostly allows for, where I could have variables. So you can plug in your variables and now we can just ignore that optimization pass and run the same query for different variables. As it turns out though, for a lot of that it's like Clojure is really fast and in a lot of these systems that optimization pass is 20 milliseconds. And at that point, it's like, "What's the point?" [laughter] I mean, until you get to the problem of, hey 20 milliseconds isn't fast enough, so much more in the system, response cycle is higher. It's gonna take longer to JSON and code the data, than run that optimizer pass.
DUSTIN: But that's what's cool about doing it after the fact, is you can just forget about it until it's a problem and then you can start injecting optimizations by toggling a check box and bypass it in this other way, and that's not expensive to do.
TIMOTHY: Yeah, yeah, exactly, that sort of abstraction. And that is the goal with GraphQL in a lot of these systems as well. It's like, "Hey, now the developers can change the back-end without changing the interface the users use." Yeah.

GraphQL - Does Clojure need it?

DUSTIN: What do you think about GraphQL overall? Is this something that you've been asked before?
TIMOTHY: No, no, I haven't. Well, not publicly. It's great. For what it's trying to do, it's great. I don't think Clojure needs GraphQL, to be honest with you.
DUSTIN: Tell me why.
TIMOTHY: Because it's... I don't know what the languages they wrote GraphQL for but... What is that? It's not JSON, what is that syntax? No, I think the biggest thing is it's not data. The response is data. The response is JSON or preferably EDN or Transit or something, but the query language itself is kind of weird. And I actually, there's two areas of things that I've hacked around GraphQL because it doesn't support it properly. One is OR, there's no logical OR in the language. So there's no way of saying, "Get me people whose name is Tim or their name is Bill." You have to either do like an array, but then you can't do arbitrary binary logic. So you can't say, "His name is Bill OR Tim" OR "They live in Chicago" because there's no way to group those OR clauses in the way I just said it. And even the way I just said it doesn't make sense in English, but you get the point. [chuckle]
DUSTIN: But you could name a collection and then have relational logic on an implementation of it?
TIMOTHY: Name a collection... Yeah or what you can do, what I've actually ended up doing is, I have a input variable that's JSON, and it's a AST as JSON. And it's gross, and it's ugly, but hey, it works. The other thing too, is there's no way to express... So if you have a GraphQL [0:23:25] tree, you can provide parameters to the children, so I can filter the parent and I can filter the children. But there's no way to filter the parent by the children. So let's say you have people, and you have vehicles that they own. You can say, "Find me all people named Tim and list their cars that are made by Ford." But what you can't say is, "Find me all people who own a car made by Ford." You actually would have to invert that in GraphQL and say, "Find me all cars made by Ford, now find me the people that own those," Which is not what the original query wanted either. You want a way of saying like "people.cars.make equals Ford" at the top level, and have that filter the people that are returned.
DUSTIN: But surely there's a way to just use GraphQL like a pull language, but not a filtering language.
TIMOTHY: Yes, and once again, that's also what I've done. I have an attribute just called Query and inside that's a string and you can give arbitrary... Well, JSON or you can do Solr, whatever syntax you want. And yeah, that works, it's just, it's funny because all of that is rectified by just using EDN or JSON as the input. And so, that's the one thing I'm not sure that I agree with, with GraphQL is that I think they should have just had JSON be the input. And I know they want it type-checked and all of this sort of stuff, but you could have type-checked JSON as well. But whatever. [chuckle] I think part of it is GraphQL does look cleaner than a blob of JSON in that the names, your labels, the keys and your maps aren't in quotes, that sort of thing. That you can say, Person: Name. But hey, machines write these things. Not humans in production. So what's the point?
DUSTIN: So I'm trying to understand what you meant when you said Clojure doesn't need it.
TIMOTHY: Yeah, so Clojure doesn't need the GraphQL syntax... Like the actual syntax. Clojure's interface could literally be EDN in and EDN out or Transit in and Transit out.

GraphQL and clojure.spec

TIMOTHY: And you don't even necessarily need a... Like spec, spec would work. Whenever we get data specs — like specs is data, where you could actually download the specs. That's something I think at that point, you basically don't need GraphQL at all, because what you have then is... You could have an end point that says, "Give me the specs that you're aware of," you download the specs into the client, adjust them, process them, whatever, and now you can type-check your request and then your request is just entity name, person, fields that you're requesting, list them out in array of values, what's the filter? That sort of thing. It becomes an AST. It's a trio...
DUSTIN: When you say spec, you mean spec and also probably the schema... If you translate this into Datomic which is more EDN-like, you've got a schema and you've got a spec and they are kind of the same thing.
TIMOTHY: Yeah, Datomic schema is much more data-dependent. You don't necessarily say in Datomic, "This needs to be a string of five characters where the first character is a capital T."
DUSTIN: But you do know it's a string and then you've got specs to EDN...
TIMOTHY: Yeah, exactly.

Clojure specs as data

TIMOTHY: The perfect future I would love to see is where we have Clojure specs as data, so you can introspect them and generate them and transfer them across the wire. And then you could derive your Datomic schema from those specs. So you could just add a new attribute to an entity and that tells you then what needs to go into the database, update the database accordingly. And then yeah, you can check your inputs based on that too. And so it's a... It's fascinating because a lot of what Rich talked about at the Conj, resonated with me as far as the features he wants to add to spec, and I've felt a lot of those pain points of; well, a person at the web level is different than a person at the database level. But there's a two-field difference, but it's more or less the same thing, I don't wanna keep creating types at every layer, as well as just a lot of the relationship stuff, he kinda hinted on this in the talk, but the example I give to people is currently in spec, it's really hard to spec HTTP. If you wanna write a spec that checks the request and response of a REST endpoint, that's really hard because the type of a child is dependent upon the error code of the parents. So if I get a 500, then I expect the body to be this. That's really kinda hard to do, that sort of polymorphic. Because the parent is an HTTP response, which is [0:28:47] body and this sort of thing. Yeah, and so hopefully that sort of stuff is improving, sounds like it will, but we'll see when it comes up.
DUSTIN: That'd be really cool. It seems to me like there's a lot of... The industry is figuring out a lot of things that weren't figured out before, like in the macro sense. Even functional programming in the last 10 years, and how to properly architect apps isn't a solved problem. It seems like everywhere outside of Clojure there's a lot of ad hoc incremental advances stacked on top of each other but Clojure is the only place I know of that has a thought-out and principled derivation of them in a way that fits together cleanly.
TIMOTHY: Yeah. I mean, with some holes right? I think we see that within Cognitect products perhaps. Well, mostly between Clojure and Datomic, but I mean, I don't know that we found the end-all be-all for like how functional app should be written, like the web stack should be written, right? The ring still revolves around a lot of, "Well it's a function, it takes a single argument." There's a lot of side-effecting and that sort of thing that's still happening. But yeah, I think Datomic is really the realization of what would happen if you put a data structure on the disk, a persons data structure on the disk.
DUSTIN: When you say web stack, you're talking about like lower level implementation stuff, right?
TIMOTHY: Yeah. What I'm saying is that like as far as... You look at Datomic and you look at Clojure and they fit together very well. It's still that succession time model that there's a very clean connection between the two, you can tell they're both written by the same person, right?
DUSTIN: Yeah, [I'd go so far as to say that they [Clojure and Datomic] are the same thing.]{.highlight}
TIMOTHY: Yeah. Yeah. Yeah exactly. But but we don't see that through the entire Clojure ecosystem... I would go so far as to say, I think we can see a glimpse of what a perfect Clojure app would be, and to be honest with you I've seen that more with... Pedestal's got its problems, but for all of that there are some very cool stuff you could do when you've basically... What Pedestal does is reify the call stack of the application, what ring has in a bunch of handlers and middleware where you walk up the stack and back down and so, "Hey I work with Liberator and stuff," where you know, you get an error and your call stack is like 200 frames deep of decision trees in Liberator, in Pedestal that's all in data, it's all in a giant hash-map that basically says, 'Here's the steps we will execute on our way out." And so it's very possible to just take the entire call stack of a Pedestal application and just do... Like just save it somewhere, you can run it on... You can go somewhere else. It's kind of like, it's the entire web request as data, including the frames, the stack frames. And it has interesting properties for like testing and introspection debugging, that sort of thing.

A future web stack for people who aren't professional software engineers?

DUSTIN: Even that's all pretty low level for me. Hyperfiddle is after more of a vision where like, what would it take for my mom or my future kids to be programmers and yet not be professional software engineers. All the web stack stuff is all like a layer underneath what matters, because if you've got specs, you've got datalog — both types of datalog, the pull one and the incremental one — Nowhere in there does it matter that it went over an HTTP request or not?
TIMOTHY: Yeah, it's interesting, it's interesting question is when I think that's been asked for years...
DUSTIN: In fact, why can't you like press a button to switch out your implementation to have it go over a websocket now or to have it whatever else there is in 20 years, same application running on a different like IO implementation strategy?
TIMOTHY: Yeah. It's fascinating because on one hand I kind of agree. I remember learning programming back when... I was doing HTTP and... HTML and PHP and that sort of thing, and that was something... I bought a book on PHP, fun story, and I learned C before this and all this and I have friends that are like, "What is up with the [0:33:33]."
DUSTIN: When were you like 15? You seem like you're about my age.
TIMOTHY: Yeah, yeah, roughly about that time, right. But I knew C, I knew C beforehand. So I pull up this book on learning PHP. I literally spent three hours reading the book and I'm like, "This is stupid... " [laughter] "I know it all now. Literally I can set this book down and write a webpage." Because all the names were just, people were like, "Why is it 'strstr'." To take a sub-string of a string in PHP, it's like what was in C, it makes sense. But yeah, so all that to say, there is a problem in programming today in that... I learned to program in GW Basic and QBasic and that sort of stuff and, there's Scratch today, there's whatever the, different Scratch that uses the Lego things? I forget. I don't know. But a lot of this stuff is just really, really complex and I think it's really hard to write something that's both powerful and useful for the average person.

Can the Javascript-GraphQL stack ever get there?

DUSTIN: I don't think that JavaScript, GraphQL, Facebook, whatever, I don't think that stack is ever gonna get there. I think there's too many holes, as you described, like GraphQL gets like 80% or something, but it'll never get all of it.
TIMOTHY: No, no, no. And I don't think that's the... Maybe they claim that's the point. I don't think it's that. For me, with GraphQL, it's more about just like inter-op with other departments, that GraphQL literally runs on anything... Any language out there that can talk to the web, there's some GraphQL client for. And so it's that idea of just being able to say, "Hey, yeah. You got a GraphQL client? Go download the schema, tell me if you have any problems." But...

Clojure has not yet reached critical mass

DUSTIN: 'Cause if you try to extrapolate what could happen with Clojure, I don't think it's reached a critical mass yet where all the benefits that we've been talking about have really reached a point where anyone can achieve these benefits. Actually achieving the benefits of Clojure takes a lot of work right now.
TIMOTHY: Yeah, and I think there is... There's something I've seen develop in Clojure recently, and that is a lot of the recent developments in Clojure are very enterprise-focused and as far as the Datomic Ions. I have friends... Hey, giving an example, I have a brother who's a mechanical engineer, I mean an EE-engineer, kind of rides the line between the two of them. And he knows enough programming to be dangerous, right. Like if he needs to test some circuitry, he can write something that targets an Adreno board or whatever, does all this stuff. He can care less about Clojure, and it's one of those things where it's like Python does anything and everything he would ever want, and I see that as one of those areas where I don't think Clojure is pushing into the accessibility, at least I haven't seen it push more into the accessibility world.
DUSTIN: It may not be time yet.
TIMOTHY: What was that?
DUSTIN: There's an order in which you have to do things and it may not be time yet.
TIMOTHY: Yeah, that could be. That could be. I don't know what the world's gonna be like too, but the area I see Clojure moving into is more the enterprise place where...
DUSTIN: But you have to go to enterprise first, right? 'Cause if they have something that's easy it also first has to be proven complete and to prove it complete, it means people are using it in real world circumstances, and validating that it is better.
TIMOTHY: I don't know that I would agree with that 'cause other languages have started at the other end of things. We could say what we want about decisions Ruby makes or Python makes or even PHP. PHP is still the most popular programming language on the planet.
DUSTIN: But it was complete. Python, PHP, all these were still all imperative languages and they didn't take any risks relative to... They were a little better than the thing that came before them, or even a lot better even, Python was a lot better than C.
TIMOTHY: Right, right. But what they focused on was ease of use and accessibility over all else and that's kind of what I'm saying is that I don't know that Clojure... Maybe someday Clojure will focus on those things. What I see it focusing on and has focused on for the last five years is very much more solving hard problems faced by enterprises, running Clojure on ions or even Datomic, all the new Datomic stuff runs mostly in AWS and... That's the running problem, is it's like, when I was a kid learning programming in C or whatever, it was like, "Where can I find a free C compiler?" We're gonna use DJ CPP, which was GPP or G++ with DOS, [chuckle] And it was like... You get all that, that's kind of where your thing is. If the barrier of entry is, you need a system that can support a two gig runtime, just to boot whatever you're running Clojure on, and it's about two gigs. It doesn't even matter what... If you're using Cursive or Emacs or whatever, you need about two gigs of free RAM to just get Clojure up and running. And then all the cool stuff, Datomic and that sort of stuff is ions, in AWS, a dollar a day is dirt cheap. But that's still more than a hobbyist will pay. Now the world is setting up something free and... Even all of that like...
DUSTIN: What if you didn't have to set it up?
TIMOTHY: Well, that's the thing, what if you didn't have to set it up? What if it was completely free? What if you didn't have to pay? What if it was open source? It's all the same sort of thing of just like, you're still kind of in that world of it's more work to just get it running. It's not just, download this one executable, click "Run" and now you can be a programmer. I'm not saying all this disparagingly towards Clojure. I think there's something that someone mentioned to me, although it was just pure economics one, it made a lot of sense to me. Is that optimizing for the ease of use has very little return at the end of the day. Here's the example. I should make this very clear, I do not work for Datomic, whatever, I do not speak for them. From my side of things, it's like if Datomic was free, what would that gain Cognitect? You'd get more users maybe, but in the end, that's more people to support that probably are just gonna putz around for a while and then not use it. [chuckle] And that's the problem with any product. It's like if I give it to you for free and you want help with it, now I'm spending my time for free, supporting something that I'm getting nothing out of. It's not economical. And even providing a free trial or an inexpensive version or "Here's a version for educational use" or whatever. It's an economics of what...
DUSTIN: To put the paradox... Tech markets are winner takes all, so it is a paradox. You can give it away for free, and then maybe you didn't win, and then you wasted your time and your money. So, you need to find a balance. And then maybe flip a switch, where you reach the point where, "Okay, now it's free."
TIMOTHY: Yeah. And there's multiple of ways of looking at that. I have never heard of this officially, but the old adage that I've heard for years, is that Adobe never prosecutes students in college who pirate their software, because they know the first thing they do when they leave, are they're gonna go and demand that employers buy it. [chuckle] So yeah, there's always that aspect of, if you can get people hooked on the software. But I guess what I'm saying is, too, there's a whole area... The whole area of enterprise is based on this idea that discs are cheap, RAM is even cheaper, I guess, to some extent. Processors are cheap. The thing that I tell new programmers that I come in contact with, it's just like, "Yeah, you have to get over the idea that writing more performant code saves the company money – it probably doesn't." And there's times when I have to sit back and say, "Wow, this is inefficient. I can optimize this code." But why? It's cheaper, it's actually cheaper... For the amount that the average developer in America is paid, pay them for three weeks or just buy a new server. And that's the unfortunate aspect. But that dynamic is different in embedded systems, in game systems, in any small desktop application, anything like that. And I think that's why... I did some work with JavaFX and stuff too, and that's really the problem with a lot of that is it's like, "I'm gonna go write an applicat... " I wrote an application in JavaFX with Clojure. Right? It takes like, it all boots up. Once the whole thing's up and running, the JVM's using like a gig of RAM or something. Then I rewrote most of that in C# and it uses 10 megs of RAM. Just because C# is optimized for this desktop environment, reduced RAM usage. It gives memory back to the operating system as soon as it's done with it. There's just... It was C# versus Clojure, mutable versus immutable and that sort of stuff. I don't know, I don't think the benefits for a... Are necessarily there for running desktop applications. But yeah. The whole market's shifting. Who knows. I don't even know what people would write desktop applications for anymore.

Functional Programming for domains other than web apps?

TIMOTHY: That's something I actually I mentioned on Twitter a while back, I would love to see somebody write something in a functional language that's NOT a web app, nor a backend service, nor an enterprise thing.
DUSTIN: What you need... Yeah... The way the system works is too imperative. The one stack has so many layers, it almost gives you more opportunities to insert abstractions.
TIMOTHY: Yeah. Yeah. Exactly. And I would love to see... You know... Hey tax season's coming. What would a tax application look like written in a functional language? You know whatever those things are written in, it's gotta be horrific. [laughter] And then there's also to apps on the desktop. But the most of what I see... Or actually I think what I actually said was, I would love to see a non-CRUD app written in a functional language 'cause there just aren't that many of 'em. Most of them are...
DUSTIN: Well most stack's also they're about information processing, whereas the desktop apps are more about effects like...
TIMOTHY: Yeah.
DUSTIN: What's on the screen or something.
TIMOTHY: And presentation and that sort of thing. Yeah. But as we've seen with React, you can put a functional veneer over that effect.
DUSTIN: React is pretty slow in the small. You couldn't... You don't write games in it? Right?
TIMOTHY: Right. Right. Yeah.
TIMOTHY: Exactly. So shoutout here to Arcadia. That's what they're trying to do with a lot of their... Their stuff... The people behind the.NET Clojure implementation targeting Unity.
DUSTIN: Yeah. I tried to use that two years ago.
TIMOTHY: Yeah. So I think they finally got funding, so now there's full-time people working on it, and I think it'll get better in the near future. But... And then if... I always have to mention this, too. There's a 2013 QuakeCon. John Carmack got waxing eloquent about Haskell and functional programming. He's the guy that wrote Doom, for anyone that's not aware, and he started talking about how you could write a functional language, write a Doom-like first-person shooter in a functional language. It's about 40 minutes in... I'm sure there's a link to it online. You know, Carmack talks about Haskell at QuakeCon 2013.
TIMOTHY: But he describes like a garbage collector and all of this, all the mechanics you would need to build the system, 'cause his whole thing was... Yeah... At the end of the day an FPS is tracking like 8 megabytes of mutable data, everything else is assets and textures and all that, and that it's really... It's very much feasible to write that sort of thing in a functional language. Except there's still problems, which is... Things you would never have thought of. The one example he gave is, in a purely functional world, two people come up to a door, and now they both wanna walk into the doorway. So you're here, next frame you're both trying to walk into the doorway. That's a problem that requires communication between the two things, they both can't walk through the doorway at once. And that's actually fairly hard to do in a functional language.
DUSTIN: I wonder, it seems to me that if you look at the way software evolves, it can evolve for stupid reasons, like OpenGL is written in an imperative style API because the hardware's imperative. It busts the layer above OpenGL. Well, because OpenGL is imperative, it takes a really smart person to figure out how to turn that into like a garbage collector of virtual DOM or something, to commit that [0:47:29]. It takes a person who's paid to do the work, in order to make another imperative layer on top of it. It doesn't require brilliance. So you get these stack of things that are just like written in terms of the things below it.
TIMOTHY: Yeah. Yeah. I think that's definitely the case.
DUSTIN: So I would wanna see... I don't know. I would wanna see turtles all the way down.
TIMOTHY: Exactly. Yeah. I would love to see that too.