Gall Middleware - Assembly 2022 presentation given by ~wicrum-wicrun

These are edited notes from a presentation given by ~wicrum-wicrun at Assembly 2022 in Miami Beach. The original slides are available here.

We at Quartus just released a new app! It’s called Keep, and it's available now from ~dister-dozzod-dalten. Keep allows you to back up the state of any agent, so that even if you breach, you can still have your Pals, and your Places, and the Gora from last Assembly.

Here I’ll present the general pattern that we used when creating Keep. This will probably make the most sense to you if you have already created a Gall agent or at the very least written some Hoon, but even if not, you will probably be able to come away with a broad-strokes picture of the idea.

The main point of this talk -- or really the underlying assumption -- is that you want to focus on one thing at a time.

For example, let’s say that you’re serving a blog from your ship. You have an agent, it stores your content and serves it to those who want to see it. Now, if you want to put some of that content behind a paywall, you do not actually want that paywall to be part of the code. You do not want a command into the agent that says "paywall this content or that content". Of course, in the frontend, you do want it to work like that, but in the backend logic, you don’t want to make the blog code more complex by adding additional pokes and conditions that need to be checked.

Because a blog should really just be a blog. It should store your content, and serve it to people who want to see it. And similarly, a paywall is also really just a paywall. You get a request in, and the paywall says "yes, you can pass" or "no, you cannot pass". There's nothing special about paywalling a blog, or paywalling a chat, or paywalling a bitcoin provider.

Architecturally, what you want is to extend an agent in such a way that neither the extension nor the agent itself knows anything about each other. This decoupling gives you code that’s much easier to understand, follow and debug. You can actually think about the different functionalities independently, instead of complecting them. Ideally, it should even be possible for the end user to extend their blog with a paywall, without having to rely on the app developer foreseeing exactly which extensions people may want to add to their agent.

Agent transformers

Interestingly, it turns out that this pattern already exists in Urbit! If you've written a Gall agent, you have probably seen dbug, which gives you the ability to view the state of any agent. There’s also verb, which prints a message in the dojo every time the agent is called. Both dbug and verb are examples of agent transformers, meaning that they are functions that take an agent in and give out an agent with some modified functionality. In Hoon, they're $-(agent:gall agent:gall).

During this spring and summer, at Quartus we have been working on two agent transformers of our own.

Keep

I already mentioned Keep. Keep is a way to back up any agent that is running on your ship. Most of you probably already know about Peat, the tool that we released to back up social content in Graph Store. Peat works great for groups and chats and notebooks, and it has already helped many martians to restore their content after breaching. But what about your pals? How do you restore those?

You could make a pull request, or ask ~paldev to implement an export-to-CSV functionality. I don’t think he would like that, and why do that when you can solve the problem once and then be done with it? That's what we've done with Keep! It allows you to take the state of any Gall agent, including its incoming and outgoing subscriptions, and store them somewhere!

You can write it to your hard drive so that it can be automatically copied to an external storage medium, or you can send it to another Urbit ship. This ship could be one of your moons, your star, or just a random person on the network who offers to host it for you. And maybe you pay them to do so!

Bank

In addition to Keep, the other agent transformer that we’ve created is called Bank. This is the paywall that I've been hinting at. It allows you to accept payments in crypto for offering any functionality to other ships on the network. We think this is a really big deal, since as far as we know, no one is really systematically charging for services on the network yet. We think that Bank is the first step towards Urbit finally developing its own internal economy.

Bank is not completely finished yet, but will be released in a week or two. We'll make sure to let you know once it's out!

Keep and Bank go really well together, and we’d like to think that getting paid for storing backups is one of the first obvious services that star operators could -- should! -- offer to their planets.

Again, both of these are examples of agent transformers. They really don’t care about the agent that you’re backing up or paywalling, to them it’s just another agent. We think this architecture is really powerful and makes it much easier to write code that we can actually understand and reason about, and also makes it easier for end users use their ships in the way that they actually want. If we want a personal server to become a useful personal tool, we believe this is a direction we have to go in.

Problems

Using agent transformers to extend agents is a very nice conceptual pattern. But in practice, there are three pretty big problems with it. To see these more clearly, let’s have a look at how a transformer actually works.

|= =agent:gall

^- agent:gall

|_ =bowl:gall

+* ag ~(. agent bowl)

::

++ on-poke

|= [=mark =vase]

?: ?=(%keep mark)

=/ poke !<(poke:keep vase)

...

=^ cards agent (on-poke:ag mark vase)

[cards this]

This is the preamble and on-poke arm of Keep. It is a gate whose sample is an agent (L1), and whose product is another agent (L2), i.e. a door that accepts a bowl (L3). But the arms in the door will be able to use the other agent using the name ag (L4) when defining their own functionality.

For example, as usual the on-poke arm accepts a mark and a vase (L7), and depending on the mark, it will either handle the poke itself (L9) or pass it onwards to the inner agent (L11).

This does work, but there are three main issues with this pattern.

1) You need to edit the agent code yourself. If the end user wants to back up an agent that the creator hasn’t already transformed with Keep, they need to go into the code, import the necessary file, and apply the function to the agent, as on lines 1-3:

/+ keep

::

%- keep

^- agent:gall

|_ =bowl:gall

++ on-init

...

--

This is obviously not user-friendly enough for a normal user who just wants to back up their stuff, or use their ship to sell services. And if the inner agent receives an update, these changes will be overwritten, and the paywall will be gone.

2) Stateful transformers can break the agent. If the transformer has any state that it needs to persist between calls to on-save and on-load (L1 below), it of course has to put this together with the inner agent's vase in on-save (L8), so that Gall can store it during reloads, and then in on-load these have to be separated manually (L12-13).

=/ state ...

::

|= =agent:gall

^- agent:gall

|_ =bowl:gall

+* ag ~(. agent bowl)

::

++ on-save !>([on-save:ag state])

::

++ on-load

|= old=vase

=^ inner state !<([vase _state] old)

=^ cards agent (on-load:ag inner)

[cards this]

This is possible, but a bit bureaucratic. Worse than bureaucratic though, if the transformer suddenly disappears (because of an update to the agent, or because the user doesn't want the wrapper anymore), after recompilation, Gall will call on-load:ag instead of the transformer's on-load, meaning that the inner agent will receive a corrupted state!

3) The agent's world will also get transformed. Dbug and Verb are fairly lightweight, they don’t have any incoming or outgoing subscriptions themselves. But if you have a transformer which itself has subscriptions or subscribers, the inner agent will see these in its bowl, as you can see on line 6 above.

So if the inner agent makes any assumptions on its subscriptions, these won’t hold, and the transformed agent might not function as it should. The transformer is not a transparent proxy.

Solutions

The second of these issues is actually a consequence of the first one, and at the moment, both of these are impossible to solve in userspace. So let’s look at the third one first.

Manual bookkeeping

This is possible to solve but the solution is kind of tedious and, again, bureaucratic. In some ways, it’s not even a complete solution, but it does show us the direction we need to go in, so let's have a look at it.

|= =agent:gall

^- agent:gall

|_ bol=bowl:gall

+* sup =- [my=(my p.-) its=(my q.-)]

%+ skid ~(tap by sup.bol)

|= [* * =path]

?= [~ %keep *] path

wex ...

bowl bol(sup my.sup, wex my.wex)

dish bol(sup its.sup, wex its.wex)

ag ~(. agent dish)

Here is a new version of the preamble to Keep. As before, it takes an agent as an argument, and returns a new agent. But now, I'm using skid to separate sup.bol (the map of incoming subscriptions) into two separate maps. One contains all subscriptions to paths that start with //keep (these are subscriptions to the Keep transformer) and I call this map my.sup. The other map contains subscriptions to the inner agent, which I call its.sup. I do the same with the outgoing subscriptions in wex, and then I use these partitioned maps to create two "fake bowls", the bowl and the dish.

This allows us to hide the transformer’s subscriptions from the inner agent, and also to clearly see which subscriptions belong to which part; the transformer uses the bowl, and the agent uses the dish.

Again, this works fine, but it's significantly more bureaucratic than managing the different states. And what about the cards that the transformer passes out?

+$ card

$% [%pass path note]

...

==

Recall that a card is passed on a path, and when they receive acks, facts or kicks in on-agent, we of course want to filter them based on the wire they’re coming on to see where they should be handled:

++ on-agent

|= [=wire =sign:agent:gall]

?. ?=([~ %keep *] wire)

(on-agent:ag wire sign)

...

We somehow want to make sure that all cards are passed on the right path so that they get responses on the right wire. But remember, the transformer just transforms the inner agent into a new, slightly different agent. From the perspective of Gall, there is no transformer:

This means that there is no clear separation between the cards produced by the transformer and the cards produced by the agent, and hence no easy way to automate it outside of manually keeping them separate in every arm of the transformer.

Pluggable middleware

So if we want to automate this, we need to separate the middleware from the inner agent. This means that we want something that looks much more this:

We want to be able to flexibly add generic middleware components to our agents, instead of faking it by stacking transformers upon transformers upon transformers. The transformer pattern is a bandaid rather than an actual solution. It works ok, and it’s the pattern we have used for Keep and Bank because it’s what exists at the moment.

However, I have already started work on the "final transformer", a transformer whose only purpose is to implement this more flexible middleware pattern in userspace. It's not done, and we have now reached the speculative part of the talk, but if I succeed with this, you will be able to use this to make your agents arbitrarily extensible in a safe and scalable way.

Earlier we identified these three main issues:

1. You need to edit the agent code yourself.

2. Stateful transformers can break the agent.

3. The agent's world will also get transformed.

Having this final transformer that could deal with subscriptions and cards would primarily solve the third issue. But if you assume that all agents always have this final transformer installed, we have actually solved the other two, too! You wouldn’t have to change the code for every piece of middleware you wanted to add, and the state of the agent could be kept separate from the states that any middleware components have, and so updates to the agent wouldn't break it!

Of course, you would have to ask your favorite app developers to add this middleware transformer to their agents. But if this architecture actually works, I believe that it will get moved into Gall eventually.

(And as an aside, I just want to say that I think that trying out these architectural changes in userspace is a very good workflow. If you feel that Urbit should be able to do something that it can’t -- see if you can fake it with some tricks in user space! The threshold is really low, you can just go out and do it without asking anyone. If and when you see that they work well and seem fundamental, you can start thinking and talking to the right people about maybe moving them into kernel space.)

Future directions

I’m really looking forward to seeing how this develops. It has already triggered quite a bit of thinking in me.

For example, circling back to the above diagram, this is how you would currently create a paywalled agent where you can back up both the agent and the paywall settings. You have the agent inside, with Bank blocking some calls, and then you use Keep to extract the states of both of these so that you can save them somewhere. This is how it has to be implemented when you’re using transformers.

But conceptually, from a namespace perspective, it looks much more like this:

The paywall sits on the outside of everything, and it may let you through to the inner agent, if you ask it nicely and pay the right bribe. Then you get through to the actual agent, which you can poke and watch as you normally do, and if you poke that the right way, then you reach the backup middleware. So in some sense we’re moving towards something that looks very much like agents with sub- and superagents.

But in my current design, these are separate things. The "real agent" sits in the middle. Super-agents aren’t really agents, but they are just middleware that sits before it, and may let you through, or not. And sub-agents aren’t really agents either, but is just middleware that sits behind the real agent.

This kind of rigid stratification into exactly three layers seems intuitively wrong to me, and I’m hoping that as I continue working on this system, this will change. It seems obviously good to me if we could make it possible to make agents arbitrarily deep, just like a file system.

This hints at a blurring of the distinction between agents and middleware. Because as soon as such a distinction exists, you run into this stratified design, where the agent sits in the middle and middleware is connected either in front of it or behind it.

I suspect that what we actually need to move towards, if we want to make agents flexible and composable even for end users, is some sort of paradigm where we put different incomplete agents together as a kind of lego so that they create a whole agent by being responsible for different points in a namespace:

So you can query an agent at different paths, depending on exactly which functionality you want,

And then it might expose other agents, as you are sent through these layers.

So there wouldn't really be any such thing as a real agent, instead you would just have many small programs that you can make responsible for a path, and if you poke them the right way, they may send you onwards to another program, and now that will be responsible for handling your call.

But this is all still very loose. I’m exploring it in real time, and I still haven’t had the time to read everything that others have already written on these topics. If you’re interested in thinking about this, please don’t hesitate to talk to me, and maybe we can figure something out!

In the meantime, Keep is available for download from ~dister-dozzod-dalten! I recommend that you install it now and ask the developers of the apps that you like that they add it to their source code, so that you can keep your data safe if you should have to breach! The frontend is not completely done but if you’re comfortable using the dojo, it’s all there. And if you install now, you’ll get the frontend in a day or two!

Thank you!