Zola 2021-01-14T12:15:00-05:00 https://thedav.is/atom.xml 2021-01-14T12:15:00-05:00 2021-01-14T12:15:00-05:00 Unknown https://thedav.is/post/crafting-semantics-2/ <p>This series of posts revolves around creating operational semantics of the Scheme programming language from the ground up, starting with the lambda calculus.</p> <p>If you have not read the introductory post, you can find it <a href="https://thedav.is/post/crafting-semantics-0/">here</a>, and see the index of this series <a href="/tags/crafting-semantics/">here</a>.</p> <h1 id="intro">Intro</h1> <p>Hi! Welcome! Today, we will be extending the CESK machine from the <a href="https://thedav.is/post/crafting-semantics-1/">previous post</a>, adding some simple features that will make our language more usable. In particular, we will implement numbers and conditionals. To accomplish that, we will be partitioning the machine into 2 different types of states. This will make future extensions to our language much simpler.</p> <h2 id="recap">Recap</h2> <p>Let's look at the machine that we have so far. These transition functions serve as the basis for the famous call-by-value lambda-calculus. If you don't remember the domains, I will reproduce them soon, when we create our new machine.</p> <script type="math/tex;mode=display">(x , \rho , \sigma , \kappa) \leadsto (v , \rho_\lambda , \sigma , \kappa) \\ \begin{aligned} \text{where } \sigma(\rho(x)) &= (v , \rho_\lambda) \\ \end{aligned}</script> <script type="math/tex;mode=display">((e_f \; e_a) , \rho , \sigma , \kappa) \leadsto (e_f , \rho , \sigma , \kappa') \\ \begin{aligned} \text{where } \kappa' &\triangleq \textbf{arg}(e_a , \rho , \kappa) \\ \end{aligned}</script> <script type="math/tex;mode=display">(lam , \rho , \sigma , \textbf{arg}(e_a , \rho' , \kappa)) \leadsto (e_a , \rho' , \sigma , \textbf{fn}(lam , \rho , \kappa)) \\</script> <script type="math/tex;mode=display">(lam , \rho , \sigma , \textbf{fn}(lam' , \rho' , \kappa)) \leadsto (e , \rho'' , \sigma' , \kappa) \\ \begin{aligned} \text{where } lam' &= (λ \; (x) \; e) \\ a &\triangleq alloc(\sigma) \\ \rho'' &\triangleq \rho'[x \mapsto a] \\ \sigma' &\triangleq \sigma[a \mapsto (lam , \rho)] \end{aligned}</script> <h2 id="roadblock">Roadblock</h2> <p>The main issue with this machine is that it is very 'lambda-centric'. For example, if you pay close attention in the last post, closure creation is actually deeply embedded into the semantics. When a variable is evaluated it finds the closure and sets it to the current environment. We want a machine that is more ... 'value independent'</p> <p>If we wanted to have values that are less syntactically visible, like <a href="https://thedav.is/post/continuations-as-return/">first-class continuations</a>, adding them to the current machine would overcomplicate our transitions. Instead, we will partition the machine into 'eval' and 'apply' states. Eval states are for <em>evaluating</em> syntax to get a value. Apply states are for <em>applying</em> the value into the current Kontinuation.</p> <h1 id="the-next-machine">The Next Machine</h1> <p>Our machine is very similar to the previous machine, with a few additions to the syntactic and semantic domains. I will point out the differences, so don't worry about cross-referencing the last post!</p> <h2 id="syntax-domains">Syntax Domains</h2> <script type="math/tex;mode=display">\begin{aligned} % expressions e \in \textsf{Exp} &::= \text{\ae} \\ &| \; (\texttt{if} \; e \; e \; e) \\ &| \; (e\;e) \\ \text{\ae} \in \textsf{AExp} &::= x \;|\; lam \;|\; n \;|\; b \\ % variables x \in \textsf{Var} &\triangleq \text{The set of variables} \\ % lambdas lam \in \textsf{Lam} &::= (λ \; (x) \; e) \\ n \in \mathbb{Z} &\triangleq \text{The set of integers } \\ b \in \textsf{Bool} &::= \texttt{\#t} \;|\; \texttt{\#f} \end{aligned}</script> <p>The main chunk of change revolves around the new <code>æ</code> domain. Through this, we have partitioned the syntax into expressions that are complex, and require further evaluation, and 'atomic expressions', which are as small as they need to be, in order to become real values.</p> <p>The set of atomic expressions include lambdas and variables, as before, and introducing integers and booleans. I also added an <code>if</code> complex-expression, which we will implement today.</p> <p>Also, quick note, booleans in Scheme are <code>#t</code> for true, and <code>#f</code> for false. Just wanted to be clear on that!</p> <h2 id="semantic-domains">Semantic Domains</h2> <p>The semantic domains are also mostly the same, but with some additions to accommodate this partition between 'eval' and 'apply' states.</p> <script type="math/tex;mode=display">\begin{aligned} % Machine \varsigma \in \Sigma &\triangleq E\langle \textit{Eval} \rangle + A\langle \textit{Apply} \rangle \\ \textit{Eval} &\triangleq \textsf{Exp} \times \textit{Env} \times \textit{Store} \times \textit{Kont} \\ \textit{Apply} &\triangleq \textit{Val} \times \textit{Env} \times \textit{Store} \times \textit{Kont} \\ % Env \rho \in \textit{Env} &\triangleq \textsf{Var} \rightarrow \textsf{Addr} \\ % Store \sigma \in \textit{Store} &\triangleq \textsf{Addr} \rightarrow \textsf{Val} \\ % Address a \in \textit{Addr} &\triangleq \mathbb{N} \\ % Value v \in \textit{Val} &\triangleq \textit{Clo} + \mathbb{Z} + \textsf{Bool} \\ % Closures clo \in \textit{Clo} &\triangleq \textsf{Lam} \times \textit{Env} \\ % Continuation \kappa \in \textit{Kont} &::= \textbf{mt} \\ & | \; \textbf{cond}(e , e , \rho , \kappa) \\ & | \; \textbf{arg}(e, \rho, \kappa) \\ & | \; \textbf{fn}(v, \kappa) \\ \end{aligned}</script> <p>The changes here were to the states, the values, and the continuations. The change to the states is the most jarring, now we can have 2 different types of sub-states. I added to the values domain, because now we have numbers and booleans! Exciting!</p> <p>We also needed a continuation type for the conditional. When is it appropriate to add a continuation type? In this machine, you have two things to look at for guidance during computation. First and foremost is the Control. When our machine starts evaluating the syntax <code>(if cond then else)</code>, we will place <code>cond</code> into the control, and place a continuation <code>cond(then, else, ρ, κ)</code> at the top of our Kontinuation stack. After the control is evaluated to a value, we need to look to the Kontinuation for the next steps. For that, we use a continuation.</p> <p>So, whenever you are adding syntax that needs some state, so the machine knows what to do after the initial control is evaluated, you probably want a new continuation frame type.</p> <p>Having 2 types of states will make writing out machine rules really easy, I promise! Now, we will have rules for evaluating the control (syntax), and for what to do after its a value (through the continuation)! It is a nice give and take relationship between the states that I really enjoy from this style of machine.</p> <h2 id="atomic-evaluation-and-other-functions">Atomic Evaluation And Other Functions</h2> <p>Last before we get to the new transition rules, we need to define an 'atomic evaluation' function. This function is the bridge from 'eval' states into 'Apply' states. When we see a <code>4</code> in our program, it is syntax, because its in the program text. We need to turn that syntax-<code>4</code> into a value-<code>4</code>.</p> <script type="math/tex;mode=display">\mathcal{A} : \textit{Eval} \rightarrow \textit{Val} \\ \begin{aligned} \mathcal{A}(\langle n , \_ , \_ , \_ \rangle) &\triangleq n \\ \mathcal{A}(\langle b , \_ , \_ , \_ \rangle) &\triangleq b \\ \mathcal{A}(\langle x , \rho , \sigma , \_ \rangle) &\triangleq \sigma(\rho(x)) \\ \mathcal{A}(\langle lam , \rho , \_ , \_ \rangle) &\triangleq (lam, \rho) \\ \end{aligned}</script> <p>This function turns syntax into values! It is straightforward in the number/bool case, and reminiscent of the machine of the last post for the other two cases.</p> <p>We use this function when the control is 'atomically evaluable', meaning we can determine its value immediately. For variables, we just need to look its value up, and for lambdas, we pair it with the current environment to form a closure. The other, 'complex', expressions need a little more work done on them before they can be atomically evaluated.</p> <p>I will also reproduce the other useful functions from the last post, for posterity:</p> <script type="math/tex;mode=display">inj : \textsf{Exp} \rightarrow \Sigma \\ inj(e) \triangleq E\langle e , \varnothing , \varnothing , \textbf{mt} \rangle \\ \; \\ alloc : \textit{Store} \rightarrow \textit{Addr} \\ alloc(\sigma) \triangleq |\sigma|</script> <h1 id="transitions">Transitions</h1> <p>Now that we have all of the supporting infrastructure, we can make the transition rules! We are going to start with all of the 'eval' rules, and then do the 'apply' rules.</p> <h2 id="eval-rules">Eval Rules</h2> <script type="math/tex;mode=display">E \langle \text{\ae} , \rho , \sigma , \kappa \rangle \leadsto A \langle v , \rho , \sigma , \kappa \rangle \\ \begin{aligned} \text{where } v &\triangleq \mathcal{A}(\varsigma) \end{aligned}</script> <p>As a refresher, you can read this rule as follows:</p> <blockquote> <p>If the current state is an Eval state, and our control is atomically evaluable, then return the following Apply state, where v is defined by using the atomic-eval function on the input state.</p> </blockquote> <p>This rule is quite simple! If the current control is an atomically evaluable expression, use our fancy-A evaluation function to make a value of it. This rule covers when the control is a number, boolean, lambda, or variable, what a multitasker!</p> <p>As I said before, this transition rule is the bridge from eval to apply states. We will make transition rules for apply states later, but for now, lets continue evaluating syntax!</p> <script type="math/tex;mode=display">E \langle (\texttt{if} \; e_c \; e_t \; e_f) , \rho , \sigma , \kappa \rangle \leadsto E \langle e_c , \rho , \sigma , \kappa' \rangle \\ \begin{aligned} \text{where } \kappa' &\triangleq \textbf{cond}(e_t , e_f , \rho , \kappa) \end{aligned}</script> <p>This is the first half of the rules required to support conditionals. When we encounter an <code>if</code> expression, we transition to start evaluating the condition. And later, once the condition is evaluated, we will check for a continuation and see the <code>cond</code> frame, and go from there! Stay tuned for the riveting conclusion to <del>this anime</del> interpreting conditionals.</p> <script type="math/tex;mode=display">E \langle (e_f \; e_0) , \rho , \sigma , \kappa \rangle \leadsto E \langle e_f , \rho , \sigma , \kappa' \rangle \\ \begin{aligned} \text{where } \kappa' &\triangleq \textbf{arg}(e_0 , \rho , \kappa) \end{aligned}</script> <p>This rule is remarkably similar to the last post's rule, which is great! I'm not trying to overcomplicate this stuff, so I hope that things don't get needlessly crazy. Although, small spoiler for the next post, I will be implementing multi-arg functions, so this will change slightly! Sorry for making you think that this rule would never ever change!</p> <p>We have now covered 100% of the syntax that can be found in our little language! But, there is a little more to do. Lets transition apply states!</p> <h2 id="apply-rules">Apply Rules</h2> <p>As I alluded to before, there are 2 things that the machine uses to right itself, the control and the continuation. The Eval states are for checking the control, and the apply states are for checking the continuation. We have exhausted control, and now we must exhaust all types of continuations!</p> <script type="math/tex;mode=display">A \langle v , \rho , \sigma , \kappa \rangle \leadsto E \langle e_f , \rho_{\kappa} , \sigma , \kappa' \rangle \\ \begin{aligned} \text{where } \kappa &= \textbf{cond}(e_f , e_f , \rho_{\kappa} , \kappa') \\ v &= \texttt{\#f} \end{aligned} \\ A \langle v , \rho , \sigma , \kappa \rangle \leadsto E \langle e_t , \rho_{\kappa} , \sigma , \kappa' \rangle \\ \begin{aligned} \text{where } \kappa &= \textbf{cond}(e_t , e_f , \rho_{\kappa} , \kappa') \\ v &\;≠ \texttt{\#f} \end{aligned}</script> <p>These are the final rules for dealing with conditionals. We refer to the value (which is the value of the condition) and check if it is equal to false. If so, we take the false branch. We 'take it' by just setting that branch to the control of the next state. It <em>is</em> that easy! The opposite goes for <code>true</code>, if the value in the condition is simply 'not false', then we take the true branch.</p> <p>(Aside: At the time of writing, it seems the rendering for ≠ is broken, but the false branch should have <code>v ≠ #f</code>)</p> <p>Note the semantics here! Many languages have many different ideas of 'falsy' and 'truthy'. Python has a whole overridable <code>__bool__</code> function, and I don't want to get <em>near</em> what JavaScript does. Scheme is simple. The only 'falsy' value is false. Everything else is truthy. Everything. Simple!</p> <p>Its also important to 'reset' the environment. During the course of evaluating the condition, the environment may have changed. We don't want to use that environment while taking one of the resultant branches, which is why we stored the original environment in the continuation.</p> <script type="math/tex;mode=display">A \langle v , \rho , \sigma , \kappa \rangle \leadsto E \langle e , \rho_{\kappa} , \sigma , \kappa'' \rangle \\ \begin{aligned} \text{where } \kappa &= \textbf{arg}(e , \rho_{\kappa} , \kappa') \\ \kappa'' &\triangleq \textbf{fn}(v , \kappa') \end{aligned} \\ % A \langle v , \rho , \sigma , \kappa \rangle \leadsto E \langle e_b , \rho_{\lambda}' , \sigma' , \kappa' \rangle \\ \begin{aligned} \text{where } \kappa &= \textbf{fn}(((\lambda \; (x) \; e_b) , \rho_{\lambda}) , \kappa') \\ a &\triangleq alloc(\sigma) \\ \rho_{\lambda}' &\triangleq \rho_{\lambda}[x \mapsto a] \\ \sigma' &\triangleq \sigma[a \mapsto v] \end{aligned}</script> <p>These two rules are all we need to interpret our function calling syntax. In the first rule, we have finished evaluating the 'function' half of the two expressions, and move to evaluate the argument. Then in the next rule, we have completely evaluated both the function and the argument, and we call the function! These rules are also exceedingly similar to the rules from the previous post, they should not surprise.</p> <p>I like to add this final rule, just to explicitly say 'this is the end':</p> <script type="math/tex;mode=display">A \langle \varsigma \rangle \leadsto A \langle \varsigma \rangle \\ \begin{aligned} \text{where } \kappa &= \textbf{mt} \end{aligned}</script> <p>Of course, this rule is vacuous, as without it, we would simply reach a stuck state in the same circumstance, and say 'thats all folks'. This way though, you can feel a little safer, having a slight distinction between 'stuck' and 'nothing left to do'.</p> <h1 id="example">Example</h1> <p>Lets evaluate a simple program utilizing our new features!</p> <script type="math/tex;mode=display">e_0 = ((λ \; (x) \; x) \;\; (\texttt{if} \; \texttt{\#f} \; 3 \; 12)) \\ \varsigma_0 = inj(e_0) = E\langle e_0 , \varnothing , \varnothing , \textbf{mt}\rangle \\ \varsigma_n = step(\varsigma_{n-1}) \\ \;\\ % \varsigma_1 = E\langle (λ \; (x) \; x) , \varnothing , \varnothing , \textbf{arg}((\texttt{if} \; \texttt{\#f} \; 3 \; 12) , \varnothing , \textbf{mt}) \rangle \\ \varsigma_2 = A\langle ((λ \; (x) \; x) , \varnothing) , \varnothing , \varnothing , \textbf{arg}((\texttt{if} \; \texttt{\#f} \; 3 \; 12) , \varnothing , \textbf{mt}) \rangle \\ \varsigma_3 = E\langle (\texttt{if} \; \texttt{\#f} \; 3 \; 12) , \varnothing , \varnothing , \textbf{fn}(((λ \; (x) \; x) , \varnothing) , \textbf{mt}) \rangle \\ \varsigma_4 = E\langle \texttt{\#f} , \varnothing , \varnothing , \textbf{cond}(3 , 12 , \varnothing , \textbf{fn}(((λ \; (x) \; x) , \varnothing) , \textbf{mt})) \rangle \\ \varsigma_5 = A\langle \texttt{\#f} , \varnothing , \varnothing , \textbf{cond}(3 , 12 , \varnothing , \textbf{fn}(((λ \; (x) \; x) , \varnothing) , \textbf{mt})) \rangle \\ \varsigma_6 = E\langle 12 , \varnothing , \varnothing , \textbf{fn}(((λ \; (x) \; x) , \varnothing) , \textbf{mt}) \rangle \\ \varsigma_7 = A\langle 12 , \varnothing , \varnothing , \textbf{fn}(((λ \; (x) \; x) , \varnothing) , \textbf{mt}) \rangle \\ \varsigma_8 = E\langle x , \{x : 0 \} , \{ 0 : 12 \} , \textbf{mt} \rangle \\ \varsigma_9 = A\langle 12 , \{x : 0 \} , \{ 0 : 12 \} , \textbf{mt} \rangle</script> <p>This example clearly shows the interchange between Eval and Apply states. You evaluate until you apply, and then you evaluate, and so on, until you reach a stuck or finished point (a fixpoint).</p> <h1 id="conclusion">Conclusion</h1> <p>So in only these few short minutes we have added some quality-of-life features to our language, making it much more usable! The semantics grew a little in breadth and depth, but is now much more capable of supporting more advanced features.</p> <p>This post will serve as a general template for this series. We take the machine we have, decide what to add, and then add it. Further than that, the addition of features is similarly patterned. You add syntax, then make sure the semantic domains support the syntax (i.e. the Val and Kont domains), and finally add transitions.</p> <p>By utilizing this method, adding new features is simple. Also by utilizing this method, I can churn out blog posts until the sun dies! Just kidding, I have better things to do than eternally add features until we accidentally formalize Perl.</p> <p>In the next post, I will be adding multi-arg functions, and a couple of other useful features. I hope to see you there!</p> 2021-01-06T12:15:00-05:00 2021-01-06T12:15:00-05:00 Unknown https://thedav.is/post/crafting-semantics-1/ <p>This series of posts revolves around creating operational semantics of the Scheme programming language from the ground up, starting with the lambda calculus.</p> <p>If you have not read the introductory post, you can find it <a href="https://thedav.is/post/crafting-semantics-0/">here</a>, and see the index of this series <a href="/tags/crafting-semantics/">here</a>.</p> <h1 id="intro">Intro</h1> <p>Hi, in this post we will be implementing a CESK Abstract Machine for the lambda calculus (or simply 'λ'). First, we need to define all of what an 'abstract machine' is, what semantics are, and the syntax we will be using. So this post may be a bit long. Future posts are likely to be shorter!</p> <p>If you know all of the intro stuff already, but dont know how to define semantics for the lambda calculus, feel free to skip to <a href="https://thedav.is/post/crafting-semantics-1/#the-machine">here</a></p> <h1 id="cesk">CESK</h1> <p>The type of abstract machine we will be creating is called 'CESK'. It stands for <strong>C</strong>ontrol, <strong>Environment</strong>, <strong>Store</strong>, and <strong>Kontinuation</strong> (C was already taken). For a language as simple as the λ-calculus, we dont need all of this machinery, but as we build features for this machine into a real scheme, it will be nice to have the extra features provided by the Store and Kontinuation pieces.</p> <p>The control of a machine is the current instruction that is being evaluated. For the λ-calculus, this will simply be the syntax. For a more complex machine, like the JVM, this could be the bytecodes and an index into the current instruction. For a CPU, it may be an instruction pointer and the binary program text. When you hear the word 'control flow' it is referring to this component.</p> <p>The environment is the set of variables that can be referenced. We will use a simple key-value mapping (dict, hash, etc.) to represent our environment. The keys of the mapping will be the variable names, and the value will be the address of where to find that variables value.</p> <p>The store is where values are actually held. If you have an address, you can access the store to get its corresponding value. We again use a key-value mapping, from address to value.</p> <p>The kontinuation is what happens 'next', after we finish evaluating an expression. We will use a special continuation frame object to track this. In many languages, this kind of thing is called a ' <a href="https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Call_stack" target="_blank">call-stack↪</a> '. When a function is called, the 'stack' is added to with the current function. In the future, We will augment it to track more things, such as conditionals, let bindings, and variable mutation.</p> <p>It is used to represent a fine grained call-stack. But instead of only keeping track of function calls, it tracks many other things, such as conditionals and let bindings.</p> <h1 id="the-l-of-computer-science">The λ of Computer Science</h1> <p>λ, spelled 'lambda' in english, is a greek letter which is ubiquitous in the computing sciences. It refers to computation itself.</p> <p>The lambda calculus is a very simple language. Created before modern computers, its goal was to express logic formally. Now, we can use it as a basis for a small and simple programming language. Its <a href="https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Lambda_calculus" target="_blank">Wikipedia Page↪</a> is chock-full of great stuff, but I will give it a quick introduction.</p> <p>The lambda calculus has 3 forms</p> <script type="math/tex;mode=display">\begin{aligned} % expressions e &::= x \\ & | \; (e\;e) \\ & | \; (\lambda \; (x) \; e) \\ x &::= \; \text{some variable} \\ \end{aligned}</script> <p>This syntax means, an expression (we are just calling it <code>e</code>) can be three things. It could be some variable, that we are calling <code>x</code>. It could also be an 'application' with 2 expressions in it. Finally, It may be an 'abstraction' that takes some variable and some expression.</p> <p>Both <code>e</code> and <code>x</code> are what are called 'metavariables', meaning they are variables to be used in the grammar, and will be replaced in a real text. An example λ-calculus program may be:</p> <script type="math/tex;mode=display">((\lambda \; (q) \; (\lambda \; (b) \; q)) \; (\lambda \; (\text{blerp}) \; \text{WORD}))</script> <p>This is a program! At this moment, it doesnt do anything, beacuse we have not given it any <em>semantics</em>! Semantics are what give program text meaning. Maybe in my semantics, I will say, "If a program looks like <code>((λ (q) (λ (b) q)) (λ (z) z))</code>, then format the Hard Drive!". I would not say thats a very moral semantics, and it places a weird dependency on something called a 'hard'... 'drive'... but it's a semantics!</p> <p>Given a reasonable (and standard, time honored) semantics, these three can perform any computation. We will implement semantics for the lambda calculus in this post, and add on to it in the future to make an actually useful language.</p> <p>So lets get started!</p> <h1 id="the-machine">The Machine</h1> <p>First, we need to define our <em>domains</em>. These are the things that we actually use to implement our machines. A simple domain that you may be familiar with is the set of Integers. starting at 0, you can go forward by 1, or backwards by 1, to infinity (and negative infinity). Our domains include 'syntactic domains', which are domains seen in the progrm text, and 'semantic domains', for determining meaning.</p> <h2 id="syntax">Syntax</h2> <p>The syntactic domain is our grammar from above, but lets expand on it a bit for our sake. We will also be using a more mathy notation, in addition to the <a href="https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Backus–Naur_form" target="_blank">BNF↪</a> above.</p> <script type="math/tex;mode=display">\begin{aligned} % expressions e \in \textsf{Exp} &::= x \\ &| \; (e\;e) \\ &| \; lam \\ % variables x \in \textsf{Var} &\triangleq \text{The set of variables} \\ % lambdas lam \in \textsf{Lam} &::= (λ \; (x) \; e) \\ \end{aligned}</script> <p>To read this notation, you need to take a look at the three parts. First, is the name of the metavariable. For the first line, the metavariable is <code>e</code>. When you see <code>e</code> in use, it is referring to this domain. Next is the name of the domain. Third, you have the definition. I use BNF style or 'definitional equality' by my own choice. the <code>∈</code> symbol means 'in', or 'is a member of domain'. So the first line reads, "<code>e</code> is a member of the domain 'Exp', which is defined as these three productions".</p> <p>There are 3 domains here, expressions, variables, and lambdas. These are the same as above, but now we have the name of the sets used, instead of just the metavariable names. If we see any <code>e</code>, then it is in the <code>Exp</code> domain. The same goes for <code>x</code> and <code>Var</code>. We may see multiple <code>e</code> metavariables used, but in this context, they can be filled in with different values. Later, when we are writing the semantics, we will differentiate metavariables of the same domain with subscripts or tick-marks.</p> <p>I also factored the lambda 'production' in the grammar into its own domain. This will be helpful in the semantic domain definitions.</p> <p>The equal sign with the delta-triangle over it just means 'is defined as'. In Computer Science and Math, <code>=</code> may mean 'is the same as' or 'is defined as', so we use the Δ to differentiate the two.</p> <p>Sometimes <code>::=</code> is used, and sometimes our <code>=</code> with <code>Δ</code> is used, why? Because using notation in this way is helpful. There arent strict rules here, and I found it useful to use BNF style for some domains, and definitional-equality for others. If you dont like it, you could perhaps give each production in the BNF grammar its own domain, and use a ' <a href="https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Tagged_union" target="_blank">sum type↪</a> ' to define everything in terms of definitional equality.</p> <h2 id="semantics">Semantics</h2> <p>The semantic domains include the definitions for the machine parts themselves. Exciting!</p> <script type="math/tex;mode=display">\begin{aligned} % Machine \varsigma \in \Sigma &\triangleq \textsf{Exp} \times \textit{Env} \\ & \times \textit{Store} \times \textit{Kont} \\ % Env \rho \in \textit{Env} &\triangleq \textsf{Var} \rightarrow \textsf{Addr} \\ % Store \sigma \in \textit{Store} &\triangleq \textsf{Addr} \rightarrow \textsf{Val} \\ % Address a \in \textit{Addr} &\triangleq \mathbb{N} \\ % Value v \in \textit{Val} &\triangleq \textit{Clo} \\ % Closures clo \in \textit{Clo} &\triangleq \textsf{Lam} \times \textit{Env} \\ % Continuation \kappa \in \textit{Kont} &::= \textbf{mt} \\ & | \; \textbf{arg}(e, \rho, \kappa) \\ & | \; \textbf{fn}(v, \rho, \kappa) \\ \end{aligned}</script> <p>So there are a lot of pieces here! Lets go through them! Dont worry if they dont make perfect sense just yet, it will make more sense when they are used in context in the next section.</p> <p>First, the state definition itself. We have 4 components as previously defined. We use a ' <a href="https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Product_type" target="_blank">product type↪</a> ' to say 'One of each'. The domain is called Σ, or 'sigma' (uppercase). When we only have one state, (much as we would have one number in the domain of 'Integer'), we use ς, or 'varsigma' (i dont understand ancient greek enough to get why its called that). To form a state, you take one of each of the constituent parts. Our machine takes a state as input, and returns a state.</p> <p>Next, we need to define the environment and store. In computer science jargon, a map is simply a 'partial function' from the input to the output. For the environment, the function takes a <code>Var</code>, and returns its <code>Addr</code>. The same is true of the store, which takes the address, and returns the value that it points to. The store isn't particularly useful in this machine, but when we implement things like mutation, it will be good to have around.</p> <p>Third, the address and value sets. In this simple machine, addresses will be very simple, we will use numbers to define them (the set with the fancy <code>N</code> is integers starting at 0 and going up to infinity).</p> <p>Values in this machine are ' <a href="https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Closure_(computer_programming)" target="_blank">closures↪</a> '. Closures are a key part of higher order languages, such as Scheme, and are defined as a syntactic function (in our case the abstraction form), and an environment. In more advanced machines, values will be able to take on many different types, including numbers, strings, even continuations!</p> <p>Finally, our definition of continuations. If you look closely, its a kind of linked list, meaning that a continuation has another continuation attached to it, until you get to the <code>mt</code> continuation, the end. Other than <code>mt</code>, there are two important 'continuation frame' types. The <code>arg</code> continuation means that after we finish the current computation, we must evaluate the argument to the application. The <code>fn</code> frame tells us that after the current computation, we need to execute the function with the argument.</p> <h2 id="injection-and-allocation">Injection And Allocation</h2> <p>As I mentioned earlier, our machine takes a state and returns a state, so how do we know what the 'first' state is? To determine that, we create an 'injection function'. This takes the program text as input, and returns the initial machine state that will evaluate it. Ours is quite simple:</p> <script type="math/tex;mode=display">inj : \textsf{Exp} \rightarrow \Sigma \\ inj(e) \triangleq (e , \emptyset , \emptyset , \textbf{mt}) \\</script> <p>The top line is the type of the function <code>inj</code>, which states that it is a function that takes an expression, and returns a state. The next line is the definition, you will see that it takes an <code>e</code>, and returns a state, with the <code>e</code> as the control, empty sets for the environment and store, meaning that there are no mappings. the starting continuation is 'mt', meaning that there is nothing left to do after it is encountered.</p> <p>We also need a way to make an address. A good allocator gives us an unused address, so that we dont step on any other variables toes. Lets define a simple alloc here:</p> <script type="math/tex;mode=display">alloc : \textit{Store} \rightarrow \textit{Addr} \\ alloc(\sigma) \triangleq |\sigma| \\</script> <p>We define alloc as simply, 'the amount of items in the store'. If the store is empty, the address return will be <code>0</code>, if there are 12 items, the address will be <code>12</code>. As long as we never implement garbage collection, this is guaranteed to always give us a clean, unused address!</p> <p>The starting state is used to create the next state, and the state after that, until we reach the end of the program (or infinite loop! Thanks halting problem!). This is called a 'state machine'. If you took a few computer science courses, you may be familiar with 'finite state machines'. A programming language can be thought of as an 'infinite state machine', because we have <em>no idea</em> if it will end! Exciting! Mysterious!</p> <h1 id="transitions">Transitions</h1> <p>The machine we will be making is based off of a multitude of 'state transition functions', that say "if the state looks like this, do that". When you combine all of these functions, you get a working machine, or what a lay-prorammer may call an 'interpreter'!</p> <p>So, without further ado, lets write out some state transition functions! We begin with something simple, variable lookup:</p> <script type="math/tex;mode=display">(x , \rho , \sigma , \kappa) \leadsto (lam , \rho_\lambda , \sigma , \kappa) \\ \begin{aligned} \text{where } \sigma(\rho(x)) &= (lam , \rho_\lambda) \\ \end{aligned}</script> <p>We have finally defined our first semantic! If our control is some variable, then we need to look it up in the environment and store. If we find that they lead to a closure (a lambda and an environment), then we use them in the resulting state. I use the funky arrow cause I think it looks funny. I call it 'drunk arrow'. You can read it as 'leads to'.</p> <p>Note that I used a regular <code>=</code> and i said 'if we find' above. In the 'where' clause, I am not stating what the variable lookup will be, but making a query. Luckily, the only thing that our store can hold at the moment is a closure, so its likely that the query succeeds.</p> <p>But there is also the case that the variable does not exist. If the variable is not found, then this transition function does not go into effect. We call such a state a 'stuck state', there is no usable transition, so the state never changes. This can mean that we are 'done', and that there is no more computation left, or it could indicate some error. You could of course implement some semantic such as:</p> <script type="math/tex;mode=display">(x , \rho , \sigma , \kappa) \leadsto \text{ERROR-STATE} \\ \begin{aligned} \text{where } \rho(x) &= \text{NO-ENTRY-FOUND} \\ \end{aligned}</script> <p>But then we would have to add error types to the state definition, and a lot more transitions to describe errors. Thats no fun, so I will leave boring stuff like that to 'real' interpreter writers. Let's just assume all programs given to our machine are 'good' and dont get into any bad stuck states.</p> <p>So if our machine consists only of the one semantic, it will never be used, because our environment and store are never added to, so no variable will ever be found. Let's add some more semantics to alleviate that issue.</p> <script type="math/tex;mode=display">((e_f \; e_a) , \rho , \sigma , \kappa) \leadsto (e_f , \rho , \sigma , \kappa') \\ \begin{aligned} \text{where } \kappa' &\triangleq \textbf{arg}(e_a , \rho , \kappa) \\ \end{aligned}</script> <p>Now we can evaluate an 'application' form. This is when well-defined semantics are really important. In our machine, we evaluate the first expression before the second. We are giving an explicit order of evaluation here. Why is the order of evaluation important? In more complex languages with mutable state, you can have undefined behavior if the order of evaluation is undefined. I will leave further research as an exercise to the esteemed reader.</p> <p>We also change the continuation in the resultant state. The new continuation contains the second expression of the application, and the environment in which to evaluate the expression. It also contains the old continuation, so we can know what to do after we finish evaluating the current control.</p> <script type="math/tex;mode=display">(lam , \rho , \sigma , \textbf{arg}(e_a , \rho' , \kappa)) \leadsto (e_a , \rho' , \sigma , \textbf{fn}(lam , \rho , \kappa)) \\</script> <p>This input state is more complex than the prior. If we have a value for the control (in our case, the control is a lambda), and if the current continuation is an <code>arg</code> frame, then we transition as shown. Wht this transition 'means', is that we are finished evaluating the left side of the application, and are now going to evaluate the right side.</p> <p>Also pay attention to the environments, and what goes where. For the resulting state, we use the env found in the continuation, but in the resulting continuation, we use the environment given in the state. This is because the closure we are creating by pairing the lambda (control) requires the env it was found in. But when we evaluate the right side of the application (e_a ), we need to use the same environment that we originally used when evaluating the left side.</p> <p>Take a little bit to think on that. The fact that we need to use the same environment for both elements of the application, and why the env that gets paired with the lambda is the one that it is.</p> <p>After you understand that, lets finish this out with the final transition function for today.</p> <script type="math/tex;mode=display">(lam , \rho , \sigma , \textbf{fn}(lam' , \rho' , \kappa)) \leadsto (e , \rho'' , \sigma' , \kappa) \\ \begin{aligned} \text{where } lam' &= (λ \; (x) \; e) \\ a &\triangleq alloc(\sigma) \\ \rho'' &\triangleq \rho'[x \mapsto a] \\ \sigma' &\triangleq \sigma[a \mapsto (lam , \rho)] \end{aligned}</script> <p>At this point, we have a lambda as the control, and a <code>fn</code> continuation frame. It is officially time to apply the application! The first part of the continuation frame is the function, and the current control is the argument. So we place the functions body as the new control, and add its argument (paired with the environment it was found in to form a closure) to the env/store.</p> <p>This is the most complex function in this machine. Make sure to internalize that in this machine, the only value is the closure. So, when we see a lambda, we have reached a value (when combined with an env). There <a href="https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Church_encoding" target="_blank">are ways↪</a> to ues closures to represent common datatypes such as numbers, or lists. These ways are how lambda calculus continues to be used as the basis for minimalistic languages such as scheme. But it makes programs hard to read, so we add integers anyways!</p> <h1 id="the-completed-machine">The Completed Machine</h1> <p>And just like that, we have the 4 transition functions that define a standard 'call-by-value' lambda calculus. Lets put them together for posterity:</p> <script type="math/tex;mode=display">(x , \rho , \sigma , \kappa) \leadsto (v , \rho_\lambda , \sigma , \kappa) \\ \begin{aligned} \text{where } \sigma(\rho(x)) &= (v , \rho_\lambda) \\ \end{aligned}</script> <script type="math/tex;mode=display">((e_f \; e_a) , \rho , \sigma , \kappa) \leadsto (e_f , \rho , \sigma , \kappa') \\ \begin{aligned} \text{where } \kappa' &\triangleq \textbf{arg}(e_a , \rho , \kappa) \\ \end{aligned}</script> <script type="math/tex;mode=display">(lam , \rho , \sigma , \textbf{arg}(e_a , \rho' , \kappa)) \leadsto (e_a , \rho' , \sigma , \textbf{fn}(lam , \rho , \kappa)) \\</script> <script type="math/tex;mode=display">(lam , \rho , \sigma , \textbf{fn}(lam' , \rho' , \kappa)) \leadsto (e , \rho'' , \sigma' , \kappa) \\ \begin{aligned} \text{where } lam' &= (λ \; (x) \; e) \\ a &\triangleq alloc(\sigma) \\ \rho'' &\triangleq \rho'[x \mapsto a] \\ \sigma' &\triangleq \sigma[a \mapsto (lam , \rho)] \end{aligned}</script> <p>The machine works by combining each of these into one big function, much like the <a href="https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Piecewise" target="_blank">piece-wise function↪</a> you learned in math class. Lets give this function the name <code>step</code>.</p> <h1 id="an-example-run">An Example Run</h1> <p>To evaluate a function, we continuously execute our transition functions until we reach a stuck-state. That final state is the return value of our program.</p> <p>Lets take our example from before, and run it by hand.</p> <script type="math/tex;mode=display">e_0 = ((\lambda \; (q) \; (\lambda \; (b) \; q)) \; (\lambda \; (\text{blerp}) \; \text{WORD})) \\ \varsigma_0 = inj(e_0) = (e_0 , \emptyset , \emptyset , \textbf{mt}) \\ \varsigma_n = step(\varsigma_{n-1}) \\ \;\\ % \varsigma_1 = ((\lambda \; (q) \; (\lambda \; (b) \; q)) , \emptyset , \emptyset , \textbf{arg}((\lambda \; (\text{blerp}) \; \text{WORD}) , \emptyset , \textbf{mt})) \\ % \varsigma_2 = ((\lambda \; (\text{blerp}) \; \text{WORD}) , \emptyset , \emptyset , \textbf{fn}((\lambda \; (q) \; (\lambda \; (b) \; q)) , \emptyset , \textbf{mt})) \\ % \varsigma_3 = ((\lambda \; (b) \; q) , \{q : 0\} , \{0 : (\lambda \; (\text{blerp}) \; \text{WORD})\} , \textbf{mt}) \\</script> <p>ς₃ is the final state, if we run <code>step</code> again, no applicable input is found. Try to write the steps out more explicitly on some paper, and see for yourself that this is the final state.</p> <p>The key insight to understand that ς₃ is the final state is that we are at an 'atomic' value with an empty continuation. Atomic in this case means we can't break it down further, like we could a variable or and application form.</p> <h1 id="whats-next">Whats next</h1> <p>Well, we have a turing complete language, what's the point in doing anything else? Language features to make this machine usable?? Fine! Stay tuned and next time we will implement 'useful' things, like numbers, and <code>if</code>. I promise that it will be shorter than this post.</p> <p>See you next time!</p> 2021-01-05T12:15:00-05:00 2021-01-05T12:15:00-05:00 Unknown https://thedav.is/post/crafting-semantics-0/ <h1 id="introduction">Introduction</h1> <p>Hi, this is the first post on a series about creating semantics for a programming language. We will be using Scheme as a reference point, and slowly building more and more features for it.</p> <p>We will be using '<em>computer science notation</em>' (specifically an operational semantics) for every post. We won't be implementing these semantics in a 'real' programming language, but you could quite easily if you wanted to.</p> <p>If you want to get started in Programming Language Theory, and want to understand how to define a programming language formally, this is the series for you! If you want to understand abstract machines better, this is the series for you!</p> <h1 id="what-does-all-of-this-mean">What does all of this mean?</h1> <p>We will be building a programming language! But instead of writing a parser, interpreter, compiler, etc. for it, we will simply define it mathematically. When you have a rigorously defined semantics, it should be quite easy to make a 'real' interpreter out of it.</p> <p>For a mostly 'finished' product, I have semantics for many forms of Scheme <a href="https://github.com/sinistersnare/aams/blob/master/latex/formalism.pdf">here in a Git repository</a>. Also included in that repo are implementations in Racket and Rust. We may or may not deviate from it as I see fit, but that will be what the semantics will look like. If this series goes far enough, I may delve into the abstract semantics section of that document, and talk about static analysis!</p> <h1 id="semantics">Semantics</h1> <p>The semantics of a language are how its syntax is interpreted by a reader (either human or machine). Like many things in Programming Language Theory, we have stolen this term from the field of linguistics (thanks!). Computer scientists use semantics to study programming languages. You can learn more about them in my <a href="https://thedav.is/post/abstract-machines/">'Abstract Machines' post</a>.</p> <h1 id="prerequisites">Prerequisites</h1> <p>There will be no 'programming' in this series, only notation. I will try to explain all notation as it is first used.</p> <p>All I ask is that you are interested in the topic at hand!</p> <h1 id="let-s-begin">Let's begin!</h1> <p>We will start after that post, with a CESK machine based on the lambda calculus. See <a href="https://thedav.is/post/crafting-semantics-1/">The next post</a> for the exciting start to this series!</p> 2020-11-30T12:15:00-05:00 2020-11-30T12:15:00-05:00 Unknown https://thedav.is/post/continuations-as-return/ <p>This post can serve as an extra-light introduction to continuations. If you have not heard of, or are confused by continuations, this post may be for you!</p> <p>This will be the first post of a series on continuations. We start with a short introduction to the idea of first-class control. In the future, I may introduce topics like <code>dynamic-wind</code>, or maybe just more interesting uses of continuations, like ambiguous functions. I eventually want to get to the downsides of these undelimited continuations, and then introduce delimited continuations. If none of that made sense, awesome! Enjoy!</p> <h1 id="first-class">First Class</h1> <p>You may be familiar with the concept of first class functions. Where you can use functions as regular-old values, and treat them like anything else in a language.</p> <pre data-lang="python" style="background-color:#282a36;color:#f8f8f2;" class="language-python "><code class="language-python" data-lang="python"><span style="font-style:italic;color:#ff79c6;">def </span><span style="color:#50fa7b;">foo</span><span>(</span><span style="font-style:italic;color:#ffb86c;">a</span><span>): </span><span> </span><span style="color:#bd93f9;">... </span><span style="font-style:italic;color:#ff79c6;">def </span><span style="color:#50fa7b;">bar</span><span>(</span><span style="font-style:italic;color:#ffb86c;">b</span><span>): </span><span> </span><span style="color:#bd93f9;">... </span><span style="color:#bd93f9;">... </span><span style="color:#ff79c6;">if </span><span>x </span><span style="color:#ff79c6;">&lt; </span><span>y: </span><span> which </span><span style="color:#ff79c6;">= </span><span>foo </span><span style="color:#ff79c6;">else</span><span>: </span><span> which </span><span style="color:#ff79c6;">= </span><span>bar </span><span> </span><span>value </span><span style="color:#ff79c6;">= </span><span style="color:#50fa7b;">which</span><span>(x) </span></code></pre> <p>Here we decide which function to use based on some condition. We can also have functions take functions as arguments.</p> <pre data-lang="python" style="background-color:#282a36;color:#f8f8f2;" class="language-python "><code class="language-python" data-lang="python"><span style="font-style:italic;color:#ff79c6;">def </span><span style="color:#50fa7b;">add1</span><span>(</span><span style="font-style:italic;color:#ffb86c;">x</span><span>): </span><span> </span><span style="color:#ff79c6;">return </span><span>x </span><span style="color:#ff79c6;">+ </span><span style="color:#bd93f9;">1 </span><span> </span><span>better </span><span style="color:#ff79c6;">= </span><span style="color:#8be9fd;">map</span><span>(add1, [</span><span style="color:#bd93f9;">1</span><span>,</span><span style="color:#bd93f9;">2</span><span>,</span><span style="color:#bd93f9;">3</span><span>,</span><span style="color:#bd93f9;">4</span><span>]) </span><span style="color:#6272a4;"># better == [2,3,4,5] </span></code></pre> <p>Functions are just like any other value in many modern languages. They are at the same status as integers, or strings. This is really nice, and programmers have decided that this is overall a nice-to-have feature.</p> <p>So, lets go further! Let's make more things first-class.</p> <h1 id="return">Return</h1> <p>What does it mean to have a first class return? It means promoting an important control-flow primitive into something that programmers have power over. By having a first class return, we turn the idea of returning into a value that can be passed around, same as an integer, or a function.</p> <p>Lets pretend our language no longer has a return statement, and we can only return by using first-class return values. Lets have <code>return</code> be the final argument to any function.</p> <pre data-lang="python" style="background-color:#282a36;color:#f8f8f2;" class="language-python "><code class="language-python" data-lang="python"><span style="color:#6272a4;"># We call the return argument `exit` </span><span style="color:#6272a4;"># in the main function. </span><span style="font-style:italic;color:#ff79c6;">def </span><span style="color:#50fa7b;">main</span><span>(</span><span style="font-style:italic;color:#ffb86c;">exit</span><span>): </span><span> </span><span style="color:#50fa7b;">compute</span><span>(</span><span style="color:#bd93f9;">12</span><span>, exit) </span><span> </span><span style="font-style:italic;color:#ff79c6;">def </span><span style="color:#50fa7b;">compute</span><span>(</span><span style="font-style:italic;color:#ffb86c;">x</span><span>, </span><span style="background-color:#ff79c6;color:#f8f8f0;">return</span><span>): </span><span> y </span><span style="color:#ff79c6;">= </span><span>x </span><span style="color:#ff79c6;">/ </span><span style="color:#bd93f9;">3 </span><span> </span><span style="color:#ff79c6;">return</span><span>(y) </span></code></pre> <p>In the <code>main</code> function, we are given a value that we named <code>exit</code> instead of <code>return</code>. Of course, these are variables so we can call them whatever we want. This <code>exit</code> argument, when called, will take you to the exited state of the program, when given a return value. Much like how in C we can return an integer at the end of the <code>main</code> procedure, and the caller of the C program can use that value however it wants.</p> <p>The <code>compute</code> function is almost identical to one in a current programming language. The difference is that we have <code>return</code> as an argument now, a value. We call the <code>return</code> argument like a function, <em><strong>but they are different</strong></em>. these values do not go back eventually like a function would. When we call <code>y = foo(x)</code>, eventually <code>foo</code> will return and <code>y</code> will be filled with some value. But in these special <code>return</code> things, we alter the 'control' of the program to somewhere else.</p> <p>Lets call these values <code>continuations</code>. As in, where to 'continue' to by calling it.</p> <p>A great thing about having continuations is that they can be passed around like any value, so we can return from some deeply nested function easily if need be.</p> <p>However, right now, the only actual continuation value we have seen is the <code>exit</code> continuation, that is provided as a way to quit the program. We have been passing it around, but, how do we get any continuation? If I want to call <code>var = foo(42, ???)</code> How do I get a continuation that will return and then fill in <code>var</code>? Lets use an operator called <code>cc</code>, the 'current continuation':</p> <pre data-lang="python" style="background-color:#282a36;color:#f8f8f2;" class="language-python "><code class="language-python" data-lang="python"><span style="font-style:italic;color:#ff79c6;">def </span><span style="color:#50fa7b;">compute</span><span>(</span><span style="font-style:italic;color:#ffb86c;">x</span><span>, </span><span style="background-color:#ff79c6;color:#f8f8f0;">return</span><span>): </span><span> y </span><span style="color:#ff79c6;">= </span><span>x </span><span style="color:#ff79c6;">/ </span><span style="color:#bd93f9;">3 </span><span> </span><span style="color:#ff79c6;">return</span><span>(y) </span><span> </span><span style="font-style:italic;color:#ff79c6;">def </span><span style="color:#50fa7b;">main</span><span>(</span><span style="font-style:italic;color:#ffb86c;">exit</span><span>): </span><span> y </span><span style="color:#ff79c6;">= </span><span style="color:#50fa7b;">compute</span><span>(</span><span style="color:#bd93f9;">12</span><span>, cc) </span><span> </span><span style="color:#8be9fd;">print</span><span>(</span><span style="color:#f1fa8c;">&quot;Got: </span><span style="color:#bd93f9;">{y}</span><span style="color:#f1fa8c;">&quot;</span><span>, cc) </span><span> </span><span style="color:#50fa7b;">exit</span><span>(</span><span style="color:#bd93f9;">9001</span><span>) </span></code></pre> <p>This <code>cc</code> operator reckons what the current computation is, and creates a continuation for that. In our line <code>y = compute(12, cc)</code>, <code>cc</code> knows that after the function ends, it is going to place the returned value into the variable <code>y</code>. The <code>cc</code> in the call to <code>print</code> is used so that the program will simply continue running, as no variable is assigned anywhere. Then we use the <code>exit</code> continuation to end the program with a nice and big number.</p> <p>With the <code>cc</code> operator, we can now fulfill our goal of 'early return' semantics.</p> <pre data-lang="python" style="background-color:#282a36;color:#f8f8f2;" class="language-python "><code class="language-python" data-lang="python"><span style="color:#6272a4;"># In the world where this program exists, </span><span style="color:#6272a4;"># 73 is a _VERY BAD NUMBER_, dont trust it. </span><span style="color:#6272a4;"># We must make sure all input does not contain 73. </span><span> </span><span style="color:#6272a4;"># If an illegal number is found, we bail </span><span style="color:#6272a4;"># out to `bad_return` with an error message </span><span style="color:#6272a4;"># Otherwise, we return to `good_return` </span><span style="color:#6272a4;"># with the value. </span><span style="font-style:italic;color:#ff79c6;">def </span><span style="color:#50fa7b;">sanitize</span><span>(</span><span style="font-style:italic;color:#ffb86c;">x</span><span>, </span><span style="font-style:italic;color:#ffb86c;">bad_return</span><span>, </span><span style="font-style:italic;color:#ffb86c;">good_return</span><span>): </span><span> </span><span style="color:#ff79c6;">if </span><span>x </span><span style="color:#ff79c6;">- </span><span style="color:#bd93f9;">1 </span><span style="color:#ff79c6;">== </span><span style="color:#bd93f9;">72</span><span>: </span><span> </span><span style="color:#50fa7b;">bad_return</span><span>(</span><span style="color:#f1fa8c;">&quot;NO EVIL ALLOWED HERE&quot;</span><span>) </span><span> </span><span style="color:#ff79c6;">else</span><span>: </span><span> </span><span style="color:#50fa7b;">good_return</span><span>(x) </span><span> </span><span style="font-style:italic;color:#ff79c6;">def </span><span style="color:#50fa7b;">get_value_from_world</span><span>(</span><span style="background-color:#ff79c6;color:#f8f8f0;">return</span><span>): </span><span> value </span><span style="color:#ff79c6;">= </span><span style="color:#50fa7b;">get_maybe_dangerous_input</span><span>(cc) </span><span> good </span><span style="color:#ff79c6;">= </span><span style="color:#50fa7b;">sanitize</span><span>(value, </span><span style="background-color:#ff79c6;color:#f8f8f0;">return</span><span>, cc</span><span style="background-color:#ff79c6;color:#f8f8f0;">)</span><span> </span><span> </span><span style="color:#ff79c6;">return</span><span>(</span><span style="color:#f1fa8c;">&quot;Got good value </span><span style="color:#bd93f9;">{good}</span><span style="color:#f1fa8c;">!&quot;</span><span>) </span><span> </span><span style="font-style:italic;color:#ff79c6;">def </span><span style="color:#50fa7b;">main</span><span>(</span><span style="font-style:italic;color:#ffb86c;">exit</span><span>): </span><span> important_val </span><span style="color:#ff79c6;">= </span><span style="color:#50fa7b;">get_value_from_world</span><span>(cc) </span><span> </span><span style="color:#50fa7b;">prin</span><span>(important_val, cc) </span><span> </span><span style="color:#50fa7b;">exit</span><span>(</span><span style="color:#f1fa8c;">&quot;nothing bad going on here&quot;</span><span>) </span></code></pre> <p>Here, we are able to exit out of our computation if we find ourselves in a tricky situation. Normally, we may accomplish this through an exception-based system. You would have a convoluted try-catch system just to watch for a single error-case... good luck getting through code review! But now, we have the power to bail out without having an extra system!</p> <h1 id="conclusion">Conclusion</h1> <p>Continuations are really powerful, and still under active research! Having first-class control is really cool. They are basically a <code>goto</code>, but more computer-sciencey.</p> <p>Current research on continuations includes making them safer and more efficient. Perhaps in the future, <code>return</code> statements will simply be a syntactic sugar for an implicit argument given to functions. That way, you can take multiple return points as arguments, and then decide where you want to go next,</p> <p>If this blog post interested you, but didnt have enough info, stay tuned!</p> 2020-11-09T12:15:00-05:00 2020-11-09T12:15:00-05:00 Unknown https://thedav.is/post/abstract-machines/ <h1 id="introduction">Introduction</h1> <p>So im a PhD student now, so I need to write about cool computer science things! As part of my learnings, I have been writing a bunch of 'Abstract Machines'. I think of them as how computer scientists do programming languages research without computers. I mean, they have to use <em>something</em> to write their super complex papers.</p> <blockquote> <p>"Computer Science is no more about computers than astronomy is about telescopes"</p> <ul> <li>Edsger Dijkstra</li> </ul> </blockquote> <p>We love computers, but they are merely a tool of computing. The real study of computation can be done without them, and the theories of how programming languages function is not excluded from that. So how do computer scientists study interpreters without a computer? The theory of abstract machines is one of the more popular ways.</p> <h1 id="how-to-compute">How to Compute</h1> <p>What does it mean to compute something? Humans are pretty good at just looking at things and formulating an answer. For example, a person does not use a sorting algorithm when matching socks after their laundry is finished. But computers can't just intuit a solution. They are given precise instructions on what to do to accomplish something. So how can we model that, and use it to inform the science behind computation?</p> <p>'Abstract Machines' were created to model real computation strategies. These are functions that take program states, and return some value. Program states can be composed of many different things. The simplest abstract machines simply use the current point that the program is at. Others include a mapping of variables to values, so we can keep state around. We will describe such machines, and what kind of languages they can describe.</p> <p>These machines, in practice, are interpreters. They are called 'abstract' because the theory on them is not specific to any exact language. You can make an abstract machine for whatever you could want: Lisp, Java bytecode, RISC-V assembly language...</p> <h1 id="digression-operational-semantics">Digression: Operational Semantics</h1> <p>Operational Semantics are how we can write semantics of a language using pen and paper. Using a simple syntax, we can get the idea of 'if the syntax looks like this, it can be evaluated into this'.</p> <p>The idea of semantics is separate from execution of a program. Semantics describe what something 'means', based on how it looks. Here are two different kinds of 'operational semantics', big-step and small-step.</p> <h2 id="big-step">Big Step</h2> <p>Big step evaluations have the type <code>State -&gt; Value</code>, meaning that you give them a state, and it will tell you which value it exactly evaluates to. These are nice and simple generally, because they give you the values in a single step.</p> <p>To use a small example expression, <code>if cond then e_true else e_false</code> there are two rules that will be used.</p> <ol> <li>If cond evaluates to <code>true</code>, then the result is what <code>e_true</code> evaluates to.</li> <li>If cond evaluates to <code>false</code>, then the result is what <code>e_false</code> evaluates to.</li> </ol> <p>These rules assume that we have some way to fully evaluate whatever <code>cond</code> is. But whatever that way is is unimportant to the rule of evaluating <code>if</code> statements. This separation is very important for operational semantics. We can write small rules for specific parts of the language, all of which get composed together into a fully formalized machine.</p> <p>I like using big step when thinking exactly about what expressions <em>do</em>.</p> <h2 id="small-step">Small Step</h2> <p>Small step semantics have the type <code>State -&gt; State</code>. They will show you what to do step by step to evaluate a term. They work iteratively, explicitly, to show how a given state is computed.</p> <p>For example, if the term is <code>if cond then e_true else e_false</code>, There will be a few different rules.</p> <ol> <li>If cond is an atomic value, and that value is <code>true</code>, the resulting state is <code>e_true</code>.</li> <li>If cond is an atomic value, and that value is <code>false</code>, the resulting state is <code>e_false</code>.</li> <li>If cond is not an atomic value, then evaluate it to <code>cond'</code> , and return <code>if cond' then e_true else e_false</code>.</li> </ol> <p>An atomic value is a value that can not be broken up any more. This means a datatype, not a complex expression. In these simple machines, the only atomic datatype is a number. Don't tell computer scientists about quarks, they may go insane.</p> <p>By going from a state to a next state, small step mechanics more closely follow how our computers work. They dont evaluate to values directly, they just... keep going.</p> <p>This may confuse a new reader, if it keeps going, how do we know its done? Big step rules directly result in a value, full stop. How do we know a state in small step semantics is the one with the value? We use what is called a 'fixpoint', or more simply, we evaluate until there isnt a meaningful change in the state after running. If we evaluate a math expression enough times, it will decompose into a single number, upon which evaluation will lead to itself. That means there is no more work to be done, and evaluation stops.</p> <p>The same is true in a language like <code>C</code>, after the final instruction in <code>main</code>, if we try to evaluate any more, nothing will happen, theres nothing left to do. That is a fixpoint.</p> <p>Small step semantics are a bit more precise in my opinion, and they are much easier to translate to <em>real</em> interpreters. But it is very useful to understand both styles, they have different uses. Using small steps, we can also more closely trace how an expression is evaluated.</p> <h1 id="a-simple-abstract-machine-c">A Simple Abstract Machine - C</h1> <p>I dont mean the the C language, especially because I would not use simple to describe it! C in this case stands for 'control'. You may remember things like 'if' are called 'control flow operators'. Control is the currently running 'thing' in your program. The <code>if</code> operator, is a way to change the control based on a condition. This machine will be called the C machine because you only need control to represent the state of the entire program.</p> <p>C machines are not capable of much, only simple rewriting of expresions, because they dont have any information other than the program's control itself to go off of. One kind of language we can formalize with a C machine is that of mathematical expressions.</p> <p>Here is a big-step semantics of type <script type="math/tex">MathExp \Downarrow Number</script> (takes a math expression and returns a number). This means that the control we choose to use is a math expression. The result of evaluating a math expression is a number, of course, so thats the value.</p> <p><script type="math/tex">Addition:</script> <script type="math/tex;mode=display">\frac{e_1 \Downarrow n_1 \;,\; e_2 \Downarrow n_2 \;,\; n_1+n_2 = n } {e_1+e_2 \Downarrow n}</script> </p> <p>This rule shows how to add expressions to end up with a number. You can understand it by reading the half under the bar as 'this is what we start and end with' and the half above the bar as 'these must be true to use these semantics'. You read <script type="math/tex">\Downarrow</script> as 'evalutes to'.</p> <p>You can read this rule like so:</p> <ol> <li>If we have some control that looks like <script type="math/tex">e_1 + e_2</script> (this is from the bottom left, before the arrow)</li> <li>if <script type="math/tex">e_1</script> evaluates to some number <script type="math/tex">n_1</script> (the first expression above the bar),</li> <li>if <script type="math/tex">e_2</script> evaluates to some number <script type="math/tex">n_2</script> (the second expression above the bar),</li> <li>if <script type="math/tex">n_1</script> added to <script type="math/tex">n_2</script> is equal to some number <script type="math/tex">n</script> (the third expression above the bar).</li> <li>THEN we know that the expression in step 1 evaluates to <script type="math/tex">n</script> .</li> </ol> <p>This may seem a bit backwards, we implement adding by adding? Well, the key is that expressions are complex, they can be composed of other expressions or just values. These rules show how to do math on expressions by first evaluating them to values. Then once they are values, it is quite easy to do math operations on them.</p> <p>Note the distinction between the arrow <script type="math/tex">\Downarrow</script> and <script type="math/tex">=</script> here. <script type="math/tex">\Downarrow</script> is saying "left evaluates to right by virtue of applying this machine's rules." and <script type="math/tex">=</script> is saying "you can substitute left for right".</p> <p>Authors of semantics like these love to use different looking arrow symbols, they all mean the same thing. Usually in big step they use a cool down-facing arrow like <script type="math/tex">\Downarrow</script> . In small step they will use a more boring arrow like <script type="math/tex">\rightarrow</script> .</p> <h3 id="example">Example</h3> <p>Lets evaluate a simple math expression to show how you can use these rules to prove that expressions are evaluated correctly using a machines rules.</p> <script type="math/tex;mode=display">\frac{\frac{7 \Downarrow 7 \; 3 \Downarrow 3 \; 7 + 3 = 10}{7 + 3 \Downarrow 10} \scriptsize{\mathbf{Addition}} \;\;\; \frac{}{4 \Downarrow 4} \;\;\; 10 + 4 = 14} {7+3+4 \Downarrow 14} \scriptsize{\mathbf{Addition}}</script> <p>Here, at the first, bottom most level, <script type="math/tex">e_1 = 7 + 3</script> and <script type="math/tex">e_2 = 4</script> . Then we need to prove that <code>7 + 3 = 10</code>, and we do that with another application of the addition rule! We could have chosen <code>3 + 4</code> to be be <script type="math/tex">e_2</script> , but it doesnt matter for addition, and we leave issues like that to a parser. I noted each level with the rule that was used to evaluate it. It is generally showed like this, but I usually dont show them. I only have so much horizontal space on this webpage!</p> <p>To evaluate simple mathematical expressions, we only need a control for the state. C machines are only capable of evaluating simple programs. What if we want to add a simple programming construct like variables? For that, we need a place to store them. And so, the CE machine is born!</p> <h1 id="environments-with-the-ce-machine">Environments with the CE machine.</h1> <p>To evaluate variables, we need to be able to keep track of what value they hold at a given program point. in a C machine, if we are given <code>a + 2</code>, we have no way of know what <code>a</code> is, because its not a number, and we can only deal with syntax as we see it. But if we gave a machine both <code>a + 2</code>, the control, and a mapping <code>{a : 4}</code>, an environment, we can evaluate the expression to 6!</p> <p>So, a big-step CE machine doesnt just have <code>MathExp</code> (C) for state anymore, we need an <code>Env</code> (E) to accompany it! The function is now of type <script type="math/tex">(MathExp \times Env) \Downarrow Number</script> . This means that our machine will take 2 arguments, a math expression and an environment, and it will return a computed number.</p> <p><script type="math/tex">Var-Lookup:</script> <script type="math/tex;mode=display">\frac{\rho[var] = n} {(var, \rho) \Downarrow n}</script> </p> <p>This simply states that if the expression is a variable, not an artithmetic expression, the value it returns is what the environment says it is. We use the greek letter rho ('ρ') (not the letter 'p') to represent the environment. This is what is used in the literature, and its always best to follow along with norms to avoid confusion! For the history on why they used <script type="math/tex">\rho</script> for this task, you will have to ask someone smarter than me.</p> <p>We dont have variable binding yet, but you can imagine an expression like: <code>m*c*c</code> with an environment <code>{m : 12 c : 299792458}</code>, which will return some number for us. We have made a calculator with predefined constants! We can put pi in there, or tau if you are a lunatic...</p> <h1 id="real-looking-programs">Real Looking Programs</h1> <p>Wow, it seems like we can do a lot with just a control and an environment. We could further extend this to things like variable assignment (again, these are big-step semantics):</p> <p><script type="math/tex">\mathbf{Assignment:}</script> <script type="math/tex;mode=display">\frac{e_1 \Downarrow n_2 \;,\; \rho_2 = (\rho + \{var : n_2\}) \;,\; (e_2, \rho_2) \Downarrow n} {(var := e_1 \;\text{in}\; e_2 \;,\; \rho) \Downarrow n}</script> </p> <p>This tells us that when we assign a variable, it will be available in the next expression.</p> <p>It seems that these two components of a state are enough to be wildly dangerous with. Soon enough, your semantics will be modeling a real language, with conditions, and inequalities, oh my! But, there are two big problems that you may face.</p> <p>First, your big step semantics may limit you. Their stringent requirements that their sub-expressions be terminating can cause issues in a turing-complete environment. And a small-step semantics can provide more granularity of control. For that reason, we will be using small-step semantics from now on.</p> <p>Also, as your language gets more and more complicated, the constructs of control and environment may be overburdened with responsibility. What can we do to simplify our understanding of these machines?</p> <h1 id="what-next-kontinuations-cek">What next? Kontinuations! CEK</h1> <p>One way to alleviate the burden is to add a new construct to our machines: The continuation. A continuation is a way of showing 'future work' to be done. For example, when we implement our assignment operation, we can delegate the work of executing the second expression to the continuation:</p> <p><script type="math/tex">\footnotesize{\mathbf{Assign-Exp}}:</script> <script type="math/tex;mode=display">\footnotesize{ \frac{} {(v := e_1 \;\text{in}\; e_2 , \rho , \kappa) \rightarrow (e_1 , \rho , \text{assign}(v, e_2, \kappa))} }</script> </p> <p><script type="math/tex">\footnotesize{\mathbf{Assign-Kont}}:</script> <script type="math/tex;mode=display">\frac{e \text{ is atomic} , \rho_2 = (\rho + \{v : \text{e} \})} {(e , \rho , \text{assign}(v, e_2 , k_{next} )) \rightarrow (e_2 , \rho_2 , k_{next})}</script> </p> <p>There is quite a bit more to unpack here!</p> <p>First, we switched to a small step semantics. Instead of returning a fully evaluated value as before, we now return states. Big step semantics are nice for simple programs, but when things get more complex, we want finer-grained control, so we switch to small-step.</p> <p>Next, our state has a third argument. Our type signature is now <script type="math/tex">(C, E, K) \rightarrow (C, E, K)</script> . The third argument is a continuation, it is useful in small step semantics, so we can focus on evaluating expressions. When we are done, we know what to do next.</p> <p>Finaly, The above semantics show that when we see an assignment expression, we simply return the <script type="math/tex">e_1</script> expression as the control for the next state. The key is that we save a new 'continuation frame' along with that control. This signifies that after that expression is fully evaluated, we continue by assigning it to a variable, and then execute the expression <script type="math/tex">e_2</script> .</p> <p>Another notable thing is that, inside of that 'assign' continuation frame, we store the old continuation. This recursive format is necessary when nesting complex expressions. Let's use an example with nested assignment expressions, with the recursive nature of the continuations being our life-jacket.</p> <h2 id="example-1">Example</h2> <p>If we had some expression:</p> <p><code>x := 4 in (x + 1)</code>, we would start with the <code>assign-exp</code> rule, which simply gives us a new state, it doesnt directly evaluate anything to a base value. The new state says that we need to evaluate <code>4</code> with the same environment as before, but we change the continuation, to store the values in the we need to use after the <code>4</code> is fully evaluated (which it is, but we cant see that far ahead! One small step at a time!). After the <code>assign-exp</code> a state that looks like <code>(4 , env , assign(...))</code> is returned. We immediately see that the <code>assign-kont</code> rule can be applied, as the control is a number, and we have an <code>assign</code> continuation in the <code>K</code> position. So by applying that, a state <code>(x + 1, env2 , old_kont)</code> is given to us, and with <code>x</code> in the environment mapped to <code>4</code>.</p> <p>What is <code>old_kont</code> in this example? As you see from the <code>assign-exp</code> rule, one of the main things that a continuation does is store the previous one, like a linked-list. Continuations are the 'rest of the program', so after the current continuation is used, we do what the old continuation was.</p> <p>For example, an program that looks like:</p> <script type="math/tex;mode=display">x := (y := (z := 3 \;\text{in}\; z) \;\text{in}\; (y + y)) \;\text{in}\; x</script> <p>Here is what the states will look like over the course of 3 applications of <code>assign-exp</code>:</p> <pre style="background-color:#282a36;color:#f8f8f2;"><code><span>(x := (y := (z := 3 in z) in (y + y)) in x , {} , NONE) </span><span>-&gt; ((y := (z := 3 in z) in (y + y)) , {} , assign(x , x , NONE)) </span><span>-&gt; ((z := 3 in z) , {} , assign(y , y + y , assign(x , x , NONE))) </span><span>-&gt; (3 , {} , assign(z , z , assign(y , y + y , assign(x , x , NONE)))) </span></code></pre> <p>Then if we apply <code>assign-kont</code> a couple times, with omitted <code>addition</code> and <code>var-lookup</code> rules:</p> <pre style="background-color:#282a36;color:#f8f8f2;"><code><span>-&gt; (z , {z : 3} , assign(y , y + y , assign(x , x , NONE))) </span><span>-&gt; (3 , {z : 3} , assign(y , y + y , assign(x , x , NONE))) </span><span>-&gt; (y + y , {z : 3 , y : 3} , assign(x , x , NONE)) </span><span>... </span><span>-&gt; (6 , {z : 3 , y : 3} , assign(x , x , NONE)) </span><span>-&gt; (x , {z : 3 , y : 3 , x : 6} , NONE) </span><span>-&gt; (6 , {z : 3 , y : 3 , x : 6} , NONE) </span></code></pre> <h3 id="interlude-injection">Interlude: Injection</h3> <p>Note above that there is an 'initial' state that we use when we start evaluating an expression. We wrap the expression with an empty environment ('{}'), and an 'empty' contination ('NONE'). This is usually done with whats called an <em>injection function</em>:</p> <script type="math/tex;mode=display">inject(e) = (e \;,\; \{\} \;,\; \text{NONE})</script> <p>This gives us the initial state from an expression. The initial state can of course be changed to your needs. adding initial environment values or constants, for example.</p> <h3 id="back-to-the-example">Back To The Example</h3> <p>We started with a NONE continuation, meaning have nothing to do next (end of program!). We then slowly build up continuations. <em>Continuations are turned to for guidance once control is atomic</em>. Atomic here means that it cant be broken down any more (computer scientists dont have quarks yet). Once we evaluate a continuation, the cycle continues, evaluate the control until it is atomic again. Eventually (if your program terminates!), we will be faced with an atomic control, and a NONE continuation. This means that we have finished evaluation, as there is nothing left to do! That is known as a fixpoint, which is a term taken from math meaning, if we run it another step, nothing will change: <code>f(x)</code> evaluated to <code>x</code>.</p> <p>This is a good example of the differences between small and big step evalaution models. Small step is a scalpel, where big is a cudgel. They may be more effective than the other, depending on what youre doing.</p> <h1 id="other-continuation-types">Other Continuation Types</h1> <p>We have demonstrated two different continuation types, "NONE", and "ASSIGN". One signifies the end of computation, and one tells us how to add variables to the environment. What if we want to add a conditional to our language? It is very easy with our language!</p> <p>Lets compute this text:</p> <script type="math/tex;mode=display">\text{if}\; c \;\text{then}\; t \;\text{else}\; f</script> <p>This will evaluate <code>c</code>, and once a value is computed, it checks its value. If the value is <code>0</code>, it will take the <code>else</code> branch, otherwise it will take the <code>then</code> branch. If we had a more complex language with booleans, we could use those. Or really, we could do almost anything! Make your own language, make it weird!</p> <p><script type="math/tex">\text{if-exp}:</script> <script type="math/tex;mode=display">\frac{} {((\text{if}\; c \;\text{then}\; t \;\text{else}\; f) \;,\; \rho , \kappa) \rightarrow (c , \rho , \text{ifk}(t , f , \kappa))}</script> </p> <p><script type="math/tex">\text{if-kont-t}:</script> <script type="math/tex;mode=display">\frac{v \text{ is atomic} , v \;\text{!=}\; 0} {(v , \rho , \text{ifk}(t , f , \kappa_{next})) \rightarrow (t , \rho , \kappa_{next})}</script> </p> <p><script type="math/tex">\text{if-kont-f}:</script> <script type="math/tex;mode=display">\frac{v \text{ is atomic} , v = 0} {(v , \rho , \text{ifk}(t , f , \kappa_{next})) \rightarrow (f , \rho , \kappa_{next})}</script> </p> <p>Here, when it is time to check the continuation for guidance, we will next evaluate one of the branches from the <code>ifk</code> continuation 'frame' based on the controls value.</p> <p>The idea of small step + continuation can be very similar to big-step. this CEK machine is basically saying 'evaluate this to a value, then do something else'. Big-step says that but in a slightly more compact way, involving more inference rules on the top half of the big bar.</p> <h2 id="another-example">Another Example</h2> <p>Lets do another simple example of continuations, just to hammer them in!</p> <p>We will show that: <script type="math/tex;mode=display">\scriptsize{ \frac{} {(x := (\text{if} \; (1 + 1) \; \text{then} \;7\; \text{else} \;12) \; \text{in} \; x \;,\; \{\} \;,\; NONE) \rightarrow (7, \{\}, NONE)} }</script> </p> <pre style="background-color:#282a36;color:#f8f8f2;"><code><span>(x := (if (1 + 1) then 7 else 12) , {} , NONE) </span><span>-&gt; (if (1 + 1) then 7 else 12 , {} , assign(x, x, NONE)) </span><span>-&gt; ((1 + 1) , {} , ifk(7, 12, assign(x, x, NONE))) </span><span>-&gt; (2 , {} , ifk(7, 12, assign(x, x, NONE))) </span><span>-&gt; (7 , {} , assign(x, x, NONE)) </span><span>-&gt; (x , {x : 7} , NONE) </span><span>-&gt; (7 , {} , NONE) </span></code></pre> <h3 id="fun-insight">Fun Insight!</h3> <p>If you have been paying close attention to the continuations, they kind of act like a fine-grained <em>stack</em>. Traditional stacks are at the level of funtions. You call a function, and a stack-frame is pushed. After a <code>return</code> is issued (in our case, reaching an atomic value), that frame is popped. The idea with continuations here is very similar, but at the level of instructions (like conditionals, assignments, etc.), not just functions.</p> <p>I didnt want to get into the weeds of <em>real progrmaming</em> in this post, but if that helps you understand more, great! If not, dont fret! But think on the recursive nature of continuations, it can really give shed new light on how to model computation!</p> <p>In practice, recursive continnuations can be made efficient by using tail-call eliminations, but thats a subject for a whole 'nother post!</p> <h1 id="the-big-idea">The Big Idea</h1> <p>The abstract machines are the C, CE, and CEK <em>ideas</em> that we have talked about. They are the types of state used when implementing an abstract machine. The resulting machine, however, is a <em>concrete interpreter</em> for your programming language.</p> <p>So what is wih all this trouble? Creating a tiny interpreter is easy! Using abstract machines offers us a way to build new features incrementally, and concisely.</p> <p>These Abstract Machines provide a way to form an independent interpreter for your language, that doesnt necessarily depend on the runtime of your host language. Also, using operational semantics can help you to formalize how your language evalautes, and be sure that your implementation is correct!</p> <p>But more than all of that, this is how computer scientists can model real languages. To accomodate that, other components can be added to the state, such as a 'store', for modeling the heap. A CESK machine changes the environment to be a mapping from variables to 'addresses', and the store becomes a mapping from addresses to the values. This is helpful for modeling mutation, and has implications in computability when doing static program analysis.</p> <h1 id="the-future">The Future</h1> <p>Adding even more complex language features is a necessity for formalizing a real language. One key feature is of exception handling: try-catch and the like. A solution used for advanced control flow such as try-catch is to bring continuations to first-class status. Continuations are very powerful constructs, and can be used to implement exception handling schemes. I am writing another post on what continuations <em>are</em>, becuase they are very poorly known in the world outside of programming panguages research.</p> <p>Abstract Machines can themselves be abstracted, to form interpreters that are non-deterministic. These interpreters can be used to see what possible states a program can reach. Using this information, static analyses of programs can be conducted, meaning optimizations, warnings, etc.. The ideas of abstract AMs is still new, and a topic of current research.</p> <h1 id="conclusion">Conclusion</h1> <p>Abstract Machines are a way to formally define computation for terms. When mapped to 'the real world' that means an interpreter for your programming language. They are a wonderful way of explicitly showing how programs are executed, and computation is performed.</p> <h1 id="appendix">Appendix</h1> <script type="math/tex"></script> <p><script type="math/tex">\rho:</script> 'rho', is an environment, used to map variables to values.</p> <p><script type="math/tex">\kappa:</script> 'kappa', is a continuation, which represents the rest of the program. We consult this after the current control is fully evaluated.</p> <p><script type="math/tex">\Sigma:</script> 'Sigma' (uppercase). Although not used in this post, this character often is used to represent the <code>State</code> type of the machine.</p> <p><script type="math/tex">\varsigma:</script> 'sigma' (lowercase in 'final word position'). This is known as 'varsigma'. You may find this used as an instance of a state. A small step machine goes from some state to a next state, so we can codify that as</p> <script type="math/tex;mode=display">\varsigma_i \rightarrow \varsigma_{i+1}</script> <p><script type="math/tex">\sigma:</script> 'sigma' (lowercase). Although not used in this post, this character is used to represent the <code>store</code> component of a machine. If we would have made a CESK machine, then the S would be this.</p> 2019-12-05T08:10:44-05:00 2019-12-05T08:10:44-05:00 Unknown https://thedav.is/post/robson-traversal/ <h1 id="this-is-a-draft">THIS IS A DRAFT</h1> <p>Hiya, this is a draft! Probably never going to finish it at this rate.</p> <h2 id="todo">TODO</h2> <ul> <li>prune images that aren't needed.</li> <li>Add pictures of tree's being traversed!!!</li> <li>StackOverflow posts: <ul> <li>https://stackoverflow.com/questions/31323283/incomplete-traversal-using-link-inversion-of-binary-tree</li> <li>https://stackoverflow.com/questions/22288074/robson-tree-traversal-algorithm</li> </ul> </li> </ul> <h1 id="introduction">Introduction</h1> <p>In this post, we will discuss a novel tree traversal algorithm. The standard traversal method operates with a <em>linear</em> spatial cost. The Robson traversal counters with a cool <em><strong>constant</strong></em> space!</p> <p>The final code is on <a href="https:&#x2F;&#x2F;github.com&#x2F;sinistersnare&#x2F;robson" target="_blank">github↪</a> , but I encourage you to read this post! The table of contents may help for people that may want to skip a bit :)</p> <h1 id="table-of-contents">Table of Contents</h1> <p>TODO</p> <h1 id="the-famous-tree">The Famous Tree</h1> <p>The tree is an incredibly important data structure. A great learning tool for beginning computer scientists, starting to understand the science. Trees are also used throughout the real-world. File systems use <a href="https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;B-tree" target="_blank">B-trees↪</a> to <a href="https:&#x2F;&#x2F;github.com&#x2F;postgres&#x2F;postgres&#x2F;tree&#x2F;master&#x2F;src&#x2F;backend&#x2F;access&#x2F;nbtree" target="_blank">store their data↪</a> . Text editors use <a href="https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Rope_(data_structure)" target="_blank">ropes↪</a> to <a href="https:&#x2F;&#x2F;github.com&#x2F;google&#x2F;xi-editor&#x2F;tree&#x2F;master&#x2F;rust&#x2F;rope" target="_blank">organize their text↪</a> . Trees power the world, inside and outside of computers. We will be using a simple binary tree for this post.</p> <p>Here is a common binary tree definition. We will be using C for the whole of this post:</p> <pre data-lang="c" style="background-color:#282a36;color:#f8f8f2;" class="language-c "><code class="language-c" data-lang="c"><span style="color:#6272a4;">/* We will use this type definition later. */ </span><span style="font-style:italic;color:#8be9fd;">typedef void </span><span>(</span><span style="color:#ff79c6;">*</span><span>VisitFunc)(Tree</span><span style="color:#ff79c6;">*</span><span>); </span><span> </span><span style="font-style:italic;color:#8be9fd;">typedef struct</span><span> Tree </span><span style="color:#ffffff;">{ </span><span> </span><span style="font-style:italic;color:#8be9fd;">int</span><span> data; </span><span> </span><span style="font-style:italic;color:#8be9fd;">struct</span><span> Tree</span><span style="color:#ff79c6;">*</span><span> left; </span><span> </span><span style="font-style:italic;color:#8be9fd;">struct</span><span> Tree</span><span style="color:#ff79c6;">*</span><span> right; </span><span style="color:#ffffff;">} </span><span>Tree; </span></code></pre> <p>We can use the tree like so:</p> <pre data-lang="c" style="background-color:#282a36;color:#f8f8f2;" class="language-c "><code class="language-c" data-lang="c"><span style="font-style:italic;color:#8be9fd;">int </span><span style="color:#50fa7b;">main</span><span>(</span><span style="font-style:italic;color:#8be9fd;">void</span><span>) </span><span style="color:#ffffff;">{ </span><span> Tree root, root_left; </span><span> root_left</span><span style="color:#ff79c6;">.</span><span style="color:#ffffff;">data </span><span style="color:#ff79c6;">= </span><span style="color:#bd93f9;">0</span><span>; </span><span> root_left</span><span style="color:#ff79c6;">.</span><span style="color:#ffffff;">left </span><span style="color:#ff79c6;">= </span><span style="color:#bd93f9;">NULL</span><span>; </span><span> root_left</span><span style="color:#ff79c6;">.</span><span style="color:#ffffff;">right </span><span style="color:#ff79c6;">= </span><span style="color:#bd93f9;">NULL</span><span>; </span><span> root</span><span style="color:#ff79c6;">.</span><span style="color:#ffffff;">data </span><span style="color:#ff79c6;">= </span><span style="color:#bd93f9;">1</span><span>; </span><span> root</span><span style="color:#ff79c6;">.</span><span style="color:#ffffff;">left </span><span style="color:#ff79c6;">= &amp;</span><span>root_left; </span><span> root</span><span style="color:#ff79c6;">.</span><span style="color:#ffffff;">right </span><span style="color:#ff79c6;">= </span><span style="color:#bd93f9;">NULL</span><span>; </span><span style="color:#ffffff;">} </span></code></pre> <p>We will specifically be using <a href="https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Binary_search_tree" target="_blank">Binary Search Trees↪</a> , so that way in-order traversal works.</p> <h2 id="how-do-we-traverse-a-tree">How Do We Traverse a tree?</h2> <p>Traversing a tree means accessing each node's data in the whole tree. This is obviously a critically important function of a tree, we should be able to access all elements if we need to.</p> <p>Luckily, traversing a tree is super easy!</p> <pre data-lang="c" style="background-color:#282a36;color:#f8f8f2;" class="language-c "><code class="language-c" data-lang="c"><span style="font-style:italic;color:#8be9fd;">void </span><span style="color:#50fa7b;">traverse</span><span>(Tree</span><span style="color:#ff79c6;">* </span><span style="font-style:italic;color:#ffb86c;">cur</span><span>) </span><span style="color:#ffffff;">{ </span><span> </span><span style="color:#ff79c6;">if </span><span>(cur </span><span style="color:#ff79c6;">== </span><span style="color:#bd93f9;">NULL</span><span>) </span><span style="color:#ff79c6;">return</span><span>; </span><span> </span><span style="color:#50fa7b;">pre_visit</span><span>(cur); </span><span style="color:#6272a4;">/* pre-order traversal */ </span><span> </span><span style="color:#50fa7b;">traverse</span><span>(cur</span><span style="color:#ff79c6;">-&gt;</span><span>left); </span><span> </span><span style="color:#50fa7b;">in_visit</span><span>(cur); </span><span style="color:#6272a4;">/* in-order traversal */ </span><span> </span><span style="color:#50fa7b;">traverse</span><span>(cur</span><span style="color:#ff79c6;">-&gt;</span><span>right); </span><span> </span><span style="color:#50fa7b;">post_visit</span><span>(cur); </span><span style="color:#6272a4;">/* post-order traversal */ </span><span style="color:#ffffff;">} </span></code></pre> <p>This algorithm is universally taught in beginning CS courses at universities. It gets the job done, but I think we can do better.</p> <p>Tree traversals can be adapted for use in memory management. Many garbage collectors represent their memory pool using tree-like structures, and when they are marking live memory, they traverse that tree.</p> <p>Pre-order traversal always visits a parent before its child. This is very useful when you want to copy a tree, maintaining its order. In-order is mostly used for sorted trees, as it will visit nodes in order! A node will be visited in post-order only after its children have been visited. This means that post-order is best used for tasks such as deleting the tree, freeing its memory without dangling pointers left over. If we made <code>free</code> our pre-visit function, then we would never be able to traverse right children!</p> <h1 id="why-is-the-standard-method-bad">Why is the 'standard' method bad?</h1> <p>To get into how algorithms are rated against eachother, we need to talk about "Big-O notation", and that is a little bit out of scope for this already long blog-post, so... Take an algorithms course I guess? I'll give a quick paragraph.</p> <p>Big-O uses a polynomial to describe how the function performs with relation to input size. If we are talking about speed, an <code>O(n)</code> function will perform linearly with regards to input. An <code>O(1)</code> function will always perform the same no matter how big the input gets. You should try to avoid <code>O(n!)</code> functions, too... These functions are used for 'time' and 'space'. If you are interested in this stuff, please check out <a href="https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Introduction_to_Algorithms" target="_blank">CLRS↪</a> chapter 3 for more information, or ask me to write a post on it!</p> <p>The 'standard' traversal method is as efficient as possible, time-wise. We need to traverse all <code>n</code>-nodes of a tree, and the algorithm has <code>O(n)</code> run-time complexity. You cant get any better than that!</p> <p>A reader may think that you also cant get better space-wise. The algorithm is like 4 lines long, how can we get better than that!? Well, if you paid attention in (or had taken an) algorithms class, you would know that stack space counts! Even though this algorithm looks small and perfect, it uses up a whole stack-frame each time the algorithm recurses! If you turn this recursive code into an iterative, explicit-stack based, solution, you would see that you need some memory for each level, and in the worst-case, we need <code>O(n)</code> memory!</p> <p>You may be saying now, "O(n) is not that bad! It seems as good as it can get!" Well I say no! We can do better! I say we can do it in <em><strong>constant O(1) time</strong></em>!</p> <p>That means we can do it using the same amount of memory, no matter how big the tree gets!</p> <p>Let's do it!</p> <h2 id="how-can-we-do-better-than-that">How can we do better than that!?!?!</h2> <p>Here is the thing about trees: they can be quite wasteful!</p> <p>Let's look at a tree-structure:</p> <div class="img-tpl"> <img src="wasteful.png" alt="TODO-SORRY!"> </div> <p>You see, the leaf-nodes have 2 pointers that are just <em><strong>null</strong></em>! What a waste! Thats so much space we are wasting! For any given tree of <code>n</code> nodes, there will be <code>n + 1</code> null pointers.</p> <p>Here is the theory. We use that freely-available space for our traversal. That way, we use as little extra memory as possible!</p> <p>That is how we are going to get to our solution, but first we need to understand some preliminary concepts.</p> <h1 id="stackless-traversals">Stackless Traversals</h1> <p>As mentioned in the introductory sections, the standard tree traversal requries a stack to traverse the tree. This is becuase we need to 'pop' back up the tree once we reach a leaf. Using a stack is super easy to understand, and it makes for a clean algorithm.</p> <p>explain stack for a minute here or in the big-O section. Chef says, at least explain a 'pop'.</p> <p>In 1968, Donald Knuth asked the computer science community if there existed a method for computing tree traversals without a stack, while also leaving the tree unmodified. Lets discuss some methods for doing just that, ending in the Robson traversal.</p> <h2 id="the-link-inversion-model">The Link-Inversion Model</h2> <p>Link Inversion is a key ingredient to our final algorithm. Link-Inversion is a process where we use a marker-bit on each node to tell if we should continue to traverse up, or traverse rightward when going up a tree.</p> <p>This method is stackless, like in the threaded tree, and solves Knuth's question. It does not solve our dilemma though, we need O(1) worst-case space complexity! Lets talk about how and why it works, which will lead us to Robson.</p> <pre data-lang="c" style="background-color:#282a36;color:#f8f8f2;" class="language-c "><code class="language-c" data-lang="c"><span style="font-style:italic;color:#8be9fd;">typedef struct</span><span> Tree </span><span style="color:#ffffff;">{ </span><span> </span><span style="font-style:italic;color:#8be9fd;">int</span><span> data; </span><span> Tree</span><span style="color:#ff79c6;">*</span><span> left; </span><span> Tree</span><span style="color:#ff79c6;">*</span><span> right; </span><span> </span><span style="font-style:italic;color:#8be9fd;">bool</span><span> went_right; </span><span style="color:#ffffff;">} </span><span>Tree; </span><span> </span><span style="font-style:italic;color:#8be9fd;">void </span><span style="color:#50fa7b;">link_inversion</span><span>(Tree</span><span style="color:#ff79c6;">* </span><span style="font-style:italic;color:#ffb86c;">cur</span><span>, VisitFunc </span><span style="font-style:italic;color:#ffb86c;">pre_order</span><span>, </span><span> VisitFunc </span><span style="font-style:italic;color:#ffb86c;">in_order</span><span>, </span><span> VisitFunc </span><span style="font-style:italic;color:#ffb86c;">post_order</span><span>) </span><span style="color:#ffffff;">{ </span><span> Tree</span><span style="color:#ff79c6;">*</span><span> prev </span><span style="color:#ff79c6;">= </span><span style="color:#bd93f9;">NULL</span><span>; </span><span> Tree</span><span style="color:#ff79c6;">*</span><span> old_prev; </span><span> Tree</span><span style="color:#ff79c6;">*</span><span> old_prev_left; </span><span> Tree</span><span style="color:#ff79c6;">*</span><span> old_cur; </span><span> </span><span> </span><span style="color:#ff79c6;">if </span><span>(cur </span><span style="color:#ff79c6;">== </span><span style="color:#bd93f9;">NULL</span><span>) </span><span style="color:#ff79c6;">return</span><span>; </span><span> </span><span> </span><span style="color:#ff79c6;">do </span><span style="color:#ffffff;">{ </span><span> </span><span style="color:#6272a4;">/* 1) Descend leftward as much as possible. */ </span><span> </span><span style="color:#ff79c6;">while </span><span>(cur </span><span style="color:#ff79c6;">!= </span><span style="color:#bd93f9;">NULL</span><span>) </span><span style="color:#ffffff;">{ </span><span> </span><span style="color:#50fa7b;">pre_order</span><span>(cur); </span><span> cur</span><span style="color:#ff79c6;">-&gt;</span><span>went_right </span><span style="color:#ff79c6;">= </span><span style="color:#bd93f9;">false</span><span>; </span><span> old_cur </span><span style="color:#ff79c6;">=</span><span> cur; </span><span> cur </span><span style="color:#ff79c6;">=</span><span> old_cur</span><span style="color:#ff79c6;">-&gt;</span><span>left; </span><span> old_cur</span><span style="color:#ff79c6;">-&gt;</span><span>left </span><span style="color:#ff79c6;">=</span><span> prev; </span><span> prev </span><span style="color:#ff79c6;">=</span><span> old_cur; </span><span> </span><span style="color:#ffffff;">} </span><span> </span><span> </span><span style="color:#6272a4;">/* 2) ascend from right as much as we can. */ </span><span> </span><span style="color:#ff79c6;">while </span><span>(prev </span><span style="color:#ff79c6;">!= </span><span style="color:#bd93f9;">NULL </span><span style="color:#ff79c6;">&amp;&amp;</span><span> prev</span><span style="color:#ff79c6;">-&gt;</span><span>went_right) </span><span style="color:#ffffff;">{ </span><span> old_prev </span><span style="color:#ff79c6;">=</span><span> prev; </span><span> prev </span><span style="color:#ff79c6;">=</span><span> prev</span><span style="color:#ff79c6;">-&gt;</span><span>right; </span><span> old_prev</span><span style="color:#ff79c6;">-&gt;</span><span>right </span><span style="color:#ff79c6;">=</span><span> cur; </span><span> cur </span><span style="color:#ff79c6;">=</span><span> old_prev; </span><span> </span><span style="color:#50fa7b;">post_order</span><span>(cur); </span><span> </span><span style="color:#ffffff;">} </span><span> </span><span> </span><span style="color:#6272a4;">/* 3) </span><span style="color:#6272a4;"> If prev is null after coming back up from the right, </span><span style="color:#6272a4;"> it means that we have finished traversal, </span><span style="color:#6272a4;"> so head back to the while-condition and get outta here! </span><span style="color:#6272a4;"> Else, we will do an exchange here, </span><span style="color:#6272a4;"> swap to right child of parent. */ </span><span> </span><span style="color:#ff79c6;">if </span><span>(prev </span><span style="color:#ff79c6;">!= </span><span style="color:#bd93f9;">NULL</span><span>) </span><span style="color:#ffffff;">{ </span><span> </span><span style="color:#6272a4;">/* Switch from the left side of prev to the right </span><span style="color:#6272a4;"> Also, mark prev as went_right so we know to traverse </span><span style="color:#6272a4;"> upwards using right pointer. */ </span><span> </span><span style="color:#50fa7b;">in_order</span><span>(prev); </span><span> old_prev_left </span><span style="color:#ff79c6;">=</span><span> prev</span><span style="color:#ff79c6;">-&gt;</span><span>left; </span><span> prev</span><span style="color:#ff79c6;">-&gt;</span><span>went_right </span><span style="color:#ff79c6;">= </span><span style="color:#bd93f9;">true</span><span>; </span><span> prev</span><span style="color:#ff79c6;">-&gt;</span><span>left </span><span style="color:#ff79c6;">=</span><span> cur; </span><span> cur </span><span style="color:#ff79c6;">=</span><span> prev</span><span style="color:#ff79c6;">-&gt;</span><span>right; </span><span> prev</span><span style="color:#ff79c6;">-&gt;</span><span>right </span><span style="color:#ff79c6;">=</span><span> old_prev_left; </span><span> </span><span style="color:#ffffff;">} </span><span> </span><span style="color:#ffffff;">} </span><span style="color:#ff79c6;">while </span><span>(prev </span><span style="color:#ff79c6;">!= </span><span style="color:#bd93f9;">NULL</span><span>); </span><span style="color:#ffffff;">} </span></code></pre> <p>The core algorithm is in the name, we must invert the links. As we push down the tree, we invert the links so that way the child points to the parent, and we can walk up the tree the same way we walked down. As we go up, we need to un-invert the links so that way the tree is back as it started. The marker bit is used so that when we go back up, we know if we are ascending from the left or from the right.</p> <p>Lets talk about each of these steps.</p> <ol> <li> <p>We must start by traversing leftwards as much as possible. Keeping pointers to the current and previously visited nodes. As we traverse, we invert the links. That means that cur-&gt;left will be changed to point to the parent.</p> </li> <li> <p>Next may not make sense, so if it doesnt make perfect sense, come to it after step 3. Here, we are done with this subtree, so we want to get to the subtree's root. We do this by ascending from the right until we get to the root of the traversed part of the tree. This ensures the tree is in a state that step 3 can deal with.</p> </li> <li> <p>Thirdly, we do an exchange. Here, we are assuredly in a left child, thanks to step 2. We can safely traverse to the parents right child and forget completely about the previous subtree forever! This exchange marks a completion of the left subtree of the new root, now we must traverse the right subtree of the new root. Back to step 1!</p> </li> </ol> <p>Here, we are skipping some of the technicalities, like what is <code>went_right</code> for? It is so that we know when we get to the end of the subtree in step 2. Hopefully, I will elucidate why its necessary with pictures very soon. After I elucidate that, I will later explain why it's not necessary (hint: Robson!).</p> <h3 id="walk-through">Walk-through</h3> <p>The algorithm is weird, but I think it'll help by showing it!</p> <div class="img-tpl"> <img src="inversion01.png" alt="TODO-SORRY!"> </div> <p>Here, the tree has been traversed almost completely down the side. Each time we step downwards, we invert the links to point to the parent. The next step will be to venture into the NULL left-child from the current point.</p> <p>This image has not even finished the first run of step 1 from above yet, so lets continue.</p> <div class="img-tpl"> <img src="inversion02.png" alt="TODO-SORRY!"> </div> <p>Okay, we have successfully finished part 1 of the algorithm: 'go leftward a lot'! Now step 2 does not apply, <code>went_right</code> has been set for exactly 0 nodes at this point in execution. We swiftly move to step 3! Here, we perform an exchange. We have finished traversing the left child of the current subtree (the leaf), and now we must traverse the left. We do some swaps and end up here:</p> <div class="img-tpl"> <img src="inversion03.png" alt="TODO-SORRY!"> </div> <p>Okay, the exchange succeeded! We are back at step 1, because <code>prev != NULL</code>. Lets ignore the red circle for just a moment and execute step 1. Done! Did you see it? Nothing! Of course, <code>cur == NULL</code>, so step 1 is not run. Time for step 2! We must go up until we reach the root! We run this until <code>prev-&gt;went_right == false</code>, in this case one time. Running this step means we are prepared to forget about the current subtree. Step 3 will exchange once more, getting us to the parents right child:</p> <div class="img-tpl"> <img src="inversion04.png" alt="TODO-SORRY!"> </div> <p>If you are keen, you may have noticed that this is a strikingly similar image to the first one in the series. The only difference is the red marker in the <code>prev</code> node. If you apply the operations I have listed since the start of this subsection, you can fully traverse this tree. Making enough images to illustrate all of that is an exercise left to the reader.</p> <p>Link inversion is relatively complex when compared to the standard method, but definitely more fun!</p> <h3 id="warning">Warning</h3> <p>This algorithm is <em>dangerous</em>! If you attempt to modify the tree while it is being traversed, pointers will be a complete mess! Make sure this algorithm completes before altering the tree any more!</p> <h3 id="analysis">Analysis</h3> <ul> <li>Space-Complexity: <code>O(n)</code> <ul> <li>This is because each tree node needs a marker, so linear cost.</li> </ul> </li> <li>Time complexity: <code>O(n)</code></li> </ul> <p>We still have not yet improved on the algorithmic cost of the standard depth-first search. We have been doing quite well on solving Knuth's challenge, but thats only a minor goal! Lets get to the real thing now, the Robson traversal.</p> <h2 id="the-threaded-tree">The Threaded Tree</h2> <p>J. H. Morris presented the threaded tree in 1979, and it involves using those wasteful null nodes at the ends of the tree. However, you will see it does not solve all of our problems.</p> <p>Threaded trees have two bits, one for each direction, telling whether the pointer is a thread. If the left pointer is a thread (<code>cur-&gt;left_thread</code>), then <code>cur-&gt;left</code> points to the in-order predecessor. The same is true for <code>cur-&gt;right_thread</code> pointing to the in-order successor. Following threads is a constant operation to the predecessor/successor, which speeds up in-order traversals.</p> <pre data-lang="c" style="background-color:#282a36;color:#f8f8f2;" class="language-c "><code class="language-c" data-lang="c"><span style="font-style:italic;color:#8be9fd;">typedef struct</span><span> Tree </span><span style="color:#ffffff;">{ </span><span> </span><span style="font-style:italic;color:#8be9fd;">int</span><span> data; </span><span> </span><span style="font-style:italic;color:#8be9fd;">struct</span><span> Tree</span><span style="color:#ff79c6;">*</span><span> left; </span><span> </span><span style="font-style:italic;color:#8be9fd;">struct</span><span> Tree</span><span style="color:#ff79c6;">*</span><span> right; </span><span> </span><span style="font-style:italic;color:#8be9fd;">bool</span><span> left_thread; </span><span> </span><span style="font-style:italic;color:#8be9fd;">bool</span><span> right_thread; </span><span style="color:#ffffff;">} </span><span>Tree; </span><span> </span><span style="color:#6272a4;">/* Takes the root of the tree (we call it cur for readability in the function itself) */ </span><span style="font-style:italic;color:#8be9fd;">void </span><span style="color:#50fa7b;">threaded_traversal</span><span>(Tree</span><span style="color:#ff79c6;">* </span><span style="font-style:italic;color:#ffb86c;">cur</span><span>) </span><span style="color:#ffffff;">{ </span><span> </span><span style="color:#6272a4;">/* Go all the way down to the smallest number in the tree. */ </span><span> </span><span style="color:#ff79c6;">while </span><span>(</span><span style="color:#ff79c6;">!</span><span>cur</span><span style="color:#ff79c6;">-&gt;</span><span>is_thread) </span><span style="color:#ffffff;">{ </span><span> cur </span><span style="color:#ff79c6;">=</span><span> cur</span><span style="color:#ff79c6;">-&gt;</span><span>left; </span><span> </span><span style="color:#ffffff;">} </span><span> </span><span> </span><span style="color:#6272a4;">/* Now all we have to do is go rightwards until the end! */ </span><span> </span><span style="color:#ff79c6;">while </span><span>(cur </span><span style="color:#ff79c6;">!= </span><span style="color:#bd93f9;">NULL</span><span>) </span><span style="color:#ffffff;">{ </span><span> </span><span style="color:#50fa7b;">inorder_visit</span><span>(cur); </span><span> cur </span><span style="color:#ff79c6;">= </span><span style="color:#50fa7b;">tree_successor</span><span>(cur); </span><span> </span><span style="color:#ffffff;">} </span><span style="color:#ffffff;">} </span><span> </span><span style="color:#6272a4;">/* Returns a successor to any given node, `node`.*/ </span><span>Tree</span><span style="color:#ff79c6;">* </span><span style="color:#50fa7b;">tree_successor</span><span>(Tree</span><span style="color:#ff79c6;">* </span><span style="font-style:italic;color:#ffb86c;">node</span><span>) </span><span style="color:#ffffff;">{ </span><span> Tree</span><span style="color:#ff79c6;">*</span><span> cur; </span><span> </span><span> </span><span style="color:#6272a4;">/* fast path! */ </span><span> </span><span style="color:#ff79c6;">if </span><span>(node</span><span style="color:#ff79c6;">-&gt;</span><span>right_thread) </span><span style="color:#ff79c6;">return</span><span> node</span><span style="color:#ff79c6;">-&gt;</span><span>right; </span><span> </span><span> </span><span style="color:#6272a4;">/* else return leftmost child of right subtree! */ </span><span> cur </span><span style="color:#ff79c6;">=</span><span> node</span><span style="color:#ff79c6;">-&gt;</span><span>right; </span><span> </span><span style="color:#ff79c6;">while </span><span>(</span><span style="color:#ff79c6;">!</span><span>cur</span><span style="color:#ff79c6;">-&gt;</span><span>left_thread) </span><span style="color:#ffffff;">{ </span><span> cur </span><span style="color:#ff79c6;">=</span><span> cur</span><span style="color:#ff79c6;">-&gt;</span><span>left; </span><span> </span><span style="color:#ffffff;">} </span><span> </span><span style="color:#ff79c6;">return</span><span> cur; </span><span style="color:#ffffff;">} </span></code></pre> <p>This <code>Tree</code> struct includes a boolean field to tell whether the current tree node has an in-order thread. When we search for the in-order successor to the current node, and find that it is a thread, we take the right node to get the immediate successor! This is always a single operation, AKA O(1) time!</p> <p>You note, of course, that to support this method, we need to add a boolean field for each and every node in the tree! This means we are stuck with a linear space cost for this traversal.</p> <p>Threaded trees are super cool, and I would love for people to know them. Luckily, there is a great <a href="https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Threaded_binary_tree" target="_blank">Wikipedia page↪</a> on the subject. If there was not, I would definitely write more.</p> <p>I won't do a full walkthrough of the threaded traversal, but here is an image of a threaded tree:</p> <div class="img-tpl"> <img src="threaded01.png" alt="TODO-SORRY!"> </div> <p>To start, go to the leftmost node, which is the minimum of an inorder traversal. To find the successor, if the right pointer is a thread, follow it, and that is the successor. If it is not, take it, and then go left as much as possible, that is the next node in-order.</p> <h3 id="analysis-1">Analysis</h3> <ul> <li>Space-Complexity: <code>O(n)</code> <ul> <li>This is because each tree node needs 2 markers, so linear cost</li> </ul> </li> <li>Time complexity: <code>O(n)</code></li> <li>Time complexity for finding a single successor: <em>Amortized</em> <code>O(1)</code>! <ul> <li>Ask for that algorithmic analysis post if you want me to explain this!</li> </ul> </li> </ul> <p>So we have found a cool algorithm that makes use of those dumb null pointers at the fringes of the tree. It does not seem like we gain much, though, as it still comes at a linear spatial cost. If you want amortized constant successor finding, then this is a great algorithm for you!</p> <h2 id="the-robson-tree-traversal">The Robson Tree Traversal</h2> <p>Here we are! The Robson Tree Traversal is a method for traversing trees in linear time, and in <em><strong>constant space</strong></em>.</p> <p>Robson learns from the threaded and link-inversion models. By using the leaf-pointers as pseudo marker-bits, we create an awesome tree-traversal.</p> <p>The downside of the threaded-tree was that it is linear-space. Yet, it successfully used those null pointers that we lamented of their waste. The downside of the link-inversion model is that it is worst-case linear-space. We had to add an extra bit of memory to each node. Robson solves both problems by playing to the strength's of both!</p> <h3 id="the-algorithm">The Algorithm</h3> <p>Lets start by talking about the memory usage. We need a few bytes of memory for this to work, here is what we will be using those bytes for:</p> <ul> <li>a <code>cur</code> pointer. This will be the current node that we are algorithmizing.</li> <li>a <code>parent</code> pointer. Fairly self explanatory, we need to know the parent of the current node at any given time. This does not need to be part of the data structure, we will just update our single pointer as the algorithm runs.</li> <li>a pointer we call <code>available</code>. This will be used to point to a node that we can use in our 'leaf-stack', that I will describe later.</li> <li>A pointer called <code>top</code>. This will be the top of the aforementioned and still mysterious leaf-stack.</li> <li>a temporary pointer used for exchanging nodes similarly to link-inversion.</li> <li>a bool used to keep some state regarding how we have been traversing the tree.</li> </ul> <p>These variables are kept in the function, and other than them, the only memory needed is the tree itself! Unlike the other algorithms mentioned in this post, this tree is completely basic. There are no changes to the underlying data structure from what you learn in CS102. Now, lets move to the algorithm itself. The core algorithm is split into two parts: pushing down the tree, and finding the next subtree.</p> <p>First, we push down the tree while inverting the pointers, a la link-inversion. We go down the tree not until we hit the left-most node, like in link-inversion, but when we hit a fully fledged leaf. The inversions themselves are a bit different than in link-inversion. There are a few couple to consider:</p> <ol> <li>There is a left child. Here, we follow the the left child. The left pointer is set to the parent of the current node, and the right is unaltered. This follows link-inversion, nothing to change here.</li> <li>There is no left child, but there is a right child. Here we do the same as case 1, but we set the current right pointer to the parent. This is very important, we leave the left child's pointer alone as null. When we go back up the tree, we will see that the parent's left is null, and know that we are ascending via the right tree. This is important in preserving the tree's structure.</li> <li>There are no children (as in cur is a leaf). This is the start of the interesting stuff! Robson uses the leaf nodes as a sort of pseudo-stack. The leaf-stack tells us where to ascend from when we dont know if we are ascending from left or right. Here, we simply will track the leaf as 'available'. Wow! We are at a leaf! That means it's time to go up!</li> </ol> <p>Lets discuss these states with pictures! If someone wants to write an interactive javascript demo of this, be my guest!</p> <p>Going up the tree, we have 4 cases, some mundane, and some interesting!</p> <ol> <li>The parent's right pointer is null. Here, we must be ascending from the left. We un-invert the pointers, as in link-inversion. No funny business yet!</li> <li>The parent's left pointer is null. This is a mirror of the previous case, so ascend using the right pointers instead of the left.</li> <li>Now, the tree has both pointers, how do we know if we are ascending from the left or right? Here we are getting a bit interesting! We are at an 'exchange point', where we do not know if we should ascend from the left or from the right. Lets move out of this list and talk about the leaf stack, and its relation to exchange points.</li> </ol> <h4 id="the-leaf-stack">The Leaf Stack</h4> <p>The leaf stack is not a real stack, but a linked list, composed of tree leaves. The <code>right</code> pointer of each leaf/stack-element points to an 'exchange point'. Each left pointer points to the next element of the list/stack. We call it a stack because we only access the top element, and therefore it is treated as a stack. Also, its a cheeky 'im not touching you' to Knuth's challenge. Anyways, these exchange points are used to tell if we are ascending from the left or right. Lets get back into the cases!</p> <p>The <code>available</code> pointer's destiny is to be the new head of the leaf-stack. Before we add it to the stack though, we must find an exchange point. I'll leave the answer for not immediately adding it to the stack as an exercise to the reader. Once we find an exchange point we will set <code>available</code> to the head of the leaf-stack.</p> <p>Okay! Now lets get back to finding the next subtree to traverse!</p> <p>TODO: does this last sentence need more reasons/usage? I feel like people will get caught up on what an exchange point is, and lose grasp of the section.</p> <!-- TODO: make this `ol` a shortcode so markdown can render inside of it. --> <!-- Have to re-start the ol from before. --> <ol start="3" class="numbered"> <li> Back at the old #3, we were discussing what to do when the parent has both children. Let's discuss the more difficult and more fun case here. When `top` is null, or when `top->right` does not point to the parent, we know we are ascending from the left. This is what we refer to as the exchange point, where the leaf stack first comes into play! We will use `available` now and then go down the right subtree. We set `available->right` to the parent, `available->left` to `top`, and then `top` to `available`. Woohoo, its a linked list! After these shenanigans, there are more shenanigans! We can not fix the pointers as in link inversion, as `parent->left` is currently pointing to `parent`'s parent! What we will do is set `parent->right` to `cur`, and `cur` to `parent->right`! It looks funky when you draw it out, but it works! Now we are on an un-traversed section of the tree, so we can resume the go-down part of the algorithm! In the next case, we will make use of the leaf stack! </li> <li> Okay! In this final case, we are pushing up the tree, and find that `parent == top->right`! This tells us we are on ascending from the right, and have thus completed the left and right side's of this subtree! Congratulations! Now, we 'pop' off of the leaf-stack, and fix the pointers. Remember how the parents right pointer is pointing to the left child? Yeah, now we re-set that! We set `parent` (the root of the subtree), to `cur` and continue pushing up the stack using the rules I just laid out! </li> </ol> <p>The algorithm really isnt that hard, but when you look at the tree mid-algorithm, it can be really funky. Pointers just completely out of place, but still kind of beautiful in a way! Oh yeah, another reminder to not edit the tree while running this algorithm! The tree is <em><strong>not</strong></em> in a safe, workable, state during these traversals!</p> <h3 id="analysis-2">Analysis</h3> <p>Amazing.</p> <h2 id="conclusion">Conclusion</h2> <p>Ending Text!</p> <h3 id="p-s">P.S.</h3> <p>On x86 systems at least, the least-significant bit of a pointer will always be 0, so you could theoretically store anything there, as long as you swap it for 0 when you need to actually use the pointer. So maybe as another way to do this stuff, you can just use one of the pointers' low bits as the follow pointer in link inversion. However, that would be really ugly to code, and then I would have had less to write about :(</p> <a href="https:&#x2F;&#x2F;stackoverflow.com&#x2F;questions&#x2F;19991506&#x2F;how-portable-is-using-the-low-bit-of-a-pointer-as-a-flag" target="_blank">relevant discussion↪</a> <h3 id="when-not-to-use-robson">When not to use robson</h3> <p>Robson only works on binary trees i think(!)</p> <p>if you need to stop the algorithm at any time the tree will be in a terrible state, you need to run the algorithm completely before doing anything else with the tree.</p> 2019-01-02T10:10:44-05:00 2019-01-02T10:10:44-05:00 Unknown https://thedav.is/post/fancy-tree-traversals/ <h1 id="introduction">Introduction</h1> <p>In this post, we will discuss a couple of novel methods for traversing trees. The threaded tree offers amortized constant access to the successor of a tree-node. The link-inversion traversal offers a stackless traversal of binary trees.</p> <p>The final code is in <a href="https://github.com/sinistersnare/robson">this repository</a>. Please read on for some history, and some novel algorithms you may have never heard of!</p> <h1 id="the-famous-tree">The Famous Tree</h1> <p>The tree is an incredibly important data structure. A great learning tool for beginning computer scientists, starting to understand the science. Trees are also used throughout the real-world. File systems use <a href="https://en.wikipedia.org/wiki/B-tree">B-trees</a> to <a href="https://github.com/postgres/postgres/tree/master/src/backend/access/nbtree">store their data</a>. Text editors use <a href="https://en.wikipedia.org/wiki/Rope_(data_structure)">ropes</a> to <a href="https://github.com/google/xi-editor/tree/master/rust/rope">organize their text</a>. Trees power the world, inside and outside of computers. We will be using a simple binary tree for this post.</p> <p>Here is a common binary tree definition. We will be using C for the whole of this post:</p> <pre data-lang="c" style="background-color:#282a36;color:#f8f8f2;" class="language-c "><code class="language-c" data-lang="c"><span style="color:#6272a4;">/* We will use this type definition later. */ </span><span style="font-style:italic;color:#8be9fd;">typedef void </span><span>(</span><span style="color:#ff79c6;">*</span><span>VisitFunc)(Tree</span><span style="color:#ff79c6;">*</span><span>); </span><span> </span><span style="font-style:italic;color:#8be9fd;">typedef struct</span><span> Tree </span><span style="color:#ffffff;">{ </span><span> </span><span style="font-style:italic;color:#8be9fd;">int</span><span> data; </span><span> </span><span style="font-style:italic;color:#8be9fd;">struct</span><span> Tree</span><span style="color:#ff79c6;">*</span><span> left; </span><span> </span><span style="font-style:italic;color:#8be9fd;">struct</span><span> Tree</span><span style="color:#ff79c6;">*</span><span> right; </span><span style="color:#ffffff;">} </span><span>Tree; </span></code></pre> <p>We can create the tree like so:</p> <pre data-lang="c" style="background-color:#282a36;color:#f8f8f2;" class="language-c "><code class="language-c" data-lang="c"><span style="font-style:italic;color:#8be9fd;">int </span><span style="color:#50fa7b;">main</span><span>(</span><span style="font-style:italic;color:#8be9fd;">void</span><span>) </span><span style="color:#ffffff;">{ </span><span> Tree root, root_left; </span><span> root_left</span><span style="color:#ff79c6;">.</span><span style="color:#ffffff;">data </span><span style="color:#ff79c6;">= </span><span style="color:#bd93f9;">0</span><span>; </span><span> root_left</span><span style="color:#ff79c6;">.</span><span style="color:#ffffff;">left </span><span style="color:#ff79c6;">= </span><span style="color:#bd93f9;">NULL</span><span>; </span><span> root_left</span><span style="color:#ff79c6;">.</span><span style="color:#ffffff;">right </span><span style="color:#ff79c6;">= </span><span style="color:#bd93f9;">NULL</span><span>; </span><span> root</span><span style="color:#ff79c6;">.</span><span style="color:#ffffff;">data </span><span style="color:#ff79c6;">= </span><span style="color:#bd93f9;">1</span><span>; </span><span> root</span><span style="color:#ff79c6;">.</span><span style="color:#ffffff;">left </span><span style="color:#ff79c6;">= &amp;</span><span>root_left; </span><span> root</span><span style="color:#ff79c6;">.</span><span style="color:#ffffff;">right </span><span style="color:#ff79c6;">= </span><span style="color:#bd93f9;">NULL</span><span>; </span><span style="color:#ffffff;">} </span></code></pre> <h2 id="how-do-we-traverse-a-tree">How Do We Traverse a tree?</h2> <p>Traversing a tree is accessing each node's data in the whole tree. This is obviously a critically important function of a tree, we should be able to access all elements if we need to.</p> <p>Luckily, traversing a tree is super easy!</p> <pre data-lang="c" style="background-color:#282a36;color:#f8f8f2;" class="language-c "><code class="language-c" data-lang="c"><span style="font-style:italic;color:#8be9fd;">void </span><span style="color:#50fa7b;">traverse</span><span>(Tree</span><span style="color:#ff79c6;">* </span><span style="font-style:italic;color:#ffb86c;">cur</span><span>) </span><span style="color:#ffffff;">{ </span><span> </span><span style="color:#ff79c6;">if </span><span>(cur </span><span style="color:#ff79c6;">== </span><span style="color:#bd93f9;">NULL</span><span>) </span><span style="color:#ff79c6;">return</span><span>; </span><span> </span><span style="color:#50fa7b;">pre_visit</span><span>(cur); </span><span style="color:#6272a4;">/* pre-order traversal */ </span><span> </span><span style="color:#50fa7b;">traverse</span><span>(cur</span><span style="color:#ff79c6;">-&gt;</span><span>left); </span><span> </span><span style="color:#50fa7b;">in_visit</span><span>(cur); </span><span style="color:#6272a4;">/* in-order traversal */ </span><span> </span><span style="color:#50fa7b;">traverse</span><span>(cur</span><span style="color:#ff79c6;">-&gt;</span><span>right); </span><span> </span><span style="color:#50fa7b;">post_visit</span><span>(cur); </span><span style="color:#6272a4;">/* post-order traversal */ </span><span style="color:#ffffff;">} </span></code></pre> <p>Pre-order traversal always visits a parent before its child. This is very useful when you want to copy a tree, maintaining its order. In-order is mostly used for sorted trees, as it will visit nodes in order! A node will be visited in post-order only after its children have been visited. This means that post-order is best used for tasks such as deleting the tree, freeing its memory without dangling pointers left over. If we made <code>free</code> our pre-visit function, then we would never be able to traverse right children!</p> <p>This algorithm is universally taught in beginning CS courses at universities. It gets the job done, and yet here we are, trying to complicate things!</p> <h1 id="stackless-traversals">Stackless Traversals</h1> <p>When we use that standard traversal algorithm, we utilize the power of stacks! Stacks are a wonderful and simple datastructure. Imagine first: a <em>stack</em> of plates in your cupboard. You can not pick a plate from the center of that stack, you must go from the top. This Last-In-First-Out ordering is great for traversing trees too! Now, lets imagine the left-edge of the tree to be our plate-stack. As we traverse down, we push nodes onto the stack. When we reach the bottom of the tree, we need a way to get back up. The solution is to simply 'pop' off the node-stack, and then we are at the <code>second-to-last</code> node!</p> <p>Next, we go right. Add the right-child of the <code>second-to-last</code> node to the stack. We now have to traverse that sub-tree, the same way as before, all the way to the left, until we reach the bottom. Eventually, after going up and right, and down and left enough, we will have traversed the entire tree. And now we have it, a semi-rigorous explanation of stackful tree traversals!</p> <p>You can re-write that code in the previous section to use an explicit stack, if you wanted. It is still there, however, implicitly. When a function recurses, it uses a computer's internal stack to store information about the current function running. When we go down the tree, we add a 'stack-frame', which we use to traverse the tree.</p> <p>Stacks are amazing! Using a stack grants quite an intuitive model for beginner programmers to grok. For a long time, we only knew how to traverse trees using stacks. It was a sad world though. Punch-cards, no Wikipedia, and Algol... I shudder at the very thought, but I digress. Computer scientists felt this was a silly limitation, and sought to fix that, creating the world we see today through their hatred of stacks.</p> <p>In 1968, famous computer scientist Donald Knuth gave his community a problem. He wanted an algorithm for traversing trees without using a stack, which does not modify the tree in any way. I will present two algorithms that were not the first, nor the best methods for traversing trees. I like them, though, and feel like they provide some good ideas for computer scientists to learn from.</p> <h2 id="the-threaded-tree">The Threaded Tree</h2> <p>J.H. Morris presented the threaded tree in 1979. It utilizes the wasteful NULL nodes at the end of trees, for great profit. Using this algorithm will allow us to perform the <code>successor</code> operation of an in-order traversal in <em>amortized constant</em> time.</p> <p>A Threaded tree has two extra bits of information. One bit informs whether the left pointer is actually a <em>thread</em>, and the other for the right pointer. A thread is <em>not</em> an OS thread, think of it more like a pointer to a seemingly random part of the tree, and not a child. If we follow a thread, what we find is the in-order successor to a node. Here is the code for our Threaded Tree. I will describe the algorithm under the code-block.</p> <pre data-lang="c" style="background-color:#282a36;color:#f8f8f2;" class="language-c "><code class="language-c" data-lang="c"><span style="font-style:italic;color:#8be9fd;">typedef struct</span><span> Tree </span><span style="color:#ffffff;">{ </span><span> </span><span style="font-style:italic;color:#8be9fd;">int</span><span> data; </span><span> </span><span style="font-style:italic;color:#8be9fd;">struct</span><span> Tree</span><span style="color:#ff79c6;">*</span><span> left; </span><span> </span><span style="font-style:italic;color:#8be9fd;">struct</span><span> Tree</span><span style="color:#ff79c6;">*</span><span> right; </span><span> </span><span style="font-style:italic;color:#8be9fd;">bool</span><span> right_thread; </span><span style="color:#ffffff;">} </span><span>Tree; </span><span> </span><span style="color:#6272a4;">/* Takes the root of the tree (we call it cur for readability in the function itself) */ </span><span style="font-style:italic;color:#8be9fd;">void </span><span style="color:#50fa7b;">threaded_traversal</span><span>(Tree</span><span style="color:#ff79c6;">* </span><span style="font-style:italic;color:#ffb86c;">cur</span><span>) </span><span style="color:#ffffff;">{ </span><span> </span><span style="color:#6272a4;">/* Go all the way down to the smallest number in the tree. */ </span><span> </span><span style="color:#ff79c6;">while </span><span>(</span><span style="color:#ff79c6;">!</span><span>cur</span><span style="color:#ff79c6;">-&gt;</span><span>is_thread) </span><span style="color:#ffffff;">{ </span><span> cur </span><span style="color:#ff79c6;">=</span><span> cur</span><span style="color:#ff79c6;">-&gt;</span><span>left; </span><span> </span><span style="color:#ffffff;">} </span><span> </span><span> </span><span style="color:#6272a4;">/* Now all we have to do is go rightwards until the end! */ </span><span> </span><span style="color:#ff79c6;">while </span><span>(cur </span><span style="color:#ff79c6;">!= </span><span style="color:#bd93f9;">NULL</span><span>) </span><span style="color:#ffffff;">{ </span><span> </span><span style="color:#50fa7b;">inorder_visit</span><span>(cur); </span><span> cur </span><span style="color:#ff79c6;">= </span><span style="color:#50fa7b;">tree_successor</span><span>(cur); </span><span> </span><span style="color:#ffffff;">} </span><span style="color:#ffffff;">} </span><span> </span><span style="color:#6272a4;">/* Returns a successor to any given node, `node`.*/ </span><span>Tree</span><span style="color:#ff79c6;">* </span><span style="color:#50fa7b;">tree_successor</span><span>(Tree</span><span style="color:#ff79c6;">* </span><span style="font-style:italic;color:#ffb86c;">node</span><span>) </span><span style="color:#ffffff;">{ </span><span> Tree</span><span style="color:#ff79c6;">*</span><span> cur; </span><span> </span><span> </span><span style="color:#6272a4;">/* fast path! */ </span><span> </span><span style="color:#ff79c6;">if </span><span>(node</span><span style="color:#ff79c6;">-&gt;</span><span>right_thread) </span><span style="color:#ff79c6;">return</span><span> node</span><span style="color:#ff79c6;">-&gt;</span><span>right; </span><span> </span><span> </span><span style="color:#6272a4;">/* else return leftmost child of right subtree! */ </span><span> cur </span><span style="color:#ff79c6;">=</span><span> node</span><span style="color:#ff79c6;">-&gt;</span><span>right; </span><span> </span><span style="color:#ff79c6;">while </span><span>(</span><span style="color:#ff79c6;">!</span><span>cur</span><span style="color:#ff79c6;">-&gt;</span><span>left) </span><span style="color:#ffffff;">{ </span><span> cur </span><span style="color:#ff79c6;">=</span><span> cur</span><span style="color:#ff79c6;">-&gt;</span><span>left; </span><span> </span><span style="color:#ffffff;">} </span><span> </span><span style="color:#ff79c6;">return</span><span> cur; </span><span style="color:#ffffff;">} </span></code></pre> <p>This <code>Tree</code> struct includes a boolean field to tell whether the current tree node has an in-order thread. When we search for the in-order successor to the current node, and find that it is a thread, we take the right node to get the immediate successor! This is always a single operation. If we traverse an entire tree this way, we find that the <code>tree_successor</code> function is running in <em>amortized <code>O(1)</code></em> speed.</p> <p>However, to support this method, we must add a boolean field for each and every node in this tree. This means that the spatial cost for this algorithm is linear.</p> <p>Threaded trees are super cool, and I would love for people to know them. Luckily, there is a great <a href="https://en.wikipedia.org/wiki/Threaded_binary_tree">Wikipedia page</a> on the subject. If there was not, I would definitely write more.</p> <p>I won't do a full walkthrough of the threaded traversal, but here is an image of a threaded tree:</p> <div class="img-tpl"> <img src="threaded01.png" alt="TODO-SORRY!"> </div> <p>To start, go to the leftmost node, which is the minimum of an in-order traversal. To find the successor, if the right pointer is a thread, follow it, and that is the successor. If it is not, take it, and then go left as much as possible, that is the next node in-order.</p> <h3 id="analysis">Analysis</h3> <ul> <li>Space-Complexity: <code>O(n)</code> <ul> <li>This is because each tree node needs 2 markers, so linear cost</li> </ul> </li> <li>Time complexity: <code>O(n)</code></li> <li>Time complexity for finding a single successor: <em>Amortized <code>O(1)</code></em>!</li> </ul> <p>So we have found a cool algorithm that makes use of those dumb null pointers at the fringes of the tree. It does not seem like we gain much, though, as it still comes at a linear spatial cost. If you want amortized constant successor finding, then this is a great algorithm for you!</p> <p>For me, the biggest downside of this algorithm is that it only works for in-order traversals. If you want pre- or post-order traversals, this algorithm is not for you.</p> <h2 id="the-link-inversion-model">The Link-Inversion Model</h2> <p>Link Inversion is a key ingredient to our final algorithm. Link-Inversion is a process where we use a marker-bit on each node to tell if we should continue to traverse up, or traverse rightward when going up a tree.</p> <p>This method is stackless, like the threaded tree traversal. The trick is that we jumble the pointers, to thwart hacking attempts. Just kidding! We only seemingly jumble pointers! Also, it is not to thwart hackers, its to show us the way back up the tree!</p> <pre data-lang="c" style="background-color:#282a36;color:#f8f8f2;" class="language-c "><code class="language-c" data-lang="c"><span style="font-style:italic;color:#8be9fd;">typedef struct</span><span> Tree </span><span style="color:#ffffff;">{ </span><span> </span><span style="font-style:italic;color:#8be9fd;">int</span><span> data; </span><span> Tree</span><span style="color:#ff79c6;">*</span><span> left; </span><span> Tree</span><span style="color:#ff79c6;">*</span><span> right; </span><span> </span><span style="font-style:italic;color:#8be9fd;">bool</span><span> went_right; </span><span style="color:#ffffff;">} </span><span>Tree; </span><span> </span><span style="font-style:italic;color:#8be9fd;">void </span><span style="color:#50fa7b;">link_inversion</span><span>(Tree</span><span style="color:#ff79c6;">* </span><span style="font-style:italic;color:#ffb86c;">cur</span><span>, VisitFunc </span><span style="font-style:italic;color:#ffb86c;">pre_order</span><span>, </span><span> VisitFunc </span><span style="font-style:italic;color:#ffb86c;">in_order</span><span>, </span><span> VisitFunc </span><span style="font-style:italic;color:#ffb86c;">post_order</span><span>) </span><span style="color:#ffffff;">{ </span><span> Tree</span><span style="color:#ff79c6;">*</span><span> prev </span><span style="color:#ff79c6;">= </span><span style="color:#bd93f9;">NULL</span><span>; </span><span> Tree</span><span style="color:#ff79c6;">*</span><span> old_prev; </span><span> Tree</span><span style="color:#ff79c6;">*</span><span> old_prev_left; </span><span> Tree</span><span style="color:#ff79c6;">*</span><span> old_cur; </span><span> </span><span> </span><span style="color:#ff79c6;">if </span><span>(cur </span><span style="color:#ff79c6;">== </span><span style="color:#bd93f9;">NULL</span><span>) </span><span style="color:#ff79c6;">return</span><span>; </span><span> </span><span> </span><span style="color:#ff79c6;">do </span><span style="color:#ffffff;">{ </span><span> </span><span style="color:#6272a4;">/* 1) Descend leftward as much as possible. */ </span><span> </span><span style="color:#ff79c6;">while </span><span>(cur </span><span style="color:#ff79c6;">!= </span><span style="color:#bd93f9;">NULL</span><span>) </span><span style="color:#ffffff;">{ </span><span> </span><span style="color:#50fa7b;">pre_order</span><span>(cur); </span><span> cur</span><span style="color:#ff79c6;">-&gt;</span><span>went_right </span><span style="color:#ff79c6;">= </span><span style="color:#bd93f9;">false</span><span>; </span><span> old_cur </span><span style="color:#ff79c6;">=</span><span> cur; </span><span> cur </span><span style="color:#ff79c6;">=</span><span> old_cur</span><span style="color:#ff79c6;">-&gt;</span><span>left; </span><span> old_cur</span><span style="color:#ff79c6;">-&gt;</span><span>left </span><span style="color:#ff79c6;">=</span><span> prev; </span><span> prev </span><span style="color:#ff79c6;">=</span><span> old_cur; </span><span> </span><span style="color:#ffffff;">} </span><span> </span><span> </span><span style="color:#6272a4;">/* 2) ascend from right as much as we can. */ </span><span> </span><span style="color:#ff79c6;">while </span><span>(prev </span><span style="color:#ff79c6;">!= </span><span style="color:#bd93f9;">NULL </span><span style="color:#ff79c6;">&amp;&amp;</span><span> prev</span><span style="color:#ff79c6;">-&gt;</span><span>went_right) </span><span style="color:#ffffff;">{ </span><span> old_prev </span><span style="color:#ff79c6;">=</span><span> prev; </span><span> prev </span><span style="color:#ff79c6;">=</span><span> prev</span><span style="color:#ff79c6;">-&gt;</span><span>right; </span><span> old_prev</span><span style="color:#ff79c6;">-&gt;</span><span>right </span><span style="color:#ff79c6;">=</span><span> cur; </span><span> cur </span><span style="color:#ff79c6;">=</span><span> old_prev; </span><span> </span><span style="color:#50fa7b;">post_order</span><span>(cur); </span><span> </span><span style="color:#ffffff;">} </span><span> </span><span> </span><span style="color:#6272a4;">/* 3) </span><span style="color:#6272a4;"> If prev is null after coming back up from the right, </span><span style="color:#6272a4;"> it means that we have finished traversal, </span><span style="color:#6272a4;"> so head back to the while-condition and get outta here! </span><span style="color:#6272a4;"> Else, we will do an exchange here, </span><span style="color:#6272a4;"> swap to right child of parent. */ </span><span> </span><span style="color:#ff79c6;">if </span><span>(prev </span><span style="color:#ff79c6;">!= </span><span style="color:#bd93f9;">NULL</span><span>) </span><span style="color:#ffffff;">{ </span><span> </span><span style="color:#6272a4;">/* Switch from the left side of prev to the right </span><span style="color:#6272a4;"> Also, mark prev as went_right so we know to traverse </span><span style="color:#6272a4;"> upwards using right pointer. */ </span><span> </span><span style="color:#50fa7b;">in_order</span><span>(prev); </span><span> old_prev_left </span><span style="color:#ff79c6;">=</span><span> prev</span><span style="color:#ff79c6;">-&gt;</span><span>left; </span><span> prev</span><span style="color:#ff79c6;">-&gt;</span><span>went_right </span><span style="color:#ff79c6;">= </span><span style="color:#bd93f9;">true</span><span>; </span><span> prev</span><span style="color:#ff79c6;">-&gt;</span><span>left </span><span style="color:#ff79c6;">=</span><span> cur; </span><span> cur </span><span style="color:#ff79c6;">=</span><span> prev</span><span style="color:#ff79c6;">-&gt;</span><span>right; </span><span> prev</span><span style="color:#ff79c6;">-&gt;</span><span>right </span><span style="color:#ff79c6;">=</span><span> old_prev_left; </span><span> </span><span style="color:#ffffff;">} </span><span> </span><span style="color:#ffffff;">} </span><span style="color:#ff79c6;">while </span><span>(prev </span><span style="color:#ff79c6;">!= </span><span style="color:#bd93f9;">NULL</span><span>); </span><span style="color:#ffffff;">} </span></code></pre> <p>The core algorithm is in the name, we must invert the links. As we push down the tree, we invert the links so that way the child points to the parent, and we can walk up the tree the same way we walked down. As we go up, we need to un-invert the links so that way the tree is back as it started. The marker bit is used so that when we go back up, we know if we are ascending from the left or from the right.</p> <p>Let's talk about each of these steps.</p> <ol> <li> <p>We must start by traversing leftwards as much as possible. Keeping pointers to the current and previously visited nodes. As we traverse, we invert the links. That means that cur-&gt;left will be changed to point to the parent.</p> </li> <li> <p>Next may not make sense, so if it doesn't make perfect sense, come to it after step 3. Here, we are done with this subtree, so we want to get to the subtree's root. We do this by ascending from the right until we get to the root of the traversed part of the tree. This ensures the tree is in a state that step 3 can deal with.</p> </li> <li> <p>Thirdly, we do an exchange. Here, we are assuredly in a left child, thanks to step 2. We can safely traverse to the parents right child and forget completely about the previous subtree forever! This exchange marks a completion of the left subtree of the new root, now we must traverse the right subtree of the new root. Back to step 1!</p> </li> </ol> <p>Here, we are skipping some of the technicalities, like what is <code>went_right</code> for? It is so that we know when we get to the end of the subtree in step 2. Hopefully, I will elucidate why its necessary with pictures very soon. After I elucidate that, I will later explain why it's not necessary (hint: Robson!).</p> <h3 id="walk-through">Walk-through</h3> <p>The algorithm is weird, but I think it'll help by showing it!</p> <div class="img-tpl"> <img src="inversion01.png" alt="TODO-SORRY!"> </div> <p>Here, the tree has been traversed almost completely down the side. Each time we step downwards, we invert the links to point to the parent. The next step will be to venture into the NULL left-child from the current point.</p> <p>This image has not even finished the first run of step 1 from above yet, so let's continue.</p> <div class="img-tpl"> <img src="inversion02.png" alt="TODO-SORRY!"> </div> <p>Okay, we have successfully finished part 1 of the algorithm: 'go leftward a lot'! Now step 2 does not apply, <code>went_right</code> has been set for exactly 0 nodes at this point in execution. We swiftly move to step 3! Here, we perform an exchange. We have finished traversing the left child of the current subtree (the leaf), and now we must traverse the left. We do some swaps and end up here:</p> <div class="img-tpl"> <img src="inversion03.png" alt="TODO-SORRY!"> </div> <p>Okay, the exchange succeeded! We are back at step 1, because <code>prev != NULL</code>. Let's ignore the red circle for just a moment and execute step 1. Done! Did you see it? Nothing! Of course, <code>cur == NULL</code>, so step 1 is not run. Time for step 2! We must go up until we reach the root! We run this until <code>prev-&gt;went_right == false</code>, in this case one time. Running this step means we are prepared to forget about the current subtree. Step 3 will exchange once more, getting us to the parents right child:</p> <div class="img-tpl"> <img src="inversion04.png" alt="TODO-SORRY!"> </div> <p>If you are keen, you may have noticed that this is a strikingly similar image to the first one in the series. The only difference is the red marker in the <code>prev</code> node. If you apply the operations I have listed since the start of this subsection, you can fully traverse this tree. Making enough images to illustrate all of that is an exercise left to the reader.</p> <p>Link inversion is relatively complex when compared to the standard method, but definitely more fun!</p> <h3 id="warning">Warning</h3> <p>This algorithm is <em>dangerous</em>! If you attempt to modify the tree while it is being traversed, pointers will be a complete mess! Make sure this algorithm completes before altering the tree anymore!</p> <h3 id="analysis-1">Analysis</h3> <ul> <li>Space-Complexity: <code>O(n)</code> <ul> <li>This is because each tree node needs a marker, so linear cost.</li> </ul> </li> <li>Time complexity: <code>O(n)</code></li> </ul> <p>We still have not yet improved on the algorithmic cost of the standard depth-first search. We have been doing quite well on solving Knuth's challenge, but thats only a minor goal! Let's get to the real thing now, the Robson traversal.</p> <h2 id="conclusion">Conclusion</h2> <p>The threaded tree is a great structure for in-order traversal. Use it when you need to find the successor to a node lickity-split. The link-inversion method is quite strange, almost alien. However, it is used as the basis for a much cooler algorithm, The Robson Traversal. This post does not cover this traversal, but maybe a future one will.</p> <p>These algorithms are quite interesting to me. Finding novel ways to do simple tasks can lead to some interesting findings. Even if we re-invent the wheel sometimes, knowing how to make a wheel is important! I hope you find a cool use for these algorithms in the future. They have provided me with quite a bit of help, I would like to think.</p> <h3 id="p-s">P.S.</h3> <p>I have another blog post in the works, detailing an even cooler tree traversal. This Robson traversal is little known, but traverses trees in <em><strong>constant space</strong></em>. Please keep an eye out for this forthcoming blog post.</p> <p>If you liked this post, and want to see the new one even sooner, shoot me an email or a tweet for encouragement! I hope you learned something great today!</p>