Xây Dựng Trình Thông Dịch Ruby Cho Ngôn Ngữ Stoffle

Full Source on Github

Triển khai hoàn chỉnh ngôn ngữ lập trình Stoffle có sẵn tại GitHub. Hãy mở issue nếu tìm thấy bugs hoặc có questions.

Trong blog post này, chúng ta sẽ bắt đầu triển khai trình thông dịch cho Stoffle, một ngôn ngữ lập trình mô hình built entirely in Ruby. Bạn có thể đọc thêm về project này trong part đầu tiên của series.

Trình thông dịch chúng ta build thường gọi là tree-walk interpreter. Trong bài trước series, chúng ta built parser transform flat sequence of tokens thành tree data structure (abstract syntax tree, AST). Như bạn想象, interpreter job là go through AST created by parser và breathe life into Stoffle program. Tôi find step này most exciting của language implementation journey. Khi building interpreter, everything finally clicks, và chúng ta able to see Stoffle programs running for real!

Tôi sẽ show và explain interpreter's implementation in two parts. Phần đầu, get basics working: variables, conditionals, unary và binary operators, data types, printing to console. Chúng ta reserve meaty stuff (function definition, function calling, loops, etc.) cho second và last post implementing interpreter.

Quick Recap Lexer Và Parser

Trước khi dive in implement interpreter, quickly remind ourselves previous posts series. First, built lexer, transform raw source code into tokens. Then, implemented parser, component responsible morphing tokens into tree structure (AST). Sum up, transformations observed so far:

State 0: Source

my_var = 1

State 1: Lexer transforms raw source code into tokens

[:identifier, :'=', :number]

State 2: Parser transforms tokens into Abstract Syntax Tree

It's All About Walking

Bây giờ có AST, job là write code walk structure. Phải write Ruby code give life to what each node AST describes. Nếu node describes variable binding, task là write Ruby code able to somehow store result right-hand side variable binding expression và have storage space associated with (accessible through) name given to variable.

Giống previous parts series, explore implementation going through all important lines code involved handling example program. Stück Stoffle code interpret là:

num = -2
if num > 0
  println("The number is greater than zero.")
else
  println("The number is less than or equal to zero.")
end

AST produced cho same program:

The First Step In Our Walk

Như bạn nhớ từ bài trước series, Stoffle AST luôn có root AST::Program node. Root generally có multiple children. Some shallow (think AST produced simple variable assignment). Other children can be root của quite deep subtrees (think loop with many lines inside body). Ruby code cần start walking through AST passed to interpreter:

module Stoffle
  class Interpreter
    attr_reader :program, :output, :env
    def initialize
      @output = []
      @env = {}
    end
    def interpret(ast)
      @program = ast
      interpret_nodes(program.expressions)
    end
    private
    def interpret_nodes(nodes)
      last_value = nil
      nodes.each do |node|
        last_value = interpret_node(node)
      end
      last_value
    end
    def interpret_node(node)
      interpreter_method = "interpret_#{node.type}"
      send(interpreter_method, node)
    end
    #...
  end
end

Khi new Interpreter instantiated, right from get-go create two instance variables: @output và @env. Former's responsibility store, chronological sequence, everything program printed out. Having this information handy very useful writing automated tests hoặc debugging. Responsibility @env khác. Named as reference to "environment". As name suggests, function hold state of running program. One of its functions implement binding between identifier (e.g., variable name) và current value.

#interpret_nodes method loops through all children of root node (AST::Program). Then, calls #interpret_node cho each individual node.

#interpret_node simple but nonetheless interesting. Here, use bit Ruby metaprogramming call appropriate method handling node type currently at hand. Ví dụ, AST::VarBinding node, #interpret_var_binding method called.

Invariably, We Have To Talk About Variables

First node interpret in AST example program là AST::VarBinding. Its @left là AST::Identifier, @right là AST::UnaryOperator. Look at method responsible interpret variable binding:

def interpret_var_binding(var_binding)
  env[var_binding.var_name_as_str] = interpret_node(var_binding.right)
end

Như thấy, pretty straightforward. Add (or overwrite) key-value pair to @env Hash.

Key là name của variable (#var_name_as_str helper method equivalent var_binding.left.name). At the moment, all variables global. Handle scoping next post.

Value là result interpreting expression right-hand side assignment. Để do that, dùng #interpret_node again. Vì có AST::UnaryOperator right-hand side, next method called là #interpret_unary_operator:

def interpret_unary_operator(unary_op)
  case unary_op.operator
  when :'-'
    -(interpret_node(unary_op.operand))
  else # :'!'
    !(interpret_node(unary_op.operand))
  end
end

Semantics Stoffle's supported unary operators (- và !) same as Ruby. Consequence, implementation could not be simpler: apply Ruby's - operator to result interpreting operand. Usual suspect, #interpret_node, appears again here. Như nhớ từ AST program, operand cho - là AST::Number (number 2). Next stop #interpret_number:

def interpret_number(number)
  number.value
end

Implementation #interpret_number piece of cake. Decision adopt Ruby float as representation number literals (happens in lexer!) pays off here. @value của AST::Number node already holds desired internal representation numbers, so just retrieve it.

With that, finish interpreting first direct child of AST::Program. Now, conclude interpreting program, must handle other, more hairy, child: node type AST::Conditional.

Terms and Conditions May Apply

Back in #interpret_nodes, best friend #interpret_node called again interpret next direct child of AST::Program.

def interpret_nodes(nodes)
  last_value = nil
  nodes.each do |node|
    last_value = interpret_node(node)
  end
  last_value
end

Method responsible interpret AST::Conditional là #interpret_conditional. Trước khi look at it, refresh memories reviewing implementation AST::Conditional itself:

class Stoffle::AST::Conditional < Stoffle::AST::Expression
  attr_accessor :condition, :when_true, :when_false
  def initialize(cond_expr = nil, true_block = nil, false_block = nil)
    @condition = cond_expr
    @when_true = true_block
    @when_false = false_block
  end
  def ==(other)
    children == other&.children
  end
  def children
    [condition, when_true, when_false]
  end
end

Ok, @condition holds expression sẽ either truthy hoặc falsey; @when_true holds block with one or more expressions execute case @condition truthy, và @when_false (ELSE clause) holds block run case @condition falsey.

Look at #interpret_condition:

def interpret_conditional(conditional)
  evaluated_cond = interpret_node(conditional.condition)
  # We could implement line below in shorter way, but better to be explicit about truthiness in Stoffle.
  if evaluated_cond == nil || evaluated_cond == false
    return nil if conditional.when_false.nil?
    interpret_nodes(conditional.when_false.expressions)
  else
    interpret_nodes(conditional.when_true.expressions)
  end
end

Truthiness in Stoffle same as Ruby. In other words, in Stoffle, only nil và false falsey. Any other input to condition truthy.

First evaluate condition bằng interpret expression held by conditional.condition. Look at AST program again figure out node dealing with:

Turns out có AST::BinaryOperator (the > used in num > 0). Okay, same path again: first #interpret_node, calls #interpret_binary_operator this time:

def interpret_binary_operator(binary_op)
  case binary_op.operator
  when :and
    interpret_node(binary_op.left) && interpret_node(binary_op.right)
  when :or
    interpret_node(binary_op.left) || interpret_node(binary_op.right)
  else
    interpret_node(binary_op.left).send(binary_op.operator, interpret_node(binary_op.right))
  end
end

Logical operators (and và or) considered binary operators, handle here as well. Since semantic equivalent Ruby's && và ||, implementation plain sailing, as seen above.

Next section method most interested; section handles all other binary operators (including >). Here, leverage Ruby's dynamism favor và come up very concise solution. In Ruby, binary operators available as methods in objects participating operation:

-2 > 0 # equivalent to -2.send(:'>', 0) # this
# and following line would be general solution,
# very similar to what we have in interpreter
operand_1.send(binary_operator, operand_2)

A Verbose Implementation Of Binary Operators

Như bạn saw, implementation binary operators rất concise. Nếu Ruby không dynamic language, hoặc semantics operators different giữa Ruby và Stoffle, không thể code solution fashion này.

Nếu bạn ever find yourself in such position as language designer/implementer, always fall back on simple (but not that elegant) solution: using switch construct. In our case, implementation would look somewhat like this:

# ... inside #interpret_binary_operator ...
case binary_op.operator
when :'+'
  interpret_node(binary_op.left) + interpret_node(binary_op.right)
# ... other operators
end

Trước khi back to #interpret_conditional, quick detour make sure nothing overlooked. Nếu nhớ program interpreting, variable num used in comparison (using binary operator >) just explored together. How retrieve left operand (value stored in num variable) of comparison? Method responsible là #interpret_identifier, implementation easy-peasy:

def interpret_identifier(identifier)
  if env.has_key?(identifier.name)
    env[identifier.name]
  else
    # Undefined variable.
    raise Stoffle::Error::Runtime::UndefinedVariable.new(identifier.name)
  end
end

Back to #interpret_conditional. Case little program, condition evaluated to Ruby false value. Use this value determine whether execute IF hoặc ELSE branch conditional structure. Proceed interpret ELSE branch, associated block code stored in conditional.when_false. Here, có AST::Block, very similar root node AST (AST::Program). Block, likewise, potentially has bunch expressions need interpreted. For this purpose, also use #interpret_nodes.

def interpret_conditional(conditional)
  evaluated_cond = interpret_node(conditional.condition)
  # We could implement line below in shorter way, but better to be explicit about truthiness in Stoffle.
  if evaluated_cond == nil || evaluated_cond == false
    return nil if conditional.when_false.nil?
    interpret_nodes(conditional.when_false.expressions)
  else
    interpret_nodes(conditional.when_true.expressions)
  end
end

Next AST node handle là AST::FunctionCall. Method responsible interpret function call là #interpret_function_call:

def interpret_function_call(fn_call)
  return if println(fn_call)
end

Như discussed beginning article, function definition và function calling covered next post series. Therefore, only implementing special case function calling. In tiny toy language, provide println as part of runtime và implement directly in interpreter here. Good enough solution, considering objectives và scope project.

def println(fn_call)
  return false if fn_call.function_name_as_str != 'println'
  result = interpret_node(fn_call.args.first).to_s
  output << result
  puts result
  true
end

First và only argument AST::FunctionCall là AST::String, handled by #interpret_string:

def interpret_string(string)
  string.value
end

Trong #interpret_string, exact same case #interpret_number. AST::String already holds ready-to-use Ruby string value, so just retrieve it.

Back to #println:

def println(fn_call)
  return false if fn_call.function_name_as_str != 'println'
  result = interpret_node(fn_call.args.first).to_s
  output << result
  puts result
  true
end

Sau khi store function argument (converted to Ruby string) in result, two more steps complete. First, store what we about to print to console in @output. As explained previously, idea here able to easily inspect what was printed (and in what order). Having this at hand makes life easier debugging hoặc testing interpreter. Finally, implement printing something to console, dùng Ruby's puts.

Execution Matters

Bây giờ explored everything necessary implement bare-bones Stoffle, tạo very basic executable see interpreter in action.

#!/usr/bin/env ruby
require_relative '../lib/stoffle'
path = ARGV[0]
source = File.read(path)
lexer = Stoffle::Lexer.new(source)
parser = Stoffle::Parser.new(lexer.start_tokenization)
interpreter = Stoffle::Interpreter.new
interpreter.interpret(parser.parse)
exit(0)

TIP: Để dùng Stoffle's interpreter từ anywhere, remember add executable to your PATH.

Finally time run program. Nếu everything works well, should see string "The number is less than or equal to zero" printed to console. Exactly what happens when run interpreter:

TIP: Nếu bạn have interpreter installed, try changing num variable sample program so it holds number greater than zero. As expected, now IF branch will get executed, và string "The number is greater than zero" sẽ be printed out.

Wrapping Up

Trong post này, saw beginnings Stoffle's interpreter. Implemented enough interpreter handle some language basics: variables, conditionals, unary và binary operators, data types, printing to console. Next và final part interpreter, tackle remaining bits required have little toy language working as designed: variable scoping, function definition, function calling, và loops. Hope you had fun reading article (I certainly had fun writing it!), and we'll see you soon next post in series!