Mặc scuba diving suite và pack stencils, chúng ta diving vào Templates hôm nay!

Phần lớn software render web pages hoặc generate emails dùng templating embed variable data vào text documents. Main structure của document thường setup trong static template với placeholders cho data. Variable data, như user names hoặc web page contents, replace placeholders khi rendering page.

Cho dive vào templating, chúng ta implement subset của Mustache, templating language available nhiều ngôn ngữ. Episode này, investigate different ways of templating. Bắt đầu string concatenation, end up writing own lexer cho more complex templates.

Sử Dụng Native String Interpolation

Bắt đầu minimal example. App cần welcome message include project name. Quickest way: dùng built-in string interpolation Ruby.

name = "Ruby Magic"
template = "Welcome to #{name}"
# => Welcome to Ruby Magic

Great! Doable. Tuy nhiên, nếu muốn reuse template multiple occasions, hoặc allow users update template?

Interpolation evaluates immediately. Không reuse template (trừ redefine trong loop) và không store template Welcome to #{name} trong database populate later mà không dùng eval dangerous.

May mắn Ruby có cách khác interpolate strings: Kernel#sprintf hoặc String#%. Methods cho interpolated string không change template. Reuse same template multiple times. Cũng không allow execution arbitrary Ruby code. Dùng nó.

name = "Ruby Magic"
template = "Welcome to %{name}"
sprintf(template, name: name)
# => "Welcome to Ruby Magic"
template % { name: name }
# => "Welcome to Ruby Magic"

Cách Tiếp Cận `Regexp` Cho Templating

Solution trên works, nhưng không fool-proof, expose more functionality than thường muốn. Xem example:

name = "Ruby Magic"
template = "Welcome to %d"
sprintf(template, name: name)
# => TypeError (can't convert Hash into Integer)

Cả Kernel#sprintf và String#% allow special syntax handle different data types. Không tất cả compatible với data we pass. Example: template expect format number nhưng nhận Hash, produce TypeError.

Có more power tools: implement own interpolation dùng regular expressions. Regexp cho phép define custom syntax, như Mustache/Handlebars inspired style.

name = "Ruby Magic"
template = "Welcome to {{name}}"
assigns = { "name" => name }
template.gsub(/{{(\w+)}}/) { assigns[$1] }
# => Welcome to Ruby Magic

Dùng String#gsub replace all placeholders (words trong double curly braces) với value trong hash assigns. Nếu không corresponding value, method remove placeholder không insert anything.

Replace placeholders trong string như này viable cho string với vài placeholders. Tuy nhiên, once things get complicated, quickly run into problems.

Giả sử cần conditionals trong template. Result khác dựa trên value của variable.

Welcome to {{name}}! {{#if subscribed}}
Thank you for subscribing to our mailing list.
{{else}}
Please sign up for our mailing list to be notified about new articles!
{{/if}}
Your friends at {{company_name}}

Regular expressions không smoothly handle use case này. Nếu try hard enough, probably still hack something together, but at this point, better build proper templating language.

Building a Templating Language

Implement templating language similar implementing other programming languages. Giống scripting language, template language cần ba components: lexer, parser, interpreter. Xem từng cái one by one.

Lexer

First task: tokenization, hoặc lexical analysis. Process rất similar identifying word categories trong natural languages.

Example: Ruby is a lovely language. Sentence consist of five words different categories. Identify category, bạn take dictionary lookup every word's category, result list: Noun, Verb, Article, Adjective, Noun. Natural language processing calls "Parts of Speech". Formal languages — như programming languages — called tokens.

Lexer works bằng đọc template và match stream of text với set of regular expressions cho each category in given order. First one match defines category của token và attach relevant data.

Với theory out of the way, implement lexer cho template language. Để easier, dùng StringScanner bằng require strscan từ Ruby standard library. (By the way, chúng ta có excellent intro StringScanner trong previous editions.) First step, build minimal version identifies everything as CONTENT.

Tạo new StringScanner instance, let nó do job dùng until loop chỉ stop khi scanner reaches end of string.

Hiện tại, let nó match every character (.*) across multiple lines (modifier m) và return one CONTENT token cho all of it. Represent token as array với token name first element và data second element. Basic lexer trông như sau:

require 'strscan'
module Magicbars
  class Lexer
    def self.tokenize(code)
      new.tokenize(code)
    end
    def tokenize(code)
      scanner = StringScanner.new(code)
      tokens = []
      until scanner.eos?
        tokens << [:CONTENT, scanner.scan(/.*?/m)]
      end
      tokens
    end
  end
end

Run code với Welcome to {{name}} get back list precisely one CONTENT token với all code attached.

Magicbars::Lexer.tokenize("Welcome to {{name}}")
=> [[:CONTENT, "Welcome to {{name}}"]]

Tiếp, detect expression. Modify code inside loop, match {{ và }} as OPEN_EXPRESSION và CLOSE.

Add conditional check different cases.

until scanner.eos?
  if scanner.scan(/\{\{/)
    tokens << [:OPEN_EXPRESSION]
  elsif scanner.scan(/\}\}/)
    tokens << [:CLOSE]
  elsif scanner.scan(/.*?/m)
    tokens << [:CONTENT, scanner.matched]
  end
end

Không added value attach curly braces vào tokens OPEN_EXPRESSION và CLOSE, drop chúng. Vì scan calls now part of condition, dùng scanner.matched attach result của last match vào CONTENT token.

Unfortunately, rerun lexer, vẫn chỉ one CONTENT token như trước. Vẫn phải modify last expression match everything up to open expression. Dùng scan_until với positive lookahead anchor cho double curly braces stop scanner right before them. Code inside loop now:

until scanner.eos?
  if scanner.scan(/\{\{/)
    tokens << [:OPEN_EXPRESSION]
  elsif scanner.scan(/\}\}/)
    tokens << [:CLOSE]
  elsif scanner.scan_until(/.*?(?=\{\{|\}\})/m)
    tokens << [:CONTENT, scanner.matched]
  end
end

Run lexer again, now four tokens:

Magicbars::Lexer.tokenize("Welcome to {{name}}")
=> [[:CONTENT, "Welcome to "], [:OPEN_EXPRESSION], [:CONTENT, "name"], [:CLOSE]]

Lexer pretty close desired result. Tuy nhiên, name không phải regular content; nó là identifier! Strings giữa double curly braces nên treat differently than strings outside.

Một State Machine

Để làm vậy, turn lexer into state machine với two distinct states. Start ở state default. Khi hit OPEN_EXPRESSION token, move to state expression và stay there until gặp CLOSE token transition back to default.



Implement state machine bằng add vài methods dùng array manage current state.

def stack
  @stack ||= []
end
def state
  stack.last || :default
end
def push_state(state)
  stack.push(state)
end
def pop_state
  stack.pop
end

state method return current state hoặc default. push_state move lexer vào new state bằng add to stack. pop_state move back previous state.

Tiếp, split conditional within loop và wrap by conditional check current state. While in default state, handle cả OPEN_EXPRESSION và CONTENT tokens. Điều này cũng nghĩa regex cho CONTENT không cần }} lookahead nữa, drop it. In expression state, handle CLOSE token và add new regex cho IDENTIFIER. Tất nhiên, implement state transitions bằng add push_state call to OPEN_EXPRESSION và pop_state call to CLOSE.

if state == :default
  if scanner.scan(/\{\{/)
    tokens << [:OPEN_EXPRESSION]
    push_state :expression
  elsif scanner.scan_until(/.*?(?=\{\{)/m)
    tokens << [:CONTENT, scanner.matched]
  end
elsif state == :expression
  if scanner.scan(/\}\}/)
    tokens << [:CLOSE]
    pop_state
  elsif scanner.scan(/[\w\-]+/)
    tokens << [:IDENTIFIER, scanner.matched]
  end
end

Với changes, lexer now properly tokenizes example.

Magicbars::Lexer.tokenize("Welcome to {{name}}")
# => [[:CONTENT, "Welcome to "], [:OPEN_EXPRESSION], [:IDENTIFIER, "name"], [:CLOSE]]

Làm Nó Khó Hơn Cho Chúng Ta
Move on advanced example. Sử dụng multiple expressions, cũng như block.

Welcome to {{name}}! {{#if subscribed}}
Thank you for subscribing to our mailing list.
{{else}}
Please sign up for our mailing list to be notified about new articles!
{{/if}}
Your friends at {{company_name}}

Không surprise lexer fail parse example. Để make it work, add missing tokens và make it handle content sau last expression. Code inside loop trông như sau:

if state == :default
  if scanner.scan(/\{\{#/)
    tokens << [:OPEN_BLOCK]
    push_state :expression
  elsif scanner.scan(/\{\{\//)
    tokens << [:OPEN_END_BLOCK]
    push_state :expression
  elsif scanner.scan(/\{\{else/)
    tokens << [:OPEN_INVERSE]
    push_state :expression
  elsif scanner.scan(/\{\{/)
    tokens << [:OPEN_EXPRESSION]
    push_state :expression
  elsif scanner.scan_until(/.*?(?=\{\{)/m)
    tokens << [:CONTENT, scanner.matched]
  else
    tokens << [:CONTENT, scanner.rest]
    scanner.terminate
  end
elsif state == :expression
  if scanner.scan(/\s+/)
    # Ignore whitespace
  elsif scanner.scan(/\}\}/)
    tokens << [:CLOSE]
    pop_state
  elsif scanner.scan(/[\w\-]+/)
    tokens << [:IDENTIFIER, scanner.matched]
  else
    scanner.terminate
  end
end

Lưu ý order of conditions important to some extent. First regex match assigned. Thus, more specific expressions phải come before more generic ones. Prime example: collection specialized open tokens cho blocks.

Dùng final version lexer, example now tokenizes into this:

[
  [:CONTENT, "Welcome to "],
  [:OPEN_EXPRESSION],
  [:IDENTIFIER, "name"],
  [:CLOSE],
  [:CONTENT, "!\n\n"],
  [:OPEN_BLOCK],
  [:IDENTIFIER, "if"],
  [:IDENTIFIER, "subscribed"],
  [:CLOSE],
  [:CONTENT, "\n Thank you for subscribing to our mailing list.\n"],
  [:OPEN_INVERSE],
  [:CLOSE],
  [:CONTENT, "\n Please sign up for our mailing list to be notified about new articles!\n"],
  [:OPEN_END_BLOCK],
  [:IDENTIFIER, "if"],
  [:CLOSE],
  [:CONTENT, "\n\nYour friends at "],
  [:OPEN_EXPRESSION],
  [:IDENTIFIER, "company_name"],
  [:CLOSE],
  [:CONTENT, "\n"]
]

Giờ finished, identified seven different token types:



Token Example


OPEN_BLOCK {{#
OPEN_END_BLOCK {{/
OPEN_INVERSE {{else
OPEN_EXPRESSION {{
CONTENT Anything outside of expressions (normal HTML or Text)
CLOSE }}
IDENTIFIER Identifiers consist of Word characters, numbers, _, and -



Next step: implement parser figure out structure of token stream và translate into abstract syntax tree, but that's for another time.

The Road Ahead
Started journey towards own templating language bằng xem different ways implement basic templating system dùng string interpolation. Khi hit limits first approaches, started implementing proper templating system.

For now, implemented lexer analyses template và figure out different token types. In upcoming edition Ruby Magic, continue journey implementing parser cũng như interpreter generate interpolated string.

Token	Example
`OPEN_BLOCK`	`{{#`
`OPEN_END_BLOCK`	`{{/`
`OPEN_INVERSE`	`{{else`
`OPEN_EXPRESSION`	`{{`
`CONTENT`	Anything outside of expressions (normal HTML or Text)
`CLOSE`	`}}`
`IDENTIFIER`	Identifiers consist of Word characters, numbers, `_`, and `-`

Xây dựng Lexer Template Ruby Mạnh mẽ cho Render Web Động

Sử Dụng Native String Interpolation

Cách Tiếp Cận Regexp Cho Templating

Building a Templating Language

Lexer

Một State Machine

Làm Nó Khó Hơn Cho Chúng Ta

The Road Ahead

Cách Tiếp Cận `Regexp` Cho Templating