自制脚本语言(1)--词法分析器
目的:为了实现一个简单的解释器并在其基础上进行修改优化使其成为一个初级的编译器.
基于千叶滋老师的stone语言进行编写,使用Java作为写作的语言,最终完成版可以运行于JVM平台之上.
既然是自制语言,首先简单介绍一下语言处理器内部的流程
- 源代码输入,编译器对其进行词法分析;
- 词法分析的结果是对语句进行了分割,产生了一系列的Token,即简短的字符串排列;
- 接下来进行语法分析,生成抽象语法树,到这一步接下来就有两种走向;
- 若进行代码生成为其他语言程序,则为编译器;
- 若直接对抽象语法树进行运算,则为解释器;本文介绍的即是基于解释执行,但到目前还是看不出来的.
词法分析器对源代码进行了分割,得到如下代码中的Token对象:
Token.java
package stone;
public abstract class Token{
public static final Token EOF = new Token(-1) {}; //end of file
public static final String EOL = "\n"; //end of line
private int lineNumber;
protected Token(int line) {
lineNumber = line;
}
public int getLineNumber() {return lineNumber;}
public boolean isIdentifier() {return false;}
public boolean isNumber() {return false;}
public boolean isString() {return false;}
public int getNumber() {throw new StoneException("not number token"); }
public String getText() {return "";}
}
StoneException简单的继承了RuntimeException,相信有Java基础的童鞋都可以看懂.
StoneException.java
package stone;
import stone.ast.ASTree;
public class StoneException extends RuntimeException {
public StoneException(String m) { super(m); }
public StoneException(String m, ASTree t) {
super(m + " " + t.location());
}
}
词法分析器主体Lexer.java借助了Java语言中的regex(正则表达式类包),关于正则表达式可以参考有关内容,本文就不再赘述.
对了,正则表达式是很强大的存在,借助其可以完成很多有趣的事情,也可以称其为瑞士军刀了吧.
Lexer.java
package stone;
import java.io.IOException;
import java.io.LineNumberReader;
import java.io.Reader;
import java.util.ArrayList;
import java.util.regex.Matcher;
import java.util.regex.Pattern;
public class Lexer {
public static String regexPat
= "\s((//.)|([0-9]+)|("(\\"|\\\\|\\n|[^"])")"+ "|[A-Z_a-z][A-Z_a-z0-9]|==|<=|>=|&&|\|\||\p{Punct})?";
private Pattern pattern = Pattern.compile(regexPat);
private ArrayList<Token> queue = new ArrayList<Token>();
private boolean hasMore;
private LineNumberReader reader;
public Lexer(Reader r) {
hasMore = true;
reader = new LineNumberReader(r);
}
public Token read() throws ParseException {
if (fillQueue(0))
return queue.remove(0);
else
return Token.EOF;
}
public Token peek(int i) throws ParseException {
if (fillQueue(i))
return queue.get(i);
else
return Token.EOF;
}
private boolean fillQueue(int i) throws ParseException {
while (i >= queue.size())
if (hasMore)
readLine();
else
return false;
return true;
}
protected void readLine() throws ParseException {
String line;
try {
line = reader.readLine();
} catch (IOException e) {
throw new ParseException(e);
}
if (line == null) {
hasMore = false;
return;
}
int lineNo = reader.getLineNumber();
Matcher matcher = pattern.matcher(line);
matcher.useTransparentBounds(true).useAnchoringBounds(false);
int pos = 0;
int endPos = line.length();
while (pos < endPos) {
matcher.region(pos, endPos);
if (matcher.lookingAt()) {
addToken(lineNo, matcher);
pos = matcher.end();
}
else
throw new ParseException("bad token at line " + lineNo);
}
queue.add(new IdToken(lineNo, Token.EOL));
}
protected void addToken(int lineNo, Matcher matcher) {
String m = matcher.group(1);
if (m != null) // if not a space
if (matcher.group(2) == null) { // if not a comment
Token token;
if (matcher.group(3) != null)
token = new NumToken(lineNo, Integer.parseInt(m));
else if (matcher.group(4) != null)
token = new StrToken(lineNo, toStringLiteral(m));
else
token = new IdToken(lineNo, m);
queue.add(token);
}
}
protected String toStringLiteral(String s) {
StringBuilder sb = new StringBuilder();
int len = s.length() - 1;
for (int i = 1; i < len; i++) {
char c = s.charAt(i);
if (c == '\\' && i + 1 < len) {
int c2 = s.charAt(i + 1);
if (c2 == '"' || c2 == '\\')
c = s.charAt(++i);
else if (c2 == 'n') {
++i;
c = '\n';
}
}
sb.append(c);
}
return sb.toString();
}
protected static class NumToken extends Token {
private int value;
protected NumToken(int line, int v) {
super(line);
value = v;
}
public boolean isNumber() { return true; }
public String getText() { return Integer.toString(value); }
public int getNumber() { return value; }
}
protected static class IdToken extends Token {
private String text;
protected IdToken(int line, String id) {
super(line);
text = id;
}
public boolean isIdentifier() { return true; }
public String getText() { return text; }
}
protected static class StrToken extends Token {
private String literal;
StrToken(int line, String str) {
super(line);
literal = str;
}
public boolean isString() { return true; }
public String getText() { return literal; }
}
}
异常类ParseException.java的实现,继承Exception
package stone;
import java.io.IOException;
public class ParseException extends Exception {
public ParseException(Token t) {
this("", t);
}
public ParseException(String msg, Token t) {
super("syntax error around" + location(t) + "." + msg);
}
private static String location(Token t) {
if(t == Token.EOF)
return "the last line";
else {
return """ + t.getText() + "" at line" + t.getLineNumber();
}
}
public ParseException(IOException e) {
super(e);
}
public ParseException(String msg) {
super(msg);
}
}
到这里,词法分析器基本完成,接下来进行测试,测试的方法是借助CodeDialog将输入的代码进行分割,并将分割后的Token以">+Token"的格式在console中打印出
LexerRunner.java
package stone;
import stone.*;
public class LexerRunner {
public static void main(String[] args) throws ParseException {
Lexer l = new Lexer(new CodeDialog());
for( Token t; (t = l.read()) != Token.EOF;)
System.out.println("=> " + t.getText());
}
}
CodeDialog.java作用如上所述
package stone;
import java.io.FileReader;
import java.io.BufferedReader;
import java.io.FileNotFoundException;
import java.io.IOException;
import java.io.Reader;
import javax.swing.JFileChooser;
import javax.swing.JOptionPane;
import javax.swing.JScrollPane;
import javax.swing.JTextArea;
public class CodeDialog extends Reader {
private String buffer = null;
private int pos = 0;
public int read(char[] cbuf, int off, int len) throws IOException {
if (buffer == null) {
String in = showDialog();
if (in == null)
return -1;
else {
print(in);
buffer = in + "\n";
pos = 0;
}
}
int size = 0;
int length = buffer.length();
while (pos < length && size < len)
cbuf[off + size++] = buffer.charAt(pos++);
if (pos == length)
buffer = null;
return size;
}
protected void print(String s) { System.out.println(s); }
public void close() throws IOException {}
protected String showDialog() {
JTextArea area = new JTextArea(20, 40);
JScrollPane pane = new JScrollPane(area);
int result = JOptionPane.showOptionDialog(null, pane, "Input",
JOptionPane.OK_CANCEL_OPTION,
JOptionPane.PLAIN_MESSAGE,
null, null, null);
if (result == JOptionPane.OK_OPTION)
return area.getText();
else
return null;
}
public static Reader file() throws FileNotFoundException {
JFileChooser chooser = new JFileChooser();
if (chooser.showOpenDialog(null) == JFileChooser.APPROVE_OPTION)
return new BufferedReader(new FileReader(chooser.getSelectedFile()));
else
throw new FileNotFoundException("no file specified");
}
}
