Shell Interpreter

JNode's line-oriented command interpreters: tokenizing, &&/||/; command sequences, pipelines, redirection, and the shared escapeWord round-trip.

Overview

JNode's shell has three independent interpreters. CommandShell picks one at startup and exposes it through the CommandInterpreter extension point:

DefaultInterpreter — single commands only. No pipelines, no redirection.
RedirectingInterpreter extends DefaultInterpreter — adds pipelines (|) and file redirection (<, >). This is the shell's initial interpreter.
BjorneInterpreter — a Bourne-compatible script interpreter with its own tokenizer, AST, and ListCommandNode; it has supported &&/||/; since long before the Java interpreters did.

// shell/src/shell/org/jnode/shell/CommandShell.java:114-116
public static final String INITIAL_INTERPRETER = "redirecting";
public static final String FALLBACK_INTERPRETER = "default";

DefaultInterpreter and RedirectingInterpreter share a single tokenizer, which lives as a nested class inside DefaultInterpreter. That is why a change to the tokenizer's operator rules immediately changes pipeline behavior too, even though the two interpreters have separate parsers.

This page covers the two Java interpreters. The argument-level parsing that runs after an interpreter has produced a CommandLine is a separate subsystem documented in Syntax and Shell-Syntax.

Key Components

Class / File	Role
`shell/src/shell/org/jnode/shell/CommandShell.java`	Interactive loop; `INITIAL_INTERPRETER = "redirecting"`, `FALLBACK_INTERPRETER = "default"`
`shell/src/shell/org/jnode/shell/DefaultInterpreter.java`	Tokenizer, `parseSequence`, `interpret`, `escapeWord`, completion (`parsePartial`)
`shell/src/shell/org/jnode/shell/RedirectingInterpreter.java`	Two-level grammar: sequence of pipelines, each a list of commands with `<`/`>`
`shell/src/shell/org/jnode/shell/BjorneInterpreter.java`	Independent Bourne-compatible script interpreter
`shell/src/shell/org/jnode/shell/syntax/SyntaxManager.java`	Validates the `CommandLine` against the command's declared syntax (see Syntax)
`shell/src/test/org/jnode/test/shell/syntax/DefaultInterpreterSequenceTest.java`	19 tests pinning sequence semantics, exit codes, and error propagation
`shell/src/test/org/jnode/test/shell/syntax/DefaultTokenizerTest.java`	20+ tests pinning token types for operator/quote/escape combinations

The Grammar

RedirectingInterpreter parses a two-level grammar; DefaultInterpreter implements only the outer level.

line    := sequence  EOF
sequence:= pipeline ( ('&&' | '||' | ';') pipeline )*
pipeline:= command ( '|' command )*          # RedirectingInterpreter only
command := name arg* ( '<' file | '>' file )*

Operators are left-associative by construction: both interpreters produce a flat List<...Entry>, not a precedence tree, and the evaluator walks it in order. There is no operator precedence between &&, || and ;.

DefaultInterpreter explicitly does not treat | as an operator — a bare | is a literal character in that interpreter.

Operator Constants

// DefaultInterpreter.java:81-89
public static final char AMP_CHAR = '&';
public static final char SEMI_CHAR = ';';
public static final String AND_IF = "&&";
public static final String OR_IF  = "||";
public static final String SEMI   = ";";
public static final String AMP    = "&";

Parsed Representation

// DefaultInterpreter.java:323-331
protected static class SequenceEntry {
    public final CommandLine commandLine;
    public final String operatorBefore;   // null for the first command
}

RedirectingInterpreter uses a parallel shape because its iteration unit is a pipeline, not a command:

// RedirectingInterpreter.java:535-546
private static class SequenceDescriptor {
    List<CommandDescriptor> pipeline;
    String operatorBefore;
}

Evaluation and Exit Status

DefaultInterpreter.interpret (DefaultInterpreter.java:194-233):

if (sequence.isEmpty()) return 0;
if (sequence.size() == 1) {
    return shell.invoke(sequence.get(0).commandLine, null, null);   // exceptions propagate
}
int rc = 0;
for (int i = 0; i < sequence.size(); i++) {
    SequenceEntry entry = sequence.get(i);
    if (i > 0) {
        String op = entry.operatorBefore;
        if (AND_IF.equals(op) && rc != 0) continue;      // skip
        else if (OR_IF.equals(op) && rc == 0) continue;  // skip
        // ';' always executes
    }
    try {
        rc = shell.invoke(entry.commandLine, null, null);
    } catch (ShellControlException ex) {
        throw ex;                                          // always rethrown
    } catch (ShellException ex) {
        shell.diagnose(ex, null);
        rc = 1;
    }
}
return rc;

Key semantics:

Rule	Behavior
Single command	Short-circuits to a bare `shell.invoke`, so `ShellException` (unknown command, script abort, test-harness `trapException` expectations) still propagates
Multiple commands	Per-command `ShellException` is swallowed → `shell.diagnose()` + `rc = 1`; the sequence continues
`ShellControlException`	Abstract subclass of `ShellException`; always rethrown even mid-sequence (used by `TimeCommand` and script abort)
Skipped command	Does not reset `rc`, so `false && true` yields 1 and `true \|\| false` yields 0 (POSIX-correct)
Return value	The exit code of the last executed command, not the last parsed
`$?`	Not available. The default interpreters only return `rc` up the call stack; there is no `lastReturnCode` equivalent to `BjorneContext.getLastReturnCode()`

RedirectingInterpreter.interpret has the identical shape but delegates each iteration to runSequence(shell, seq.pipeline), where runSequence chooses runCommand for a single command and runPipeline for two or more.

`>>` Is Rejected as a Unit

The tokenizer treats >> as one SPECIAL token (ddfd75de5, #650/#669), and RedirectingInterpreter rejects it explicitly:

$ echo 'x' >> /tmp/f
unsupported '>>' redirection: use '>' instead

The old tokenizer split the two > characters, so the parser accepted > with an empty filename or misread the second > as a new redirection. The fix appends the second > to the same token in DefaultInterpreter.Tokenizer and adds the explicit branch in RedirectingInterpreter. The command is not executed (shell.executed stays empty), and the shell recovers with a fresh prompt instead of exiting.

`Pipeline.skip()` Returns 0, Not -1

Pipeline.skip(long n) returned -1 whenever it consumed nothing (n <= 0 or already at EOF). -1 is the read() EOF sentinel, not the InputStream.skip() contract. It now returns the actual count (0), so callers can distinguish "nothing to skip" from "end of stream". PipelineStreamsAndFanoutWriterTest covers skip(0) and negative counts on both live and drained streams.

Tokenizer Rules

DefaultInterpreter.Tokenizer.tokenize (DefaultInterpreter.java:499-683) produces tokens of type LITERAL=0, STRING=1, CLOSED=2, SPECIAL=4. REDIRECTS_FLAG = 0x01 switches | from literal to operator.

Input	Tokenizing	Note
`a && b`, `a \|\| b`, `a ; b`	`&&`/`\|\|`/`;` as `SPECIAL`	recognized regardless of `REDIRECTS_FLAG`
`a\|b` (no flag)	one `LITERAL a\|b`	escaping beats the `\|\|` operator rule
`'a&&b'`, `"c\|\|d"`, `'e;f'`	`STRING` token	quoting makes operator chars literal
`a\&\&b`, `a\|\|b`, `a\;b`	one `LITERAL`	backslash escape
`a \| b` (no flag)	`\|` is `LITERAL`	only `RedirectingInterpreter` sets the flag
`a \| b` (with flag)	`\|` is `SPECIAL`	pipeline
`http://x/?a=1&b=2`	one `LITERAL`	embedded single `&` stays literal (URL compatibility)
`echo a & echo b`	`&` is `SPECIAL`	standalone, so the parser can diagnose it
`a &&& b`	`&&` `SPECIAL`, then `&` `SPECIAL`	parser error
`a && b # c && d`	comment truncates the line	tokenizer still emits a zero-length trailing `LITERAL`

The tokenizer is a nested class of DefaultInterpreter, so RedirectingInterpreter reaches it as DefaultInterpreter.Tokenizer and constructs it with REDIRECTS_FLAG.

`escapeWord` and the Round-Trip

CommandShell.escapeWord(String) must invert tokenization so that a command's output can be fed back into the shell. Since &, ; and | can now introduce operators, all three are escaped unconditionally:

// DefaultInterpreter.java:371-379 (escapeWord(String, boolean))
case AMP_CHAR:
case SEMI_CHAR:
case PIPE_CHAR:
    // '&', ';' and '|' can introduce operators ...
    sb.append(ESCAPE_CHAR).append(ch);
    break;

Previously | was only escaped when escapeRedirects == true, which is the path used by RedirectingInterpreter.escapeWord at RedirectingInterpreter.java:166-168. DefaultInterpreter.escapeWord(String) delegates to escapeWord(word, false) (:334-336).

Callers: CommandShell.escapeWord (:960), CommandCompletions (:79), BjorneContext (:1066), and the black-box test harness harness/CommandTestRunner (:44, :46). DefaultInterpreterSequenceTest.testEscapeWordRoundTrip (:334-345) pins the round-trip.

Pathname Glob Expansion

Both DefaultInterpreter and RedirectingInterpreter expand pathname patterns in argument tokens, as BjorneInterpreter already did. Before this, rm /jnode/tmp/*.b64 passed the literal * to rm, which rejected it ("not a flag set").

protected static final int EXPANSION_FLAGS = PathnamePattern.SORT_MATCHES
        | PathnamePattern.HIDE_DOT_FILENAMES
        | PathnamePattern.INCLUDE_DOT_AND_DOTDOT
        | PathnamePattern.CHARACTER_CLASSES;

Expansion runs on the dequoted tokens and uses the current directory as the base for relative patterns (compilePathPattern(...).expand(new File("."))).
Only tokens the tokenizer marked as purely LITERAL are expanded. A quoted pattern ("*.txt") stays literal, and the tokenizer now also marks a token as STRING when a backslash escapes *, ?, [ or ], so an escaped meta-character is never expanded.
PathnamePattern.isPattern decides whether the token needs expanding at all; IllegalArgumentException from a bad pattern leaves the token untouched.
A pattern that matches nothing is passed through unchanged (bash's non-nullglob behaviour), not replaced by the empty list.
Expansion is skipped in completing mode (parsePartial/completion), so tab completion still sees the raw pattern; help parsing is untouched.
RedirectingInterpreter applies the same expandPathPatterns to each pipeline command's argVec.

DefaultInterpreterGlobTest (216 lines) pins the rules: quoted patterns, escaped metacharacters, no-match passthrough, sorting, hidden files, character classes, and the pipeline path.

Syntax Errors

All raised as ShellSyntaxException:

Message	Raised when
`Misplaced '<op>': expected a command name`	`&&`/`\|\|`/`;` appears where a command name was expected
`no command after '<op>'`	sequence operator at end of line
`unsupported '&': use '&&' for conditional execution`	standalone `&` reaches the parser
`unrecognized symbol: '<text>'`	any other `SPECIAL` token
`no command after '\|'`	trailing pipe (`RedirectingInterpreter` only)
`no filename after '<'/'>'`	redirection with no target (`RedirectingInterpreter` only)
`misplaced '<tok>'`	special token in an illegal position (`RedirectingInterpreter` only)
`empty '<special>' file name`	empty redirection target (`RedirectingInterpreter` only)

Errors are suppressed while the parser runs in completing mode (tab completion): a trailing operator yields an empty CommandLine rather than an exception.

Tab Completion

DefaultInterpreter.parsePartial (DefaultInterpreter.java:147-172) has to double-tokenize because parseSequence consumes the whole line:

Tokenizer tokenizer = new Tokenizer(line);
List<SequenceEntry> sequence = parseSequence(tokenizer, true);
if (sequence.isEmpty()) return new CommandLine("", null);
tokenizer.seek(0);                    // re-scan from the beginning
CommandLine.Token last = null;
while (tokenizer.hasNext()) {
    CommandLine.Token t = tokenizer.next();
    if (t.text.length() > 0) last = t;   // skip the empty token a #comment leaves behind
}
if (last != null && last.tokenType == SPECIAL
        && (AND_IF.equals(last.text) || OR_IF.equals(last.text) || SEMI.equals(last.text))) {
    return new CommandLine("", null);     // completing after a trailing operator
}
CommandLine res = sequence.get(sequence.size() - 1).commandLine;
return res == null ? new CommandLine("", null) : res;

This requires SymbolSource.seek(int) (DefaultInterpreter.java:708-713). RedirectingInterpreter.parsePartial (:124-129) accumulates SequenceDescriptors instead, and getLastCommand(List<SequenceDescriptor>) (:155-163) walks backwards skipping empty pipelines — it is also used by help().

Gotchas & Non-Obvious Behavior

| is a literal in DefaultInterpreter. A bare | only becomes SPECIAL when the REDIRECTS_FLAG is set, which only RedirectingInterpreter does.
Embedded & stays literal for URL compatibility. Only a standalone & (token buffer empty) is special. This is deliberate and covered by DefaultTokenizerTest.testTokenizerUrlWithAmp.
& is recognized but rejected. The tokenizer must surface it as SPECIAL so interpret can produce a helpful message instead of silently treating it as a literal.
No operator precedence. a || b && c evaluates strictly left to right, not as a || (b && c).
A single command still throws. The sequence.size() == 1 short-circuit is load-bearing for the black-box test harness and for script abort; removing it changes observable behavior of testSingleUnknownCommandRethrows and testSingleCommandFailurePropagates.
setArgumentAnticipated is only set on the last command of a sequence (DefaultInterpreter.java:313, RedirectingInterpreter.java:288-289). Earlier entries never get it, so "argument expected" completion state is only meaningful for the final command.
A trailing #comment leaks an empty argument. The comment token is zero-length but still lands in tokenList, so the last command of a sequence gains an empty-string argument. parsePartial works around it; interpret does not. Pre-existing quirk, preserved.
Sequence operators are recognized even without REDIRECTS_FLAG. Only | is flag-gated.
>> is one token and is rejected. It is not append mode; split tokens would otherwise parse as two > redirections.
Pipeline.skip() follows InputStream.skip(). Returning -1 for a zero-byte skip looked like EOF and broke callers that loop until a negative result.
A syntax error must leave the shell runnable. The >> regression test runs a subsequent command and asserts two prompts; a rejected line must not tear down the interactive loop.
Globs are expanded by the interpreter, not the command. *, ? and [] become matching pathnames before the CommandLine is built; quoting or escaping the metacharacter suppresses it, and a no-match pattern is passed through unchanged.
Expansion is skipped while completing. Tab completion must see the pattern, so parsePartial and RedirectingInterpreter.parsePartial pass completing through to the expander.
Testability requires subclasses. parseSequence, Tokenizer and the entry types are protected/private, so DefaultInterpreterSequenceTest drives them through TestableDefault, TestableRedirecting and a StubShell extends TestShell.
No $?, no && in Bjorne-parsed $(( )) expressions. The arithmetic evaluator (BJorne-Arithmetic-Evaluator) has its own tokenizer and is unaffected.

Related Pages

Shell-Commands — Parent hub for the command framework, invokers, and built-in commands
Syntax — Argument-level parsing (SyntaxManager, MuParser, Argument types) that runs after interpret
Shell-Syntax — Declarative argument syntax framework and tab-completion model
BJorne-Arithmetic-Evaluator — The independent Bourne-compatible interpreter's expression evaluator
JNode-Serial-Skill — Driving the shell over the serial agent console; notes which operators the interpreter supports
Testing — The two host-JVM test tiers: JUnit AllTests.java and black-box XML specs via TestHarness

Shell Interpreter

Shell Interpreter

Overview

Key Components

The Grammar

Operator Constants

Parsed Representation

Evaluation and Exit Status

>> Is Rejected as a Unit

Pipeline.skip() Returns 0, Not -1

Tokenizer Rules

escapeWord and the Round-Trip

Pathname Glob Expansion

Syntax Errors

Tab Completion

Gotchas & Non-Obvious Behavior

Related Pages

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

`>>` Is Rejected as a Unit

`Pipeline.skip()` Returns 0, Not -1

`escapeWord` and the Round-Trip