VoidScript Codex

Table of Contents

• 1. Project Structure
• 2. Build System
• 3. Modules
• 4. Lexer
• 5. Parser
• 6. AST & Interpreter
• 7. OperationsFactory & Container
• 8. Adding New Syntax
• 9. Testing & Debugging

1. Project Structure

/cli                  # CLI binary entrypoint (main.cpp)
/src
  /Lexer              # Tokenizer: Lexer.cpp, Operators.hpp/.cpp, Token.hpp
  /Parser             # Parser: Parser.hpp/.cpp, ParsedExpression.hpp
  /Interpreter        # AST nodes (ExpressionNode, StatementNode, OperationsFactory, etc.)
  /Modules            # Built‑in modules (Array, File, JSON, Print, String, VariableHelpers)
  /Symbols            # SymbolTable, SymbolContainer, Value, Types
/Modules              # External/plugin modules (e.g., CurlModule)
test_scripts          # Reference scripts (.vs) for tests/examples
build                 # Out‑of‑source build directory

2. Build System

• Uses CMake (top‑level CMakeLists.txt).
• Configure: cmake -S . -B build
• Build: cmake --build build (or ninja -C build).
• Binaries & libraries under build/:
• build/voidscript (CLI)
• build/libvoidscript.so
• build/Modules/<ModuleName>

3. Modules

Built‑in

Located in src/Modules/BuiltIn/. They export C++ functions to the interpreter via BuiltinModule interface.

Plugins

Under /Modules/<YourModule>/:
1. Create CMakeLists.txt next to PluginInit.cpp.
2. Implement hook VS_MODULE_INIT() to register functions.
3. Build installs to build/Modules/<YourModule>.

4. Lexer

• Files: src/Lexer/Lexer.hpp/.cpp, Operators.hpp/.cpp, Token.hpp.
• Operators.hpp defines token categories (arithmetic, logical, relational, assignment, punctuation).
• In Lexer::matchOperatorOrPunctuation, longest‑match two‑char operators first.
• To add a token:

1. Update the appropriate vector in Operators.cpp.
2. If it’s a literal/identifier change, adjust matchIdentifierOrKeyword or matchNumber.

5. Parser

• Files: src/Parser/Parser.hpp/.cpp.
• Entry: Parser::parseScript(), which loops parseStatement() until EOF.
• Statement handlers:
• Top‑level: parseVariableDefinition(), parseAssignmentStatement(), parseIfStatement(), parseWhileStatement(), parseForStatement(), parseCallStatement(), parseReturnStatement().
• In‑block: parseStatementNode(), plus parseIfStatementNode(), parseForStatementNode().
• Expression parsing: parseParsedExpression() implements a shunting‑yard algorithm; outputs ParsedExpression AST fragments.

6. AST & Interpreter

• AST node interfaces under src/Interpreter/:
• ExpressionNode (evaluate returns Symbols::Value)
• StatementNode (interpret for side‑effects)
• Mapping from ParsedExpression to concrete ExpressionNode in Parser/ExpressionBuilder.hpp.
• Each node implements evaluate() or interpret().
• To add a node:

1. Create header & source in Interpreter/.
2. Update ExpressionBuilder (for expr nodes) or parser (for stmt nodes).

7. OperationsFactory & Container

• Interpreter/OperationsFactory.hpp/.cpp: factories to record operations (Assignment, Call, Conditional, Loop, Return).
• Operations::Container holds a list of operations per scope; executed by Interpreter.
• Top‑level parser uses OperationsFactory to enqueue AST StatementNode into the container.

8. Adding New Syntax

1. Lexer: add or extend token definitions in Operators.hpp and adjust Lexer.cpp matching logic.
2. Parser:

• Register new keyword in Parser::keywords (if needed).
• Write a new parseXxx() or extend parseStatementNode()/parseParsedExpression().
• For expressions, integrate into the shunting‑yard switch.

1. AST: implement a new ExpressionNode or StatementNode subclass with semantics.
2. Interpreter: define evaluate() or interpret() in the node.
3. OperationsFactory: if top‑level, add a factory method to emit the operation.
4. Tests: write a .vs script under test_scripts/ and run with debug flags.

9. Testing & Debugging

• Sample scripts in test_scripts/.
• Run interpreter with debug levels:
• Lexer: voidscript --debug=lexer script.vs
• Parser: voidscript --debug=parser script.vs
• Interpreter: voidscript --debug=interpreter script.vs
• Use printnl(...) functions in scripts to observe runtime behavior.

---

Maintenance

This document must be updated whenever changes are made to the codebase (lexer, parser, AST nodes, modules, build system, etc.) to keep the guide current.

Generated by Codex CLI Agent
Always keep this file in sync with project changes.