Sometimes, simple things are just very effective abstractions that are very good at hiding the giant rabbit hole underneath.
Like an innocent little literal value for example:
Debug.Print 42
This 42 is just that, a plain integer value – isn’t it?
One could reasonably set out and write a parser that produces an abstract syntax tree (AST) here that would include some LiteralExpression node, that necessarily evaluates to 42.
But the node itself isn’t the value. In fact – perhaps surprisingly – it doesn’t even know what or where that value is. It knows where it is defined in the source code, but that’s not what I meant with where that value is.
So where does the 42 actually live? Where is it stored? And why does it even matter?
The answer is… not in the expression node: the node evaluates to a value, but is not that value.
For starters, the specifications dictate that 42 must be an Integer literal. Not a Long, not a Double – although a different value might yield either semantic data type. The grammar however, only distinguishes integer numbers from floating-point numbers: it’s token semantics specified in MS-VBAL that ultimately determines the semantic data type.
It also states very plainly that an integer value should be a 16-bit signed integer, and also defines such internal representations for each intrinsic data type: clearly we can’t just make everything a double and run with it, or we’ll almost certainly break something downstream.
Why is that?
More Than Values
If we stop and think about the type system in VBA, we quickly realize that we don’t just need values for literals, and that many other things evaluate to, or represent a value – like functions, parameters, and if we squint a little even a class instance (object) could be coerced into a value.
And that’s a problem if we model a semantic type system that embeds values directly, because then the assignment of a ByRef parameter can only behave as a value, not as a reference. Besides we also want to be able to represent values that exist and must be usable but aren’t defined anywhere in source code.
This is why RDCore introduced binding handles as an indirection layer between a VBTypedValue and the underlying binary value it represents.
Back to the literal 42 integer value: we have a value binding that yields the 42, and so the VBIntegerValue type knows it has a binding that can get the value, and it doesn’t need to know where that value actually is.
An integer parameter passed by reference would also be a VBIntegerValue, but its binding would be a reference binding rather than a value one, likely pointing to a symbol defined somewhere in the call stack: changing the value of this reference binding would necessarily affect the value everywhere it’s referenced, because its runtime value is associated to its declaration symbol within the program memory.
The RDCore semantic type system explicitly differentiates types and values, such that a VBIntegerValue represents a value of type VBInteger. This is slightly different from other models, where a value might be an instance of a given data type. Beyond that, a type might also be described by a value that represents a data type: VBTypeDescValue. This special meta-value is useful in a number of scenarios (notably TypeOf operators), and enables reflection-like capabilities and semantics that may or may not end up exposed at the language level, possibly through platform extensions.
This differentiation is necessary to correctly model arrays, UDT, and custom class types – that’s a whole other rabbit hole that deserves its own post, but for now this is what matters:
VBInteger:VBType
👇describes
VBIntegerValue:VBTypedValue
👇bound to
42:System.Int16
If an instance of a given VBType represented a value of that type, then we would need to generate the complex types on the fly, and that would be way more complicated than things need to be: we don’t need dynamic types when everything is statically defined – what we need is just a model that makes it work, and conceptually separating data types from typed values does exactly that.
Runtime Values
The RDCore SDK defines an entire type system that supports everything (and more!) VBA can do, using .net types to represent all underlying runtime values.
While several data types work pretty well as-is, others like Boolean, Currency, and Decimal don’t really map 1:1 and need a special type that correctly represents the value. For example an Integer value easily maps to a .net Int16, but mapping a .net Boolean to a VBA Boolean would introduce bit alignment issues (notably in UDT types), because it’s a 16-bit integer under the hood but a .net Boolean is only a single byte! Decimal is another, perhaps less consequential since it’s not declarable, but MS-VBAL specifies it as fitted across 14 bytes (3x Int32 +Int16), yet the corresponding .net type is 16 bytes wide (4x Int32); Variant is another data type that needs its own interop value, a 24-byte struct that can hold anything, value or reference.
The Debug.Print 42 instruction parses the 42 as a literal expression that evaluates to a VBIntegerValue that has a ValueBindingHandle that holds the managed Int16 value 42.
If it were Debug.Print “Hello, world!”, we would’ve had a VBStringValue with a ReferenceBindingHandle instead, holding a reference to a managed string holding the literal value. Concretely, this means a reference binding is really just a wrapper around a managed object; this makes it possible to box value types and use them as references in ByRef scenarios.
But it’s just a literal!
And it is. But to be useful, the semantic model must be able to tell a Variant from a Long from a Double, and a unified semantic type system that doesn’t care whether a value came from source code or is the result of an operation, and isn’t concerned about the internal memory representation of that value: that’s what the semantic model abstraction level does, and that’s where the AST belongs.
Constantly wrapping and unwrapping runtime values into semantic ones would be terribly inefficient, for run-time though. If it doesn’t sound so bad, try looping a million times into an array and adding one to each element, and the overhead should become obvious: it’ll interpret nodes all right, but something else must be able to actually run things without caring for the semantic layer.
Progress Status
As of this writing, the parser produces an AST containing all the expected nodes in an initial declaration pass: module directives and attributes, as well as everything in the declarations section, and procedure member signatures (including parameters) – but no executable statement nodes yet, although many of these have already been defined.
These nodes will be introduced in the AST alongside their respective evaluation semantics in an evaluation engine that’ll become the interpreter that will eventually evaluate nodes in break mode, while a separate execution engine will operate at a lower level, working directly with the underlying runtime types.
This will all be tested and benchmarked (and adjusted as needed) in due time, of course.
For now the development focus is turning to the LSP client/server SDK, which is what underpins cross-process communications across the entire RDCore platform. Once that’s in place, the language server can start implementing LSP handlers that an IDE can use, for everything a declaration pass already unlocks – but that work will be ticketed and not completed right away (not by me anyway), because then focus will have to shift to the semantic evaluation engine and the statement nodes, because GoTo and GoSub/Return, not to mention error states, are an interesting twist on what would otherwise be a rather straightforward AST traversal.
Rubberduck News 