Tag: thought-process

Simple Things Pt.1: Values

Rubberduck VBA2 Comments

Sometimes, simple things are just very effective abstractions that are very good at hiding the giant rabbit hole underneath.

Like an innocent little literal value for example:

Debug.Print 42

This 42 is just that, a plain integer value – isn’t it?

One could reasonably set out and write a parser that produces an abstract syntax tree (AST) here that would include some LiteralExpression node, that necessarily evaluates to 42.

But the node itself isn’t the value. In fact – perhaps surprisingly – it doesn’t even know what or where that value is. It knows where it is defined in the source code, but that’s not what I meant with where that value is.

So where does the 42 actually live? Where is it stored? And why does it even matter?

The answer is… not in the expression node: the node evaluates to a value, but is not that value.

For starters, the specifications dictate that 42 must be an Integer literal. Not a Long, not a Double – although a different value might yield either semantic data type. The grammar however, only distinguishes integer numbers from floating-point numbers: it’s token semantics specified in MS-VBAL that ultimately determines the semantic data type.

It also states very plainly that an integer value should be a 16-bit signed integer, and also defines such internal representations for each intrinsic data type: clearly we can’t just make everything a double and run with it, or we’ll almost certainly break something downstream.

Why is that?

More Than Values

If we stop and think about the type system in VBA, we quickly realize that we don’t just need values for literals, and that many other things evaluate to, or represent a value – like functions, parameters, and if we squint a little even a class instance (object) could be coerced into a value.

And that’s a problem if we model a semantic type system that embeds values directly, because then the assignment of a ByRef parameter can only behave as a value, not as a reference. Besides we also want to be able to represent values that exist and must be usable but aren’t defined anywhere in source code.

This is why RDCore introduced binding handles as an indirection layer between a VBTypedValue and the underlying binary value it represents.

Back to the literal 42 integer value: we have a value binding that yields the 42, and so the VBIntegerValue type knows it has a binding that can get the value, and it doesn’t need to know where that value actually is.

An integer parameter passed by reference would also be a VBIntegerValue, but its binding would be a reference binding rather than a value one, likely pointing to a symbol defined somewhere in the call stack: changing the value of this reference binding would necessarily affect the value everywhere it’s referenced, because its runtime value is associated to its declaration symbol within the program memory.

The RDCore semantic type system explicitly differentiates types and values, such that a VBIntegerValue represents a value of type VBInteger. This is slightly different from other models, where a value might be an instance of a given data type. Beyond that, a type might also be described by a value that represents a data type: VBTypeDescValue. This special meta-value is useful in a number of scenarios (notably TypeOf operators), and enables reflection-like capabilities and semantics that may or may not end up exposed at the language level, possibly through platform extensions.

This differentiation is necessary to correctly model arrays, UDT, and custom class types – that’s a whole other rabbit hole that deserves its own post, but for now this is what matters:

VBInteger:VBType
 👇describes
VBIntegerValue:VBTypedValue
 👇bound to
42:System.Int16

If an instance of a given VBType represented a value of that type, then we would need to generate the complex types on the fly, and that would be way more complicated than things need to be: we don’t need dynamic types when everything is statically defined – what we need is just a model that makes it work, and conceptually separating data types from typed values does exactly that.

Runtime Values

The RDCore SDK defines an entire type system that supports everything (and more!) VBA can do, using .net types to represent all underlying runtime values.

While several data types work pretty well as-is, others like Boolean, Currency, and Decimal don’t really map 1:1 and need a special type that correctly represents the value. For example an Integer value easily maps to a .net Int16, but mapping a .net Boolean to a VBA Boolean would introduce bit alignment issues (notably in UDT types), because it’s a 16-bit integer under the hood but a .net Boolean is only a single byte! Decimal is another, perhaps less consequential since it’s not declarable, but MS-VBAL specifies it as fitted across 14 bytes (3x Int32 +Int16), yet the corresponding .net type is 16 bytes wide (4x Int32); Variant is another data type that needs its own interop value, a 24-byte struct that can hold anything, value or reference.

The Debug.Print 42 instruction parses the 42 as a literal expression that evaluates to a VBIntegerValue that has a ValueBindingHandle that holds the managed Int16 value 42.

If it were Debug.Print “Hello, world!”, we would’ve had a VBStringValue with a ReferenceBindingHandle instead, holding a reference to a managed string holding the literal value. Concretely, this means a reference binding is really just a wrapper around a managed object; this makes it possible to box value types and use them as references in ByRef scenarios.

But it’s just a literal!

And it is. But to be useful, the semantic model must be able to tell a Variant from a Long from a Double, and a unified semantic type system that doesn’t care whether a value came from source code or is the result of an operation, and isn’t concerned about the internal memory representation of that value: that’s what the semantic model abstraction level does, and that’s where the AST belongs.

Constantly wrapping and unwrapping runtime values into semantic ones would be terribly inefficient, for run-time though. If it doesn’t sound so bad, try looping a million times into an array and adding one to each element, and the overhead should become obvious: it’ll interpret nodes all right, but something else must be able to actually run things without caring for the semantic layer.

Progress Status

As of this writing, the parser produces an AST containing all the expected nodes in an initial declaration pass: module directives and attributes, as well as everything in the declarations section, and procedure member signatures (including parameters) – but no executable statement nodes yet, although many of these have already been defined.

These nodes will be introduced in the AST alongside their respective evaluation semantics in an evaluation engine that’ll become the interpreter that will eventually evaluate nodes in break mode, while a separate execution engine will operate at a lower level, working directly with the underlying runtime types.

This will all be tested and benchmarked (and adjusted as needed) in due time, of course.

For now the development focus is turning to the LSP client/server SDK, which is what underpins cross-process communications across the entire RDCore platform. Once that’s in place, the language server can start implementing LSP handlers that an IDE can use, for everything a declaration pass already unlocks – but that work will be ticketed and not completed right away (not by me anyway), because then focus will have to shift to the semantic evaluation engine and the statement nodes, because GoTo and GoSub/Return, not to mention error states, are an interesting twist on what would otherwise be a rather straightforward AST traversal.

“I’m not a programmer”

Rubberduck VBA7 Comments

If you’re an accountant, a sales analyst, or in any other office position where writing VBA code helps you do your job faster, you may have uttered the words “I’m not a programmer” before, and that wouldn’t have been wrong: once the code is written, you’d tweak it every once in a while to fix a bug here or there, and then move on to do your actual job as soon as things look like they work as they should. If you’re finding that as weeks and months pass, you’re spending more and more time debugging that code, it might pay off to learn a bit more about how “actual programmers” do things, but be warned: it’s a bit of a rabbit hole – in a good way, but still, it goes as deep as you’re willing to go.

If you’re a consultant delivering advanced Excel solutions to your business clients, you may have thought, said, or written the same words, too. But you like your worksheets efficient, flexible, reliable, easy to maintain; over the years you’ve become an expert at dissecting and modeling a business problem into a fine solution that will live on and grow with the client. You are a “power user”, a professional, and it shows.

Writing code involves exactly the same identical problem-solving thought process: dissecting a problem into small steps, and modeling it into …a sequence of executable statements. When you use a SUM function in a worksheet, you assess whether the range of cells you’re adding up will need to grow over time, and you make sure it’s as simple as possible to add or insert a new row without breaking the integrity of the worksheet: you’re not just solving the problem at hand, you’re anticipating the extension points, facilitating them, making it harder to break things. You shade the background of cells intended for data entry, use borders around fields, conditional formatting, and data validation to ensure everything is obvious and remains consistent; you source data validation lists from named ranges pointing to a column in a table, so that adding new possible valid entries is easy as pie and requires no other step than… adding the new possible valid entries in the table. Next to another table that requires a particular sort order because it’s being used by dynamic named ranges that source data validation lists for co-dependent dropdowns, you might put up a very obvious formatted shape with an inner text that explains why that table needs to be sorted by this column then that column, and what happens to the associated validation dropdowns when the sort is broken.

See, you are a programmer. Worksheets are programs – even more intensely so with the amazing new features coming to Excel: dynamic arrays are changing the entire paradigm and turning the very thinking of worksheet functions into something that really isn’t very far from the mindset you’d have in functional programming.

So why is it that VBA code is so often seen as merely a sequence of executable statements then? Why is it that “it works, therefore it’s good enough” is so often where the bar is? You could have made that data validation list work off a standard range of cells in some (hidden?) column somewhere off-screen, and that would have worked too… but “well, it works” isn’t where the bar should be at, and you know it.

Depending on what the code needs to do, VBA code can become much more than just a macro once you start not just solving the problem at hand, but also anticipating the extension points, facilitating them, making it easier to maintain, and harder to break things. If writing code is part of what you do for a living, then you might as well write good code. Good code isn’t just code that works. It’s code that adheres to a number of language-agnostic principles and modern-day guidelines that even plain procedural code should follow. It’s code that Joe from accounting probably couldn’t have written by themselves, but that they could likely read and understand (at least at the higher abstraction levels), and if they know enough VBA to be dangerous, they could likely even maintain and extend it!

I’m not saying every piece of VBA code needs OOP and dependency injection and inversion of control and 20 class modules with 1 method each ought to get involved in sorting a ListObject. Just that maybe, VBA code would be a little less dreadful to the eventual IT staff inheriting it, if instead of saying “I’m not a programmer”, we cared about the quality of our code in the exact same way we care about the quality of our worksheets and dashboards, or Access databases and reports, or whatever it is that we’re doing.