Back in the summer of 2017, we ran a rather successful GoFundMe campaign to collect funds to get some swag made, and shipped to the project’s contributors. It was great to see all that support pouring in, but it was bittersweet… for I would have loved to have been able to materially thank back our supporters. Alas, making custom printed gear quickly gets expensive, and crazy shipping fees essentially doubled it up.
So this time around I decided to bite the bullet and make a more significant initial investment, and actually start an online shop and keep an inventory in stock that can be shipped to supporters whenever, without needing to wait for prints, or paying too much for too few items being made, severely limiting profitability and driving prices up.
I guess I’ll know soon enough if I still have my old merchandise planner’s touch! If the inventory turns too fast, I’ll miss opportunities and may have to delay shipments to replenish. But if the markup is too high and stock isn’t moving quickly enough, I’ll probably need to take markdowns.
I’ll be writing an article about the particular workbook I’m using to track all that next – I want a bit of VBA code to simplify tracking sales and invoicing.. but for now I’m just thrilled to start shipping all the goodies!
I’m very happy with how everything turned out. That beautiful yellow ducky mug will probably only ever be printed in small batches, for they are quite expensive and I’m unfortunately not going to be able to have a better price for them unless I make them in very large batches. Everything else (even t-shirts, to a degree) has the potential to eventually get to a very decent price point – the goal is obviously to collect some funds to help finance the costs of hosting and securing the blog and website, but I want this inventory to move and don’t mind taking a lower margin if there’s enough volume to drive enough sales to just keep the shop afloat – time will tell!
I’ve embarked on a journey to take Rubberduck to the next major version, making it the add-in we’ve always wanted to build. These monthly updates provide a sneak peek at what’s coming, and how it’s coming to be.
Quick catch-up
Rubberduck 3.0 will run a LSP server process in the background
A separate process will host a local SQLite database
Telemetry will be opt-in, fully configurable, and transparent
Several quite important low-level changes since last time: we’re now looking at named pipes rather than sockets for the JSON-RPC communications between the editor and the language server, and I’m now using Microsoft’s StreamJsonRpc library for this. Named pipes are inherently local, so they’re less of a concern than sockets, and they don’t seem to trip up Windows Defender, so we’ll take it!
I spent the better part of last month tidying up and documenting the code off the language server protocol (LSP) specifications, moving things around and splitting up responsibilities, writing abstractions that will be shared by all server processes: LSP and SQLite, but also a separate/dedicated server process for telemetry, so even constant writes couldn’t interfere much with LSP server activities, or with the add-in client.
At the time of this writing, I’m still somewhat struggling with the RPC communications, but that won’t remain stuck for too long – the plan is to merge a rather large structural PR and the whole RPC infra by the end of this week.
The Rubberduck.Server.LocalDb console at startup. This process will normally run in the background, hidden.
I’ve taken a number of important decisions about the project in the last few weeks.
GitHub Repository Issues
Since the project’s beginning, Rubberduck was pretty much ad-hoc development. I remember in the first few days after creating the GitHub repository, going on an issue-creation spree to write down everything I could dream the thing could do. A lot of it was implemented, but the oldest open issues in the repository are from 2014, 2015:
2,857 issues closed is quite something. 958 open feels daunting though. Using issues as a backlog might not have been the best of ideas…
I’m not going through nearly a thousand issues to sort out what’s already implemented/fixed, what’s unrealistic after all, what’s a good idea that got buried under a million others, etc. Implementing LSP isn’t magically going to clean this up, and when 3.0 releases we’re not going to be maintaining two distinct, massive code bases: one of them isn’t going to make it, and it’s sadly going to have to be the one with 1.7K stars and 284 forks and 97 watchers. I can hardly express how I feel about these numbers, let alone those:
As of 2023/02/14, release tag 2.5.2.1 (build 2.5.2.5906) has been downloaded 27,320 times… that’s crazy!
The repository isn’t going anywhere though – it’s just that at some point in the [somewhat-near] future, it’s going to be made read-only and essentially archived, and the Rubberduck3 repository will become Rubberduck’s new home on GitHub (you’ll still find it under the rubberduck-vba organization).
Rubberduck is still accepting pull requests for v2.x and will continue to do so until further notice.
Methodology Upgrade
If building Rubberduck up to v2.5 was pretty much ad-hoc (and that’s fine!), I don’t think the same strategy would work with v3.0; we can’t just go and create a thousand issues to churn through, or pick a feature to implement because it looks like it’s going to be fun to do. Rubberduck 3.0 is still an embryo at this point, and while all the DNA is there and we know exactly (at least in large fuzzy outlines) what we want this add-in to do, this time things need to happen in order, for technical reasons mostly, but also for project management.
By adopting a different development methodology, we’re going to better control the backlog and project progression. We can better track what’s in progress and determine what the next logical steps should be.
Instead of making a ton of issues, we’ll be drafting them, sizing them, prioritizing them, refining them until they’re small enough to be realistically achievable within a week or two of part-time contributions. Work items will now have a life cycle like this:
New items/ideas not yet fleshed out, not yet planned, and/or not yet prioritized.
Backlog for work items being documented.
Ready for documented work items that are ready to be worked on; items are assigned a sprint (not necessarily the next one), and convert into issues at this stage.
In Progress is work in progress; a branch is created for resolving that issue.
In Review is work items ready to be peer reviewed; a pull request is opened at that stage.
Done is when the work is merged into dev/next.
Delivered status is set when the work is merged into main.
Items/issues will be assigned a priority level:
Urgent is the highest priority level, for things that should be worked on before anything else.
High is for work that’s directly aligned with the objectives of the sprint it’s in.
Medium priority work could be delayed a sprint or two.
Low priority work doesn’t need to be in the current sprint, but would be nice to deliver anyway.
The priority level of any given issue likely evolves over time, particularly the lower-level ones.
In addition to status and priority, each draft issue / work item gets sized. Again this is meant to evolve over time: issues should become smaller over time as they are refined and documented and broken down into smaller tasks.
X-Large items represent a large development that should be broken down into smaller tasks.
Large items represent a significant development effort that can realistically be completed within a sprint by a single developer.
Medium items represent perhaps up to 2-3 days of effort.
Small items represent small tasks that can be completed in a few hours.
Tiny items represent tasks that should only take a few minutes: fixing a typo, adding a column to a database view or table, a configuration tweak, etc.
As of this writing, Sprint 1 is in its second half, and I’m still working on the RPC infrastructure:
“Project Cucumber” – it just had to be named that.
There’s a bit of history around the cucumber thing, and it involves two major contributors we lost and think about fondly all the time.
Comintern simply disappeared one day after a brief return, after a year under terms that essentially prevented him from working on open-source projects and hanging out in a public chat. We’re all hoping he’s all right and will eventually pop back.We lost our beloved ThunderFrame to a fierce cancer in late 2018. Rest in peace, my friend. I got the Ctrl+Alt+Delete cushion covers you wanted me to have.
Discord Server
Rubberduck’s dev chat was always in a Stack Exchange room under the Code Review site. In fact it’s just a Code Review chat room we ended up [ab]using for this purpose. Back in 2014 I was very active on CR, and as a moderator on that site in 2015-2018 it made a lot of sense to keep it there.
But with 2-week sprints and a living backlog, we’re going to need more than SE chat to pull this off, and this is where Discord shines.
The general chat is fun, but a nice thing about Discord is that you can schedule events, like a public Sprint Review presentation every two weeks, followed by a Sprint Planning conversation among developers.
I’ll be hosting these events regularly, whether there’s an audience or not, whether other contributors are present or not.
Sprint Review
At each end of a sprint, we’re going to be going over what was done in the previous two weeks, and developers will present/demo their work. Since I’m doing sprint 1 by myself the first review will be me going over the solution structure and explaining the mechanisms and abstractions involved at a high level; reviews for sprint 2 and onward will likely involve more contributors, and things will get more and more exciting to present every time.
Sprint Planning
After the review concludes, developers convene to plan a realistically deliverable workload for the upcoming sprint. If we overshoot and under-deliver, we can always adjust the next sprint. If we over-deliver, we can always pull work items from upcoming sprints into the current one. So this conversation is about the work itself, whether there’s enough information in an issue for anyone at the table (or not!) to pick up and complete that task within two weeks, and whether the backlog is healthy or falling behind; if it’s falling behind, we take the time to talk about what needs to happen and outline work items to be drafted and refined during the sprint (I’ll be doing that backlog maintenance).
So yeah, Rubberduck3 is starting to feel very much like it’s just about to officially kick off, and Rubberduck as a project is entering a whole new phase, in continuous delivery mode.
One of the objectively coolest features in Rubberduck is the Fakes API. Code that pops a MsgBox for example, needs a way to work without actually popping that message box, otherwise that code cannot be unit tested… without somehow hijacking the MsgBox function. The Fakes API does exactly that: it hooks into the VBA runtime, intercepts specific internal function calls, and makes it return exactly what your test setup …set up.
This API can stop time, or Now can be told to return 1:59AM on first invocation, 1:00AM on the next, and then we can test and assert that some time-sensitive logic survives a daylight savings time toggle, or how Timer-dependent code behaves at midnight.
Let’s take a look at the members of the IFakesProvider interface.
Fakes Provider
Fakes for many of the internal VBA standard library functions exist since the initial release of the feature, although some providers wouldn’t always play nicely together – thanks to a recent pull request from @tommy9 these issues have been resolved, and a merry bunch of additional implementations are now available in pre-release builds:
Name
Description
Parameter names
MsgBox
Configures VBA.Interaction.MsgBox calls
Fakes.Params.MsgBox
InputBox
Configures VBA.Interaction.InputBox calls
Fakes.Params.InputBox
Beep
Configures VBA.Interaction.Beep calls
Environ
Configures VBA.Interaction.Environ calls
Fakes.Params.Environ
Timer
Configures VBA.DateTime.Timer calls
DoEvents
Configures VBA.Interaction.DoEvents calls
Shell
Configures VBA.Interaction.Shell calls
Fakes.Params.Shell
SendKeys
Configures VBA.Interaction.SendKeys calls
Fakes.Params.SendKeys
Kill
Configures VBA.FileSystem.Kill calls
Fakes.Params.Kill
MkDir
Configures VBA.FileSystem.MkDir calls
Fakes.Params.MkDir
RmDir
Configures VBA.FileSystem.RmDir calls
Fakes.Params.RmDir
ChDir
Configures VBA.FileSystem.ChDir calls
Fakes.Params.ChDir
ChDrive
Configures VBA.FileSystem.ChDrive calls
Fakes.Params.ChDrive
CurDir
Configures VBA.FileSystem.CurDir calls
Fakes.Params.CurDir
Now
Configures VBA.DateTime.Now calls
Time
Configures VBA.DateTime.Time calls
Date
Configures VBA.DateTime.Date calls
Rnd*
Configures VBA.Math.Rnd calls
Fakes.Params.Rnd
DeleteSetting*
Configures VBA.Interaction.DeleteSetting calls
Fakes.Params.DeleteSetting
SaveSetting*
Configures VBA.Interaction.SaveSetting calls
Fakes.Params.SaveSetting
Randomize*
Configures VBA.Math.Randomize calls
Fakes.Params.Randomize
GetAllSettings*
Configures VBA.Interaction.GetAllSettings calls
SetAttr*
Configures VBA.FileSystem.SetAttr calls
Fakes.Params.SetAttr
GetAttr*
Configures VBA.FileSystem.GetAttr calls
Fakes.Params.GetAttr
FileLen*
Configures VBA.FileSystem.FileLen calls
Fakes.Params.FileLen
FileDateTime*
Configures VBA.FileSystem.FileDateTime calls
Fakes.Params.FileDateTime
FreeFile*
Configures VBA.FileSystem.FreeFile calls
Fakes.Params.FreeFile
IMEStatus*
Configures VBA.Information.IMEStatus calls
Dir*
Configures VBA.FileSystem.Dir calls
Fakes.Params.Dir
FileCopy*
Configures VBA.FileSystem.FileCopy calls
Fakes.Params.FileCopy
*Members marked with an asterisk are only available in pre-release builds for now.
Parameter Names
The IVerify.ParameterXyz members make a unit test fail if the specified parameter wasn’t given a specified value, but the parameter names must be passed as strings. This is a UX issue: the API essentially requires hard-coded magic string literals in its users’ code; this is obviously error-prone and feels a bit arcane to use. The IFakesProvider interface has been given a Params property that gets an instance of a class that exposes the parameter names for each of the IFake implementations, as shown in the list above, and the screenshot below:
Picking the correct parameter name from a drop-down completion list beats risking a typo, doesn’t it?
Note: the PR for this feature has not yet been merged at the time of this writing.
Testing Without Fakes (aka Testing with Stubs)
Unit tests have a 3-part structure: first we arrange the test, then we act by invoking the method we want to test; lastly, we assert that an actual result matches the expectations. When using fakes, we configure them in the arrange part of the test, and in the assert part we can verify whether (and/or how many times) a particular method was invoked with a particular parameterization.
Let’s say we had a procedure we wanted to write some tests for:
Public Sub TestMe()
If MsgBox("Print random number?", vbYesNo + vbQuestion, "Test") = vbYes Then
Debug.Print Now & vbTab & Rnd * 42
Else
Debug.Print Now
End If
End Sub
If we wanted to make this logic fully testable without the Fakes API, we would need to inject (likely as parameters) abstractions for MsgBox, Now, and Debug dependencies: instead of invoking MsgBox directly, the procedure would be invoking the Prompt method of an interface/class that wraps the MsgBox functionality. Unit tests would need a stub implementation of that interface in order to allow some level of configuration setup – an invocation counter, for example. A fully testable version of the above code might then look like this:
Public Sub TestMe(ByVal MessageBox As IMsgBox, ByVal Random As IRnd, ByVal DateTime As IDateTime, ByVal Logger As ILogger)
If MessageBox.Prompt("Print random number?", "Test") = vbYes Then
Logger.LogDebug DateTime.Now & vbTab & Random.Next * 42
Else
Logger.LogDebug DateTime.Now
End If
End Sub
The method is testable, because the caller controls all the dependencies. We’re probably injecting an IMsgBox that pops a MsgBox, an IRnd that wraps Rnd, a DateTime parameter that returns VBA.DateTime.Now and an ILogger that writes to the debug pane, but we don’t know any of that. I fact, we could very well run this method with an ILogger that writes to some log file or even to a database; the IRnd implementation could consistently be returning 0.4 on every call, IDateTime.Now could return Now adjusted to UTC, and IMsgBox might actually display a fancy custom modal UserForm dialog – either way, TestMe doesn’t need to change for any of that to happen: it does what it needs to do, in this case fetching the next random number and outputting it along with the current date/time if a user prompt is answered with a “Yes”, otherwise just output the current date/time. It’s the interfaces that provide the abstraction that’s necessary to decouple the dependencies from the logic we want to test. We could implement these interfaces with stubs that simply count the number of times each member is invoked, and the logic we’re testing would still hold.
We could then write tests that validate the conditional logic:
'@TestMethod
Public Sub TestMe_WhenPromptYes_GetsNextRandomValue()
' Arrange
Dim MsgBoxStub As StubMsgBox ' implements IMsgBox, but we want the stub functionality here
Set MsgBoxStub = New StubMsgBox
MsgBoxStub.Returns vbYes
Dim RndStub As StubRnd ' implements IRnd, but we want the stub functionality here too
Set RndStub = New StubRnd
' Act
Module1.TestMe MsgBoxStub, RndStub, New DateTimeStub, New LoggerStub
' Assert
Assert.Equals 1, RndStub.InvokeCount
End Sub
'@TestMethod
Public Sub TestMe_WhenPromptNo_DoesNotGetNextRandomValue()
' Arrange
Dim MsgBoxStub As StubMsgBox
Set MsgBoxStub = New StubMsgBox
MsgBoxStub.Returns vbNo
Dim RndStub As StubRnd
Set RndStub = New StubRnd
' Act
Module1.TestMe MsgBoxStub, RndStub, New DateTimeStub, New LoggerStub
' Assert
Assert.Equals 0, RndStub.InvokeCount
End Sub
These stub implementations are class modules that need to be written to support such tests. StubMsgBox would implement IMsgBox and expose a public Returns method to configure its return value; StubRnd would implement IRnd and expose a public InvokeCount property that returns the number of times the IRnd.Next method was called. In other words, it’s quite a bit of boilerplate code that we’d usually rather not need to write.
Let’s see how using the Fakes API changes that.
Using Rubberduck.FakesProvider
The standard test module template defines Assert and Fakes private fields. When early-bound (needs a reference to the Rubberduck type library), the declarations and initialization look like this:
'@TestModule
Option Explicit
Option Private Module
Private Assert As Rubberduck.AssertClass
Private Fakes As Rubberduck.FakesProvider
'@ModuleInitialize
Public Sub ModuleInitialize()
Set Assert = CreateObject("Rubberduck.AssertClass")
Set Fakes = CreateObject("Rubberduck.FakesProvider")
End Sub
The Fakes API implements three of the four stubs for us, so we still need an implementation for ILogger, but now the method remains fully testable even with direct MsgBox, Now and Rnd calls:
Public Sub TestMe(ILogger Logger)
If MsgBox("Print random number?", vbYesNo + vbQuestion, "Test") = vbYes Then
Logger.LogDebug Now & vbTab & Rnd * 42
Else
Logger.LogDebug Now
End If
End Sub
With an ILogger stub we could write a test that validates what’s being logged in each conditional branch (or we could decide that we don’t need an ILogger interface and we’re fine with tests actually writing to the debug pane, and leave Debug.Print statements in place), but let’s just stick with the same two tests we wrote above without the Fakes API. They look like this now:
'@TestMethod
Public Sub TestMe_WhenPromptYes_GetsNextRandomValue()
' Arrange
Fakes.MsgBox.Returns vbYes
' Act
Module1.TestMe New LoggerStub ' ILogger is irrelevant for this test
' Assert
Fakes.Rnd.Verify.Once
End Sub
'@TestMethod
Public Sub TestMe_WhenPromptNo_DoesNotGetNextRandomValue()
' Arrange
Fakes.MsgBox.Returns vbNo
' Act
Module1.TestMe New LoggerStub ' ILogger is irrelevant for this test
' Assert
Fakes.Rnd.Verify.Never
End Sub
We configure the MsgBox fake to return the value we need, we invoke the method under test, and then we verify that the Rnd fake was invoked once or never, depending on what we’re testing. A failed verification will fail the test the same as a failed Assert call.
The fakes automatically track invocations, and remember what parameter values each invocation was made with. Setup can optionally supply an invocation number (1-based) to configure specific invocations, and verification can be made against specific invocation numbers as well, and we could have a failing test that validates whether Randomize is invoked when Rnd is called.
API Details
The IFake interface exposes members for the setup/configuration of fakes:
Name
Description
AssignsByRef
Configures the fake such as an invocation assigns the specified value to the specified ByRef argument.
Passthrough
Gets/sets whether invocations should pass through to the native call.
RaisesError
Configures the fake such as an invocation raises the specified run-time error.
Returns
Configures the fake such as the specified invocation returns the specified value.
ReturnsWhen
Configures the fake such as the specified invocation returns the specified value given a specific parameter value.
Verify
Gets an interface for verifying invocations performed during the test. See IVerify.
The members of Rubberduck.IFake
The IVerify interface exposes members for verifying what happened during the “Act” phase of the test:
Name
Description
AtLeast
Verifies that the faked procedure was called a specified minimum number of times.
AtLeastOnce
Verifies that the faked procedure was called one or more times.
AtMost
Verifies that the faked procedure was called a specified maximum number of times.
AtMostOnce
Verifies that the faked procedure was not called or was only called once.
Between
Verifies that the number of times the faked procedure was called falls within the supplied range.
Exactly
Verifies that the faked procedure was called a specified number of times.
Never
Verifies that the faked procedure was called exactly 0 times.
Once
Verifies that the faked procedure was called exactly one time.
Parameter
Verifies that the value of a given parameter to the faked procedure matches a specific value.
ParameterInRange
Verifies that the value of a given parameter to the faked procedure falls within a specified range.
ParameterIsPassed
Verifies that an optional parameter was passed to the faked procedure. The value is not evaluated.
ParameterIsType
Verifies that the passed value of a given parameter was of a type that matches the given type name.
The members of Rubberduck.IVerify
There’s also an IStub interface: it’s a subset of IFake, without the Returns setup methods. Thus, IStub is used for faking Sub procedures, and IFake for Function and Property procedures.
When to Stub Standard Library Members
Members of VBA.FileSystem not covered include EOF and LOF functions, Loc, Seek, and Reset. VBA I/O keywords Name, Open, and Close operate at a lower level than the standard library and aren’t covered, either. VBA.Interaction.CreateObject and VBA.Interaction.GetObject, VBA.Interaction.AppActivate, VBA.Interaction.CallByName, and the hidden VBA.Interaction.MacScript function, aren’t implemented.
Perhaps CreateObject and GetObject calls belong behind an abstract factory and a provider interface, respectively, and perhaps CallByName doesn’t really need hooking anyway. In any case there are a number of file I/O operations that cannot be faked and demand an abstraction layer between the I/O work and the code that commands it: that’s when you’re going to want to write stub implementations.
If you’re writing a macro that makes an HTTP request and processes its response, consider abstracting the HttpClient stuff behind an interface (something like Function HttpGet(ByVal Url As String)): the macro code will gain in readability and focus, and then if you inject that interface as a parameter, then a unit test can inject a stub implementation for it, and you can write tests that handle (or not?) an HTTP client error, or process such or such JSON or HTML payload – without hitting any actual network and making any actual HTTP requests.
Until we can do mocking with Rubberduck, writing test stubs for our system-boundary interfaces is going to have to be it. Mocking would remove the need to explicitly implement most test stubs, by enabling the same kind of customization as with fakes, but with your own interfaces/classes. Or Excel’s. Or anything, in theory.
The next major version of Rubberduck is currently in [very] early development stages – saying that there is a lot of work ahead would be quite an understatement, but the skeleton is slowly taking shape, and things are looking very, very good.
Since the beginning of the project, Rubberduck’s user interface components (other than dialogs) have always been hosted in traditional, native dockable toolwindows. We built everything on top of the VBIDE editor, using Office CommandBar UI to simulate a status bar and make up for the lack of in-editor integration. Over the years this early design decision slowly became a burden: tearing down the many dockable toolwindows contributed to a pesky access violation crash on exit, low-level hooks for keyboard shortcuts constantly need to detach and re-attach as focus switches between the VBE main window and other applications, autocompletion/self-closing pairs was a nightmare to implement, and while the all-or-nothing approach to parsing made it so that we could always assume we were looking at valid VBA code that could be compiled, it also painted us into a corner where actually moving towards what we wanted Rubberduck to achieve by v3.0 would be extremely difficult, if not impossible.
Behold, the Rubberduck Editor
Rubberduck’s input was always driven by the Visual Basic Editor – now the code in the VBE is going to be output by Rubberduck. Of course, the code will go both ways, but now hidden attributes probably won’t need to be hidden anymore, and the editor can now be exactly what we envision it to be.
There will only be a single toolwindow that will host the editor and UI components like the Code Explorer. At this early stage my focus is entirely on the editor itself, but the idea is ultimately to get actual document tabs and a more practical and friendly docking manager.
Here’s what it looks like as of this writing:
The dropdowns don’t have a real item source yet, but the mock data gives a good idea of what it’s going to be like to edit VBA code with Rubberduck in the future.
Typing “Sub” and hitting the spacebar immediately completes the block and places a new folding node:
The faint dotted underline under “Sub” is a text marker; the editor has the ability to display various such markers at the exact desired position in the document, so we will be using them to show inspection results right there – with tooltips:
Hint-level results will be denoted with this dotted underline indicator; suggestion level will be a green squiggly underline, warnings a blue squiggle, and error level results will appear as red squiggles:
There will also be a new “ducky button” that pops up when the caret is on one such marker, and lets you pick a quick-fix in-place to address an inspection result:
The indenter still needs to be wired up, but this editor will ultimately indent your code as you type it. All the autocompletion features also need to be ported over to work here, and then we’ll want searchable and filterable IntelliSense, parameter info tooltips, and we’ll need to simulate the VBIDE “prettification” that occurs when a line is validated, so that public sub becomes Public Sub and identifiers take the casing they’re declared with.
We get an undo stack that can handle much more than 20 steps, and did I mention the status bar?
For now, all it does is report the current caret position in the editor, but Rubberduck 3.0 will be using it to report parsing progress, instead of the CommandBar button/label we’ve been abusing forever.
There will probably still be a command bar of some sort, but it will be part of the WPF/XAML managed UI; the old Rubberduck CommandBar will be decommissioned.
The one thing that’s 100% guaranteed to not happen in the new Rubberduck editor, is everything that needs to happen beyond design-time: there is no hook into the VBIDE debugger, so Rubberduck has no way of tracking the current instruction. As a result, the editor will be sadly useless in debug mode.
The editor work is just the beginning: Rubberduck 3.0 currently doesn’t even have a parser, let alone any inspections. In the next few months, the very heart of Rubberduck will be reworked to function with the new editor. It’s essentially like rewriting Rubberduck, but with an editor we fully control instead of one we constantly need to fight with.
Meanwhile v2.5.2 is approaching 25K downloads, and there’s quite a bit of work in 2.5.x that hasn’t been “officially” released yet, including everything that happened during a very successful Hacktoberfest 2022: we’ll be releasing v2.5.3 in the near future – stay tuned!
As I wrote last July, I’ve started to get more time for myself lately, and that means I get to tackle a number of long-standing projects that have been on the backburner for way too long. One of them is the rewrite of the project’s website, which has been “under construction” ever since it was published as an ASP.NET MVC website, a few years ago already.
If you missed it, I tweeted a sneak-peek link last week:
Tweeted 09/28: “A couple of things need a bit of work still, but this website rewrite is coming along nicely – have a peek here: https://test.rubberduckvba.com“
Why a rewrite?
For the longest time, I wouldn’t have considered myself a web developer. I have well over a decade of experience in C# desktop development, but the web stuff essentially scared me to death. The version of the website that’s currently live was pretty much my first time doing anything like it. The site itself wouldn’t write to the database; it was another application that pulled the tag metadata, downloaded the xml-doc assets, parsed the documentation and examples, and wrote them to the database.
One of the biggest issues with the current model, is that the database is made to contain HTML that is needlessly difficult to modify:
Unreachable code is certainly unintended, and is probably either redundant, or a bug.
<div><h5>Quick-Fixes</h5>
<p>The following quick-fixes are available for this inspection:</p>
<ul style="margin-left: 8px; list-style: none;">
<li>
<span class="icon icon-ignoreonce"></span>
<a href="https://rubberduckvba.com/QuickFixes/Details/IgnoreOnce">IgnoreOnce</a>
: Adds an '@Ignore annotation to ignore a specific inspection result. Applicable to all inspections whose results can be annotated in a module.</li>
<li>
<span class="icon icon-tick"></span>
<a href="https://rubberduckvba.com/QuickFixes/Details/IgnoreInModule">IgnoreInModule</a>
: Adds an '@IgnoreModule annotation to ignore a inspection results for a specific inspection inside a whole module. Applicable to all inspections whose results can be annotated in a module.
</li>
</ul>
</div>
Having this HTML markup, CSS classes, and inline styles as part of the data meant the data was being responsible for its own layout and appearance on the site. With the new JSON objects serialized into this Properties column, I could easily keep everything strongly typed and come up with separate view models for inspections, quick-fixes, and annotations, that each did their own thing and let the website in charge of the layout and appearance of everything.
Separation of Concerns
The solution architecture could be roughly depicted like this – I suppose I meant the arrows to represents “depends on” but note that this doesn’t necessarily mean a direct project reference: the Client/API relationship is through HTTPS, and no project in the solution references the Rubberduck.Database SQL Server database project, but ContentServices connects to a rubberduckdb database that you can deploy locally using that database project:
You could draw a thick red line between Rubberduck.Client and Rubberduck.API (actually that’s Rubberduck.WebApi now), and it would perhaps better illustrate the actual wall between the website and the data: the website project doesn’t need a connection string, only a base URL for the API!
Authentication is assured with GitHub’s API using OAuth2: if you authorize the rubberduck-vba OAuth application to your profile, the HttpContext.User is cast as a ClaimsPrincipal and claims the GitHub login as a name, and a rubberduck-orgroleclaim is added when organization membership is validated; an additional rubberduck-admin role claim is added if the user is also a member of the WebAdmin org team.
The website packages the HttpContext.User into a Json Web Token (JWT), an encrypted string that encapsulates the claims; this token is passed as a bearer token in authenticated API requests. The API accepts an Authorize header with either such a bearer token, or a valid GitHub personal access token (PAT).
The API receives a request, and given an Authorization header, either decrypts the JWT or queries GitHub to validate the provided access token and attach the appropriate role claims, before any controller action is invoked.
Another authentication filter performs a similar task to authorize an incoming webhook payload: the rubberduck-webhook role is set and tag metadata and xml-doc content can get updated automatically whenever a new tag/release gets created.
Performance
This new website performs much, much better than the current one. It sends asynchronous (ajax) requests to the MVC controller to render partial views, fetching only enough information to paginate the data and present a decent preview. Since most pages are presenting markdown content, an asynchronous request is also sent to format the markdown and, if applicable, apply syntax highlighting to code blocks. At this stage static content isn’t being cached yet, and screenshots should be loaded dynamically – still, performance is quite decent:
Home page scores 94, but then both Code Inspections and Inspections pages (two pages with extensive content, lots of markdown, code blocks, etc.) score a full 100 with Google Lighthouse, so things are looking very good performance-wise.
Another detail: the code examples no longer trigger a page load when you select a tab, so everything just feels much smoother now. Note, as of this writing the example records have been wiped from the database while I work on fixing a problem with the xml-doc processing, so annotations, inspections, and quick-fixes aren’t showing any examples on the test site for now.
Online Indenter
This feature once worked, but then my inexperienced past self, went and broke it in an attempt to make it asynchronous. Well, it’s back online and running Rubberduck.SmartIndenter.dll version 2.5.2:
You can paste VBA code into the box there, click the Indent button, then copy the indented code back into the clipboard.
The code can be indented as per the default indenter settings (which are also used for indenting all syntax-highlighted code blocks on the site), or if you expand the Indenter Settings panel you can tweak every knob Rubberduck’s Smart Indenter port has to offer.
It wouldn’t be too hard to include a “download these settings” button here, to serialize the settings into a .xml file that Rubberduck can then import to update indenter settings.
Content Administration
Users with the appropriate claims will be able to see additional buttons and commands on the site:
A modal dialog allows authenticated users to add and edit markdown content directly on the site.
Content administration features still need a little bit of work, but they are already being used to document how to use each and every single feature in Rubberduck – once this documentation is completed, the site will be a huge user manual, and ready for launch!
What’s Next?
Once everything works as it should (getting very close now!) and all that’s left to do is to take screenshots and generate more content, I’ll shift my focus to the Rubberduck3 project, the ownership of which I’ve now transferred over to the rubberduck-vba organization – the repo remains private for now, but all Rubberduck contributors have access to it. Uploading the RubberduckWebsite solution as a public repository isn’t a priority at this point; I feel like dealing with the implications of having API secrets in a .config file would be a distraction that I don’t need right now. When the time comes, it’ll be properly setup with continuous integration and deployment, but there are other priorities for now.
Version 2.5.1 was released August 22, 2020. Since then, the installer was downloaded over 11,600 times; we are now 420 commits and 650 modified files later, and the time has come to deliver all that work into a convenient little installer package and move on to the next dev/release cycle.
What’s New?
If you’ve kept up with latest pre-release builds (especially in the last few weeks), nothing much. If you’ve been patiently waiting for the next release, you’re in for a treat!
The first thing you’ll probably notice is the shiny new splash screen design:
It’s the same old yellow splash made with Paint.NET, with a tiled reflection distortion effect against the background, a semi-transparent white bottom panel, and a finer font. Do you like it?
50-some issues labelled “bug” were closed between 2020-08-22 and mid-April 2021, many of them thanks to flicking the switch on leveraging our internal ITypeLib API for user code – thanks to earlier invaluable contributions from the amazing Wayne Phillips (vbWatchdog, twinBASIC), Rubberduck is now able to tap into the actual in-memory COM type library compiled from the VBA code and, eventually, fill the remaining the gaps in Rubberduck’s understanding of the code: Rubberduck now understands enough to be able to tell that ThisWorkbook has a _Workbook subtype, and that Sheet1 has a _Worksheet subtype, …and that’s enough to identify the ThisWorkbook module at long last, and as a result Rubberduck’s ImplicitActiveSheetReference and ImplicitActiveWorkbookReference inspections get to work exactly as intended, and the door is now opened for so many interesting things…
New Inspections
A Rubberduck release wouldn’t be a Rubberduck release without at least a handful of new inspections. The IllegalAnnotation inspection is being replaced by InvalidAnnotation, UnrecognizedAnnotation, and together with the new AnnotationInIncompatibleComponentType inspection they allow Rubberduck to better convey exactly what’s wrong with a given “illegal” annotation comment.
Some annotations cannot be used in certain types of modules. For example, attribute-related annotations cannot be used in document modules (because Rubberduck cannot import back the modified modules), and a @TestModule annotation is only meaningful in a standard module.
Note that the @Description, @ModuleDescription and @VariableDescription annotations do work in document modules now, because Rubberduck is now reading docstrings off annotations rather than hidden attributes.
Code in the ThisWorkbook module (Excel) referring to members of the Workbook class, have an implicit Me qualifier. This makes an unqualified Worksheets(1) retrieval in ThisWorkbook refer to ThisWorkbook.Worksheets(1), but an identical statement in any other module would be (implicitly) referring to ActiveWorkbook. By qualifying such member calls with Me, the intent is clarified.
Code in a worksheet module (Excel) referring to members of the Worksheet class, have an implicit Me qualifier. This makes an unqualified Range member call in the Sheet2 module refer to Sheet2, but an identical statement in any other module would be (implicitly) referring to ActiveSheet. By qualifying such member calls with Me, the intent is clarified.
Flags unbound annotations; that is, annotation comments that were correctly parsed as Rubberduck annotations but that could not be associated with a target element. This would happen when a module annotation is used in local scope, or a member annotation at module level. This inspection only flags annotation comments that parsed as Rubberduck annotations.
The RHS/Value parameter of a Property Let procedure is always passed by value. As such, an explicit ByRef modifier on such a parameter definition is misleading. From MS-VBAL (VBA language specifications) section 5.3.1.7 Property Declarations:
§ If the <value-param> of a <property-LHS-declaration> does not have a <parameter-mechanism> element or has a <parameter-mechanism> consisting of the keywordByRef, it has the same meaning as if it instead had a <parameter-mechanism> element consisting of the keywordByVal. § The <value-param> of a <property-LHS-declaration> always has the runtime semantics of a ByVal parameter.
This inspection flags comments that parsed like a Rubberduck annotation, but aren’t recognized or supported. It picks up typos in Rubberduck annotations, and annotation-like comments that aren’t Rubberduck annotations but parse as such. Splicing this specific scenario from other invalid annotations is particularly useful when you want to mute inspection results for non-Rubberduck annotations while still validating the supported ones.
New Quick Fixes
This release also introduces a handful of new quick-fixes:
This fix is now available for ProcedureNotUsed inspection results in standard and document modules; it simply annotates a member with the new @EntryPoint annotation which specifically instructs ProcedureNotUsed to ignore that member. Use this quick-fix for UDFs and macro procedures that are attached to document objects and don’t need an Excel hotkey. If your project is hosted in an Excel workbook, macros annotated with @ExcelHotkey are also considered as entry points now.
VariableTypeNotDeclared inspection results could always be “fixed” by making the declared type an explicitVariant; this new quick-fix makes Rubberduck infer the declared type from usage where possible, which is objectively awesome.
This new quick-fix is available for the new implicit containing workbook/worksheet reference inspections, making the reference to the containing module explicit.
The Write-Only Property inspection gets a new quick-fix with this release; this iteration does not try to infer the backing field, so further manual edits are needed, but it’s a start.
New UI Language: Italian
Thanks to a timely contribution by @PhilCattivocaratere, we are thrilled to announce that this release introduces Italian as a UI language:
Every single UI string in Rubberduck comes from a localized resource file. Translating all the resources for a new language can take 3-5 hours, and then it’s only a matter of keeping the translations up-to-date by creating a small pull request when new resource strings are added for new features.
In a nutshell
Here’s a quick summary of the most significant pull requests and commits merged this cycle:
Encapsulate Field enhancements
We are now leveraging our internal ITypeLib API
We are now building Rubberduck with the latest version of Visual Studio 2019
Precompiler directives now parse correctly with line continuations
Internal CodeBuilder API honors indenter settings when generating code
Fixed a number of issues with name conflict validation
Test methods now support a @TestIgnore annotation to ignore a test
Specific projects can now be ignored by the parser
Users no longer need to accept the GPLv3 as if it were an End User License Agreement (EULA)
Custom templates extensions is changing from .rdt to .template
Implicit Variant inspection quick-fix will now infer the best type from usage instead of just making the variable an explicitVariant
For...Next loop variables no longer trigger a variable not used inspection
Implicit Public Member inspection will now flag Enum types and Type structures
Branch “master” was renamed to “main”
New Property Group indenter settings
Arrays declared with ReDim now correctly resolve the declared type
@Description, @VariableDescription, and @ModuleDescription can now be used in document modules (cannot be synchronized)
Documentation strings are now read from annotations when missing from attributes
Start menu link to website now uses https
Fixed context menu positioning
New @EntryPoint annotation marks a standard or document module member as invoked from outside the code; as such the Procedure Not Used inspection will no longer flags members annotated with @EntryPoint or @ExcelHotkey (Excel only).
Several other opportunistic fixes left & right, improved overall stability.
Shiny new splash screen; debug builds now indicate “debug” instead of a meaningless local build number (build version# is controlled by the AppVeyor CI build server; local builds are all .0).
Expand/collapse all in Code Explorer
Rubberduck CommandBar label will now show the corresponding parameter declaration for a selected argument, and Find all References will now include arguments at call sites for parameter declarations (previous versions would only count named arguments).
Find Symbol navigation tool works again.
Find all References search results will now highlight the target reference in its context.
Added Italian UI resources.
Possible (Silent) Crash on Exit
I haven’t personally experienced it in a long time in Excel, but Rubberduck may run into issues tearing down, sometimes causing an AccessViolationException when it unloads, which can either crash the host process or leave it hanging as a ghosted process that will interfere with reloading: verify that the host process (e.g. ACCESS.EXE) has shut down completely using Task Manager when you close everything, and make sure to kill any such ghosted processes before loading Rubberduck in a new process.
Sounds familiar? If you’ve been following the project all along, you probably remember similar behavior in earlier releases – at one point during this development cycle we thought the problem was finally under control, but the cure was worse than the disease and there was a chance that the host document / project gets completely corrupted and impossible to open in the VBE: because we think it’s much better to sometimes crash on teardown than to corrupt our users’ host documents forever, we have reverted that “fix” and will have to come up with something else.
What’s Next?
Lots of good stuff, including a new peek definition command to the code pane, Code Explorer, and the VBE’s own Project Explorer‘s context menus – the feature was developed too late to make the cut for this release, but will be available in 2.5.2.x pre-release builds very soon:
Peek Definition commands pop a panel that shows you the syntax-highlighted source code for a type or member. The pop-up panel can then be dragged around to keep it in sight while editing.
In the Unit Testing department, a mocking framework is about to debut as an experimental feature with a number of technical limitations.
I’m going to be turning my attention towards code path analysis this cycle; this internal API is needed to implement the more advanced inspection ideas, and an Extract Method refactoring needs it too.
Here we are again, some 580+ commits and 1000+ modified files later, with 10 contributors involved (with particular thanks to @MDoerner and @BZngr, and honorable mentions to @IvenBach and @testingoutgith1) in over 60 pull requests since the last release: time to look back at what was done and call it version 2.5.1! If you’ve been keeping up with pre-release builds, none of this is going to be news to you, but with over 9.1K downloads of v2.5.0 a lot of you seem to prefer to upgrade less often but more significantly, so here’s a timely recap.
But first, let’s get the known problems out of the way.
Known Issues
Making a VBIDE add-in means we can’t know or assume what our host application is going to be, and different hosts sometimes wire things up differently – and this can spell trouble under certain circumstances. Making a VBIDE add-in in .NET has even further implications: while it’s how we can extend a 64-bit VBE, it’s also causing various type cast errors/exceptions when other add-ins are loaded.
Possible Crash
The Visual Basic Editor has a peculiar way of loading its add-ins: Rubberduck’s (and any other VBIDE add-in’s) entry point is invoked by the VBE before the VBE has completely finished constructing itself – accessing the object model too early can throw COM exceptions that take down Rubberduck as it initializes.
Normally Rubberduck initializes itself, then proceeds to parse the project (if it’s an empty project then the bulk of that is Rubberduck loading everything defined in VBA7.DLL and the type library for the host application’s object model) – normally if the VBE isn’t ready for this yet, we bail out and don’t access any objects and the “Refresh” button says “Pending” instead of “Ready”, and by the time you manually run that command the VBE has finished initializing and the only annoyance is that the initial parse isn’t automatic.
But in certain host applications (Microsoft Access being a known one, but I’ve seen it happen in Excel as well, although not with a recent build), sometimes the VBE actually isn’t ready to take member calls against its own object model, and the result is a COM exception that is either caught and then Rubberduck says it can’t initialize, or thrown several layers deeper, uncaught, and then everything goes up in flames.
Loading Rubberduck manually from the VBE’s Add-Ins Manager is sure annoying, but is really the only 100% sure-shot way to load any VBIDE add-in with a properly initialized VBE, regardless of the host application. Note that the installer registers Rubberduck as a VBE add-in with the LoadBehavior flag set to load at startup. If Rubberduck blows up at startup or fails to initialize, consider editing this configuration to make it load manually (exception details should normally be logged for the first start-up).
Heavy on Memory (RAM)
Rubberduck has always used a lot of memory to keep all the code metadata handy and cache a lot of things to improve processing performance. Working on a large legacy project that generates lots of inspection results can grind the main thread of the host process to a halt as the toolwindow renders the many objects (whether the toolwindow is displayed or not).
Unless you are discovering Rubberduck with a new, empty VBA project, consider first reviewing the settings – can’t hurt to review them either way:
Disable “run inspections automatically on successful parse”, so that they only run if you explicitly refresh them from the Inspection Results toolwindow;
Set inspection severity to “Do not Show” for inspections that could produce thousands upon thousands of results, like “use meaningful names” if you’re into Hungarian Notation for example, or “use of bang operator” if that’s the only way you’re ever accessing recordset fields in Access;
Other general performance tips:
Rubberduck parses per-module, so when you leave a module after modifying it, trigger a parse – by the time you’re in the other module and have scrolled to where you want to be and are in that mindset, the modified module will have processed.
Reduce coupling: the more modules are inter-dependent, the more modifying a module requires re-resolving identifier references in the dependent modules.
Avoid complex grammar: bang operators, among other code constructs, are somewhat ambiguously defined and ultimately parse in two passes, with the first one failing. The standard member call syntax parses faster, in a single parser pass.
Undesirable Interactions
If you are using the free but rather old 32-bit MZ-Tools 3.x productivity add-in, this section shouldn’t be a concern. However MZ-Tools 8.x was rewritten from the ground up, ported from VB6 into .NET-land, and while its author Carlos Quintero took extraordinary steps to isolate MZ-Tools from other in-process .NET add-ins and has issued recommendations for Rubberduck to do the same, …there is still a chance the two add-ins bump into each other; if MZ wins, RD is essentially bricked.
MZ-Tools normally runs inside its own .NET AppDomain, except when hosted in AutoDesk products (Inventor, AutoCAD), which implement VBE initialization in a way that breaks MZ-Tools’ startup mechanics – up until recently it was assumed this collision only happens in AutoDesk hosts, but a recent support ticket involving Microsoft Access was filed and implicates interactions with MZ-Tools.
This issue manifests itself with InvalidCastException being thrown at various points, often during initialization, or later during parse: the exception message involves attempting to cast COM objects like Microsoft.Vbe.interop._VBProject into types such as VBClassicExtensibility.VBProjectClass, where VBClassicExtensibility is defined by MZ-Tools, not Rubberduck.
One thing that can be attempted to mitigate this problem, would be to set MZ-Tools to not load on start-up, and manually load it after Rubberduck has initialized… but sadly this cross-add-in confused COM marshaling is simply not supposed to happen given MZ-Tools’ AppDomain mechanics, and we don’t really have any solutions for this – same as we don’t really have any solution for cases where COM registrations are broken (e.g. when multiple Microsoft Office product versions are running side-by-side but were not installed in chronological order – that’s an officially unsupported scenario, per Microsoft).
As a result, using Rubberduck together with other .NET-based add-ins cannot be considered a completely fail-safe scenario, and we have to treat this as a “known issue” here, and the work-around sucks and boils down to “drop other add-ins, or drop Rubberduck”. This is actually probably true at various degrees of all .NET-based VBIDE add-ins.
On the bright side, we have taken several steps in this release cycle to prepare the ground not only to get Rubberduck to build correctly in the latest & greatest Visual Studio 2019, but also to get most of our build process ready for .NET Core – so when .NET Core 5 is released in a few weeks, we can try to get Rubberduck to run on the shiny new Core framework, which theoretically makes AppDomain completely moot, and so we have very little incentive to work on getting Rubberduck to load its own AppDomain the way MZ-Tools does: if we can make Rubberduck build and run on .NET Core 5, then this problem should simply disappear… in theory.
Enhancements & New Features
This release does not introduce any new top-level Rubberduck features, but makes a number of very useful user-facing additions nonetheless, on top of the many under-the-hood enhancements made this cycle.
Surfacing Annotations
One of the most useful and powerful features of Rubberduck, annotations are special comments that use a particular but relatively simple syntax – these are all grammatically valid:
'@AnnotationName("text")
'@AnnotationName "text"
'@AnnotationName("text", 123) : there can be comments here
'@AnnotationName "text", 123
'@AnnotationName Identifier1, Identifier2, ...IdentifierN
While the syntax itself is reasonably simple to use, the problem was that unless you knew every supported annotation, well then the @AnnotationName part kind of had to be a guess.
Rubberduck uses these annotations for various purposes, from identifying Rubberduck test modules to keeping hidden module/member attributes in sync with these comments (this includes the ability to document and literally map Excel hotkeys using VBA comments). You can read everything we’ve documented about them on the project’s website.
In Rubberduck 2.5.1.x builds, we finally get new commands in the code pane and Code Explorer context menus, that bring up a dialog that gives us all the options to easily and safely annotate everything that can be annotated, using the correct syntax and arguments every time:
Select the annotation to add, and supply the argument values. Magic!
Adding an @Ignore annotation to ignore a specific inspection, can now be done without needing to know the exact name we decided to call that inspection class in Rubberduck’s source code!
Encapsulate Field Enhancements
This particular refactoring has seen a terrific enhancement that makes it very easy to cleanly and quickly turn a set of public fields into Property Get/Let members, with a Private Type TClassName and a module-scope Private this As TClassName instance variable – and all properties automatically reading/writing from it. You can see this feature in action in the previous article.
The new Wrap fields in Private Type functionality leverages the very useful Private Type pattern.
Unit Testing
The Test Explorer now makes it easier to ignore one or more specific selected tests, or all tests under a given category/group, by exposing the context menu commands that add or remove the @TestIgnore annotation as appropriate; having this command in the Test Explorer makes it possible to annotate a test method while a completely different and unrelated module is maximized in the code editor.
Hmm, …these icons are out of control, aren’t they… expect that to change soon-ish…
Running tests while results are regrouped by outcome is still a known issue (tests run painfully slow because the UI thread is busy re-sorting and re-rendering the list every time a test finishes running), but everything works much more smoothly when the tests are regrouped any other way.
Code Inspections
Ok the logic for that is currently broken on the website (working on that… somewhat) so this is much harder than it will be in the future when I’ll just look at the [New] tab on the inspections page of the website and every inspection that is in [next] but not in [main] will be listed right there. From skimming through every pull request merged since the last release:
Function return value is always discarded inspection is the old “return value not used” logic targeting the function itself, when none of the call sites capture the function’s return value.
Implicitly typed const inspection was added to flag Const declarations without an As clause to specify an explicit type.
Assignment not used inspection now correctly handles an assignment that is overridden in the next statement but first read in the RHS expression of the assignment.
Not user-facing but critically important nonetheless, is all the behind-the-scenes work done to simplify inspecting VBA code as much as possible. This cycle saw a tremendous amount of technical debt paid in the code inspections department, that pave the way for future enhancements like, say, having the ability to run inspections per-module; as the number of implemented inspections continues to grow, the ability to scope inspections in a more granular way is going to be very useful for our plan to eventually report inspection results in a custom code pane, with colored squiggly lines (that’s v3.x stuff, though).
Applying Quick-Fixes
The Code Inspections toolwindow has been updated with a context menu that makes it much simpler to apply a quick-fix to one or more inspection results; all available quick-fixes appear in the context menu, each with various options to apply to the selection. This menu is also shown by clicking the “Fix” drop-down menu from the toolwindow’s toolbar.
The weird “fix all occurrences in procedure/module/project” link buttons in the bottom panel are now gone, completely replaced with a more flexible menu system.
Code Explorer Enhancements
The Code Explorer toolwindow context menu now includes a move to folder command to easily organize your project components, and there’s a new setting to enable drag and drop in Code Explorer (disabled by default):
A new setting enables dragging & dropping capabilities in the Code Explorer toolwindow.
With that setting enabled, you can now move a code file to an existing folder, simply by dragging it from its location and dropping it onto a folder node! The setting was made to require being explicitly enabled, to avoid discovering that feature by accidentally dragging a code file somewhere (that did happen in beta/testing).
Website Integration
You may have noticed rubberduckbva.com is “under reconstruction”. Before that, the site’s content was mostly static, with only the inspections list assembled from content parsed from the Rubberduck.CodeAnalysis.xml, downloaded periodically off GitHub through unauthenticated REST API requests. That worked relatively well until a spike in traffic occurred following the release of Excel Insights, a book collectively authored by 24 Microsoft MVP Award recipients, including myself: suddenly a bug in the caching mechanism became very apparent when the site’s home page started getting served as a wonderful HTTP 500 error page.
Since the website hosting came with a SQL Server database that I wasn’t using, I decided to start using it and make the site pull the content from there rather than directly off the GitHub API. I wrote a small console application, got myself a private API key to make properly authenticated REST API requests, and now there’s a scheduled task running on a virtual machine in my garage, that runs this application every 30 minutes to update the installer asset download counts and verify whether the XML documentation assets are up-to-date for the latest pre-release build, and then proceeds to parse the XML docs and generate/update the database records: the website simply pulls the data from the database at every request, and now the website couldn’t bust GitHub’s REST API limits even if it tried.
Documenting Rubberduck is challenging: there are a lot of features, and there isn’t really any user guide that’s constantly being kept up-to-date. The wiki on the repository is terribly outdated in several parts, and the feature announcements on this blog are nice when you’re following the project along its journey, but in a dream world using Rubberduck would be content found on the website, and contributing to Rubberduck would be content found in the repository’s wiki.
Parsing the xml-docs into website content is a step in that direction. Nobody wants to maintain documentation, but xml-doc comments are part of the source code, and we even put source code analyzers in place that will break the build if we try to introduce an inspection, quick-fix, or annotation, without properly documenting it with xml-docs.
Every single inspection, quick-fix, and annotation has thorough documentation, including code examples that may span multiple modules. But best of all, every single page generated from source code includes an “edit this page” link that points to a GitHub page where you can literally edit the xml documentation for the inspection you were looking at (and review its source code if you like – it’s the same file!) – and just like that, all you need to contribute to Rubberduck (yes, single-character typo fixes and additional useful code examples are welcome!) is a GitHub login!
Every page generated from xml documentation includes an “Edit this page” link at the bottom.
Because of how the request routing on the website was setup, it was easy to make Rubberduck link in-app inspection results to this website content – you can now click a URL at the bottom of the inspection results toolwindow (this will likely change one way or another in the future) to bring up the details page with the xml documentation and code examples:
The URL at the bottom of the adjustable panel (it’s possible you need to scroll the content or make the bottom panel tall enough to show it) opens that URL in a new tab in whatever your default browser is.
The in-app content exists as localized resources, lovingly translated by our international contributors; the website content however, is only available in English, because we’re absolutely not going to start translating XML comments in the source code. But the processed content actually resides in a database, so it wouldn’t be impossible to eventually localize it at that level, as well – we’re just not there yet at all at the moment.
The website content is often different than the in-app content, and over time it should be expected to grow more and more in-depth, thorough and descriptive.
The revamped rubberduckvba.com domain will ultimately span 3 sites, including api.rubberduckvba.com, which will eventually expose REST endpoints for various purposes, including Rubberduck’s “check for newer version at startup” feature; for example something like api.rubberduckvba.com/indenter.json that might accept some VBA code in a JSON object in the request’s body, and respond with a JSON object in the response body containing the indented VBA code. Or api.rubberduckvba.com/inspect.json that might also accept some VBA code (presumably along with some metadata about the module type) in a JSON object in the request’s body, but could respond with a JSON object representing all inspection results for it. It’s still all just brewing ideas at this stage. The other subdomain, admin.rubberduckvba.com, is going to host a web-based, GitHub-authenticated version of the VBA program I’m going to present in my next article: a tool for managing and editing most of the website’s content.
Moving Forward
Rubberduck is becoming a pretty mature code base and has an ever-increasingly better abstracted internal framework/API to understand and manipulate VBA code. The project now builds with the latest version of Microsoft Visual Studio 2019, and we’re hoping COM Automation support in .NET Core 5 will allow us to build an increasing number of the project’s components with it; I’m thinking the “main” type library is better off under the old tech, but I’ll be more than happy to be proven wrong here!
A rough roadmap for v2.5.1.x might include…
More resolver capabilities unlocked by fully leveraging our internal ITypeLib API
Syntax-highlighted preview of the changes for all refactorings (and quick-fixes?)
Some Code Path Analysis API, to help implement the more complex inspection ideas
The Moq wrapper mocking framework
Block Completion, maybe
Anything else anyone feels like contributing to the project!
The goal for the rest of the 2.x cycle is to prepare everything that needs to happen in order to implement our own custom code editor window – giving us full, complete control over every single token and everything that can possibly happen in that custom code pane. We’re talking code folding, custom theming, that kind of thing.
‘Main’ vs ‘Master’ – Why it Matters
You may have noticed (or not) that the website is now labeling “main” the branch formerly known as “master”. As a French native, “master/slave” terminology in any non-actual master/slave context has always sounded a bit weird to me, but I’m a white man in North America (although not in the US) and I get the luxury to read these words and decide that they don’t affect me, and reflecting on the events of this summer has taught me that this is part of what white privilege is.
I don’t do political & editorial commenting, I prefer to leave that space to others – but I warmly recommend watching 13th on Netflix and, if you can handle it, When They See Us. Black lives matter, it’s simple – and no, it doesn’t say “but white lives don’t” anywhere between the lines.
So yes, we’re going to be taking steps to alter the language in Rubberduck a bit in this cycle. The “master” branch will be renamed to “main”, yes, but we’ll also come up with a better term for “white-listing” identifier names. It won’t stop racism, no, indeed. But it won’t hurt anyone, either.
As was shared a week or two ago on social media, Rubberduck contributor and supporter Andrew “ThunderFrame” Jackson passed away recently – but his love for VBA, his awesomely twisted ways of breaking it, his insights, the 464 issues (but mostly ideas, with 215 still open as of this writing) and 30 pull requests he contributed to Rubberduck, have shaped a large part of what this project is all about, and for this release we wanted to honor him with a special little something in Rubberduck, starting with the splash screen.
Last December we lost a very valued contributor and friend. Next release the Rubberduck splash screen will be in his memory: that devil ducky was his online avatar.
— Rubberduck VBIDE add-in (@rubberduckvba) March 14, 2019
Andrew joined the project very early on. He gave us the signature spinning duckies and the SVG icon of the project; he once implemented a very creative way to make unit testing work in Outlook (and I know a certain duck that had to eat their hat because of it!), before the feature was made host-agnostic. He gave us the weirdest, most completely evil-but-still-legal VBA code we could possibly test Rubberduck’s parser/resolver with – and we’re very proud to have a ThunderCode-proof parser now!
What’s New?
This isn’t an exhaustive list. See the release notes for more information.
¡Rubberduck ahora habla español!
This release introduces Spanish language support. German, French, and Czech translations have also been updated.
Rubberduck doesn’t speak your language yet? Nothing would make us happier than helping you help us translate Rubberduck! See contributing.md for all the details, and don’t hesitate to ask any questions you have – go on, fork us!
The project’s many resource files are easily handled with the ResX Manager add-in for Visual Studio.
UI Enhancements
The Test Explorer has had a rather impressive facelift, Inspection Results are now much easier to review, navigate and filter. There is a known issue about the GroupingGrid control expanding/collapsing all groupings together, but we weren’t going to hold back the release for this – we will definitely address it in a near-future release though.
Toggle buttons in the toolbar now allow filtering inspection results by severity, and grouping by inspection type, by module, by individual inspection, or by severity. Similar toggle buttons in the Test Explorer allow grouping tests by outcome, module, or category. Tests can be categorized by specifying a category name string as an argument to the @TestMethod annotation.
Parser performance has improved, especially for the most common usages of “bang” (foo!bar) notation, which remain a difficult language construct to handle. But they’re also late-bound, implicit, default member calls that would probably be better off made explicit and early-bound.
Self-Closing Pair completion works rather nicely now, with only two known, low-priority edge cases that don’t behave quite as nicely as they should.
Easter Is Coming
And with Easter comes… White WalkersEaster Eggs, so all I’m going to say, is that they’ll be flagging ThunderCode – the kind of code our friend loved to test & push the limits of Rubberduck’s parser with. If your code trips a ThunderCode inspection, …nah, it can’t happen.
Woopsie, might happen after all. We’ll eventually figure out a way to hide them from the settings!
Also it’s apparently not impossible that there’s no way noother Easter Eggs werenever notadded to Rubberduck. For the record I don’t know if this means what I think I mean it to say, and that’s perfect.
What’s Next?
Some very important changes have been waiting for this release and will be merged in the next few weeks – these changes won’t necessarily be visible from a user standpoint, but they will greatly enhance our internal API – refactorings, COM object management, and we’ll be leveraging more of the TypeLibs API, which in turn should end up translating into greatly enhanced user experience and feature set.
Next release will include a few new inspections, including one that locates obsolete While...Wend loops, and suggests to rewrite them as Do While...Loop blocks, which can be exited with an Exit Do statement, whereas While loops can only be prematurely exited (without throwing an error) by an inelegant GoTo jump.
We really want to tighten our release cycle, so we’ll be shooting for the end of April for what should be version 2.4.2.
Unlike quite a number of Rubberduck releases, this time we’re not boasting a thousand commits though: we’re looking at well under 300 changes, but if the last you’ve seen of Rubberduck was 2.3.0 or prior, …trying this release you’ll quickly realize why we originally wanted to release it around Christmas.
So, here’s your belated Christmas gift from the Rubberduck dev team!
VBE Project References: CURED!
You may have seen the Introducing the Reference Explorer announcement post last autumn – well, the new feature is now field-tested, works beautifully, instinctively, and is ready for prime time. It’s a beauty!
The add/remove references dialog has seen a number of enhancements since its pre-release: thanks everyone for your constructive feedback!
Quickly locate any type library by name and/or description.
Pin your favorite references, and Rubberduck will keep them handy for all your VBA/VB6 projects.
You’ll never want to use the vanilla-VBE references dialog again!
If you’ve been following the Rubberduck project for quite some time, you may remember something about using annotations together with inspections and quick-fixes to document the presence of module & member attributes. You may also remember when & why the idea was dropped. Keeping in tradition with including new inspections every release… Surprise, it’s coming back!
German, French, and Czech translations have been updated, a number of bugs were fixed in a few inspections, the Code Explorer has seen a number of subtle enhancements, and WPF binding leaks are all but gone.
Code Explorer Enhancements
Adding the Reference Explorer made a perfect opportunity to revisit the Code Explorer toolwindow – our signature navigation feature. Behold, the new Code Explorer:
The new ‘Sync with code pane’ toolbar button (the left/right arrows icon) selects the treeview node closest to the current code pane selection.
There’s a new ‘Library References’ node that shows your project’s library dependencies …whether they’re in use or not:
Find all references can now be used to locate all uses of a given type library – including the built-in standard libraries! Note that rendering lots of search results in a toolwindow will require confirmation if there are too many results to display.
The project reference nodes get new icons:
Classes with a VB_PredeclaredID attribute set to True now have their own icon too (and their names now say (Predeclared) explicitly), and class modules marked with an @Interface annotation now appear with an “interface” icon, like IGameStrategy here:
Annotations & Attributes
They’re back, and this time it does work, and it’s another game changer: Rubberduck users no longer need to export any code file to modify module & member attributes!
Module & Member Annotations
At module level, the @ModuleDescription annotation can be given a string argument that controls the value of the module’s VB_Description attribute; the @Exposed annotation controls the value of the VB_Exposed attribute; the presence of a @PredeclaredId annotation signals a VB_PredeclaredId attribute with a value of True.
At member level, @Description annotations can be given a string argument that controls the value of the member’s VB_Description attribute.
Through inspections, Rubberduck is now able to warn about attributes that don’t have a corresponding annotation, and annotations that don’t have a corresponding attributes. Look for inspection results under the “Rubberduck Opportunities” category.
v2.4.x
The months to come will see further enhancements in several areas; there are several pull requests lined up already – stay tuned, and keep up with the pre-release builds by watching releases on GitHub!
Back in the 2.1.x announcement post over a whole year ago, one of the bullet points about the upcoming roadmap said we were going to “make you never want to use the VBE’s Project References dialog ever again“; it took a bit longer than expected, but as far as we can tell, this feature does exactly that.
If you’ve been following the project on social media recently, you already know that the next version of Rubberduck will introduce a very exciting, unique new feature: the Reference Explorer dialog, and the addition of a references node in the Code Explorer tree.
Vanilla-VBE
Since forever, adding a reference to the active project in the VBE is a rather… vanilla experience. Functional, but somewhere between bland and tedious.
What’s wrong with it?
Regardless of what we think of the very 1998-era buttons docked on the side, the dialog works. There’s a list of available libraries (sorted alphabetically), we can browse for unlisted ones, cancel or accept changes, and the libraries that are selected when the dialog is displayed, are conveniently shown at the top of the list!
On a closer look though…
The vanilla-VBE project references dialog
The list of available libraries has the available libraries listed in alphabetical order. You can’t resize the dialog to show more, but you get first-key search. The Scripting runtime’s library name starts with “Microsoft”… which happens to also be the case for a few other libraries; this makes the extremely useful Scripting.Dictionary and Scripting.FileSystemObject classes pretty much hidden until you stumble upon a blog post or a Stack Overflow answer that introduces them.
The selected libraries show up at the top of the list, in priority order. Locked libraries are stacked at the top. You use the up/down arrow buttons to move the selected library up or down, but you can’t move the locked ones.
The priority buttons are used to determine the identifier resolution order; if an identifier exists in two or more libraries, VBA/VB6 binds to the type defined in the library with the highest priority. There’s no visual cue in the list itself to identify the locked-in type libraries, so the Enabled state of these buttons is used to convey that information: you can’t move the locked-in, default references.
The bottom panel is useful… but the path gets cropped if it’s longer than the rather narrow dialog can fit, and you can’t select or copy the text. The actual library version number isn’t shown.
Visual Studio
Let’s take a look at what adding a project reference using the latest version of Microsoft Visual Studio feels like:
The Microsoft Visual Studio 2017 Reference Manager dialog
The dialog can be resized, search is no longer limited to a single character, but still limited to the beginning of the [Name]. The library info is now richer; it moved to the right side, and a panel on the left side determines the contents of the list. Other than that, besides a new [Version] column and a nice dark theme, …the mechanics are pretty much the same as they were 20 years ago: check boxes in a list. Priority is no longer relevant in .NET though – namespaces fixed that.
Rubberduck
This screenshot was taken shortly before the pull request was opened:
The Rubberduck ‘Add/Remove References’ dialog (work in process: release build may differ)
Available libraries appear in a list on the left-hand side of the dialog. Like in Visual Studio, the version number appears next to the library name, and the list is sorted alphabetically. There is no checkbox: instead, the selected library can be moved into the list of referenced libraries.
Referenced libraries appear in a list on the right-hand side of the dialog. Since there is no checkbox, the selected library can be moved back into the list of available libraries.
Priority up/down buttons appear for the selected referenced library, unless it’s locked.
Icons differentiate locked libraries, libraries that were already referenced when the dialog was shown, and libraries that were newly added. In the list of available libraries, recent and pinned libraries have an icon too.
Search works on a “contains” basis, and matches the library name, description, and path. It immediately filters the list of available libraries.
Tabs for quickly accessing type libraries recently referenced, or pinned libraries, or registered. Host-specific project types are in a separate tab, as applicable.
Bottom panel displays the full name and path of the selected type library. The text can be selected and copied into the clipboard.
Browse button allows referencing any project/library that isn’t listed anywhere. If a library can’t be loaded, it will appear in the list as a broken reference, before it’s even tentatively added to the project.
If you haven’t seen it in action yet, here’s a sneak peek:
Of course that’s just the beginning: layout is not completely final, drag-and-drop functionality remains to-do, among other enhancements.
A first iteration of this feature will likely be merged some time next week, and since this is a major, completely new feature, we’ll bump the minor version and that will be Rubberduck 2.4.0, to be released by the end of 2018…
…not too long after the imminent 2.3.1 hotfix release.
If you think this is one of the coolest things a VBE add-in could possibly do, you’re probably not alone. Share the news, and star us on GitHub!
Rubberduck News 