Posts

Breaking Changes – Part 2: Rubberduck Menus

RUBBERDUCK 2.0 FLIPS EVERYTHING AROUND.

When numbering versions, incrementing the “major” digit is reserved for breaking changes – and that’s exactly what Rubberduck 2.0 will introduce.

I have these changes in my own personal fork at the moment, not yet PR’d into the main repository.. but as more and more people fork the main repo I feel a need to go over some of the changes that are about to happen to the code base.

If you’re wondering, it’s becoming clearer now, that Rubberduck 2.0 will not be released until another couple of months – at this rate we’re looking at something like the end of winter 2016… but it’s going to be worth the wait.


Inversion of Control

In Rubberduck 1.x we had a class called RubberduckMenu, which was responsible for creating the add-in’s menu items. Then we had a RefactorMenu class, which was in theory responsible for creating the Refactor sub-menu under the main Rubberduck menu and in the code pane context menu. As more and more features were added, these classes became cluttered with more and more responsibilities, and it became clear that we needed a more maintainable way of implementing this, in a way that wouldn’t require us to modify a menu class whenever we needed to add a functionality.

In the Rubberduck 2.0 code base, RubberduckMenu and RefactorMenu (and every other “Menu” class) is deprecated, and all the per-functionality code is being moved into dedicated “Command” classes. For now everything is living in the Rubberduck.UI.Command namespace – we’ll eventually clean that up, but the beauty here is that adding a new menu item amounts to simply implementing the new functionality; take the TestExplorerCommand for example:

public class TestExplorerCommand : CommandBase
{
    private readonly IPresenter _presenter;
    public TestExplorerCommand(IPresenter presenter)
    {
        _presenter = presenter;
    }

    public override void Execute(object parameter)
    {
        _presenter.Show();
    }
}

Really, that’s all there is to it. The “Test Explorer” menu item is even simpler:

public class TestExplorerCommandMenuItem : CommandMenuItemBase
{
    public TestExplorerCommandMenuItem(ICommand command)
        : base(command)
    {
    }

    public override string Key { get { return "TestMenu_TextExplorer"; }}
    public override int DisplayOrder { get { return (int)UnitTestingMenuItemDisplayOrder.TestExplorer; } }
}

The IoC container (Ninject) knows to inject a TestExplorerCommand for this ICommand constructor parameter, merely by a naming convention (and a bit of reflection magic); the Key property is used for fetching the localized resource – this means Rubberduck 2.0 will no longer need to re-construct the entire application when the user changes the display language in the options dialog: we simply call the parent menu’s Localize method, and all captions get updated to the selected language. …and modifying the display order of menu items is now as trivial as changing the order of enum members:

public enum UnitTestingMenuItemDisplayOrder
{
    TestExplorer,
    RunAllTests,
    AddTestModule,
    AddTestMethod,
    AddTestMethodExpectedError
}

The “downside” is that the code that initializes all the menu items has been moved to a dedicated Ninject module (CommandbarsModule), and relies quite heavily on reflection and naming conventions… which can make things appear “automagic” to someone new to the code base or unfamiliar with Dependency Injection. For example, ICommand is automatically bound to FooCommand when it is requested in the constructor of FooCommandMenuItem, and we now have dedicated methods for setting up which IMenuItem objects appear under each “parent menu”:

private IMenuItem GetRefactoringsParentMenu()
{
    var items = new IMenuItem[]
    {
        _kernel.Get<RefactorRenameCommandMenuItem>(),
        _kernel.Get<RefactorExtractMethodCommandMenuItem>(),
        _kernel.Get<RefactorReorderParametersCommandMenuItem>(),
        _kernel.Get<RefactorRemoveParametersCommandMenuItem>(),
    };
    return new RefactoringsParentMenu(items);
}

The end result, is that instead of creating menus in the VBE’s commandbars and handling their click events in the same place, we’ve now completely split a number of responsibilities into different types, so that the App class can now be injected with a very clean AppMenu object:

public class AppMenu : IAppMenu
{
    private readonly IEnumerable<IParentMenuItem> _menus;

    public AppMenu(IEnumerable<IParentMenuItem> menus)
    {
        _menus = menus;
    }

    public void Initialize()
    {
        foreach (var menu in _menus)
        {
            menu.Initialize();
        }
    }

    public void EvaluateCanExecute(RubberduckParserState state)
    {
        foreach (var menu in _menus)
        {
            menu.EvaluateCanExecute(state);
        }
    }

    public void Localize()
    {
        foreach (var menu in _menus)
        {
            menu.Localize();
        }
    }
}

These changes, as welcome as they are, have basically broken the entire application… for the Greater Good. Rubberduck 2.0 will be unspeakably easier to maintain and extend.

To be continued…

Breaking Changes – Part 1: Parser

Rubberduck 2.0 flips everything around.

When numbering versions, incrementing the “major” digit is reserved for breaking changes – and that’s exactly what Rubberduck 2.0 will introduce.

I have these changes in my own personal fork at the moment, not yet PR’d into the main repository.. but as more and more people fork the main repo I feel a need to go over some of the changes that are about to happen to the code base.

If you’re wondering, it’s becoming clearer now, that Rubberduck 2.0 will not be released until another couple of months – at this rate we’re looking at something like the end of winter 2016… but it’s going to be worth the wait.


 

Parser State

Parsing in Rubberduck 1.x was relatively simple:

  • User clicks on a command that requires a fresh parse tree;
  • Parser knows which modules have been modified since the last parse, so only the modified modules are processed by the ANTLR parser;
  • Once we have a parse tree and a set of Declaration objects for everything (modules, procedures, variables, etc.), we resolve the identifier usages we encounter as we walk the parse tree again, to one of these declarations;
  • Once identifier resolution is completed, the command can run.

The parse results were cached, so that if the Code Explorer processed the entire code base to show up, and then the user wanted to run code inspections or one of the refactor commands, they could be reused as long as none of the modules were modified.

Parsing in Rubberduck 2.0 flips this around and completely centralizes the parser state, which means the commands that require a fresh parse tree can be disabled until a fresh parse tree is available.

We’ve implemented a key hook that tells the parser whenever the user has pressed a key that’s changed the content of the active code pane. When the 2.0 parser receives this message, it cancels the parse task (wherever it’s at) for that module, and starts it over; anytime there’s a “ready” parse tree for all modules, the expensive identifier resolution step begins in the background – and once that step completes, the parser sends a message to whoever is listening, essentially saying “If you ever need to analyze some code, I have everything you need right here”.

Sounds great! So… What does it mean?

It means the Code Explorer and Find Symbol features no longer need to trigger a parse, and no longer need to even wait for identifier resolution to complete before they can do their thing.

It means no feature ever needs to trigger a parse anymore, and Rubberduck will be able to disable the relevant menu commands until parser state is ready to handle what you want to do, like refactor/rename, find all references or go to implementation.

It means despite the VBE not having a status bar, we can (read: will) use a command bar to display the current parser state in real-time (as you type!), and let you click that parser state command button to expand the parser/resolver progress and see exactly what little ducky’s working on in the background.


To be continued…

VariableNotUsedInspection: the false positives of v1.4, upcoming fixes of v2.0

One of my favorite features since we started working on this project, is the Code Inspections.

rd-code-inspections

I like it because, well, it does find things.

The problem is that, sometimes, under specific circumstances, it makes false claims. Take this code for example:

Public Sub FormatChart(cht As ChartObject)
    Dim ax As Axis
    Set ax = cht.Axes(xlValue)
    ax.MajorGridlines.Border.Color = RGB(200, 200, 200)
    ax.MinorGridlines.Border.Color = RGB(230, 230, 230)
    ax.Crosses = xlAxisCrossesMinimum
End Sub

Here Rubberduck 1.4.3 would say “Variable ‘ax’ is never used” – and suggest a quick-fix to remove the supposedly unused declaration. A quick-fix which, of course, would break the code. Is there a bug in the VariableNotUsedInspection code?

Believe it or not, there isn’t. What makes the inspection fire up false positives, is a bug in the identifier reference resolver that causes member calls to ignore the “parent” reference.

Another common case, is caused by – again – the resolver treating For and For Each loops as assignments, but not as usages. So in code like this:

 Dim fYear As Integer
 Dim fQuarterOfYear As Integer
 Dim fMonthOfQuarter As Integer
 Dim fWeekOfMonth As Integer
 Dim fDayOfWeek As Integer
 
 For fYear = fStartYear To fStartYear + years - 1
     Set current = FiscalCalendarDate.Create(currentDate, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, fYear, vbNullString)
 
     For fQuarterOfYear = 1 To 4
 
         current.FiscalDayOfQuarter = 1
         current.FiscalWeekOfQuarter = 1
         current.FiscalMonthOfQuarter = 1
 
         For fMonthOfQuarter = 1 To 3
 
             current.FiscalDayOfMonth = 1
             current.FiscalWeekOfMonth = 1
 
             If IIf(IsLeapYear(current.calendarYear) And current.FiscalMonthOfYear = 12, True, fMonthOfQuarter Mod 2 = 0) Then
 
                 For fWeekOfMonth = 1 To 5
 ...

You get a “Variable ‘fWeekOfMonth’ is not used” and “Variable ‘fQuarterOfYear’ is not used”, because the only place they’re ever used is in For loops.

Then you have the forgotten edge cases:

Private WithEvents indicator As ProgressIndicator

WithEvents variables are likely to be assigned a reference, but not necessarily to be themselves referenced anywhere. And if they aren’t, well then, they’re reported as “never used”. Which is a problem, because you don’t want a quick-fix to go and delete the WithEvents declaration that provides all these events you’re handling. So we’re going to be ignoring WithEvents variables.


Okay… Got any Good news?

Totally. Ducky 2.0 code inspections are being completely revamped. And all these false positives are being addressed, which means…

…most inspections will support “Fix all” options. Of course one shouldn’t “fix all occurrences in project” without actually reviewing the inspection results. But so far, it’s looking very, very good. This UI is still preliminary: we’re happy to hear (and integrate) your feedback!

Blue Print of 2.x

The foundation of the project needed a bit of clean-up and restructuring. The Big Refactoring is now pretty much completed, and with it, summer 2015’s feature freeze:

Feature-freeze until this is completed. We need to implement proper DI/IoC and make the code testable, if not tested. No ifs, no buts.

So we’re ready to start implementing the new features of 2.0. But before, we need to reconnect the pieces, by moving code from the 1.x “RubberduckMenu” and “RefactorMenu” classes, and into these “ICommand” implementations.


 

So, What’s Cooking?

Tons. The new architecture allows us to write code that is aware of the host application. This means Unit Testing commands can be disabled when the host is Outlook, for example (it seems executing VBA code on-demand from a VBE add-in isn’t possible in Outlook – ideas welcome). It also means we can write code inspections that warn about implicit references to Application.ActiveWorkbook in Excel, but that don’t run when the host application is Access, Word, or PowerPoint. Things like that, and…

Regex Search & Replace is coming. We’ll need a way to hijack Ctrl+F and Ctrl+H!

Smart Indenter is coming. The owners of this awesome add-in have graciously offered their source code to the Rubberduck project a while ago already; 2.0 isn’t releasing without an embedded Smart Indenter!

More grammar fixes on the way. This means fewer parser errors, more accurate inspections, navigation and refactorings, i.e. a more reliable tool.

Source Control Integration continues to improve, and early minor releases of 2.x will likely see a new WPF/XAML UI.

More translations have been completed since the last release: 2.0 will speak English, French, Swedish, German, Japanese… and every other translation we receive a PR for in the next.. uh… …6-8 weeks.

Shiny new UI. Docked gridviews are turning into WPF treeviews; the “Rubberduck” menu has been revamped, and under the hood, everything changed. Might as well make 2.0 look like as much change happened!

Code Inspections and Quick-Fixes that take the host application into account, giving recommendations tailored for a given host API. Also, some of the coolest inspections we envisioned as fantasies a year ago, are now possible to implement.

More refactoring tools like inline method, encapsulate field and promote local. The early minor releases of 2.x will likely see a new WPF/XAML UI for the refactorings issued in 1.x.


 

Something really good is cooking.

We’re also looking at implementing a more VBA-like Assert class, that would be more permissive with AreEqual and AreNotEqual than the current (C#-like strict) implementation is. Test results will be copied to clipboard or exported/serialized to XML with a simple click.

If you write some C# and would like to contribute to Rubberduck, note that our GitHub repository has a bunch of up-for-grabs issues opened, a lot of which are critical (i.e. no fix, no release) – the faster all functionalities work off the new command architecture, the faster we can deliver a pre-release…

Stay tuned!

Resolver: still not perfect

I just found another bug in the rewritten 1.4.x resolver, and as much as I want to fix it (and I will)… part of me can’t help thinking if you encounter this bug, your code has greater problems than mine.

Picture this code:

Sub Foo()
     Dim Foo As New Foo
     With Foo
         With .Foo
             .Foo = 42
         End With 
         Bar .Foo.Foo
     End With
End Sub

Believe it or not, it compiles… and the 1.4.1 resolver works perfectly and correctly identifies who’s who and who’s calling what. Now what if we added a recursive call?

Sub Foo()
     Dim Foo As New Foo
     With Foo
         With .Foo
             .Foo = 42
         End With 
         Bar .Foo.Foo
     End With
     Foo
End Sub

The bug here, is that this Foo gets resolved as a call to the local variable, so renaming the procedure to DoSomething would leave that Foo unchanged.

Not too bad. I mean, nobody in their right minds would ever do that. (right?)

However, it gets worse: if we change Sub to Function, in the name of being able to identify a function’s return value, a matching identifier within the body of a function (or property getter) is resolved to the function (or getter) itself… so this part needs to be refined at bit.

Function Foo()
     Dim Foo As New Foo
     With Foo
         With .Foo
             .Foo = 42
         End With 
         Bar .Foo.Foo
     End With
     Foo
End Function

Every single Foo here, with the sole exception of the local declaration, resolves to a reference to the function.

I hope this bug doesn’t affect your code… for your sake – and that of whoever will be maintaining your code in the future.

If it does… v1.4.2 with certainly include a fix for this issue.


Update

The non-resolving recursive call isn’t a bug:

it's a local!

VBA and Git the Sequel: This Time it’s Integrated

There’s some information about the new beta Source Control Integration feature over at my blog.

Christopher J. McClellan's avatarChristopher J. McClellan

I’ve talked before about using Git with VBA and also about the importance of using source control and other tools for our work. Truthfully though, using source control with VBA is still hard. This is mostly because getting the code modules into and out of the VB Project is hard, and harder to do right. Well, you may have heard that there’s a new duck in town. It’s taken me about 6 months of spare time, but Rubberduck v1.4 not only has a source control COM library that you can use right in VBA to work with Git VBA repositories, but you can also now branch, commit, pull, push, and merge right from inside of the editor.

View original post 545 more words

There’s a new duck in town!

…and it rocks.

The past few months have been tough. We were facing some serious ANTLR grammar bugs, and our identifier resolver had even more serious bugs, which meant false positives in code inspections, a rename refactoring that wouldn’t always rename all references, and a StackOverflowException if you were unlucky… which blew up Rubberduck, the VBE, and the Office host app with it, without warning.

That’s why 1.3 remained a “pre-release”.

Rubberduck 1.4 has none of these issues. Oh, our ANTLR grammar is still far from perfect (line numbers, and “precompiler” #IF statements come to mind) – but a number of bugs were fixed, and the resolver was completely rewritten. It’s not as perfect as we’d like it to be, but it correctly resolved everything we threw at it, without a single exception.

So, what’s so cool about it?


 

#Refactor

Extract Method has been around since version 1.2, and Rename was pre-released (albeit with some issues) with 1.3; Rubberduck 1.4 introduces two new refactorings:

  • Reorder Parameters in a procedure, and automatically adjust all call sites.
  • Remove Parameters from a procedure’s signature, and automatically adjust all call sites.

Rename can be used from the Project Explorer, Code Pane or Form Designer context menus, as well as from the Code Explorer to rename a project, a component (form, module, class), a procedure, variable, line label, …anything. This completely eliminates all excuses to keep meaningless identifiers in your code.


 

#Navigate

The Code Explorer was already a nice feature in 1.2, but Rubberduck 1.4 takes navigation to a whole new level.

  • Find Symbol lets you search for anything – and go to its declaration.
  • Find all references displays all call sites of any identifier.
  • Go to implementation lets you navigate implementations of an interface or interface members. Rubberduck considers any class referenced by an Implements statement as an interface.

One cool thing is that we’ve created declarations for pretty much everything in the Standard VBA library, so you can use find all references to list all usages of, say, MsgBox, or Err.Raise.


#Git

That’s right. We’ve talked about it for a while. Well, it’s here. Integrated source control, right there in the VBE. Commit, push, pull, merge, branch, to and from local or remote repositories on GitHub.


And so much more…

If you’ve never used a previous version of Rubberduck, now is the time.

Every VBA programmer needs a Rubberduck.

DOWNLOAD

Rubberduck, or Gummianka, le petit canard en caoutchouc

Next release rocks. Like, if you’re running 1.22, you’re not going to believe everything that’s coming at you in 1.4 – and if you’re running 1.3x, …you’re not going to believe it either.

The feature set is becoming massive, and that’s thoroughly awesome. We are currently in the final stages of stabilizing the next release and polishing everything. There’s a ton of amazing stuff coming, but one of the coolest things is that, thanks to the awesomeness of Code Review community members…

…we are proud to announce localization in French and Swedish, and a German translation is under way.

If a language isn’t supported on your system, it won’t show up in the display language dropdown:

Options-Localized

More announcements to come, stay tuned!

What happened to “release early, release often”?

Since the project started, we’ve been releasing pretty much every 100 commits or so – actually more than that, given 13 releases in 1,003 commits. Our “next” branch stands at 1,375 commits at the time of this writing, and the next release will likely be well beyond the 1,400th commit. What’s up, duck?

Quite a lot actually.

Architecture changes

Since the 1.3 pre-release, we have moved the unit testing feature into its own project. This means starting with Rubberduck 1.4, upgrading your Rubberduck will not break any references because the .tlb will remain the same unless there are changes to the unit testing feature.

Localization

We’ve been working on extracting every single hard-coded UI string literal into resource files, and proceeded to translate them. The French translation is pretty much done, and we’ve been promised a Swedish translation. Surely more translations will be added in future versions. Rubberduck’s UI language will be selectable from a dropdown in the options dialog.

Grammar is fun

Our ANTLR grammar has undergone yet another little tune-up, which fixes a number of annoyances. And when our grammar fails, we’re now going to be showing you exactly where and why it’s failing, and you can double-click the error to navigate there.

Navigation

We already had the Code Explorer doing a great job with navigation, but v1.4 takes navigation totally elsewhere, by letting you locate all references to any identifier (including built-in constants, enums, functions, modules, etc.), and all implementations of any implemented interface class – this latter functionality is an awesome tool if you’re into coding against abstractions… which happens to be a great way to write unit-testable code.

But that’s not all: a find symbol functionality lets you search for literally anything, from variables to constants, procedures, modules, function parameters, …even subroutine line labels.

UX

Parsing a relatively large VBA project, and resolving all rerences to all identifiers, can take a little while. Instead of just freezing up the IDE while Rubberduck is working on that, we’re now going to be displaying a little progress dialog, showing you exactly what’s going on – and we only parse and resolve references for modules that have been modified since the last parse, so we’re doing everything we can to keep that waiting time to a minimum.

Moreover, the Code Explorer and Code Inspections now work on a background thread, so you can continue browsing your project while Rubberduck is parsing/resolving/inspecting – keep in mind though, that if you modify the code while it’s being inspected, you’re likely to get stale inspection results… and that’s not a bug.

Refactorings

1.3 introduced a rename refactoring, on top of the extract method introduced in 1.2; well guess what, 1.4 fixes a number of issues in both, and introduces brand new ones! You will be able to reorder parameters in any procedure, and automatically update all usages; or remove a parameter in a signature, and automatically update all call sites.

Source Control

It’s almost done. Most of it is already committed to our [next] branch, so 1.4 isn’t shipping without this feature – GitHub integrated source control for VBA. No, it’s not a joke.


So yeah, there’s a lot going on, and it’s all coming in the next release of Rubberduck. And more awesome features are coming up for 2.0; follow us on Twitter, and stay tuned!

You’ve waited long enough.

The wait is over!

I have to say that this release has been… exhausting. Correctly resolving identifier references has proven to be much, much more complicated than I had originally anticipated. VBA has multiple ways of making this hard: with blocks are but one example; user-defined-type fields are but another.

But it’s done. And as far as I could tell, it works.

Why did you tag it as “pre-release” then?

Because resolving identifier references in VBA is hard, and what I released is not without issues; it’s not perfect and still needs some work, but I believe most normal use cases are covered.

For example, this code will blow up with a wonderful StackOverflowException:

Class1

Public Function Foo() As Class2
    Set Foo = New Class2
End Function

Class2

Public Sub Foo()
End Sub

ThisWorkbook

Public Sub DoSomething()
    Dim Foo As New Class1
    With Foo
            .Foo
    End With
End Sub

It compiles, VBA resolves it. And it’s fiendish, and nobody in their right minds would do anything near as ambiguous as that. But it’s legal, and it blows up.

That’s why I tagged it as a “pre-release”: because there are a number of hair-pulling edge cases that just didn’t want to cooperate.

See, finding all references to “foobar” works very well here:

Public Sub DoSomething()
    Dim foobar As New Class1
    With foobar
        With .Foo
            .Bar
        End With
    End With
End Sub

…and finding all references to “Foo” in the below code will not blow up, but the “.Foo” in the 2nd with block resolves as a reference to the local variable “Foo”:

Public Sub DoSomething()
    Dim Foo As New Class1
    With Foo
        With .Foo
            .Bar
        End With
    End With
End Sub

And of course, there are a number of other issues still.

Here’s a non-exhaustive list of relatively minor known issues we’re postponing to 1.31 or other future release – please don’t hesitate to submit a new issue if you find anything that doesn’t work as you’d expect.

There can be only one

Rubberduck doesn’t [yet] handle cross-project references; while all opened VBA projects are parsed and navigatable, they’re all “silos”, as project references aren’t taken into account; this means if project A is referencing project B, and project A contains the only usage of a procedure defined in project B, then that procedure will fire up a code inspection saying the procedure is never used.

It also means “find all references” and “rename” will miss it.

self-referencing Parameters

“Find all references” has been seen listing the declaration of a parameter in its list of references. Not a biggie, but not intended either. There’s no explanation for that one, yet – in fact it’s possible you never even encounter this issue.

Selection Glitches From Code Inspection Results

We know what causes it: the length of the selection is that of the last word/token in the parser context associated with the inspection result. That’s like 80% fixed! Now the other 80% is a little bit tricky…

Performance

Code inspections were meant to get faster. They got more accurate instead. This needs a tune-up. You can speed up inspections by turning off the most expensive ones… although these are often the most useful ones, too.

Function return vAlue not assigned

There has been instances of [hard-to-repro] object-typed property getters that fire up an inspection result, when they shouldn’t. Interestingly this behavior hasn’t been reproduced in smaller projects. This inspection is important because a function (or property getter) that’s not assigned, will not return anything – and that’s more than likely a bug waiting to be discovered.

Parameter can be passed by value

This inspection is meant to indicate that a ByRef parameter is not assigned a value, and could safely be passed ByVal. However if such a parameter is passed to another procedure as a ByRef parameter, Rubberduck should assume that the parameter is assigned. That bit is not yet implemented, so that inspection result should be taken with a grain of salt (like all code inspection results in any static code analysis tool anyway).

This inspection will not fire up a result if the parameter isn’t a primitive type, for performance reason (VBA performance); if performance is critical for your VBA code, passing parameters by reference may yield better results.