(Update: 2.0 is out!)
Fix in Progress
There is a fix in progress in Wingman Jr., it changes quite a bit of code, but it should be out in a bit. I have a fully working solution and am testing it. If you upgrade, you will likely see a note about hidden tabs; this is expected and is due to the nature of the solution.
What Went Wrong
First, it's important to know that to make the AI work, the computer does a lot of math for each image it scans. A typical computer has more than one type of chip on it that knows how to do math. There's the CPU, which is for general purpose math. Then - on most modern computers - there's the GPU, which is for large numbers of parallel calculations of the same kind - which works great for both graphics/video as well as for AI.
Having a GPU - and indeed a fast GPU - can make a big difference. In some cases, it scans images 10x faster or even more than the CPU. The AI library I use for helping run calculations, Tensorflow.js, is careful to ensure that it goes as fast as possible.
So what happened with Firefox 83?
Prior to Firefox 83, there was bug in Firefox in certain cases. Basically, there is a special call that you can make that says "give me access to the GPU, but if there's a big performance problem because something about loading the GPU isn't quite right, don't give me access to the GPU at all - just let me know and fail". For the most part, this function was working correctly in Firefox 82 and prior. However, it didn't work in all cases. In some cases, it would give access to the GPU even when it maybe was taking a performance hit.
In Firefox 83, the Mozilla team fixed the glitch. So wouldn't this make things better?
Not quite. Basically, the fact that Wingman was trying to load the GPU in the background in the addon wasn't fully supported in some cases. So when Tensorflow.js tried to get access to the GPU in this way, it would now correctly fail and say, "I won't give you access to the GPU because there is a bit of a performance hit". This meant that Tensorflow.js would fall back to doing all the calculations on the CPU, even if the performance-hit GPU would still be much faster.
If you're one of the unfortunate users like myself that encountered this, it made the browsing experience basically unusable. The whole browser would seem to lock up on loading pages and things would take forever to load.
Partial Mitigation
This spring, Tensorflow.js also released another method of running AI models, the "WASM backend". It still used the CPU, but it did some advanced tricks that leveraged some things basically all modern CPU's can do, and it made the CPU case much faster. So much faster, in fact, that in some cases it was as good as the GPU or maybe even a tiny bit better. (See here for Google's blog post on the matter).
I added this as a fallback method for calculation, and it helped some users. But for some users (like myself), the performance with this method is still unbearably slow.
Options
One option I pursued for fixing this was to have Tensorflow.js use the GPU even if there were performance issues noted by Firefox. This loading option is not exposed by Tensorflow.js, but they were kind enough to consider adding it as an option.
While this might work for some, it might end up being the wrong choice for others. If it was the wrong choice, then the system should by all rights fallback to the "WASM backend" but would not if we forced it to use the GPU. Likely then the right thing to do would be to expose an option in Wingman to pick which method to use, but this makes for a potentially poor default experience.
As the true nature of the bug unfolded as the excellent team at Mozilla looked into my bug report, it started to become clearer that 1) a real bug had been fixed and that 2) existing performance may have been suboptimal already! Additionally, there was a critical realization: it wasn't that the GPU couldn't necessarily be loaded quickly - it's that the addon background setting wasn't programmed quite correctly to allow it do so. However, this meant that if you could load the GPU in a different setting, it might work as expected - for example, in a "normal" web page setting. So how could this be accomplished?
New Architecture
In the past, there was more or less one place where code would run: the background of the addon. This approach is simple, light, and generally works great. But now we needed to do processing in a "normal" web page. An addon can also create and load web pages, too.
So the solution was to split the code into two parts: the "background" and a "processor" running on a normal page. The two parts need to talk back and forth in deep conversation in order to work. The "background" says things like "here's a request and the data flowing in for it" and the "processor" says things like "here are the scan results you asked for". The addon ecosystem makes this straightforward to accomplish, but it's a lot of plumbing and a large rip up of the existing code.
I've finished the rewrite and am ensuring the changes are stable. While there is some overhead in this approach (due to the two sides being in conversation), there are also some advantages. One of those is that it is much easier to load more than one processor if needed. So far I have not yet been able to see a performance gain out of this, but in the future I may be able to use the GPU and WASM backends together to see a bit of a performance boost.
It is probably apparent now why there might be a warning about hidden tabs. Wingman creates the "processor" tabs as web pages so that they work properly, but they're not helpful for the user to see, so it immediately hides them. That's all the tab hiding that Wingman does, but it still drives needing the "tabHide" permission and will prompt a new message after the upgrade.
Final Notes and 2.0
This is a big change in the overall architecture - large enough that I plan to change the version to 2.0 to reflect what has happened.
I may try to squeeze in a couple other features or fixes, but stay tuned for a new release soon.
