Welcome, Linux Users!

 Recently there was a bump in new users - and the Firefox stats seem to indicate that many of you use Linux. Welcome friends!

Now, I've got a question for you. I don't use Linux desktop as my daily driver, so I need your help in understanding how the performance is looking so far. This addon relies on some level of hardware acceleration ... which hasn't always had the best support on Linux. If you've got a second, I'd love to hear from you over at this poll on Google Forms - can you please help me out?

One last note: if you are finding that the default WebGL backend isn't working well on your machine, it might be worth it to try the WASM backend, which you can pick in the options for the addon.

Hope you enjoy Wingman!

 

Update: Poll results from 4 users - glad to hear it's working well for you!

Mattermost Breaks?

 Hello! I got a report from a helpful user "pakxo" about the web app Mattermost (redacted a bit):

I have an issue while using Mattermost. The extension becomes so so slow and sometimes break, often times break on Mattermost. ... I don't think it's a performance issue. It's more like a bug or something.

Pakxo also indicated that it breaks more on his PC, which has a decent graphics card.

I don't regularly use Mattermost, but I did try out the community site here: https://community.mattermost.com

Unfortunately, I wasn't able to reproduce any specific issues like pakxo had listed. If anyone else has problems with the web app - please let me know! I'm tracking the issue on GitHub here.

On/Off Switch

 Just got some great feedback from "someuser":

Loving the extension so far, but I'd like to ask if it's possible to add a toggle to quickly enable and disable the extension, either on a per-page basis or globally. While the "trusted" toggle seems to work fine, sometimes I'd prefer to temporarily disable the extension without going through firefox's menu. Thanks.

Well, "someuser" - you're not the only one who's wanted to do this, and it's actually already a feature!

If you go into options, you'll see this:

 

Then when you go to the main button's dropdown, you get a new "On?" checkbox at the top:

 

It works globally, not per-page like say, uBlock Origin does. As a bonus, turning the addon on and off this way is much faster than disabling and re-enabling it not only because of the extra clicks, but because it also keeps the AI model loaded so it doesn't have to do the long several-second startup.

As a note, the reason this is not on by default is that some folks use this for their kids so I don't want it be shown by default. But for those that do want to use this feature, they tend to want to use it all the time, so making it an option seemed like the best balance.

Thanks for the feedback "someuser"! As a quick note: I always appreciate feedback as it helps me know how people are using the addon. There's a quick feedback form in the menu there, but any time there's an upgrade, I've got a short little survey. Please, if you would, consider filling it out sometime! One big thing it helps me understand is how folks are using the plugin so I can better understand where to focus my efforts.

Thumbnail Scanning Size

 Let's talk about thumbnails here for a bit.

Usually when you're dealing with an image classification problem, if you can figure out what the thing is, then you can classify it, which is easy. But for NSFW content, this turns into a bit more of a grey area. Sure, there are some things that - no matter how small - are clearly NSFW. But for others, as an image either scales down in size or becomes more abstract, it starts to cross a line where it goes from R to PG or even G. Where exactly one should draw this line is quite subjective.

I just received a feature request from a helpful user, Sneed:

A setting to optionally block small thumbnails or set an arbitrary minimum/maximum would be nice. The thumbnail for the image on this webpage does not get blocked, when you click it to expand, it IS blocked.

I've definitely run into this too. The current logic is around this area of the code and basically has two conditions to skip blocking: if an image is less than 36x36 pixels or the total image size is less than 1024 bytes.  The size in bytes is perhaps less obvious, but the idea is that an offensive image generally needs a certain amount of complexity, so if I can indicate that the image is too small by just the number of bytes, I can skip even decoding the image to find the true dimensions which helps with performance.

This is an area I haven't turned my attention to in quite a while, so I'm going to give it some thought. I'm tracking it over at this issue on Github, so feel free to join the conversation there.

Firefox 83 Fix - Get 2.0!

(Update: 2.0 is out!)

Fix in Progress

There is a fix in progress in Wingman Jr., it changes quite a bit of code, but it should be out in a bit. I have a fully working solution and am testing it. If you upgrade, you will likely see a note about hidden tabs; this is expected and is due to the nature of the solution.

What Went Wrong

First, it's important to know that to make the AI work, the computer does a lot of math for each image it scans. A typical computer has more than one type of chip on it that knows how to do math. There's the CPU, which is for general purpose math. Then - on most modern computers - there's the GPU, which is for large numbers of parallel calculations of the same kind - which works great for both graphics/video as well as for AI.

Having a GPU - and indeed a fast GPU - can make a big difference. In some cases, it scans images 10x faster or even more than the CPU. The AI library I use for helping run calculations, Tensorflow.js, is careful to ensure that it goes as fast as possible.

So what happened with Firefox 83?

Prior to Firefox 83, there was bug in Firefox in certain cases. Basically, there is a special call that you can make that says "give me access to the GPU, but if there's a big performance problem because something about loading the GPU isn't quite right, don't give me access to the GPU at all - just let me know and fail". For the most part, this function was working correctly in Firefox 82 and prior. However, it didn't work in all cases. In some cases, it would give access to the GPU even when it maybe was taking a performance hit.

In Firefox 83, the Mozilla team fixed the glitch. So wouldn't this make things better?

Not quite. Basically, the fact that Wingman was trying to load the GPU in the background in the addon wasn't fully supported in some cases. So when Tensorflow.js tried to get access to the GPU in this way, it would now correctly fail and say, "I won't give you access to the GPU because there is a bit of a performance hit". This meant that Tensorflow.js would fall back to doing all the calculations on the CPU, even if the performance-hit GPU would still be much faster.

If you're one of the unfortunate users like myself that encountered this, it made the browsing experience basically unusable. The whole browser would seem to lock up on loading pages and things would take forever to load.

Partial Mitigation

This spring, Tensorflow.js also released another method of running AI models, the "WASM backend". It still used the CPU, but it did some advanced tricks that leveraged some things basically all modern CPU's can do, and it made the CPU case much faster. So much faster, in fact, that in some cases it was as good as the GPU or maybe even a tiny bit better. (See here for Google's blog post on the matter).

I added this as a fallback method for calculation, and it helped some users. But for some users (like myself), the performance with this method is still unbearably slow.

Options

One option I pursued for fixing this was to have Tensorflow.js use the GPU even if there were performance issues noted by Firefox. This loading option is not exposed by Tensorflow.js, but they were kind enough to consider adding it as an option.

While this might work for some, it might end up being the wrong choice for others. If it was the wrong choice, then the system should by all rights fallback to the "WASM backend" but would not if we forced it to use the GPU. Likely then the right thing to do would be to expose an option in Wingman to pick which method to use, but this makes for a potentially poor default experience.

As the true nature of the bug unfolded as the excellent team at Mozilla looked into my bug report, it started to become clearer that 1) a real bug had been fixed and that 2) existing performance may have been suboptimal already! Additionally, there was a critical realization: it wasn't that the GPU couldn't necessarily be loaded quickly - it's that the addon background setting wasn't programmed quite correctly to allow it do so. However, this meant that if you could load the GPU in a different setting, it might work as expected - for example, in a "normal" web page setting. So how could this be accomplished?

New Architecture

In the past, there was more or less one place where code would run: the background of the addon. This approach is simple, light, and generally works great. But now we needed to do processing in a "normal" web page. An addon can also create and load web pages, too.

So the solution was to split the code into two parts: the "background" and a "processor" running on a normal page. The two parts need to talk back and forth in deep conversation in order to work. The "background" says things like "here's a request and the data flowing in for it" and the "processor" says things like "here are the scan results you asked for". The addon ecosystem makes this straightforward to accomplish, but it's a lot of plumbing and a large rip up of the existing code.

 I've finished the rewrite and am ensuring the changes are stable. While there is some overhead in this approach (due to the two sides being in conversation), there are also some advantages. One of those is that it is much easier to load more than one processor if needed. So far I have not yet been able to see a performance gain out of this, but in the future I may be able to use the GPU and WASM backends together to see a bit of a performance boost.

 It is probably apparent now why there might be a warning about hidden tabs. Wingman creates the "processor" tabs as web pages so that they work properly, but they're not helpful for the user to see, so it immediately hides them. That's all the tab hiding that Wingman does, but it still drives needing the "tabHide" permission and will prompt a new message after the upgrade.

Final Notes and 2.0

This is a big change in the overall architecture - large enough that I plan to change the version to 2.0 to reflect what has happened.

I may try to squeeze in a couple other features or fixes, but stay tuned for a new release soon.