@lxoliva had some compelling words about this at LP2019:

https://media.libreplanet.org/u/libreplanet/m/who-s-afraid-of-spectre-and-meltdown/

I don't know if your comment related at all to Spectre, but---if all the software running on your system is free software, what is there to fear? And I agree.

The biggest trouble is that people often run non-free and untrusted code all of the time in their web browsers, and don't see it as a software freedom or security issue. It's important to recognize it for what it is---untrusted, unsigned, ephemeral software---if you're going to consider security tradeoffs when it comes to certain mitigations. I personally don't run JS at all, even if it's free, with very few exceptions, because it's unsigned.