The issue is in Apple’s Transparent Proxy Network Extension framework. This framework, which we have to use on Big Sur as the old kext interface is no longer working, is still quite buggy unfortunately.
To explain how Transparent Proxy works: we specify ports we want to intercept to scan web traffic. When a connection is being established, a callback is invoked in our system extension. Since the callback does not originate from a browser in this case, we answer “we are not interested in it, please pass it though!” return value. However, the resulting connection is not good enough for flutter and the error occurs. I am not sure why exactly flutter fails.
We have reported similar issue to Apple previously (https://forum.avast.com/index.php?topic=239798.0) and they have seemed to fix it in MacOS 11.2 beta2, however this issue is a different one, as it occurs even when our system extension is the only Network Extension system extension on the machine. It is also NOT fixed in 11.2 beta2.
Please note that this seems to happen with multiple Antivirus products and other security products which use Transparent Proxy Network Extension.
I have filed a bug to Apple. The only workaround is to disable either Web Shield, or scanning of ssl connections in web shield. There is unfortunately no way for us how to work around this issue, we will have to wait for Apple to fix this.