We now know what the cause of the 502 errors was. The backends of the mastodon application bind only to ipv4 by default. We used the name localhost on the nginx reverse proxy to refer to the backend. This, as usual first resolves to ::1 the local ipv6 address. As there is no backend listening under this address we need to fallback to ipv4 and we think this was to slow in some cases leading to the user facing 502 error.
chaos.social – a Fediverse instance for & by the Chaos community