For reverse engineering, you still need access to the FPGA tools provided by the vendor, to see what changes in the bitstream when you change the design.

If the bitstream is encrypted, you will not see the changes, so the only way is to reverse engineer the Vivado executables.

You do not need only the bitstream, but you also need a huge amount of timing parameters. In theory, they could be obtained by fuzzing, but that would require a huge amount of executions of the Vivado tools. So again the most plausible method is to reverse engineer the Vivado executables, to get the timing parameter database.

In some countries that should be legal, as such reverse engineering might become the only way to use the AMD FPGAs that one buys legally.

The difficult part is the place and route algorithm, not the bitstream. The proprietary ones already take quite a long time to solve: I regularly have 12-24h runs. Perhaps an open source one could do better? But it's not quite as straightforward as reverse engineering a proprietary bitstream.