To either of the brown wires. Ground being connected internally isn't an issue, however you only want one path to ground, so either one (but not both) of the brown wires can become ground. If you connect both brown wires to ground you have what is called a ground loop (essentially two parallel paths to ground) which can cause problems. The same is true when connecting the various ground and power busses to each other, you only want one path through the whole system no places with parallel connections.
Peter
