To get the GUI of wireshark running on Mac OS X please note that the latest version requires QT and uses this for it’s GUI.
They are no longer using X / X11 (because it is no longer free?) so the with-x option is not a good solution.
You have probably already installed Wireshark with
brew install wireshark so what you need to do to get the GUI working is run:
brew reinstall wireshark --with-qt
From their blog:
Today I released the next development version of Wireshark, 1.11.0. This marks a major change in the direction of the project. We’re switching our user interface library from GTK+ to Qt. Both libraries make it easy for developers write applications that will run on different platforms without having to rewrite a lot of code. GTK+ has had a huge impact on the way Wireshark looks and feels and on its popularity but it doesn’t cover our supported platforms as effectively as it should and the situation is getting worse as time goes on.
If you’re a power user you should probably keep using the GTK+ flavor for the time being. You should also test the Qt flavor from time to time to see how your workflow will change. Both are installed by default on Windows and Linux.