Required Software

OpenSolaris (and therefore also Solaris 11) can build Mozilla with a few extra packages installed:

$ pfexec pkg install SUNWhea SUNWmercurial SUNWaudh SUNWxwinc SUNWxorg-headers SUNWgm4 SUNWgnome-common-devel SUNWcvs SUNWgmake sunstudio rust
$ pfexec pkg install x11/header mercurial header-audio x11/header header-xorg gnu-m4 gnome/gettext cvs gnu-make sunstudio rust

Build Environment

Get source code from hg repo or source tarball.

Set the environment variables as follows:

export PATH=$PATH:/opt/sunstudio12.1/bin:{gcc PATH}:{autoconf-2.13 PATH}
export CC=[g]cc
export CXX=[CC | g++]

You can use a typical .mozconfig file. For current HEAD these mozconfigs are recommended:

For Firefox debug build:

mk_add_options MOZ_OBJDIR=@TOPSRCDIR@/obj-debug
ac_add_options --enable-application=browser
ac_add_options --enable-tests
ac_add_options --disable-jemalloc
ac_add_options --enable-dtrace
ac_add_options --enable-debug
ac_add_options --disable-optimize
ac_add_options --disable-ipc
ac_add_options --disable-libxul
ac_add_options --disable-crashreporter
ac_add_options --disable-verify-mar

For Firefox release build:

mk_add_options MOZ_OBJDIR=@TOPSRCDIR@/obj-release
ac_add_options --enable-application=browser
ac_add_options --enable-tests
ac_add_options --disable-jemalloc
ac_add_options --enable-dtrace
ac_add_options --disable-debug
ac_add_options --enable-optimize
ac_add_options --disable-ipc
ac_add_options --disable-crashreporter
ac_add_options --disable-verify-mar

For Thunderbird debug build:

mk_add_options MOZ_OBJDIR=@TOPSRCDIR@/obj-debug
ac_add_options --enable-application=mail
ac_add_options --enable-calendar
ac_add_options --enable-tests
ac_add_options --disable-jemalloc
ac_add_options --enable-dtrace
ac_add_options --enable-debug
ac_add_options --disable-optimize
ac_add_options --disable-ipc
ac_add_options --disable-libxul
ac_add_options --enable-shared
ac_add_options --disable-static
ac_add_options --disable-crashreporter
ac_add_options --disable-verify-mar

For Thunderbird release build:

mk_add_options MOZ_OBJDIR=@TOPSRCDIR@/obj-release
ac_add_options --enable-application=mail
ac_add_options --enable-calendar
ac_add_options --enable-tests
ac_add_options --disable-jemalloc
ac_add_options --enable-dtrace
ac_add_options --disable-debug
ac_add_options --enable-optimize
ac_add_options --disable-ipc
ac_add_options --disable-shared
ac_add_options --enable-static
ac_add_options --disable-crashreporter
ac_add_options --disable-verify-mar

Then run gmake -f client.mk to start to build.

Note that the standard Solaris make tool doesn't work. Use gmake instead. (For OpenSolaris, /usr/gnu/bin/make is gmake.)

Patches

See attached files for patches needed to build with Sun Studio 12.1. (The only remaining binary package build for Solaris 2.11.x is built with this set of patches. If you are building as part of a Solaris distribution, perhaps the upstream Oracle patches may be of interest?)

You will also need to verify that python/mozbuild/mozbuild/mozconfig_loader is a /bin/bash script, not /bin/sh. Make it /bin/bash, if it isnt already.
Otherwise, you may see errors such as

mozconfig_loader[57]: .[39]: local: not found [No such file or directory]

Troubleshooting

$ make -f client.mk build
make: Fatal error in reader: client.mk, line 126: Unexpected end of line seen

$ gmake -f client.mk build
(this one works!)
prefix=/usr/openwin
exec_prefix=${prefix}
libdir=${exec_prefix}/lib
includedir=${prefix}/include

xthreadlib=

Name: X11
Description: X Library
Version: 0.0
Requires: xproto kbproto inputproto
Requires.private: xau xdmcp
Cflags: -I${includedir} -D_REENTRANT -D_POSIX_PTHREAD_SEMANTICS -DXTHREADS -DXUSE_MTSAFE_API
Libs: -L${libdir} -R${libdir} -lX11
Libs.private:
prefix=/usr/X11
exec_prefix=${prefix}
libdir=${exec_prefix}/lib
includedir=${prefix}/include

Name: Xrandr
Description: X Randr Library
Version: 0.9.4
Requires: xproto renderproto >= 0.9 x11

Cflags: -I${includedir}
Libs: -L${libdir} -R${libdir} -lXrandr
ac_add_options --disable-auto-deps
export CXXFLAGS="-xannotate=no"