Imprudence talk:Compiling

I just bought a Mac Mini for the purposes of compiling Imprudence releases on it, and to help testing out Mac specific issues. Here I'll keep track of progress so that I can update the Mac building instructions.

goals
There are two goals. The first is to figure out how to produce a proper Mac OSX package. The second is to figure out how to script the entire process. The scripted version will be added to my release building script that already knows how to build releases for Linux and Windows. Hopefully the script can drive the Mac build process via ssh. At the moment the Mac build instructions are largely GUI driven, so this might be a challenge. On the other hand, the Windows build script makes use of my Linux build system, oddly enough, so there's hope, since Mac OSX is Unix at heart.

side goals
Mac OSX seems to have some sort of remote desktop thingy. See if I can send that to my Linux main box. See if it's good enough for 3D. Apparently it's some variation of VNC, some people report good and bad results using standard VNC clients. Ordinary VNC works fine so far. I can get rid of my second mouse and keyboard. Though a proper HDMI KVM would be better, but that's for next time I have some spare cash.

See if I can get mc running without too much trouble, coz I use it for everything. Seems it's not in Homebrew, but it is in MacPorts, so I'll go with MacPorts. I managed to get it running under Win XP. B-)

Mac OSX has some system for installing and running Windows. See if it's good enough for 3D. And building the Impy release. B-)

one more goal
I've offered Nicky the use of my Mac Mini for building Kokua, so the next goal is to set things up so we can both build. This requires two Xcodes installed at once, switching between them, and making sure only one of us uses the Mac at any one time, since that's a license thing that it's strictly single user at a time.

the build box
This is a 2012 unibody Mac Mini 2.3GHz quad core i7, with 4GB of RAM and a 1TB hard drive. The graphics is Intel HD 4000 with 512MB shared memory. Apparently this is not so good for 3D graphics and video. It comes installed with Mac OSX 10.8.3 Mountain Lion. It was chosen coz it's cheap, and I can hook it up to existing monitor, keyboard, and mouse. Building viewers tends to be RAM intensive, 4GB should be enough for a decent build speed. I expect maybe 15 to 30 minutes based on previous experience.

I don't know off the top of my head which version of Mac OSX is the minimum that should be supported. The release building script uses Ubuntu 10.04 LTS and Windows XP as the minimum supported versions. I'll try to keep the OS as virgin as possible, only adding and tweaking as needed to actually do the build.

It seems that just owning a Mac these days means you need an Apple ID, but you need an Apple Developer ID to get the quicktime stuff for building on Windows, and it seems that ID can be reused. Also helps to get XCode it seems.

VM
First up, the real reason for paying for over priced Mac hardware in the first place - you can't legaly run Mac OS X on a VM unless that VM is running on actual Apple hardware (yes, I double checked the licence). From what I can tell, you can only install 10.5 or 10.6 server versions if 10.7 or later versions are the host. Hopefully server versions are good enough for building software. Word is that 10.5 server is the only one you can legally run on a VM, but I could only find the license for 10.6 and 10.6 server, neither of which mention VMs, the 10.8 license says (section 2B) -

"(iii) to install, use and run up to two (2) additional copies or instances of the Apple Software within virtual operating system environments on each Mac Computer you own or control that is already running the Apple Software, for purposes of: (a) software development; (b) testing during software development; (c) using OS X Server; or (d) personal, non-commercial use."

10.5 server costs $999, 10.6 server costs $499. Later server version where included as part of the desktop OS, so I already have 10.8 server. If you can buy them, can't find them on Apple web sites. Soooo, costs too much and can't be done anyway. B-(

Useful links -

http://stackoverflow.com/questions/39159/is-it-possible-to-run-mac-os-x-in-a-virtual-machine?rq=1 https://discussions.apple.com/thread/3252928?start=0&tstart=0

bootcamp
Not a VM, it helps to resize an existing disk and putting Windows on the empty space. On 10.8 it only supports Windows 7.

parallels
VM specifically for Mac. Runs on 10.8 and can host 10.6. US$50 - 80

qemu
Open source. It's what I'm used to at least. B-)

There's a Mac specific version called Q.

virtualbox
Open source.

vmware fusion
US$40 - 80

supported versions
Apples support web site only goes back as far as 10.5. A lot of this is taken from https://en.wikipedia.org/wiki/Xcode, some of it is guesses.

See the note for Xcode 4.0. In that two year old discussion is this "Apple has made this difficult enough that I suspect most developers will quickly drop support for 10.5 now that Lion is out. This is certainly what we'll be doing.". Sounds like a sane idea. So to support 10.5 might be better to install 10.6 on a VM (download size unknown, perhaps 3GB) and install Xcode 3.2.6 on it (4.14GB download). Xcode 4.0 isn't available as a download, 4.1 (2.96GB) is for Lion, and dropped gcc 4.0. 4.5 seems to be the earliest that runs on 10.8 and is still available, but no support for 10.6 SDK. Currently it looks like LLVM gcc 4.2 / LLVM Clang 4.2 is not working for Impy builds. I'm not sure yet if Impy will build on those later compilers at all, but to support people building on 10.7 and up, will be needed.

release builds
For releaseses install OS X 10.6 and install Xcode 3.2.6 on it. Currently releases are built on Ubuntu 10.04 LTS and Windows XP, so this makes sense as the way things are done. Building actual releases on the oldest supported OS versions built in VMs. This should in theory mean that they can run on later OS versions as well as the old ones. This is good for releases. Well, that was the theory until I started looking at how to actually do that, soooo ....

plan B
Install 3.2.6 under my existing 10.8 using evil hacks. sigh

http://anatomicwax.tumblr.com/post/7906770311/installing-xcode-3-2-6-on-lion and the follow up http://anatomicwax.tumblr.com/post/8064949186/installing-xcode-3-2-6-on-lion-redux

Another follow up http://carleeto.tumblr.com/post/21606695553/anatomic-wax-installing-xcode-3-2-6-on-lion-redux mentions -

A couple of additional things:


 * If your installation fails, reset your system date to before March 1, 2012.
 * Make sure you don’t have any iTunes helper processes running before you start the install

plan C
If all else fails, then we would have to drop support for 10.5, 10.6, and get the code to a state where it compiles with LLVM 4.2 compilers, then just use the Xcode 4.6.3.

10.5 builds
Xcode 3.1.

10.6 builds
Xcode 3.2.6, if that's what is used for the releases, that's what gets the most support. Xcode 4.0 to 4.3.3 could be used to, but 4.1 dropped gcc 4.0 support.

10.7 builds
Xcode 4.1 to 4.6.3. No gcc 4.0 support on any of those versions.

10.8 builds
Xcode 4.4 and up.

10.9 builds
Xcode 5.0.1 and up.

requisites
The original page says XCode 4.1 or greater for this version of OSX, latest is 5.0.2. Python 2.5, though claims this OSX should have one (2.7.6 or 3.3.3 latest). Perhaps git, though my release build scripts do the git part on the host, which for me is Linux. Cmake 2.4.8 (2.8.12.1 is latest). Apparently some versions of XCode install cmake, some don't. If not, we can use Homebrew. Homebrew is git+ruby scripts from what I can tell. So git might be needed after all, which apparently is in XCode anyway. Ruby 1.8.7 is installed.


 * Install Homebrew
 * It will complain that XCode is outdated, though you can't install the suggested one anyway.
 * 2.8.12.1 got installed
 * It will complain that XCode is outdated, though you can't install the suggested one anyway.
 * 2.8.12.1 got installed
 * 2.8.12.1 got installed

An alternative to homebrew is MacPorts, which include mc that homebrew doesn't. Some notes from Nicky -


 * When you install macports it will update to xcode 5. If this happens rename the xcode.app to xcode and it will then turn it into a folder.
 * You may need to reinstall XCode 4.6.3.
 * http://wiki.secondlife.com/wiki/Compiling_the_viewer_(Mac_OS_X)
 * This outlines the procedure to go to an earlier sdk.
 * I did this in a virtual machine a couple weeks back and added a note to SL wiki.

Mac OSX is based on BSD, and BSD uses ports for package management, and I never liked ruby anyway. When I did this, there was no update of XCode as Nicky mentions. The problem here is that MacPorts requires Xcode 4.6.2 or later, and we are using Xcode 3.2.6. So gotta install the later Xcode anyway. sigh


 * Install MacPorts using the .pkg file downloaded from their web page.
 * cmake 2.8.12 got installed
 * lftp 4.4.13 got installed
 * mc 4.8.1.4 got installed B-)
 * pkgconfig 0.28 got installed
 * lftp 4.4.13 got installed
 * mc 4.8.1.4 got installed B-)
 * pkgconfig 0.28 got installed

Ssh server side naturally, it's installed already, disabled by default. From what I understand the components for Mac OSX server are all installed in ordinary OSX.

You can have multiple copies of Xcode installed. 3.2.6 installs into /Developer, and is special. 4.6.3 is an ordinary app and installs into /Applications/Xcode.app.

the procedure
(Please don't change this, this is the steps I'm actually doing so that I can reproduce it later.)


 * Turn on ssh server. System preferences -> Internet & Wireless -> Sharing -> tick "Remote Login" on the left hand side, setup users as you wish.
 * Install XCode 4.6.3 from https://developer.apple.com/downloads/ or the Mac App Store if you prefer.
 * Install command line tools from Xcode menu -> Preferences -> Downloads.
 * Install cmake via Homebrew or MacPorts, I used MacPorts.
 * Create a source directory in my home directory, change into it.
 * Use the linux scripts to build it.
 * error: There is no SDK with the name or path '/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.5.sdk'
 * Hmm, it only has 10.7 and 10.8. The "fixes" mostly involve installing an earlier XCode and symlinking to the old one.  Messy.
 * As far as I can tell, basically Apple dropped support for those older ones, perhaps we should to?
 * Commented out Variables.cmake's reference to the SDK to let it pick one, it seems to have picked 10.8
 * Errors about unknow GCC 4.0
 * Commented out Variables.cmake's reference to the gcc, let it pick one.
 * error: Error Domain=NSPOSIXErrorDomain Code=2 "Non-zero exit code 255 returned from shell command: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang -v -E -dM -arch ppc -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.8.sdk -x c++ -c /dev/null 2>&1" UserInfo=0x401efa860 {NSLocalizedDescription=Non-zero exit code 255 returned from shell command: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang -v -E -dM -arch ppc -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.8.sdk -x c++ -c /dev/null 2>&1, NSLocalizedFailureReason=No such file or directory}
 * Ewww, try the GUI version now.
 * "arch ppc" isn't good.
 * Using XCode GUI to build it.
 * Start up Xcode, wait for the indexing to finish.
 * Search for this mythical "Info button".
 * Try to find this other stuff the notes mention.
 * Make note to actually learn this IDE, and give up searching for that stuff.
 * On the left is a bunch of icons under the big Run button, first one is Project Navigator, hit it.
 * Right click Imprudence -> Open as -> Xcode project.
 * Build settings -> change from Basic to All.
 * Architectures -> Architectures -> 32 bit Intel. (Note 3)
 * Architectures -> Valid Architectures -> i386 (not i386 ppc)
 * Architectures -> Base SDK -> OS X 10.7 (Note 2, though that says to use 10.5, not an option)
 * Build Options -> Compiler ... -> LLVM GCC 4.2 (Note 1, though that says to use 4.0, not an option)
 * Can't see the option for Note 4.
 * Hit the Run button.
 * Enable developer mode.
 * Actual compile time errors.
 * Architectures -> Base SDK -> OS X 10.7 (Note 2, though that says to use 10.5, not an option)
 * Build Options -> Compiler ... -> LLVM GCC 4.2 (Note 1, though that says to use 4.0, not an option)
 * Can't see the option for Note 4.
 * Hit the Run button.
 * Enable developer mode.
 * Actual compile time errors.

take two

 * uninstall Xcode 4.6.3
 * update your software. In my case this bumps the OS up to 10.8.5
 * reboot
 * set the date on the clock back to 2011-01-01 (set it back later)
 * System Preferences -> Personal -> Security & Privacy -> Allow applications downloaded from -> Anywhere (set this back again afterwards for safety)
 * install xCode 3.6.2
 * download it from https://developer.apple.com/downloads/
 * install it using evil hacks
 * the repair script mentioned in http://hamstergene.github.io/blog/2012-11-13-xcode3-mountain-lion/ helped
 * the instructions from http://anatomicwax.tumblr.com/post/7906770311/installing-xcode-3-2-6-on-lion is what mostly worked
 * create a repair script /Users/my-user-name/repair-script
 * convert the Xcode 3.2.6 disk image to a read-write disk image with Disk Utility
 * mount the new read-write image
 * edit /Volumes/ where ever you mounted it /Contents/iPhoneSDKSL.dist
 * remove the string ' ' from line 148
 * run the installer
 * disable the installation of System Tools
 * you may get kernel panic and your system won't boot anymore
 * turn the power off
 * turn the power back on and hit Cmd+S once gray screen appears, then run the repair script
 * restore date, and security settings
 * reboot
 * use the linux scripts to build it
 * SUCCESS! And in the predicted half an hour.  B-)
 * Install it, run it, log onto 3rd Rock Grid, watch the film in the Team Purple meeting area. All seems to work.  B-)
 * SUCCESS! And in the predicted half an hour.  B-)
 * Install it, run it, log onto 3rd Rock Grid, watch the film in the Team Purple meeting area. All seems to work.  B-)
 * SUCCESS! And in the predicted half an hour.  B-)
 * Install it, run it, log onto 3rd Rock Grid, watch the film in the Team Purple meeting area. All seems to work.  B-)
 * Install it, run it, log onto 3rd Rock Grid, watch the film in the Team Purple meeting area. All seems to work.  B-)

Running a viewer through VNC sucks. Also, this is a bit of chicken and egg for getting cmake into the system. MacPorts needs a later Xcode I think.

release script setup
For the release script -


 * create the builder account
 * copy .profile to it, changing owner

The package step keeps failing for the builder account, when it works fine for my own account.

According to this old thread https://discussions.apple.com/thread/1201327?start=15&tstart=0 even Apple is not sure what it means. None of the stuff mentioned there seems to apply here.

Even after making the builder account a system admin, and adding an Apple ID to them, this step still fails with that erorr message on that account, or a similar one that says Finder is not running. There's some mystic incantation needed to get that to work, this is way to fragile. Also, the result of running that script has the icons in odd places. Let's see if I can find a less fragile method that does not involve mysterious magic buttons.

http://stackoverflow.com/questions/5088872/error-code-10810-when-calling-open-from-applescript says that the "open" command in that script wont work unless there's an actual GUI running. This one at least seems to match what is going on. YES this seems to have done the trick. Though "have a GUI up and running" is no way to deal with an automated shell script. Still need a better way. Looks like the better way is buried elsewhere in the source code, but not being invoked like it should.

Finally seems to be working.

In app_settings/windlight/skies the files like A%2D12AM.xml and %2EAnaLu%2E%20default1.xml don't seem to be working, though the others with %00 style bits seem to work fine. None of the others specifically includes %2D and %2E. Viewer start up complains about the first ones, so likely other OSes are not having this problem, but I should check that. Linux has no problem with them. Seems that it's using curl_unescape to do this, and I see no reason why that should fail on Mac OS X. %2D is an ordinary hyphen, and %2E is an ordinary period. "-" and ".". All three OSes can handle those in file names, we have been using them in the Impy release file names since the beginning of time. I'm just gonna change the file names. Pffft

dual Impy / Kokua setup
xcodebuild -license port selfupdate port upgrade outdated port install mercurial
 * move /Developer to /Xcode/Xcode_3.2.6 and create a link back to /Developer
 * I found out that 3.2.6 REALLY wants that /Developer, the build failed without it.
 * Install Xcode 4.6.3 as above, but not the command line tools.
 * I may have missed this step somehow -
 * move /Applications/Xcode.app to /Applications/Xcode_4.6.3.app
 * Link /Developer/SDKs/MacOSX10.5.sdk to /Applications/Xcode_4.6.3.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.5.sdk
 * Link /Developer/SDKs/MacOSX10.6.sdk to /Applications/Xcode_4.6.3.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.6.sdk
 * install mercurial

That last one failed, leaving log file in /opt/local/var/macports/logs/_opt_local_var_macports_sources_rsync.macports.org_release_tarballs_ports_devel_mercurial/mercurial/main.log Also note it installed python27-2.7.6_0 -

To make python 2.7 the default (i.e. the version you get when you run 'python'), please run: sudo port select --set python python27

port install wget wget http://mercurial.berkwood.com/binaries/Mercurial-2.6.2-py2.7-macosx10.8.zip unzip Mercurial-2.6.2-py2.7-macosx10.8.zip
 * install pre built mercurial
 * Then find it in Finder and install the package.

To switch command line tools between the two Xcodes (have to do this as root, dammit) -

xcode-select -switch /Xcode/Xcode_3.2.6

xcode-select -switch /Applications/Xcode_4.6.3.app/Contents/Developer

And to check which is selected (root not needed) -

xcode-select -print-path

For future reference, to uninstall Xcode 3.2.6 (might be best to undo the move and link from above first) -

sudo /Developer/Library/uninstall-devtools –mode=all

When creating new accounts for building, make sure they have a copy of the .profile file, though I should probably merge it's contents into the system wide version.